I am new to AWS Glue and I am encountering an issue while trying to use the "google bigquery connector 0.24.2 for aws glue 3.0" in an AWS Glue Job. The goal is to establish a connection between my RDS database (configured as a JDBC source) and Google BigQuery (as the data destination). However, when I run the Glue Job, I receive the following error message: "LAUNCH ERROR | Glue ETL Marketplace - failed to download connector. Please refer to the logs for details."
Upon investigating the logs, I discovered the following error:
2023-06-05 09:27:16,800 - main - ERROR - Glue ETL Marketplace - boto3 clientError raised in function get_ecr_authorization_tokenClientError('An error occurred (AccessDeniedException) when calling the GetAuthorizationToken operation: User: arn:aws:sts:::assumed-role/AWSGlueServiceRole/GlueJobRunnerSession is not authorized to perform: ecr:GetAuthorizationToken on resource: * because no identity-based policy allows the ecr:GetAuthorizationToken action')
I have already confirmed that I successfully subscribed to the "google bigquery connector 0.24.2 for aws glue 3.0" and established the necessary connections. Furthermore, I have used a crawler to connect to the data source in the Data Catalog, and it successfully returns all the tables. However, when monitoring the Job Run, it displays an "Error loading data" message.
I would appreciate any guidance or insights on resolving this issue. Thank you in advance for your assistance!
AWS OFFICIALUpdated 3 years ago
After attaching AWSEC2ContainerRegistryFullAccess, I am facing GET request error in logs:
2023-06-09 10:07:05,453 - main - INFO - Glue ETL Marketplace - Start downloading connector jars for connection: data_warehouse 2023-06-09 10:07:05,742 - main - INFO - Glue ETL Marketplace - using region: us-east-1, proxy: 169.254.76.0:8888 and glue endpoint: https://glue.us-east-1.amazonaws.com to get connection: data_warehouse 2023-06-09 10:07:06,012 - main - INFO - Glue ETL Marketplace - Requesting ECR authorization token for registryIds=709825985650 and region_name=us-east-1. 2023-06-09 10:07:06,048 - main - INFO - Glue ETL Marketplace - Calling ECR HTTP API to get manifest of https://709825985650.dkr.ecr.us-east-1.amazonaws.com/amazon-web-services/glue/bigquery:0.24.2-glue3.0. 2023-06-09T15:07:06.134+05:00 2023-06-09 10:07:06,134 - main - INFO - Glue ETL Marketplace - Ran into exception in GET request. Retrying...
2023-06-09T15:07:06.205+05:00 2023-06-09 10:07:06,205 - main - INFO - Glue ETL Marketplace - Ran into exception in GET request. Retrying...
2023-06-09T15:07:06.291+05:00 2023-06-09 10:07:06,290 - main - INFO - Glue ETL Marketplace - Ran into exception in GET request. Retrying...
2023-06-09T15:07:06.370+05:00 2023-06-09 10:07:06,369 - main - INFO - Glue ETL Marketplace - Ran into exception in GET request. No more retries
Probably you are running in a VPC without access to that ecr url (no NAT or internet access). If it was firewalled, it would take much longer in each failure