- Newest
- Most votes
- Most comments
It seems you're encountering an issue when trying to preview data from Google BigQuery in AWS Glue. While your connection test was successful, the error you're seeing during data preview suggests there might be a problem with data retrieval or compatibility. Here are some steps you can take to troubleshoot and potentially resolve the issue:
-
Verify your Google Cloud Platform (GCP) credentials: Make sure the service account you're using has the necessary permissions to access and read data from the BigQuery table you're trying to connect to. The service account should have at least the BigQuery Job User and BigQuery Data Viewer roles.
-
Check your AWS Glue connection configuration: Ensure that the connection details in AWS Glue, including the secret containing your GCP credentials, are correctly set up. Double-check that you're using the correct project ID, dataset, and table name in your connection configuration.
-
IAM role permissions: Verify that the IAM role associated with your AWS Glue job has the necessary permissions to read the secret containing your GCP credentials from AWS Secrets Manager.
-
Connector version compatibility: The Google BigQuery Connector 0.24.2 for AWS Glue 3.0 that you're using should be compatible, but you might want to check if there's a more recent version available that could potentially resolve this issue.
-
Simplify your query: If you're using a complex query in your Google BigQuery (read) node, try simplifying it to a basic SELECT statement to rule out any query-related issues.
-
Check for empty dataset: The error message suggests an empty collection. Although you mentioned there is data in your BigQuery table, double-check to ensure the specific query or table you're accessing isn't returning an empty result set.
-
Network configuration: Ensure that your AWS Glue environment has the necessary network access to reach Google Cloud services. This might involve configuring VPC endpoints or NAT gateways if you're running in a private subnet.
-
Logging and monitoring: Enable detailed logging for your AWS Glue job and check the CloudWatch logs for any additional error messages or clues that might not be visible in the Visual ETL interface.
If the issue persists after trying these steps, you may need to reach out to AWS Support with the details and job runID for further assistance as they can provide more in-depth troubleshooting based on your specific setup and configuration by checking the logs
Sources
Creating a BigQuery connection - AWS Glue
AWS Glue Error Category: UNCLASSIFIED_ERROR; Failed Line Number: 177; An error occurred while calling o104.getDynamicFrame. empty.reduceLeft | AWS re:Post
answered 2 years ago
Relevant content
asked 3 years ago
asked 8 months ago
