- Newest
- Most votes
- Most comments
Based on the information provided, it appears that there's a discrepancy between what the Glue job is experiencing and what you're seeing in the Athena UI. The error message suggests that the table 'awsdatacatalog.our_database.our_table' cannot be found when the Glue job attempts to run the query, despite the same query working in the Athena UI.
This situation could be caused by a few potential issues:
-
Temporary Glue Data Catalog inconsistency: There might be a temporary synchronization issue between the Glue Data Catalog and Athena. The Glue job might be attempting to access the catalog at a moment when the table information is not available or properly synced.
-
Permissions mismatch: While you mentioned it doesn't look like an access issue, it's worth double-checking that the IAM role used by the Glue job has the same permissions as your user account when accessing Athena UI. The Glue job might be missing necessary permissions to access the Glue Data Catalog or the specific table.
-
Region mismatch: Confirm that the Glue job is set to run in the same region (eu-west-1) as the Athena table and the S3 bucket containing the data.
-
Glue job configuration: There might be an issue with how the Glue job is configured to interact with Athena. Check if the job is using the correct database name and table name in its configuration.
-
Recent changes: Although you mentioned nothing was changed, it's worth investigating if there were any recent updates to the Glue Data Catalog, S3 bucket permissions, or IAM policies that might affect the Glue job's ability to access the table.
To troubleshoot this issue:
- Verify the table exists in the Glue Data Catalog for the specific region.
- Double-check the IAM permissions for the Glue job's execution role.
- Ensure the Glue job is configured to use the correct region.
- Try running the Glue job again, as it might have been a temporary issue with the Glue Data Catalog.
- If the problem persists, you may need to refresh the table metadata in the Glue Data Catalog or recreate the table definition.
If these steps don't resolve the issue, you may need to contact AWS support for further assistance, providing them with the Query ID (6cc354a3-9850-4cea-8e6b-7e6f79c8c388) for their investigation.
Sources
Use AWS Glue to connect Athena to your data in Amazon S3 - Amazon Athena
Community | Unleashing Data Analytics on S3 Data lake with AWS Glue Crawler and Amazon Athena
Community | What happens when you run a query in Amazon Athena?
Athena error catalog - Amazon Athena
As you mentioned you can run the query successfully in Athena UI, the first thing I believe is to make sure your Glue job execution role has necessary permission to access the AWS Glue Data Catalog. Also, you can check the CloudTrail event around your failure time to see if there is any 'AccessDenied' error code, so that you can check if this issue is related to access.
To answer your question, we require details that are non-public information. Please open a technical support case with AWS by signing into the AWS Console for the AWS Account that is running the Glue Job, then following this link.
In the technical support case, please be sure to include all resource detail such as Glue Job Run Ids, Athena Query Ids, and relevant error messaging.
Relevant content
- asked 2 years ago
