Athena ThrottlingException Errors

Question

We seeing errors with Athena Iceberg tables.  One error is:

ICEBERG_CANNOT_OPEN_SPLIT: Error opening Iceberg split s3://us-east-1-prod-fifa-athena-aggregation/ec2_usage_fraud_nounmet/data/15db801c/week_date=2022-09-25/region=MXP/20230822_210918_00039_fcdie-9eda1dff-09c5-4b21-b433-2ad3020e52dc.parquet (offset=0, length=3656): com.amazonaws.services.lakeformation.model.AWSLakeFormationException: Rate exceeded (Service: AWSLakeFormation; Status Code: 400; Error Code: ThrottlingException; Request ID: 6145a9a0-b16f-48a6-be62-031cb2e1ad9b; Proxy: null)

And another error is:

ICEBERG_COMMIT_ERROR: Failed to commit Iceberg update to the table: . If a data manifest file was generated at 's3://us-east-1-prod-fifa-core-aggregation/athena_temp/00000000-0003-4f24-83df-aee5e538d7d7_1711065409/b0b4acf6-bee6-461b-8589-ced10f36eba8-manifest.csv', you may need to manually clean the data from locations specified in the manifest. Athena will not delete data in your account.

This isn't a massive system batch oriented system, but it does read around 10 trillion rows and writes out 1 trillion rows over the course of a week.  We have multiple processes updating different Iceberg partitions (not the same partition) on the same table.

The end result is for on table has 219,768 versions in Glue, meaning we have done many different updates to this table.  From the error messages, it seems like Athena can't handle this many updates to the the table, but I would like to confirm that.

What are options for dealing with these errors?

And do we need to delete the dead manifest file that is mentioned in the second error?

On a side note, is there an Athena logging table that tracks all the errors, like Redshift's stl_errors table?

Answer

The errors you're encountering with Athena Iceberg tables are related to various issues such as rate limits being exceeded and failed commits during updates

These checks below should be able to help in troubleshooting the errors:-

Throttling and Rate Limiting:

The error ThrottlingException indicates that you're exceeding rate limits, likely imposed by Lake Formation or other AWS services involved in your data processing pipeline.
Check if you're making too many concurrent requests or exceeding service limits, and adjust your workload accordingly.
Consider reaching out to AWS support to request a limit increase if necessary.

Handling Commit Errors:

The error ICEBERG_COMMIT_ERROR suggests that commits to the Iceberg table are failing. This could be due to various reasons such as conflicts in concurrent updates or issues with data integrity.
Ensure that your update processes are designed to handle concurrent updates properly and implement retry mechanisms for failed commits.
Monitor the logs and error messages to identify any patterns or specific conditions that lead to commit failures.

Cleaning Up Dead Manifest Files:

The manifest file mentioned in the error message is likely a temporary file generated during the commit process.
It's generally a good practice to clean up any temporary or dead files to avoid cluttering your storage and potential confusion in the future.
You can safely delete the dead manifest file referenced in the error message. However, make sure to review its contents and ensure that any necessary data is persisted elsewhere before deletion.

About the Logging 
Currently, Athena doesn't have a built-in logging table similar to Redshift's stl_errors table.
However, you can enable CloudTrail logging for Athena to track API calls and activities, including errors and failures. CloudTrail logs can be analyzed to identify errors and troubleshoot issues in your Athena queries and operations.

to address the errors you're encountering with Athena Iceberg tables, you'll need to optimize your workload, handle commit errors gracefully, clean up temporary files, and leverage AWS CloudTrail for logging and monitoring. Additionally, consider reaching out to AWS support for assistance with resolving rate limiting and other service-related issues.

Thanks

Athena ThrottlingException Errors

Relevant content