How do I prevent "ResourceNumberLimitExceededException" errors when I update my Glue Data Catalog table version in DynamoDB?

Short description

To help service performance, AWS Glue API requests are throttled for each AWS account per AWS Region. When you update an AWS Glue Data Catalog table, the following errors might appear:

Account Level Exception: 'org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table. Number of TABLE_VERSION resources exceeds the account limit 1000000 (Service: AWSGlue; Status Code: 400; Error Code: ResourceNumberLimitExceededException; Request ID: c6baeb9d-45a9-489c-bd72-1fb86a873495; Proxy: null);'

Table Level Exception: "java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to alter table. Number of TABLE_VERSION resources exceeds the limit 100000 per TABLE (Service: AWSGlue; Status Code: 400; Error Code: ResourceNumberLimitExceededException; Request ID: e61d0518-70f5-42c9-a058-586347fababb; Proxy: null)"

By default, versioning is turned on for AWS Glue tables to track the schema changes of the table. An update to the table schema creates a new schema version. Also, there's a soft limit on the number of AWS Glue table versions at the table level or for the account. For more information, see AWS Glue endpoints and quotas.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

To prevent ResourceNumberLimitExceededException errors when you update your AWS Glue Data Catalog table version, complete one or more of the following actions:

• Increase the max table versions per table quota for the AWS Glue Data Catalog Region.
• Turn off table versioning to prevent any new schema versions.
• Update the table with the SkipArchive parameter to avoid schema versioning.
• Retrieve and delete earlier versions of the table schema. Run the following commands:

To retrieve all the versions of a table, run the get-table-versions command:

get-table-versions --database-name example-database-name --table-name example-table-name

Note: Replace example-database-name with the database name of the required table. Replace example-table-name with the name of the required table.

To delete a table version, run the delete-table-version command:

delete-table-version --database-name example-database --table-name example-table-name --version-id example-table-version-id

To delete multiple versions of a table, run the batch-delete-table-versions command:

batch-delete-table-version --database-name example-database-name --table-name example-table-name --version-ids example-table-version-ids

Note: Replace example-database-name with the database name of the required table. Replace example-table-name with the name of the required table. Replace example-table-version-ids with the versions of the required table that you want to delete.

Use an open source cleanup utility command to delete earlier versions of AWS Glue tables. For more information, see aws-glue-table-versions-cleanup-utility on the GitHub website.