How do I prevent "ResourceNumberLimitExceededException" errors when I try to update my AWS Glue Data Catalog table version?

Short description

AWS Glue API requests are throttled for each AWS account per AWS Region to help service performance. When you update an AWS Glue Data Catalog table, the following errors might appear:

• Account Level Exception: 'org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table. Number of TABLE_VERSION resources exceeds the account limit 1000000 (Service: AWSGlue; Status Code: 400; Error Code: ResourceNumberLimitExceededException; Request ID: c6baeb9d-45a9-489c-bd72-1fb86a873495; Proxy: null);'

• Table Level Exception: "java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to alter table. Number of TABLE_VERSION resources exceeds the limit 100000 per TABLE (Service: AWSGlue; Status Code: 400; Error Code: ResourceNumberLimitExceededException; Request ID: e61d0518-70f5-42c9-a058-586347fababb; Proxy: null)"

By default, AWS Glue tables have versioning turned on to track the schema changes of the table. An update to the table schema creates a new schema version. Also, there's a soft limit on the number of AWS Glue table versions at the table level or for the account. For more information, see AWS Glue endpoints and quotas.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

To prevent ResourceNumberLimitExceededException errors when you update your AWS Glue Data Catalog table version, complete one or more of the following actions:

• Increase the max table versions per table quota for the AWS Glue Data Catalog Region.
• Turn off table versioning to prevent any new schema versions.
• Update the table with the SkipArchive parameter to avoid schema versioning.
• Retrieve and delete older versions of the table schema. Run the following commands:

Retrieve all the versions of a table:

Note: Replace example-database-name with the database name of the required table. Replace example-table-name with the name of the required table.

get-table-versions --database-name example-database-name --table-name example-table-name

To delete a table version, run the delete-table-version command:

Note: Replace example-database-name with the database name of the required table. Replace example-table-name with the name of the required table. Replace example-table-version-id with the version of the required table that you want to delete.

delete-table-version --database-name example-database --table-name example-table-name --version-id example-table-version-id

To delete multiple versions of a table, run the batch-delete-table-versions command::

Note: Replace example-database-name with the database name of the required table. Replace example-table-name with the name of the required table. Replace example-table-version-ids with the versions of the required table that you want to delete.

batch-delete-table-version --database-name example-database-name --table-name example-table-name --version-ids example-table-version-ids

• Use an open source cleanup utility command to delete old versions of AWS Glue tables. For more information, see aws-glue-table-versions-cleanup-utility on the GitHub website.