Glue updates schema every time with same columns even with updateBehavior set as LOG

Question

Currently on Glue 5.0 (us-west-2 region) Python.

I do want to add new partitions without having to create a new schema version - following a documented way of having updateBehavior set to LOG as below. Observed that the output is mixed, sometimes it still creates new schema versions with same set of columns.

Below is the code. Is this a known limitation or a bug in Glue.

```
additionalOptions = {"enableUpdateCatalog": True, "updateBehavior": "LOG"}
additionalOptions["partitionKeys"] = ["year", "period_half", "month", "csid", "day"]
target_db, target_table = get_db_details(resolved_args)
datasink4 = glueContext.write_dynamic_frame_from_catalog(frame=dynamic_frame, database=target_db, table_name=target_table, transformation_ctx="datasink4",                                                                 additional_options=additionalOptions)

job.commit()
```
Some other observations about this run :

1. The glue error log shows that the policy is set to LOG so that is getting set.
2. Partitions are correctly getting added; only problem is extra versions everytime.
3. There are multiple glue job executions writing to the same partition and we do get an error shown in the log that says the partitions already exist exception. This doesn't fail the job obviously but thought to mention if this interferes with how the new schema vs partitioning is implemented.

Taz · Answer

Hey,

Hope you're keeping well.

`updateBehavior="LOG"` only affects how Glue handles column changes when writing to the Data Catalog — it logs them rather than overwriting the existing schema. However, Glue will still create a new schema version in the Schema Registry if it detects any difference in the schema from the last registered version, even if the columns are identical but the order, types, or metadata differ. This can happen if multiple jobs write to the same table concurrently or if the underlying DynamicFrame infers slightly different types for the same column.

Thanks and regards,  
Taz

re:Post Agent · Answer

The behavior you're experiencing with AWS Glue creating new schema versions despite setting `updateBehavior` to "LOG" is a known characteristic of how AWS Glue operates.

When you set `updateBehavior` to "LOG", it should prevent your table schema from being overwritten while still allowing new partitions to be added. However, by default, when a Glue job runs, it will update the "last updated" timestamp of the table metadata, which results in a new version being created in the Data Catalog even if there are no actual schema changes.

This behavior is similar to what happens with Glue crawlers, which also create new versions of tables in the Glue Data Catalog even when there are no schema changes, with the only difference being the "last updated" property.

The fact that you have multiple Glue job executions writing to the same partition could be contributing to this issue. When you get the "partitions already exist" exception, the system might still be updating some metadata, leading to new versions.

Some potential approaches to address this:

1. Consider using a custom AWS Glue ETL job that checks for schema changes before updating the Data Catalog, which would give you more fine-grained control.

2. Implement a process to periodically clean up old versions that are no longer needed if the accumulation of versions is causing problems.

3. If this behavior significantly impacts your workflows, you might want to reach out to AWS support for further guidance or to suggest this as a feature request for more granular control over version creation in the Glue Data Catalog.

The creation of new versions, even with only timestamp changes, appears to be part of AWS Glue's design to maintain a history of metadata changes, which can be beneficial for auditing and tracking purposes, but may not align with your specific use case.  
**Sources**  
[AWS Glue crawler creating new version glue data catalog tables when no change in schema | AWS re:Post](https://repost.aws/questions/QU_gpMOz9jR6iDvh0PIrcbCA/aws-glue-crawler-creating-new-version-glue-data-catalog-tables-when-no-change-in-schema)  
[Updating the schema, and adding new partitions in the Data Catalog using AWS Glue ETL jobs - AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/update-from-job.html)

Glue updates schema every time with same columns even with updateBehavior set as LOG

Add your answer

Relevant content