Schema was not updated after the glue job run by following the code in this page:
https://docs.aws.amazon.com/glue/latest/dg/update-from-job.html
In the "Updating table schema" sesssion, it says
When the job finishes, view the modified schema on the console right away, without having to rerun the crawler. You can enable this feature by adding a few lines of code to your ETL script, as shown in the following examples. The code uses enableUpdateCatalog set to true, and also updateBehavior set to UPDATE_IN_DATABASE, which indicates to overwrite the schema and add new partitions in the Data Catalog during the job run.
In my case, the schema of the source is changing over time. For example, there might be a new column added to the source table. However, in the data catalog, the schema of the destination table does not get updated, meaning that the new column is not added, after the glue job finishes.
What is going wrong?
As a result, it throws "HIVE_CURSOR_ERROR: Failed to read Parquet file" when querying the table in the Athena.
