Glue Crawler has an option to use for schema evolution scenario. To be able to add new columns whenever new columns are added into the files, you need to follow the following steps:
Make sure you choose "Crawl all folders"
Choose Add new columns or Update Table definition in Catalog under advanced options in "Set output and scheduling"
In case your column names are updated correctly, you may also need to create a classifier and link that classifier with this crawler. Here are the options you need to choose for the classifier:
Thanks for bring this scenario up. Is it possible for you to try to do perform few tests using parquet files and see if that works for your use case?
backslash in CSV with glueasked 8 months ago
Cannot query Athena tables as the Glue crawler generated data having = instead of : in the column valuesasked 2 months ago
How could we have Glue to get data from csv as String?Accepted Answerasked 5 months ago
crawled data can not be queried in athenaAccepted Answerasked 5 years ago
Amazon Athena query outputting all columns in the csv as VARCHARsAccepted Answerasked 2 years ago
Can't get Partitions to work with my Glue Data CatalogAccepted Answerasked 4 months ago
AWS Glue with CSV source data that changes over timeasked 6 days ago
AWS Glue not properly crawling s3 bucket populated by "Resource Data Sync" -- specifically, "AWS: InstanceInformation" is not made into a tableasked 5 months ago
No 'Time with time zone' data type in AWS Glue Crawlerasked 2 months ago
I need to read S3 data, transform and put into Data Catalog. Should I be using a Crawler?Accepted Answerasked 6 months ago