2 réponses
- Le plus récent
- Le plus de votes
- La plupart des commentaires
0
You cannot just read the new columns, for that you would need a columnar format like parquet.
Also incremental ingestion normally refers to loading new files, for that you could use Glue bookmarks (running a Glue job instead of Spectrum) or putting new files on different folders(partitions) and telling Spectrum to load just that)
0
Have you configured an ETL job to merge data? https://github.com/sinemozturk/INCREMENTAL-DATA-LOADING-FROM-AWS-S3-BUCKET-TO-REDSHIFT-BY-USING-AWS-GLUE-ETL-JOB
We want to explore the options to load the data without flattening the json using AWS Glue job to reduce the billing
Contenus pertinents
- demandé il y a un an
- demandé il y a 4 mois
- demandé il y a 6 mois
- demandé il y a 9 mois
- AWS OFFICIELA mis à jour il y a 2 ans
- AWS OFFICIELA mis à jour il y a un an
- AWS OFFICIELA mis à jour il y a 2 ans
- AWS OFFICIELA mis à jour il y a 2 ans
How to dynamically change the partition values so that we could automate this job !
If you mean filtering partitions, you would need to build your query with the values you need, for instance using the current date for date related columns