1 Risposta
- Più recenti
- Maggior numero di voti
- Maggior numero di commenti
1
I believe I got this figured out. I was missing a step, where I had to add additionalOptions to my sink. As explained here: https://medium.com/analytics-vidhya/add-new-partitions-in-aws-glue-data-catalog-from-aws-glue-job-79b0442b17af
# parameter "enableUpdateCatalog" tells the aws glue job to update #the glue data catalog as the new partitions are created
additionalOptions = {"enableUpdateCatalog": True}
# define the partition keys
additionalOptions["partitionKeys"] = ["year", "month","day"]
After doing this, I saw that the partitions (nested directories) were being created. Now I am having some other isuses with querying, but I will post that as a separate question on here.
con risposta 2 anni fa
Contenuto pertinente
- AWS UFFICIALEAggiornata un anno fa
- AWS UFFICIALEAggiornata 2 anni fa
- AWS UFFICIALEAggiornata 2 anni fa