1 Antwort
- Neueste
- Die meisten Stimmen
- Die meisten Kommentare
1
I believe I got this figured out. I was missing a step, where I had to add additionalOptions to my sink. As explained here: https://medium.com/analytics-vidhya/add-new-partitions-in-aws-glue-data-catalog-from-aws-glue-job-79b0442b17af
# parameter "enableUpdateCatalog" tells the aws glue job to update #the glue data catalog as the new partitions are created
additionalOptions = {"enableUpdateCatalog": True}
# define the partition keys
additionalOptions["partitionKeys"] = ["year", "month","day"]
After doing this, I saw that the partitions (nested directories) were being created. Now I am having some other isuses with querying, but I will post that as a separate question on here.
beantwortet vor 2 Jahren
Relevanter Inhalt
- AWS OFFICIALAktualisiert vor 2 Jahren
- AWS OFFICIALAktualisiert vor 3 Jahren
- AWS OFFICIALAktualisiert vor 2 Jahren