AWS Glue generating multiple files instead of one

0

Hello,

I'm using a glue studio to do some custom transformation that I have in csv files and it's basically having a scheme as I'm sending here. Glue Scheme

After run the job, I notice that the logic is correct but is generating multiple files with the naming like "part" etc. I suppose it's due some parallel processing. But could I configure to have one output file only, for each document I have in my input folder?

Cheers, Tassio

tassio
gefragt vor einem Jahr455 Aufrufe
1 Antwort
0

From the looks of it your DynamicFrames are partitioning your files, You can repartition them.

Try the following: https://repost.aws/knowledge-center/glue-job-output-large-files

AWS
vtjean
beantwortet vor einem Jahr
  • Inside that "Custom transform", you can just call "repartition(1)" before you return the DF

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen