AWS Glue generating multiple files instead of one

0

Hello,

I'm using a glue studio to do some custom transformation that I have in csv files and it's basically having a scheme as I'm sending here. Glue Scheme

After run the job, I notice that the logic is correct but is generating multiple files with the naming like "part" etc. I suppose it's due some parallel processing. But could I configure to have one output file only, for each document I have in my input folder?

Cheers, Tassio

tassio
preguntada hace un año456 visualizaciones
1 Respuesta
0

From the looks of it your DynamicFrames are partitioning your files, You can repartition them.

Try the following: https://repost.aws/knowledge-center/glue-job-output-large-files

AWS
vtjean
respondido hace un año
  • Inside that "Custom transform", you can just call "repartition(1)" before you return the DF

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas