AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
AWS Glue generating multiple files instead of one
Hello,
I'm using a glue studio to do some custom transformation that I have in csv files and it's basically having a scheme as I'm sending here.
After run the job, I notice that the logic is correct but is generating multiple files with the naming like "part" etc. I suppose it's due some parallel processing. But could I configure to have one output file only, for each document I have in my input folder?
Cheers, Tassio
- 最新
- 投票最多
- 评论最多
From the looks of it your DynamicFrames are partitioning your files, You can repartition them.
Try the following: https://repost.aws/knowledge-center/glue-job-output-large-files
已回答 3 年前
Inside that "Custom transform", you can just call "repartition(1)" before you return the DF