AWS Glue generating multiple files instead of one
0
Hello,
I'm using a glue studio to do some custom transformation that I have in csv files and it's basically having a scheme as I'm sending here.
After run the job, I notice that the logic is correct but is generating multiple files with the naming like "part" etc. I suppose it's due some parallel processing. But could I configure to have one output file only, for each document I have in my input folder?
Cheers, Tassio
1回答
- 新しい順
- 投票が多い順
- コメントが多い順
0
From the looks of it your DynamicFrames are partitioning your files, You can repartition them.
Try the following: https://repost.aws/knowledge-center/glue-job-output-large-files
回答済み 1年前
関連するコンテンツ
- 質問済み 9ヶ月前
AWS公式更新しました 1年前- AWS公式更新しました 3年前
- AWS公式更新しました 3年前
Inside that "Custom transform", you can just call "repartition(1)" before you return the DF