By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

AWS Glue generating multiple files instead of one

0

Hello,

I'm using a glue studio to do some custom transformation that I have in csv files and it's basically having a scheme as I'm sending here. Glue Scheme

After run the job, I notice that the logic is correct but is generating multiple files with the naming like "part" etc. I suppose it's due some parallel processing. But could I configure to have one output file only, for each document I have in my input folder?

Cheers, Tassio

Topics
AnalyticsDatabase
Tags
AWS GlueExtract Transform & Load Data
Language
English
tassio
asked a year ago403 views
1 Answer
0

From the looks of it your DynamicFrames are partitioning your files, You can repartition them.

Try the following: https://repost.aws/knowledge-center/glue-job-output-large-files

AWS
vtjean
answered a year ago
  • Gonzalo Herreros EXPERT
    a year ago

    Inside that "Custom transform", you can just call "repartition(1)" before you return the DF

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content