Passer au contenu
En utilisant AWS re:Post, vous acceptez les AWS re:Post Conditions d’utilisation
AWS re:Postre:Post
À propos de re:Post

AWS Glue generating multiple files instead of one

0

Hello,

I'm using a glue studio to do some custom transformation that I have in csv files and it's basically having a scheme as I'm sending here. Glue Scheme

After run the job, I notice that the logic is correct but is generating multiple files with the naming like "part" etc. I suppose it's due some parallel processing. But could I configure to have one output file only, for each document I have in my input folder?

Cheers, Tassio

Sujets
AnalyseBase de données
Balises
Extraire, transformer et charger des donnéesAWS Glue
Langue
English

demandé il y a 3 ans1,5 k vues

1 réponse
0

From the looks of it your DynamicFrames are partitioning your files, You can repartition them.

Try the following: https://repost.aws/knowledge-center/glue-job-output-large-files

AWS

répondu il y a 3 ans

  • Gonzalo Herreros EXPERT
    il y a 3 ans

    Inside that "Custom transform", you can just call "repartition(1)" before you return the DF

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Contenus pertinents