How do I get the output of an AWS Glue DataBrew job to be a single CSV file?

Question

I'm running a test AWS Glue DataBrew job to run a simple **Group By** and **Sum** operation on a CSV dataset that contains total costs by product each day. The desired output is a single CSV file with three columns: Day (timestamp), total cost, and product ID.

After I make the changes in the DataBrew project and select **Run Job**, the job runs successfully, but the output is numerous CSV files. Is there a way for me to specify that I want the output to just be one file, rather than multiple files?

Accepted Answer

You have two options:

* Download the processed recipe directly from the project workspace. (**Note:** This option is limited to the sample data on screen and has a maximum row size of 5000.)
* [Write an AWS Glue extract, transform, and load (ETL) job](https://lakeformation.workshop.aws/glue-basics/glue-etl.html) to repartition the data before writing a [DynamicFrame](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame.html) to Amazon S3.

How do I get the output of an AWS Glue DataBrew job to be a single CSV file?

相关内容