Glue slow on JSON to Parquet

Question

Hi,

I have a couple of glue jobs to convert JSON to Parquet from one S3 bucket to another.
Average JSON file size between 1-2 KB, total files so far 1.6 million, the total size of all files so far is 1.6 GB.
My glue jobs take between 3-4 minutes to process and run every 10 minutes.
How can I improve the execution time of these glue jobs? Is keeping a large number of these small files in the source S3 bucket is an issue, thought of moving prior 30 days files to the glacier, will it improve run time?

acmanjon · Answer

Hi,
Did you look at Spark UI to see whether the job is spending most of its time?  If you are not processing all the files every 10 minutes it would be great to move those processed objects to another bucket to improve listing time or use partitions in order to improve the reading of those files.

Please read this link (Handle large number of small files and partition sections)
[Glue Best Practices](https://aws.amazon.com/blogs/big-data/best-practices-to-scale-apache-spark-jobs-and-partition-data-with-aws-glue/)

Bests

Glue slow on JSON to Parquet

Add your answer

Relevant content