AWS Glue repartition with certain file size in python

Question

I have below Python script where currently it generates several gz files with size 4MB in S3 bucket. Its bydeafult what AWS glue has created. But now i want to create multiple files of each file size 100MB in s3 bucket. I have tried below logic in python script but it did not work and still creates several gz files with size 4MB. I dont know how we can repartition here with to generate file with certain file size.

`
```
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
import datetime

args = getResolvedOptions(sys.argv, ['target_BucketName', 'JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

outputbucketname = args['target_BucketName']

timestamp = datetime.datetime.now().strftime("%Y%m%d")
filename = f"tbd{timestamp}"
output_path = f"{outputbucketname}/{filename}"

# Script generated for node AWS Glue Data Catalog
AWSGlueDataCatalog_node075257312 = glueContext.create_dynamic_frame.from_catalog(database="ardt", table_name="_ard_tbd", transformation_ctx="AWSGlueDataCatalog_node075257312")

# Script generated for node Amazon S3
AmazonS3_node075284688 = glueContext.write_dynamic_frame.from_options(frame=AWSGlueDataCatalog_node075257312, connection_type="s3", format="csv", format_options={"separator": "|"}, connection_options={"path": output_path, "compression": "gzip",  "recurse": True, "groupFiles": "inPartition", "groupSize": "100000000"}, transformation_ctx="AmazonS3_node075284688")

job.commit()
```
`

Yaniv Rozenboim · Answer

Have you followed the [How can I configure an AWS Glue ETL job to output larger files?](https://repost.aws/knowledge-center/glue-job-output-large-files) article?
It contains several suggestions to control the output file sizes.

Alexander B · Answer

The provided example is using groupSize in the 'from_options' method. I can clarify that the groupFiles method enables 'the grouping of files within an Amazon S3 data partition', this method is specific to the reading of files from S3. [1][2]
It is true that repartitioning data will involve shuffling of data, but the number of files you write in Spark is determine by the number of partitions. We can reduce the impact of such partition management in the case of controlling the size of written files by pushing the repartitioning action close to your write operation to ensure your other operations transforming your data parallelize effectively.

References
[1] https://docs.aws.amazon.com/glue/latest/dg/grouping-input-files.html
[2] https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-s3-home.html#aws-glue-programming-etl-connect-s3

AWS Glue repartition with certain file size in python

Relevant content