How G.1X worker type behaves in Glue when write to S3 for parquet files

When using the G.1X worker type in AWS Glue, each worker maps to 1 DPU (4 vCPUs, 16 GB of memory) with a 94GB disk, and provides 1 executor per worker. This worker type is recommended for workloads such as data transforms, joins, and queries, offering a scalable and cost-effective way to run most jobs.

For your specific scenario where you're reading parquet files from S3 and writing them back as parquet and CSV files to different locations, the behavior depends on your configuration and data size:

1. By default, with a G.1X worker type, all the processing and writing would happen through that single worker/executor if you only have one worker allocated.

2. If you want to split the files during write operations, you have several options:

• Increase the number of workers in your Glue job configuration to parallelize the write operations
• Use the repartition() or coalesce() functions in your PySpark code to control the number of output files
• Set the maxRecordsPerFile option when writing to limit the number of records per file

For example, to repartition your data before writing:

df = df.repartition(10)  # Creates 10 output files
df.write.parquet("s3://your-bucket/output-path/")

Or to limit records per file:

df.write.option("maxRecordsPerFile", 100000).parquet("s3://your-bucket/output-path/")

These approaches will help distribute the write operations even with a single G.1X worker, as the worker will process the partitions sequentially but create multiple files as output.

Note that when developing locally, some Glue-specific optimizations like the Glue Parquet writer may not be available, and you might need to use standard Spark writers instead.
Sources
Job runs - AWS Glue
Jobs - AWS Glue
AWS Glue: local dev container and useGlueParquetWriter not working | AWS re:Post