Getting No space left on device error in Glue Etl

Hey, this is a pretty common issue in Glue, especially when writing large CSV outputs. Even though you're using G.2X workers, each worker only gets 64 GB of local disk, and CSV writes are disk-heavy (way more than Parquet).

Here’s what usually causes the No space left on device error:

You might be using .coalesce(1) or .repartition(1) before the write that forces everything onto one node = disk full.

Writing to /tmp or local disk instead of S3

Huge shuffle or wide transformation during repartition

Fixes that work: Let Spark write multiple CSV part files; avoid trying to merge them in the job

Write directly to s3:// paths, not local /tmp

Use Parquet if possible; it’s far more efficient.

If needed, upgrade to G.4X workers; they have 128 GB of disk

Let me know if you want help tweaking the code; happy to look at it.

The "No space left on device" error you're encountering in your AWS Glue ETL job is a common issue when processing large volumes of data. This error typically occurs when the Spark executor nodes run out of disk space during write operations.

Here's what's happening and how to fix it:

1. Root Cause: When writing large datasets to CSV or Parquet formats, Spark needs temporary disk space to stage the data before writing it to the final destination. With large volumes of data, the default disk allocation can be insufficient.

2. Solutions:

a) Increase worker capacity: Since you're already using G.2X workers (which have more memory and compute), consider increasing the number of workers to distribute the workload better.

b) Partition your data: Break down your write operations into smaller chunks by partitioning your data. This reduces the amount of data each executor needs to handle at once.

c) Use Glue's native optimizations: If writing to Parquet, consider using AWS Glue's optimized Parquet writer which can handle large datasets more efficiently.

d) Adjust Spark configurations: You can modify Spark's temporary storage settings by adding these configurations to your Glue job:

• Set --conf spark.driver.extraJavaOptions=-Djava.io.tmpdir=/tmp
• Set --conf spark.executor.extraJavaOptions=-Djava.io.tmpdir=/tmp

e) Optimize your write operations: Instead of writing the entire dataset at once, consider writing in batches or using coalesce/repartition to control the number of output files.

The error doesn't necessarily mean the writes are running on the driver, but rather that the executors handling the write operations are running out of disk space. By implementing these solutions, you should be able to overcome the disk space limitation and successfully complete your ETL job.
Sources
Introducing generative AI troubleshooting for Apache Spark in AWS Glue (preview) | AWS Big Data Blog
Debugging OOM exceptions and job abnormalities - AWS Glue