No space left on device in Glue 4.0 with 50 DPUs G.1X worker

The "No space left on device" error you're encountering in your AWS Glue job is a common issue that can occur even when there appears to be sufficient storage available. This error typically happens when a specific filesystem or partition runs out of space during the job execution, rather than being related to the overall storage capacity.

To address this issue, you can try the following solutions:

1. Increase the number of partitions: By repartitioning your data, you can distribute the workload across more executors, potentially reducing the space required on each individual executor.

2. Optimize your code: Look for ways to reduce memory usage in your PySpark code. This might include using more efficient data structures or processing data in smaller batches.

3. Adjust Glue job parameters: You can try increasing the number of workers or changing the worker type to provide more resources to your job. However, since you're already using 50 DPUs with G.1X workers, this might not be the primary issue.

4. Use Glue Parquet writer: If you're not already using it, consider using the AWS Glue Parquet writer, which is optimized for writing Parquet files in Glue jobs.

5. Monitor disk usage: Add logging to track disk usage as your job progresses to identify where the space is being consumed most rapidly.

6. Clean up temporary files: Ensure that your job is properly cleaning up any temporary files it creates during processing.

7. Consider using Amazon Elastic File System (EFS): For large-scale data processing, you might want to explore using EFS, which provides scalable storage separate from your job's local storage.

If the issue persists after trying these solutions, you may need to reach out to AWS support for more detailed troubleshooting specific to your Glue job configuration and data processing pipeline. They can provide more targeted advice based on your specific use case and job characteristics.
Sources
Introducing generative AI troubleshooting for Apache Spark in AWS Glue (preview) | AWS Big Data Blog
No space left on device - Error during training a 6.8 GB dataset on llama 3.1 8B instruct | AWS re:Post