An error occurred while calling o1109.pyWriteDynamicFrame - AWS Glue Job

Question

I have a job that reads data from multiple sources in multiple S3 directories and performs some processing and aggregation and saves it back to S3, but when I try to write to S3, I get the following error:

```
An error occurred while calling o1109.pyWriteDynamicFrame. Job aborted due to stage failure: Task 43 in stage 3863.0 failed 4 times, most recent failure: Lost task 43.3 in stage 3863.0 (TID 20419) (10.31.54.34 executor 68): ExecutorLostFailure (executor 68 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
```

The job is running with 25 workers and worker type G 2.x

Writing to the bucket is done as follows:

```
context_job.glue_context.write_dynamic_frame.from_options(
frame=DynamicFrame.fromDF(dataframe, self.context_job.glue_context, "final_glue_df"),
connection_type="s3",
format="parquet",
connection_options={
"path": path,
"partitionKeys": ['anomesdia']
}
```

I also added some settings:

```
spark.sql.adaptive.fetchShuffleBlocksInBatch=false,
spark.sql.sources.partitionOverwriteMode=dynamic,
spark.driver.memory=10g,
spark.driver.maxResultSize=2g,
spark.sql.shuffle.partitions=50,
spark.sql.execution.arrow.enabled=true
```

re:Post Agent · Answer

The error you're encountering suggests that there might be resource constraints or network issues causing the Glue job to fail. Here are some potential solutions and things to consider:

1. Resource allocation: Your job is running with 25 workers of type G 2.x, which might not be sufficient for the amount of data you're processing. Consider increasing the number of workers or using a more powerful worker type.

2. Memory issues: The error mentions "containers exceeding thresholds," which could indicate memory problems. You've set the driver memory to 10g, but you might need to increase this further. Also, consider adjusting executor memory if needed.

3. Partitioning: You're using 'anomesdia' as a partition key. Ensure that this doesn't create too many small partitions, which can lead to performance issues. You might want to adjust your partitioning strategy.

4. Network issues: The error mentions possible network issues. Check if there are any connectivity problems between your Glue job and the S3 bucket.

5. Job bookmarks: If you're not already using them, consider implementing Glue job bookmarks to help track processed data and potentially avoid reprocessing large datasets.

6. Timeout settings: Verify if your job is hitting any timeout limits. You may need to increase the timeout settings for your Glue job if it's processing a large amount of data.

7. Data skew: If your data is heavily skewed, it could cause certain tasks to take much longer than others. Consider implementing strategies to handle data skew, such as salting or repartitioning.

8. Glue version: Ensure you're using a compatible and up-to-date version of Glue for your script and dependencies.

To troubleshoot further:

1. Review the full Glue job logs for any additional error messages or stack traces.
2. Monitor the job's resource utilization in CloudWatch to identify any bottlenecks.
3. Try running the job with a smaller subset of data to see if it completes successfully, which could help isolate whether it's a data volume issue.

If the issue persists after trying these steps, you may need to further optimize your transformation logic or consult AWS support for more detailed troubleshooting specific to your ETL job configuration.  
**Sources**  
[Error in AWS Glue ETL pipeline | AWS re:Post](https://repost.aws/questions/QUXdFwO32ST6-Oxvyx0dYtHg/error-in-aws-glue-etl-pipeline)  
[Troubleshooting errors Spark errors - AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/glue-troubleshooting-errors.html)

An error occurred while calling o1109.pyWriteDynamicFrame - AWS Glue Job

Add your answer

Relevant content