By using AWS re:Post, you agree to the Terms of Use

Glue ETL PySpark Job Fails after Upgrade from Glue Version 2.0 to 3.0 error occurred while calling pyWriteDynamicFrame EOFException occurred while reading the port number from pyspark.daemon's stdout


A PySpark Glue ETL job fails after upgrading from Glue Version 2.0 to 3.0.

The job fails while writing a DynamicFrame to s3 in parquet format.

error snippet from log: py4j.protocol.Py4JJavaError: An error occurred while calling o433.pyWriteDynamicFrame. : org.apache.spark.SparkException: Job aborted

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 30.2 failed 4 times, most recent failure: Lost task 1.3 in stage 30.2 (TID 2007594) ( executor 8): org.apache.spark.SparkException: EOFException occurred while reading the port number from pyspark.daemon's stdout

2 Answers

I changed the worker_type from G.1X to G.2X and the job completed successfully, albeit in 38 hours. So then I tuned the Spark code so that the 3 Dataframes are all partitioned on the same value .repartition("attribute_name"), and also doubled the number of workers from 5 to 10. Then the job completed successfully in 1 hr 20 mins. The partitioning helped the JOIN that was being done to create the final dataset being written to s3.

answered 7 months ago
  • thank you for feedback. How long was it taking with Glue 2.0? were you using the same number of nodes?



have you looked at the documentation about migrating Glue from version 2.0 to 3.0 ? and Spark 2 to Spark 3?

Do you use external libraries?

Contacting AWS Support might be the fastest way to resolve your issue if you cannot find any indication in the documentation shared, without seeing the job itself it is difficult to provide more prescriptive guidance.

hope this helps

answered 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant questions