I changed the worker_type from G.1X to G.2X and the job completed successfully, albeit in 38 hours. So then I tuned the Spark code so that the 3 Dataframes are all partitioned on the same value
.repartition("attribute_name"), and also doubled the number of workers from 5 to 10. Then the job completed successfully in 1 hr 20 mins. The partitioning helped the JOIN that was being done to create the final dataset being written to s3.
Do you use external libraries?
Contacting AWS Support might be the fastest way to resolve your issue if you cannot find any indication in the documentation shared, without seeing the job itself it is difficult to provide more prescriptive guidance.
hope this helps
An error occurred while calling o95.pyWriteDynamicFrame.asked 8 months ago
An error occurred while calling o187.pyWriteDynamicFrame. Unable to execute HTTP request:asked 5 months ago
AWS Glue Studio Job Failed An error occurred while calling o106.pyWriteDynamicFrame. Data truncation: Data too long for column [insertcolumnname] at row 1asked a month ago
Glue ETL PySpark Job Fails after Upgrade from Glue Version 2.0 to 3.0 error occurred while calling pyWriteDynamicFrame EOFException occurred while reading the port number from pyspark.daemon's stdoutasked 8 months ago
An error occurred while calling o280.pyWriteDynamicFrameasked 4 years ago
An error occurred while calling o110.pyWriteDynamicFrame. nullasked 2 months ago
An error occurred while calling o128.pyWriteDynamicFrame. Record is incomplete.asked 6 months ago
AWS Glue gives error: "An error occurred while calling o285.save. File already exists:"asked 9 months ago
Glue Job Error (An error occurred while calling o100.pyWriteDynamicFrame. Illegal empty schema)Accepted Answerasked 6 months ago
An error occurred while calling o352.pyWriteDynamicFrame. Job 1 cancelled because SparkContext was shut down caused by threshold for consecutive task creation reachedasked 3 months ago