1 Answer
- Newest
- Most votes
- Most comments
0
I think, the issue seems due to the way AWS Glue handles concurrent runs of the same job. When you run the same job multiple times with different input data, AWS Glue will reuse the same executors and resources from the previous run, which can lead to memory issues. You may consider 1) Increase the worker memory, 2) Try Glue's built-in data frame capabilities , 3) Try Glue's data catalog partitioning, 4) Try Glue's bounded execution. You may try troubleshooting using this reference - https://docs.aws.amazon.com/glue/latest/dg/monitor-profile-debug-oom-abnormalities.html
answered 2 years ago
Relevant content
asked 2 years ago
- AWS OFFICIALUpdated a year ago
