Skip to content

Executor node OOM issues

0

lets say there is no data skew at the time of join or at the time of aggregation, so what will be the case when executor memory may result in out of memory errors considering data is partitioned to 128MB size, what are all other factors that will consume executor memory in this case that will result in executor node out of memory issues

asked a year ago462 views

2 Answers
0
Accepted Answer

Great question, even when you've eliminated data skew and are using reasonable partition sizes (128MB), executor memory OOM (Out of Memory) errors in Spark can still happen. Here's a sure-shot, human-style breakdown of what else can cause this:

Common Reasons for Executor OOM (Even Without Skew):

  1. Wide Transformations (Shuffle Explosion) Even if partitions are balanced, wide transformations like join, groupByKey, or distinct can cause shuffle files to accumulate in memory, especially if:

The join keys have many unique values You're joining two large datasets at once There’s a large number of shuffle partitions

Example: A hash join might pull in large hash tables to memory. If multiple stages try to materialize those at once, you hit OOM.

  1. Too Many Simultaneous Tasks per Executor If your executor is running too many tasks concurrently, each one gets a slice of the memory pie. That means even balanced partitions can push the limits.

Fix: Tune spark.executor.cores and spark.task.cpus to avoid running too many memory-hungry tasks at once.

  1. Caching / Persisting Too Much If you're caching large DataFrames or persisting them (df.cache() or persist()), they live in memory until evicted.

If you don't have enough space in spark.storage.memoryFraction, this eats into execution memory and causes OOM.

  1. High Cardinality / Exploding Data Operations like explode(), flatMap(), or large JSON/XML parsing can create many more records than expected per input row. Looks like no skew, but output size is massive = memory blowup.

  2. UDFs or Complex Serialization User-defined functions (especially Python UDFs) or large nested structures (e.g., deeply nested structs or maps) may cause:

Huge serialization overhead GC pressure Unpredictable memory spikes

  1. Improper Memory Tuning Default Spark memory fractions may not suit your workload:

spark.memory.fraction controls how much of executor memory is usable (default: 0.6)

spark.memory.storageFraction further limits what's used for storage vs execution

You may need to tweak these if you’re doing caching, aggregation, and joining in the same pipeline.

  1. Spill Fails or Slow Disk Even when memory fills up, Spark should spill to disk. But if:

Disk is slow Spill location is misconfigured Disk quotas are reached

Then Spark can't offload the memory and OOM occurs.

answered a year ago

AWS
SUPPORT ENGINEER

reviewed a year ago

EXPERT

reviewed a year ago

  • Thank you Manvitha for this wonderful answer, this helps!

0

When dealing with executor out-of-memory (OOM) issues in AWS Glue Spark jobs, even with properly partitioned data (128MB size) and no data skew during joins or aggregations, several other factors can still consume executor memory:

  1. JDBC Data Source Reading: By default, JDBC data source reads are not parallelized and use only one executor to read the complete table sequentially. If the JDBC driver's fetch size is set to zero (the default for some drivers), it will attempt to cache all rows in memory, leading to OOM errors.

  2. Large-Scale Transformations: Distributed in-memory execution of complex transformations can consume significant memory, even with properly sized partitions.

  3. Large Data Shuffles: Operations that require shuffling large amounts of data across executors can lead to memory pressure.

  4. Processing Backlog Data: When processing huge amounts of historical or backlog data, executors may run out of memory even with proper partitioning.

  5. Memory-Intensive Operations: Certain operations in your Spark code might be particularly memory-intensive.

  6. JVM Heap Usage: The JVM heap usage for executors can be monitored using the glue.ALL.jvm.heap.usage or specific executor metrics like glue.executorId.jvm.heap.usage. If this metric shows a positive slope crossing 50% and reaching high values (like 92%), it indicates memory exhaustion.

  7. Executor Replacement Cycle: When an executor runs out of memory, it gets stopped by Apache Hadoop YARN, and a new executor is launched to replace it. This cycle can continue if the underlying issue isn't addressed.

To address these issues, you can:

answered a year ago

  • I'm aware it happens because of the above reasons, my question is specific to why it happens, what are the factors affecting the memory

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.