Memory management in Glue

Memory management in AWS Glue is indeed an important aspect to understand for optimizing your ETL jobs. Let me address your questions:

1. In the Spark UI for a G.1x DPU (Data Processing Unit), you're seeing 10GB of executor memory out of the total 16GB per DPU. This 10GB is typically the heap memory allocated for Spark executors. The remaining 6GB is not wasted but is used for various purposes:

   • A portion is reserved for the operating system and other system processes.
   • Some memory is allocated for off-heap usage by Spark itself (e.g., for certain internal data structures).
   • Memory is also set aside for the driver process, which manages the overall Spark application.

2. Regarding off-heap memory, it is indeed used in AWS Glue jobs, but it's not as explicitly visible as heap memory. Off-heap memory is utilized for various purposes:

   • Spark's internal operations that benefit from direct memory access.
   • Certain data processing tasks that can leverage off-heap storage for improved performance.

While you can't directly enable or disable off-heap memory in AWS Glue (as it's a managed service), it is used automatically as needed. You can indirectly observe its usage through memory utilization metrics.

To monitor memory usage more comprehensively, AWS Glue provides observability metrics that you can access through Amazon CloudWatch. These metrics include:

• glue.driver.memory.heap.[available | used]
• glue.driver.memory.non-heap.[available | used]
• glue.driver.memory.total.[available | used]
• glue.ALL.memory.heap.used.percentage
• glue.ALL.memory.non-heap.used.percentage
• glue.ALL.memory.total.used.percentage

These metrics can give you insights into both heap and non-heap memory usage across your Glue job's driver and executors.

To optimize memory usage in your Glue jobs, consider techniques such as:

• Using efficient data storage formats like Parquet or ORC
• Implementing partitioning strategies
• Leveraging AWS Glue's grouping feature for processing large numbers of small files
• Adjusting Spark configurations when necessary (though be cautious as Glue manages many of these automatically)

Remember, AWS Glue is designed to handle memory management efficiently in most cases, but understanding these concepts can help you troubleshoot and optimize your jobs when needed.
Sources
Best practices - AWS Prescriptive Guidance
Debugging OOM exceptions and job abnormalities - AWS Glue
Monitoring with AWS Glue Observability metrics - AWS Glue