Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Is there a memory leak with Sagemaker Jupyperlab python kernel?

0

I am trying to process a large amount of S3 data in batches with a Sagemaker Jupyterlab instance. I have had to resort to restarting the kernel at the end of each batch, painful. Memory seems to grow unbounded and the kernel eventual restarts.

The same Python code runs memory-neutral on my desktop, but within Jupyterlab memory grows unbounded. Is there some additional memory management instrumentation integrated into Sagemaker Jupyperlab? This problem has really make Sagemaker Jupyper lab clunky and difficult to use for what seems like a common data/analytics workload.

Topics
Machine Learning & AIStorage
Tags
Amazon SageMaker
Language
English
asked a year ago367 views
1 Answer
0
Accepted Answer

The issue could be attributed to either Python code or the sagemaker setting(instance type selection etc) . Please try below (High level tips based on limited context.)

Python Code Issues:

  1. Global Variables: Unintended modifications to global variables can lead to memory accumulation.

  2. Object References: Objects that are no longer needed might still be referenced, preventing garbage collection.

  3. Inefficient Data Structures: Using inappropriate data structures can consume excessive memory.

  4. Library-Specific Issues: Certain libraries might have memory leaks or inefficient implementations.

** Use Python's memory_profiler or tracemalloc to identify memory-intensive parts of your code. OR Consider using del or gc.collect() to explicitly release memory.

SageMaker JupyterLab Environment:

1/ Kernel Configuration: The kernel's memory limits and configuration might be affecting behavior.

2/ Instance Type: The chosen instance type might not have sufficient resources for your workload.

3/ Environment Variables: Incorrectly set environment variables could impact memory usage.

AWS
answered a year ago
EXPERT
reviewed 10 months ago
  • mbest
    a year ago

    I doubled back to the code, and pulled things apart more. Turns out memory growth was related to use of Pandas concat(). My test code is using csv input into pandas, the subject code was using ndjson. Not sure what is different inside of Pandas between those two, but they seem to handle memory differently. My test code was fine but the subject code showed continual memory growth. I had to put some explicit 'del <concatted df>' code into the cell and that has stabilized memory usage. Thanks for the guidance.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content