Issues Launching Custom JupyterLab Image in SageMaker Studio (SageMaker AI)

Question

Hello,
I am trying to attach a custom JupyterLab image to my SageMaker Studio domain, but when I launch a space with this image, it fails to start, and I get this error

ContainerExecutionFailedError: SageMaker is unable to launch the app because the container entrypoint or Dockerfile for the provided image is not configured correctly. Please review the command output and update the image or use a different image to launch the app. Output: [App container prematurely exited with status [exited], exit code [255], and error [].]

**My Dockerfile:**

```
FROM python:3.9-slim

ENV SHELL=/bin/bash
ENV NB_USER=sagemaker-user
ENV NB_UID=1000
ENV NB_GID=100

RUN useradd -m -s /bin/bash -N -u ${NB_UID} ${NB_USER}

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    curl \
    tini \
    && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir \
    jupyterlab \
    ipykernel \
    jupyter-activity-monitor-extension

WORKDIR /home/${NB_USER}
USER ${NB_USER}

ENTRYPOINT ["jupyter-lab"]
CMD ["--ServerApp.ip=0.0.0.0", "--ServerApp.port=8888", "--ServerApp.allow_origin=*", "--ServerApp.token=", "--ServerApp.base_url=/jupyterlab/default"]
```

How can I fix this error, please? And is there a minimal working example Dockerfile that AWS recommends for custom JupyterLab images in Studio?

Ethan_S · Accepted Answer

Hello Hamza!

I understand that you are experiencing issues with using your custom JupyterLab image in SageMaker Studio, receiving an error message that indicates the configuration is incorrect.

-- Findings --

After replicating your Dockerfile, I found that I was able to build it and use is as a JupyterLab image for SageMaker Studio. I would definitely recommend ensuring that you are using the latest version of your image when creating your JupyterLab space. [1]

To check this, you can:

Open the Amazon SageMaker AI console at https://console.aws.amazon.com/sagemaker/.
On the left navigation pane, choose Admin configurations.
Under Admin configurations, choose Images.
Select your Image.
Choose Create version.
For Image source, enter the registry path to the Amazon ECR container image, and ensure the path is using the ':latest' tag.
Select Create.

Once the Image Version is created, you can then go to the 'Environment' settings of your Domain, and attach the latest version for JupyterLab spaces.

From my side, I did create a Shell Script that details the steps for creating and and attaching an image to a SageMaker AI Domain. I used your Dockerfile configuration for reference. As a prerequisite, I would recommend ensuring that Docker is installed in your environment, and that you have the appropriate IAM permissions. [2] [3]

The following script can be ran inside a Notebook Instance with Docker installed:

-- AttachImage.sh --

#!/usr/bin/env bash

REPO_NAME="byoi-jupyter-lab-image"
FILE="/opt/ml/metadata/resource-metadata.json"

if [ -f ${FILE} ]; then
    if [ $(cat ${FILE} | jq .ResourceArn) = *":app/"* ]; then
        echo "Running in a SageMaker Studio Space"
        ACCOUNT_NUMBER=$(aws sts get-caller-identity --query Account --output text)
        REGION=${AWS_DEFAULT_REGION}
        DOMAIN_ID=$(cat /opt/ml/metadata/resource-metadata.json | jq .DomainId)
        EXECUTION_ROLE=$(cat /opt/ml/metadata/resource-metadata.json | jq .ExecutionRoleArn)
        DOCKER_NET="--network sagemaker"
        # Check if in VPCOnly Mode
        if [ -f "/opt/.sagemakerinternal/internal-metadata.json" ]; then
            NETWORK=$(cat /opt/.sagemakerinternal/internal-metadata.json | jq -r .AppNetworkAccessType)
            if [ $NETWORK = 'VpcOnly' ];then
                echo "Cannot pull image when in VPCOnly Mode. Please consider running outside of this domain."
                exit 1
            fi
        fi
    else
        echo "Running in Notebook Instance"
        ACCOUNT_NUMBER=$(aws sts get-caller-identity --query Account --output text)
        REGION=$(aws configure get region)
        read -p "Enter your Studio Domain ID " DOMAIN_ID
        EXECUTION_ROLE=$(aws sagemaker describe-notebook-instance --notebook-instance-name $(cat /opt/ml/metadata/resource-metadata.json | jq -r .ResourceName) | jq -r .RoleArn)
        DOCKER_NET=""
    fi
    echo "Validating Inputs"
    read -p "Is this the correct DomainID? ${DOMAIN_ID} [Y/n]" RES
    RES=${RES:-Y}
    if [[ ! "$RES" =~ ^[Yy]$ ]]; then 
        read -p "Enter your Studio Domain ID" DOMAIN_ID
    fi

    read -p "Is this the correct SageMaker Execution Role? ${EXECUTION_ROLE} [Y/n]" RES
    RES=${RES:-Y}
    if [[ ! "$RES" =~ ^[Yy]$ ]]; then 
        read -p "Enter your SageMaker Execution Role" EXECUTION_ROLE
    fi
    
else
    echo "Not running inside Studio Space or NB Instance. Need to set the following: "
    read -p "Enter your AWS Account ID: " ACCOUNT_NUMBER
    read -p "Enter your AWS Region: " REGION
    read -p "Enter your Studio Domain ID: " DOMAIN_ID
    read -p "Enter your SageMaker Execution Role: " EXECUTION_ROLE
    DOCKER_NET=""
fi


cat > Dockerfile <<- "EOF"
FROM python:3.9-slim

ENV SHELL=/bin/bash
ENV NB_USER=sagemaker-user
ENV NB_UID=1000
ENV NB_GID=100

RUN useradd -m -s /bin/bash -N -u ${NB_UID} ${NB_USER}

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    curl \
    tini \
    && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir \
    jupyterlab \
    ipykernel \
    jupyter-activity-monitor-extension

WORKDIR /home/${NB_USER}
USER ${NB_USER}

ENTRYPOINT ["jupyter-lab"]
CMD ["--ServerApp.ip=0.0.0.0", "--ServerApp.port=8888", "--ServerApp.allow_origin=*", "--ServerApp.token=", "--ServerApp.base_url=/jupyterlab/default"]
EOF

docker build $DOCKER_NET -t $REPO_NAME .

aws ecr create-repository --repository-name $REPO_NAME --region $REGION

docker tag $REPO_NAME $ACCOUNT_NUMBER.dkr.ecr.$REGION.amazonaws.com/$REPO_NAME

aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $ACCOUNT_NUMBER.dkr.ecr.$REGION.amazonaws.com

docker push $ACCOUNT_NUMBER.dkr.ecr.$REGION.amazonaws.com/$REPO_NAME

aws sagemaker create-image --image-name $REPO_NAME --role-arn $EXECUTION_ROLE

sleep 5

aws sagemaker create-image-version --base-image $ACCOUNT_NUMBER.dkr.ecr.$REGION.amazonaws.com/$REPO_NAME:latest --image-name $REPO_NAME

aws sagemaker create-app-image-config --app-image-config-name $REPO_NAME

CUSTOM_IMAGE='{"ImageName":"$REPO_NAME","ImageVersionNumber":1,"AppImageConfigName":"$REPO_NAME"}'
JSON_ARRAY=$(aws sagemaker describe-domain --domain-id ${DOMAIN_ID} --query DefaultUserSettings.JupyterLabAppSettings.CustomImages)
COMBINED=$(echo $JSON_ARRAY | jq '. +=  [{"ImageName":"'${REPO_NAME}'","ImageVersionNumber":1,"AppImageConfigName":"'${REPO_NAME}'"}]')

cat > new-user-settings.json <<- EOF
{
    "DefaultUserSettings": {
        "JupyterLabAppSettings": { 
           "CustomImages": ${COMBINED}
        }
    }
}
EOF

aws sagemaker update-domain --domain-id $DOMAIN_ID --cli-input-json file://new-user-settings.json

-- Examples --

In regards to your question on base examples for custom images, the SageMaker AI documentation provides two examples using different distributions. I have included those below. [4]

-- AMAZON LINUX 2023 --

FROM public.ecr.aws/amazonlinux/amazonlinux:2023

ARG NB_USER="sagemaker-user"
ARG NB_UID=1000
ARG NB_GID=100

# Install Python3, pip, and other dependencies
RUN yum install -y \
    python3 \
    python3-pip \
    python3-devel \
    gcc \
    shadow-utils && \
    useradd --create-home --shell /bin/bash --gid "${NB_GID}" --uid ${NB_UID} ${NB_USER} && \
    yum clean all

RUN python3 -m pip install --no-cache-dir \
    'jupyterlab>=4.0.0,<5.0.0' \
    urllib3 \
    jupyter-activity-monitor-extension \
    --ignore-installed

# Verify versions
RUN python3 --version && \
    jupyter lab --version

USER ${NB_UID}
CMD jupyter lab --ip 0.0.0.0 --port 8888 \
    --ServerApp.base_url="/jupyterlab/default" \
    --ServerApp.token='' \
    --ServerApp.allow_origin='*'

-- SAGEMAKER DISTRIBUTION --

FROM public.ecr.aws/sagemaker/sagemaker-distribution:latest-cpu
ARG NB_USER="sagemaker-user"
ARG NB_UID=1000
ARG NB_GID=100

ENV MAMBA_USER=$NB_USER

USER root

RUN apt-get update
RUN micromamba install sagemaker-inference --freeze-installed --yes --channel conda-forge --name base

USER $MAMBA_USER

ENTRYPOINT ["jupyter-lab"]
CMD ["--ServerApp.ip=0.0.0.0", "--ServerApp.port=8888", "--ServerApp.allow_origin=*", "--ServerApp.token=''", "--ServerApp.base_url=/jupyterlab/default"]

If you still experience issues, I would recommend reaching out to AWS Premium Support through the Support Console, so that we can assist with your resources directly.

References:

re:Post Agent · Answer

Based on your error message and Dockerfile, there are a few issues that need to be addressed to make your custom image work with SageMaker Studio.

The main issue is that your container configuration doesn't match what SageMaker Studio expects. Here are the key changes needed:

1. User configuration: SageMaker Studio JupyterLab applications run as `sagemaker-user` with UID:1001 and GID:101 (not UID:1000 and GID:100 as in your Dockerfile).

2. JupyterLab installation: You need to ensure JupyterLab is properly installed and configured for SageMaker Studio.

3. Entrypoint configuration: Your entrypoint might not be properly configured for SageMaker Studio.

Here's a corrected version of your Dockerfile:

```
FROM python:3.9-slim

ENV SHELL=/bin/bash
ENV NB_USER=sagemaker-user
ENV NB_UID=1001
ENV NB_GID=101

RUN useradd -m -s /bin/bash -N -u ${NB_UID} -g ${NB_GID} ${NB_USER}

RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
curl \
tini \
&& rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir \
jupyterlab \
ipykernel \
jupyter-activity-monitor-extension

WORKDIR /home/${NB_USER}
USER ${NB_USER}

# Don't set the entrypoint here - let SageMaker handle it
```

Alternatively, you could extend a SageMaker Distribution image which already has JupyterLab and other required components pre-configured:

```
FROM public.ecr.aws/sagemaker/sagemaker-distribution:latest-cpu

# Add your custom packages
RUN pip install --no-cache-dir jupyter-activity-monitor-extension
```

After creating your image, you'll need to:
1. Push it to Amazon ECR
2. Create an AppImageConfig
3. Attach the image to your domain

If you need to customize the entrypoint, you can do so through the ContainerConfig when creating your AppImageConfig rather than hardcoding it in the Dockerfile.  
**Sources**  
[Resolve image errors in SageMaker AI Studio Jupyterlab | AWS re:Post](https://repost.aws/knowledge-center/sagemaker-custom-image-errors)  
[(Optional) Migrate custom images and lifecycle configurations - Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-updated-migrate-lcc.html)  
[Update container configuration - Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-updated-byoi-how-to-container-configuration.html)

Hamza · Answer

Unfortunately, this isn't helpful. The Dockerfile fails to build, and even after applying fixes, the underlying issue persists.

lwha · Answer

> User configuration: SageMaker Studio JupyterLab applications run as sagemaker-user with UID:1001 and GID:101 (not UID:1000 and GID:100 as in your Dockerfile).

That's simply not true none of the citations support that.

1. https://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/byoi-specifications.html per the example Dockerfile the UID:1000 & GID:100.
2. Per the Dockerfile as well https://github.com/aws/sagemaker-distribution/blob/main/build_artifacts/v3/v3.6/v3.6.2/Dockerfile