Troubleshooting instance type changes and pipeline execution failures in Amazon SageMaker Unified Studio

Overview

When teams move Amazon SageMaker Unified Studio (SMUS) projects from exploration to production-adjacent machine learning (ML) workloads, two failure classes appear that rarely show up during initial notebook work: blocked instance type changes, and SageMaker Pipelines steps that fail even though the same code runs interactively. This article maps each symptom to its root cause and gives the commands to confirm and fix it.

Symptoms

• Changing a JupyterLab or Code Editor space to a larger or GPU (graphics processing unit) instance type fails, or the space does not start after the change.
• ResourceLimitExceeded when starting a space or a training job on an accelerated instance type in a new account.
• AccessDeniedException on sagemaker:CreateTrainingJob, sagemaker:CreateApp, or sagemaker:UpdateSpace only for certain instance types.
• A pipeline step that runs Spark on Amazon EMR Serverless fails because the referenced application no longer exists.
• Amazon Simple Storage Service (Amazon S3) reads succeed from a notebook but fail with Access Denied inside a pipeline step.

First, separate space compute from job compute

The instance type of a SMUS space (JupyterLab or Code Editor) and the instance type of a training, processing, or transform job are independent settings. A space running on ml.m5.xlarge can submit a training job on ml.g5.12xlarge — the job requests its own instances. If a job fails on instance type, the space configuration is not the cause; go to root causes 2 and 3.

from sagemaker.estimator import Estimator

est = Estimator(
    image_uri=image_uri,
    role=execution_role_arn,          # job role, set explicitly
    instance_type="ml.g5.12xlarge",   # job compute, independent of the space
    instance_count=1,
    output_path=f"s3://{bucket}/output/",
)

Root cause 1 — Space instance type change blocked

A space instance type can be changed in place. Stop the space (or notebook compute), choose Configure, select the new instance type, and start it again. See Manage compute environments and Managing configurations. Increased Amazon Elastic Block Store (Amazon EBS) volume size cannot be decreased later.

When the change fails, the cause is almost always one of the following:

1. App quota for the target instance type. Spaces have their own per-instance-type quotas, separate from training job quotas. A value of 0 for the target type returns ResourceLimitExceeded when the space starts.

   aws service-quotas list-service-quotas --service-code sagemaker \
     --query "Quotas[?contains(QuotaName,'JupyterLab') && contains(QuotaName,'ml.g5')].[QuotaName,QuotaCode,Value]" \
     --output table

2. AWS Identity and Access Management (IAM) condition on instance type. A policy using sagemaker:InstanceTypes on sagemaker:CreateApp or sagemaker:UpdateSpace blocks the change for the project role. See root cause 2 for diagnosis.
3. Space still running. The configuration change applies only after the space is stopped and restarted.

Root cause 2 — IAM policy restricts instance families

Regulated environments commonly limit instance families in development with the sagemaker:InstanceTypes condition key. When the same role later requests ml.g5.* or ml.p4d.*, the call fails with AccessDeniedException, not a capacity error. A typical guardrail looks like this:

{
  "Effect": "Deny",
  "Action": ["sagemaker:CreateTrainingJob", "sagemaker:CreateProcessingJob", "sagemaker:CreateApp"],
  "Resource": "*",
  "Condition": {
    "ForAnyValue:StringNotLike": { "sagemaker:InstanceTypes": ["ml.m5.*", "ml.t3.*"] }
  }
}

Confirm which identity-based policy is responsible with the IAM policy simulator:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::<ACCOUNT_ID>:role/<EXECUTION_ROLE> \
  --action-names sagemaker:CreateTrainingJob \
  --context-entries "ContextKeyName=sagemaker:InstanceTypes,ContextKeyValues=ml.g5.12xlarge,ContextKeyType=stringList" \
  --query "EvaluationResults[].[EvalActionName,EvalDecision,MatchedStatements[].SourcePolicyId]"

An explicitDeny or implicitDeny result identifies the policy to update. Also read the full error message: if it states the request was denied by a service control policy (SCP), the restriction is set at the AWS Organizations level, and changing the role policy alone will not resolve it.

Root cause 3 — Training quota not raised for accelerated instances

Accelerated instance quotas for training jobs are frequently 0 or very low in newly created accounts. The first pipeline run that requests one fails immediately with ResourceLimitExceeded, regardless of SMUS configuration. See Troubleshoot the ResourceLimitExceeded error in Amazon SageMaker.

1. List the instance types the pipeline uses:

aws sagemaker describe-pipeline --pipeline-name <PIPELINE_NAME> \
  --query PipelineDefinition --output text \
  | jq -r '.. | objects | .InstanceType? // empty' | sort -u

2. Look up the quota code and current value for each type (quota codes differ per instance type, so look them up rather than hard-coding):

aws service-quotas list-service-quotas --service-code sagemaker \
  --query "Quotas[?QuotaName=='ml.g5.12xlarge for training job usage'].[QuotaCode,Value]" \
  --output table

3. Request an increase before promoting the pipeline:

aws service-quotas request-service-quota-increase \
  --service-code sagemaker --quota-code <QUOTA_CODE> --desired-value 2

4. Add a pre-flight check so the pipeline fails fast before any job is created. A Lambda step can call Service Quotas:

import boto3

sq = boto3.client("service-quotas")

def handler(event, context):
    missing = []
    for itype in event["instance_types"]:
        name = f"{itype} for training job usage"
        paginator = sq.get_paginator("list_service_quotas")
        quota = next((q for page in paginator.paginate(ServiceCode="sagemaker")
                      for q in page["Quotas"] if q["QuotaName"] == name), None)
        if quota is None or quota["Value"] < 1:
            missing.append(itype)
    if missing:
        raise RuntimeError(f"No training quota for: {missing}")
    return {"status": "ok"}

The Lambda function's role needs servicequotas:ListServiceQuotas.

Root cause 4 — Pipeline step references a deleted EMR Serverless application

Pipeline steps that run Spark on EMR Serverless reference a specific application ID. If that application is deleted (for example, by an automated cleanup job) or replaced when project compute is re-added, the step fails because the ID no longer resolves.

aws emr-serverless get-application --application-id <APPLICATION_ID> \
  --query "application.[name,state,releaseLabel]"

aws emr-serverless list-applications \
  --query "applications[?state!='TERMINATED'].[id,name,releaseLabel]" --output table

Fix: If the application is missing or terminated, add the EMR Serverless compute to the project again from the project's compute page, then update the application ID in the pipeline definition and re-run with upsert(). Avoid hard-coding the ID; pass it as a pipeline parameter so it can be changed without editing step code:

from sagemaker.workflow.parameters import ParameterString

emr_app_id = ParameterString(name="EmrServerlessAppId")
# reference emr_app_id in the step, then:
pipeline.start(parameters={"EmrServerlessAppId": "<CURRENT_APPLICATION_ID>"})

Root cause 5 — Notebook and pipeline step use different IAM roles

Interactive notebook code runs with the project role. Each pipeline step job runs with the role passed to its estimator, processor, or transformer. The pipeline's own role_arn (supplied to upsert()) is the role SageMaker Pipelines uses to create the step jobs, and it needs iam:PassRole on each step role. If the cross-account S3 bucket trusts only the project role, reads succeed interactively and fail in the pipeline.

1. Compare the identities actually used:

# Interactive identity
import boto3
print(boto3.client("sts").get_caller_identity()["Arn"])

# Role the failing step job ran with
sm = boto3.client("sagemaker")
print(sm.describe_training_job(TrainingJobName="<JOB_NAME>")["RoleArn"])
# use describe_processing_job(...)["RoleArn"] for processing steps

2. Grant the step role on both sides of the cross-account boundary. The bucket policy in the data account must allow the step role, and the step role's identity policy in the ML account must allow the bucket. If the bucket uses server-side encryption with AWS Key Management Service (SSE-KMS), the AWS KMS key policy must also allow the step role.

{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::<ML_ACCOUNT_ID>:role/<STEP_EXECUTION_ROLE>" },
  "Action": ["s3:GetObject", "s3:PutObject"],
  "Resource": "arn:aws:s3:::<BUCKET>/<PREFIX>/*"
}

3. Set role= explicitly on every estimator and processor rather than relying on get_execution_role(), so the identity used in production is deliberate and auditable.

Diagnostic checklist

1. Is the error AccessDeniedException (IAM or SCP, root cause 2) or ResourceLimitExceeded (quota, root causes 1 and 3)? The fixes differ.
2. Is the failure on a space or on a job? Space and job instance types and quotas are separate.
3. Does the error message mention a service control policy?
4. Is the quota for the exact instance type and usage (training job, space app) greater than 0?
5. Does the EMR Serverless application ID referenced by the step still exist?
6. Which role did the failing step job run with (RoleArn on the job), and does the bucket, identity, and KMS key policy allow it?

Production-readiness checklist

Check	How to verify
Training quota > 0 for every instance type in the pipeline	describe-pipeline + list-service-quotas
Space app quota covers the target space instance type	list-service-quotas filtered on JupyterLab or Code Editor
Step roles allowed for the target instance families	simulate-principal-policy with sagemaker:InstanceTypes
No SCP blocks the target instance families	Review SCPs attached to the account's organizational unit (OU)
EMR Serverless application ID passed as a parameter and active	emr-serverless get-application
role= set explicitly on every step	Review pipeline code; confirm RoleArn on a test job
Cross-account bucket, identity, and KMS policies allow each step role	Run one test step that reads and writes the target prefix

Related resources

• Manage compute environments — Amazon SageMaker Unified Studio User Guide
• Troubleshoot the ResourceLimitExceeded error in Amazon SageMaker
• CreateTrainingJob — Amazon SageMaker API Reference
• Testing IAM policies with the IAM policy simulator

Authors

Sandip Samanta (AWS), Krish Balaraman (AWS), Kiran Khambete (AWS)