Skip to content

SageMaker Pipeline creation stuck at pipeline.upsert() without error

0

Hi, I'm working on to create an SageMaker Pipeline with two processing phases in a jupyter notebook. In the phases, my goal is to process files with an generative ai model. When I run my pipeline creation code, the process gets stuck indefinitely at two lines:

definition = json.loads(pipeline.definition())
print(definition)

or

pipeline.upsert(role_arn=role)

Here's my code (without sensitive information):

!pip install sagemaker
!pip install boto3

import sagemaker
from sagemaker.processing import ScriptProcessor, ProcessingInput, ProcessingOutput
from sagemaker.workflow.steps import ProcessingStep
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.parameters import ParameterString
from sagemaker.workflow.pipeline_context import PipelineSession
import boto3

sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()
pipeline_session = PipelineSession()

S3_BUCKET_NAME = "my-s3-bucket"

PHASE1_INSTANCE_TYPE = "ml.g5.2xlarge"
PHASE1_IMAGE_URI = "<PHASE1_IMAGE_URI>"
PHASE1_SCRIPT_FILE_PATH = "phase1_script.py"

PHASE2_INSTANCE_TYPE = "ml.t3.medium"
PHASE2_IMAGE_URI = "phase2_image_uri"
PHASE2_SCRIPT_FILE_PATH = "phase2_script.py"

input_data = ParameterString(
    name="InputDataUri",
    default_value=f's3://{S3_BUCKET_NAME}/raw_pdf_files/'
)
phase1_output = ParameterString(
    name="Phase1OutputUri",
    default_value=f's3://my-s3-bucket/phase1_md_outputs/'
)
phase2_output = ParameterString(
    name="Phase2OutputUri", 
    default_value=f's3://my-s3-bucket/phase2_json_outputs/'
)

# Phase 1:
phase1_processor = ScriptProcessor(
    image_uri=PHASE1_IMAGE_URI,  
    command=["python3"],
    instance_type=PHASE1_INSTANCE_TYPE,
    instance_count=1,
    role=role,
    sagemaker_session=pipeline_session
)

phase1_job_args = phase1_processor.run(
    code=PHASE1_SCRIPT_FILE_PATH,
    inputs=[
        ProcessingInput(
            source=input_data,
            destination="/opt/ml/processing/input",
            s3_data_type="S3Prefix",
            s3_input_mode="File"
        )
    ],
    outputs=[
        ProcessingOutput(
            output_name="phase1_output",
            source="/opt/ml/processing/output",
            destination=phase1_output,
            s3_upload_mode="EndOfJob"
        )
    ],
    wait=False
)

phase1_processing_step = ProcessingStep(
    name="Phase1",
    step_args=phase1_job_args
)

# Phase 2
phase2_processor = ScriptProcessor(
    image_uri=PHASE2_IMAGE_URI,
    command=["python3"],
    instance_type=PHASE2_INSTANCE_TYPE,
    instance_count=1,
    role=role,
    sagemaker_session=pipeline_session
)

phase2_job_args = phase2_processor.run(
    code=PHASE2_SCRIPT_FILE_PATH, 
    inputs=[
        ProcessingInput(
            source=phase1_output,
            destination="/opt/ml/processing/input",
            s3_data_type="S3Prefix",
            s3_input_mode="File"
        )
    ],
    outputs=[
        ProcessingOutput(
            output_name="phase2_output",
            source="/opt/ml/processing/output",
            destination=phase2_output,
            s3_upload_mode="EndOfJob"
        )
    ],
    wait=False
)

phase2_processing_step = ProcessingStep(
    name="Phase2",
    step_args=phase2_job_args
)

pipeline = Pipeline(
    name="My_Pipeline_Name",
    parameters=[input_data, phase1_output, phase2_output],
    steps=[phase1_processing_step, phase2_processing_step],
    sagemaker_session=pipeline_session,
)

import json
definition = json.loads(pipeline.definition())
print(definition)

pipeline.upsert(role_arn=role)

Thanks for your time.

asked a year ago461 views

1 Answer
1

It seems like you're experiencing an issue where your SageMaker Pipeline creation is getting stuck without producing an error. This can happen for a few reasons, and there are several steps you can take to troubleshoot and potentially resolve the issue:

  1. Check your IAM role permissions: Ensure that the IAM role you're using (specified by the 'role' variable) has the necessary permissions to create and update SageMaker pipelines. It should have appropriate access to SageMaker resources and the S3 buckets you're using.

  2. Validate your pipeline definition: Before calling 'upsert', try to print out the pipeline definition to make sure it's correctly formed. You can do this with:

    print(pipeline.definition())

    This will show you the raw JSON definition, which you can inspect for any obvious issues.

  3. Use explicit create or update: Instead of 'upsert', try explicitly creating or updating the pipeline. If the pipeline doesn't exist, use:

    pipeline.create(role_arn=role)

    If it already exists and you want to update it:

    pipeline.update(role_arn=role)

  4. Check for existing pipelines: Make sure there isn't already a pipeline with the same name that might be causing conflicts.

  5. Examine CloudWatch Logs: If the operation is actually running but taking a long time, you might find more information in the CloudWatch logs.

  6. Use a unique client request token: When creating or updating the pipeline, you can provide a unique client request token to ensure idempotency. This might help if there are any underlying issues with repeated requests.

  7. Check your network connection: Ensure you have a stable network connection to AWS services.

  8. Verify your AWS credentials: Make sure your AWS credentials are correct and haven't expired.

If none of these steps resolve the issue, you may want to try creating a minimal pipeline definition to isolate the problem. Start with a single step and gradually add complexity until you identify where the issue occurs.

Remember to handle any exceptions that might be raised during the pipeline creation or update process. This can provide more information about what's going wrong.
Sources
update_pipeline - Boto3 1.34.113 documentation
create_pipeline - Boto3 1.34.113 documentation
Troubleshooting Amazon SageMaker Pipelines - Amazon SageMaker AI
Tutorial - Automate Machine Learning Workflows - Amazon Web Services

answered a year ago

EXPERT

reviewed a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.