How do I troubleshoot issues that occur with my Sagemaker AI scheduled notebook jobs?

Resolution

Your notebook job doesn't run during the scheduled time

Scheduled notebook jobs use Amazon EventBridge and SageMaker AI training and pipeline services. If your notebook doesn't run during the scheduled time, then your AWS Identity and Access Management (IAM) role might not have the required permissions.

The IAM role that you used to create and schedule the notebook job is either the SageMaker AI Studio domain role or the role that's attached to the individual user profile in the domain.

To grant permissions to manage Amazon CloudWatch events for your scheduled notebook jobs, add the following policy to your scheduled notebook jobs:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "events:TagResource",
                "events:DeleteRule",
                "events:PutTargets",
                "events:DescribeRule",
                "events:PutRule",
                "events:RemoveTargets",
                "events:DisableRule",
                "events:EnableRule"
            ],
            "Resource": "*",
            "Condition": {
              "StringEquals": {
                "aws:ResourceTag/sagemaker:is-scheduling-notebook-job": "true"
              }
            }
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::*:role/*",
            "Condition": {
                "StringLike": {
                    "iam:PassedToService": "events.amazonaws.com"
                }
            }
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "sagemaker:ListTags",
            "Resource": "arn:aws:sagemaker:*:*:user-profile/*/*"
        }
    ]
}

Note: The preceding policy allows list tags on user profiles to identify notebooks that are tagged as a scheduled job.

The "Create" button is greyed out when you try to create a scheduled notebook job

If the Create button is greyed and the Additional options parameter shows ! when you create a notebook job, then retrieve the HAR file. Review the network capture during the time that your issue occurred to identify the cause. Check for Amazon Virtual Private Cloud (Amazon VPC) misconfigurations or runtime role permission issues.

If you use Amazon VPC to run your notebook job, then make sure that you have the following Amazon VPC endpoints:

• SageMaker AI: To connect to SageMaker AI, see Connect to SageMaker AI through an Amazon VPC interface endpoint.
• Amazon Simple Storage Service (Amazon S3): To connect to Amazon S3, see Gateway endpoints for Amazon S3.
• Amazon Elastic Compute Cloud (Amazon EC2): To connect to Amazon EC2, see Access Amazon EC2 using an interface VPC endpoint.
• EventBridge: Use an EventBridge endpoint only when you set up a scheduled notebook. To connect to EventBridge, see Using Amazon EventBridge with interface Amazon VPC endpoints.

Specify at least one private subnet and security group. If you don't use private subnets, then use another configuration option. For more information, see Requirements to use VPC only mode.

If you configured the preceding endpoints or don't use an Amazon VPC to run the notebook job, then configure the IAM permissions and runtime role permissions.

To configure the IAM permissions, complete the following steps:

1. Open the IAM console.
2. In the navigation pane, choose Users.
3. Select the IAM user that’s associated with your notebook job.
4. On the dropdown menu, choose Add Permissions, and then choose Create inline policy.
5. Choose the JSON tab, and then add the following policy:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "EventBridgeSchedule",
               "Effect": "Allow",
               "Action": [
                   "events:TagResource",
                   "events:DeleteRule",
                   "events:PutTargets",
                   "events:DescribeRule",
                   "events:EnableRule",
                   "events:PutRule",
                   "events:RemoveTargets",
                   "events:DisableRule"
               ],
               "Resource": "*",
               "Condition": {
                   "StringEquals": {
                       "aws:ResourceTag/sagemaker:is-scheduling-notebook-job": "true"
                   }
               }
           },
           {
               "Sid": "IAMPassrole",
               "Effect": "Allow",
               "Action": "iam:PassRole",
               "Resource": "arn:aws:iam::*:role/*",
               "Condition": {
                   "StringLike": {
                       "iam:PassedToService": [
                           "sagemaker.amazonaws.com",
                           "events.amazonaws.com"
                       ]
                   }
               }
           },
           {
               "Sid": "IAMListRoles",
               "Effect": "Allow",
               "Action": "iam:ListRoles",
               "Resource": "*"
           },
           {
               "Sid": "S3ArtifactsAccess",
               "Effect": "Allow",
               "Action": [
                   "s3:PutEncryptionConfiguration",
                   "s3:CreateBucket",
                   "s3:PutBucketVersioning",
                   "s3:ListBucket",
                   "s3:PutObject",
                   "s3:GetObject",
                   "s3:GetEncryptionConfiguration",
                   "s3:DeleteObject",
                   "s3:GetBucketLocation"
               ],
               "Resource": [
                   "arn:aws:s3:::sagemaker-automated-execution-*"
               ]
           },
           {
               "Sid": "S3DriverAccess",
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetObject",
                   "s3:GetBucketLocation"
               ],
               "Resource": [
                   "arn:aws:s3:::sagemakerheadlessexecution-*"
               ]
           },
           {
               "Sid": "SagemakerJobs",
               "Effect": "Allow",
               "Action": [
                   "sagemaker:DescribeTrainingJob",
                   "sagemaker:StopTrainingJob",
                   "sagemaker:DescribePipeline",
                   "sagemaker:CreateTrainingJob",
                   "sagemaker:DeletePipeline",
                   "sagemaker:CreatePipeline"
               ],
               "Resource": "*",
               "Condition": {
                   "StringEquals": {
                       "aws:ResourceTag/sagemaker:is-scheduling-notebook-job": "true"
                   }
               }
           },
           {
               "Sid": "AllowSearch",
               "Effect": "Allow",
               "Action": "sagemaker:Search",
               "Resource": "*"
           },
           {
               "Sid": "SagemakerTags",
               "Effect": "Allow",
               "Action": [
                   "sagemaker:ListTags",
                   "sagemaker:AddTags"
               ],
               "Resource": [
                   "arn:aws:sagemaker:*:*:pipeline/*",
                   "arn:aws:sagemaker:*:*:space/*",
                   "arn:aws:sagemaker:*:*:training-job/*",
                   "arn:aws:sagemaker:*:*:user-profile/*"
               ]
           },
           {
               "Sid": "ECRImage",
               "Effect": "Allow",
               "Action": [
                   "ecr:GetAuthorizationToken",
                   "ecr:BatchGetImage"
               ],
               "Resource": "*"
           }
       ]
   }

To configure the runtime role permissions, complete the following steps:

1. Open the IAM console.
2. In the navigation pane, choose Roles.
3. Select the job runtime role that's associated with your notebook job.
4. Choose the Trust relationships tab.
5. Choose Edit trust policy, and then add the following policy:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Service": [
                       "sagemaker.amazonaws.com",
                       "events.amazonaws.com"
                   ]
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }

To add permissions to access resources, complete the following steps:

1. Open the IAM console.

2. In the left navigation pane, choose Roles.

3. Select the job runtime role that's associated with your notebook job.

4. On the dropdown menu, choose Add Permissions, and then choose Create inline policy.

5. Choose the JSON tab, and then add the following policy:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "PassroleForJobCreation",
               "Effect": "Allow",
               "Action": "iam:PassRole",
               "Resource": "arn:aws:iam::*:role/*",
               "Condition": {
                   "StringLike": {
                       "iam:PassedToService": "sagemaker.amazonaws.com"
                   }
               }
           },
           {
               "Sid": "S3ForStoringArtifacts",
               "Effect": "Allow",
               "Action": [
                   "s3:PutObject",
                   "s3:GetObject",
                   "s3:ListBucket",
                   "s3:GetBucketLocation"
               ],
               "Resource": "arn:aws:s3:::sagemaker-automated-execution-*"
           },
           {
               "Sid": "S3DriverAccess",
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetObject",
                   "s3:GetBucketLocation"
               ],
               "Resource": [
                   "arn:aws:s3:::sagemakerheadlessexecution-*"
               ]
           },
           {
               "Sid": "SagemakerJobs",
               "Effect": "Allow",
               "Action": [
                   "sagemaker:StartPipelineExecution",
                   "sagemaker:CreateTrainingJob"
               ],
               "Resource": "*"
           },
           {
               "Sid": "ECRImage",
               "Effect": "Allow",
               "Action": [
                   "ecr:GetDownloadUrlForLayer",
                   "ecr:BatchGetImage",
                   "ecr:GetAuthorizationToken",
                   "ecr:BatchCheckLayerAvailability"
               ],
               "Resource": "*"
           }
       ]
   }

   Note: In the preceding policy, you can add permissions to other resources that your notebook job must have access to.

6. Choose Review policy. Then, enter a name for your policy.

7. Choose Create policy.

You receive the "Unable to find metadata for image" error

You receive the "Unable to find metadata for image arn in region: example-region" error message.

The preceding error occurs when you don't associate or select the user's image when you create the notebook job. This error can also occur when you try to attach the user's image to the notebook job.

To resolve this issue, reschedule your notebook job at a later time. If the error still occurs, then contact AWS Support.

Related information

Operationalize your Amazon SageMaker Studio notebooks as scheduled notebook jobs

Install policies and permissions for local Jupyter environments