How do I troubleshoot issues that occur when I use Systems Manager to connect to my Amazon SageMaker endpoint?

Short description

SageMaker uses Systems Manager to provide a secure method to connect to Docker containers that host deployed models for inference. This feature grants shell-level access to the containers and uses Amazon CloudWatch to allow users to debug processes that run within them.

Also, users can establish an AWS PrivateLink connection to the machine learning instances that host their containers, and allow private access to the containers from Systems Manager. This method enhances a user's ability to monitor, troubleshoot, and manage their deployed models with a high level of security.

Note: Turning on SSM access might affect your endpoint performance. It's a best practice to use this feature with your dev or test endpoints instead of your production endpoints. For more information, see the Warning message on Access containers through SSM.

Resolution

Troubleshoot issues that occur when you use Systems Manager to connect to SageMaker based on the following errors:

ValidationException: SSM access is not allowed for your account in the requested region. Please contact customer support to enable this feature

This error occurs when your account isn't allowlisted to use the SageMaker endpoint Systems Manager feature in the requested AWS Region. To resolve this error, contact AWS Support and request your account to be allowlisted to use this feature. If your account isn't allowlisted for access, then you can't create an endpoint with Systems Manager access turned on.

An error occurred (TargetNotConnected) when calling the StartSession operation: <example-endpoint-id>_<example-instance-id> is not connected

This error occurs when you try to connect to a SageMaker endpoint and access the model container with the endpoint's target ID. To troubleshoot this error, complete the following steps:

• Check the AWS Identity and Access Management (IAM) permissions of the IAM user that's connecting to the endpoint. Also, check the permissions of the runtime role that the endpoint is associated with. For more information, see IAM configuration.
• Check the AWS account or Region of the managed node that you want to start a session on. This error occurs if the managed node is located in a different account or Region.
• Check if the instance ID is incorrect or outdated. Instances are regularly replaced throughout the lifecycle of an endpoint. To resolve an incorrect or outdated instance, locate the current instance behind your endpoint and establish a Systems Manager connection to that instance.

Related information

Connecting to SageMaker inference endpoints with SSM on the GitHub website