Skip to content

Failed to create endpoint in sagemaker without any reason and logs

0

I am trying to constantly deploy my model to sagemaker using sagemaker-python-sdk. Everything is working perfect except the cloud side. For the first 20 minutes, the endpoint is hanging in the "creating" state. Than it is falling into "failed" state with the following error:

The primary container for production variant AllTraffic did not pass the ping health check. Please check CloudWatch logs for this endpoint.

I am pressing CloudWatch link and I am seeing multiple spammed errors:

Please specify --force/-f option to overwrite the model archive output file. See -h/--help for more details./.sagemaker/mms/models/model ERROR - %s already exists.

Those lines are copied more than 1000 times. No logs from sagemaker or from machine that is going to boot up. What happened?

My sagemaker endpoint: pytorch-inference-2025-03-12-10-11-20-638, endpoint logs: /aws/sagemaker/endpoints/pytorch-inference-2025-03-12-10-11-20-638.

Why the sagemaker service is completely not working? Why there are no logs for both sagemaker and instance machine? Why there are no answers to this question starting from 2023 in sagemaker-python-sdk repo?

See also https://github.com/aws/sagemaker-python-sdk/discussions/3996.

1 Answer
1

The issue you're experiencing with your SageMaker endpoint creation failing is likely related to problems with your container configuration or model deployment process. Let's address your concerns and provide some troubleshooting steps:

  1. Ping Health Check Failure: The error message indicating that the primary container failed the ping health check suggests that there's an issue with your container or the inference code. This could be due to various reasons such as incorrect file structure, permissions issues, or problems with the container startup process.

  2. CloudWatch Logs: The logs you're seeing about specifying the --force/-f option to overwrite the model archive output file indicate that there might be a problem with how your model artifacts are being packaged or deployed. This could be causing conflicts during the model loading process.

  3. Lack of Logs: The absence of more detailed logs from SageMaker or the instance machine can make troubleshooting challenging. This could be due to the container crashing before it can write logs, or issues with log configuration.

To address these issues and improve your endpoint creation process:

  1. Check your container structure: Ensure that your inference code (typically 'inference.py') is located in a 'code/' subfolder within your model tarball.

  2. Review model artifact packaging: When creating your model tarball, make sure it extracts correctly to the current directory and doesn't create unexpected subfolders.

  3. Test locally: Before deploying to SageMaker, test your inference code locally using SageMaker Local Mode with 'instance_type='local''. This can help identify issues before cloud deployment.

  4. Increase logging verbosity: Add more detailed logging in your inference code to capture any errors occurring during container startup or model loading.

  5. Check IAM roles and permissions: Ensure that the execution role has all necessary permissions, including access to required AWS services like S3 for model artifacts.

  6. Verify container health check: Make sure your container can respond to the health check requests SageMaker sends.

  7. Review resource allocation: For multi-container endpoints, ensure you've chosen an instance type with sufficient memory and CPU to handle the load.

If the issue persists, you may need to engage AWS support for more in-depth troubleshooting. The lack of recent answers in the sagemaker-python-sdk repo doesn't necessarily indicate a widespread issue with the service, but rather that your specific problem might be unique or require more context to diagnose.

Sources
Sagemaker endpoint creation failing | AWS re:Post
SageMaker Python SDK Troubleshooting Guide - Amazon SageMaker AI
Troubleshoot multi-container endpoints - Amazon SageMaker AI
Troubleshoot Amazon SageMaker AI model deployments - Amazon SageMaker AI

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.