Request to service failed. If failure persists after retry, contact customer support.- sage maker

When a SageMaker serverless endpoint creation fails while the same model works fine with a provisioned endpoint, there are several potential causes to investigate:

1. Container permissions issues: Containers that work for instance-based endpoints may lack necessary permissions in the serverless environment. This can lead to ping health check failures. To fix this, you may need to modify your Dockerfile to add read, write, and execute permissions for "other" on relevant files or directories using: RUN chmod o+rwX <file or directory name>, then rebuild and upload the container to ECR.

2. Container structure problems: Ensure your inference code (typically 'inference.py') is correctly located in a 'code/' subfolder within your model tarball. Incorrect file structure is a common cause of deployment failures.

3. Missing required files: Check if essential files like '/usr/bin/serve' are present in your container. This was identified as one of the common issues in similar cases.

4. Model artifact packaging: Verify that your model tarball extracts correctly and doesn't create unexpected subfolders. A missing "/" in paths can prevent models from being properly copied to /opt/ml/model.

5. Script location: Ensure your custom script is in /opt/ml/code rather than just /opt/ml.

6. Resource limits: You might be hitting account-level service limits for serverless endpoints. Check your quotas in the Service Quotas console and request an increase if needed.

7. Logging issues: Enable more verbose logging by setting the environment variable "PYTHONUNBUFFERED" to "1" when creating your model to help diagnose the issue.

If the problem persists, consider testing your inference code locally before deploying to save time, and ensure your container can properly respond to SageMaker's health check requests.
Sources
Sagemaker endpoint creation failing | AWS re:Post
Resolve errors when creating a model evaluation job in Amazon SageMaker AI - Amazon SageMaker AI
Troubleshooting - Amazon SageMaker AI
Failed to create endpoint in sagemaker without any reason and logs | AWS re:Post

Hello,

Thank you for reaching out.

I understand you're encountering difficulties while attempting to create a SageMaker endpoint.

The error you're experiencing could be attributed to several factors, including:

 1. Account service limits.
 2. Transient issue.
 3. Dependency failures.
 4. Container configuration problems.
 5. Other deployment-related settings.

To provide you with the most effective assistance, I would appreciate if you could share the code in question for replication purposes.

Given the technical nature of this issue, I recommend opening a support ticket if you have an active AWS Support plan. This will help us to conduct a thorough investigation and provide more targeted assistance in resolving your deployment concerns.