OCI format unsupported when creating a Sagemaker model

Question

When attempting to create a model from a custom container using the SDK, sagemaker returns an error that the OCI manifest media type is not supported:
```
software.amazon.awssdk.services.sagemaker.model.SageMakerException: Unsupported manifest media type application/vnd.oci.image.manifest.v1+json for image /<img>:1.0.0. Ensure that valid manifest media type is used for specified image. (Service: SageMaker, Status Code: 400, Request ID: *****)
```

The source image is built by a Kaniko process, which (as far as I know) doesn't provide an option to force the image to use Docker format.

Testing an image that I have built locally with `podman build --format Docker` results in success - no errors from Sagemaker.

Why doesn't ECR handle translation between manifest media types transparently, if Sagemaker requires a docker-formatted manifest?

Answer

Good Day @mclark,

Thanks for reaching out to us at AWS re:Post. Ramneek this side from ECR Support Team and here to assist further on the question you asked here.

As from the post, I can understand that you're trying to create sagemaker model to use custom image built using Kaniko [1] and while doing so, model returns below error when tried building your custom image via SDK. You want to understand why ECR didn't handle translation between manifest media types, if Sagemaker requires a docker-formatted manifest. The same is working locally when using podman tool and model is successfully built on sagemaker.

``
software.amazon.awssdk.services.sagemaker.model.SageMakerException: Unsupported manifest media type application/vnd.oci.image.manifest.v1+json for image /<img>:1.0.0. Ensure that valid manifest media type is used for specified image. (Service: SageMaker, Status Code: 400, Request ID: *****)
``

Please correct me if I have misunderstood your query.

As I have explored [1], I also can't see any explicit flag to format the image to Docker format, however please note that when you push and pull images to and from Amazon ECR, your container engine client (for example, Docker) communicates with the registry to agree on a manifest format that is understood by the client and the registry to use for the image.

And there are some compatible translation of manifests only available at Amazon ECR Service as listed at [2]. As I can note that there's no translation available at the moment where if image is pushed for OCI.

I hope I was able to address your query on the post, in case of any follow-up queries/clarifications. Please proceed to open a case with AWS Support Engineering at [3]. Thanks for your patience & cooperation!

References:

[1] https://github.com/GoogleContainerTools/kaniko

[2] https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-manifest-formats.html#image-manifest-conversion

[3] https://console.aws.amazon.com/support/home#/case/create?issueType=technical

OCI format unsupported when creating a Sagemaker model

Relevant content