Error when uploading to s3 bucket from sagemaker training job

The issue you're facing is related to Amazon S3 Object Lock and the requirements for uploading objects to buckets with this feature enabled. Let me explain the situation and provide some guidance:

Error Cause: The error occurs because the bucket you're trying to upload to has Object Lock enabled. When Object Lock is active, S3 requires additional integrity checks for uploads.

Content-MD5 or Checksum Requirement: For buckets with Object Lock, S3 mandates either a Content-MD5 header or an x-amz-sdk-checksum-algorithm header for object uploads. This is to ensure data integrity during the upload process.

Multipart Upload: SageMaker typically uses multipart uploads for large files like model artifacts. This is an efficient way to upload large objects, but it also means each part of the upload needs to meet the Object Lock requirements.

SageMaker Configuration: It appears that SageMaker's default configuration for uploading model artifacts doesn't automatically include the required headers for Object Lock-enabled buckets.

To resolve this issue and successfully upload your model artifact, consider the following options:

Disable Object Lock: If possible and if it aligns with your data protection requirements, you could disable Object Lock for the specific bucket used for model artifacts.

Use a Different Bucket: Create a separate S3 bucket without Object Lock for storing model artifacts.

Custom Upload Script: Implement a custom script that handles the model artifact upload with the required headers. This would involve modifying your training script to perform the upload manually using boto3 with the appropriate headers.

SageMaker Configuration: Check if there are any SageMaker configuration options to include the required headers. This might involve working with AWS Support to find a solution, as it's not a commonly documented scenario.

IAM Permissions: Ensure that the IAM role used by SageMaker has the necessary permissions to perform the upload with the required headers.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-managing.html