How do I resolve Amazon S3 AccessDenied errors in Amazon SageMaker training jobs?

3 minute read
0

My Amazon SageMaker training job failed with an AccessDenied error, even though the AmazonSageMakerFullAccess policy is attached to the execution role.

Short description

AccessDenied errors indicate that your AWS Identity and Access Management (IAM) policy doesn't allow one or more the following Amazon Simple Storage Service (Amazon S3) actions:

  • s3:ListBucket
  • s3:GetObject
  • s3:PutObject

The permissions that you need depend on the SageMaker API that you're calling. For example, the only Amazon S3 action that the CreateModel API requires is s3:GetObject. However, the CreateTrainingJob API requires s3:GetObject, s3:PutObject, and s3:ListObject. For more information about the permissions that are required for each API, see SageMaker roles.

Resolution

AccessDenied errors commonly happen in the following scenarios.

Encrypted input bucket

If the data in the S3 bucket is encrypted with AWS Key Management Service (AWS KMS):

  • Be sure that the IAM policy that's attached to the execution role allows the kms:encrypt and kms:decrypt actions. For more information, see SageMaker roles.
  • Be sure that the AWS KMS key policy grants access to the IAM role. For more information, see Using key policies in AWS KMS.
  • If using an AWS KMS key for the machine learning (ML) storage volume in the resource configuration of your job, the IAM policy must allow kms:CreateGrant action. For more information, see Using grants. For more information about encrypting the ML storage volume, see Protect data at rest using encryption.
  • When using the Python SDK and implementing an abstraction of the estimator.EstimatorBase class, you must pass the output_kms_key and volume_kms_key parameters through kwargs keyword arguments. This must be done regardless of their documented presence in the inheriting class. For more information, see Estimators.

Permissions boundaries

If you define permissions boundaries for the execution role, then SageMaker can execute only the actions that are allowed by both the IAM policy and the permissions boundaries. Be sure that the IAM policy and the permissions boundaries allow the required Amazon S3 actions.

Bucket policies

If the input bucket uses a bucket policy, then be sure that the bucket policy allows the execution role to perform the required Amazon S3 actions. For more information about bucket policies, see Policies and permissions in Amazon S3.

Here's an example of a bucket policy that denies access to the SageMaker execution role and causes an AccessDenied error:

{
  "Version": "2012-10-17",
  "Id": "ExamplePolicy01",
  "Statement": [
    {
      "Sid": "ExampleStatement01",
      "Effect": "Deny",
      "Principal": {
        "AWS": "arn:aws:iam::Account-ID:role/SageMakerExecutionRole"
      },
      "Action": [
        "s3:GetObject",
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::awsdoc-example-bucket/*",
        "arn:aws:s3:::awsdoc-example-bucket"
      ]
    }
  ]
}

Cross-account Amazon S3 access

If a different AWS account owns the Amazon S3 data:

  • Be sure that both accounts have access to the AWS KMS key. If you don't specify an AWS KMS key for the training job, then SageMaker defaults to an Amazon S3 server-side encryption key. A default Amazon S3 server-side encryption key can't be shared with or used by another AWS account.
  • Be sure that the IAM policy for the SageMaker execution role and the S3 bucket policy have cross-account permissions.

For more information, see How can I deploy an Amazon SageMaker model to a different AWS account?


Related information

How do I troubleshoot 403 Access Denied errors from Amazon S3?

AWS OFFICIAL
AWS OFFICIALUpdated a year ago