How can I determine the SageMaker notebook instance that made a particular API call if all the instances use the same IAM role?

Short description

When you have multiple SageMaker instances with the same IAM role, a CloudTrail event doesn't identify the instance that performed an API action.

The following example CloudTrail event log shows the principalId as "SageMaker". By default, principalId doesn't identify specific SageMaker instances.

{    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AAAAAAAAAAAAAAAAAA:SageMaker",

Resolution

To create a notebook script that distinguishes between instances, take the following steps:

1. Create an IAM execution role for the SageMaker notebook instance. Or, use an execution role that already exists. In the following steps, the Amazon Resource Name (ARN) for the execution role is arn:aws:iam::111122223333:role/service-role/AmazonSageMaker-ExecutionRole. Your ARN for the execution role will differ.

2. Attach an IAM policy that includes sts:AssumeRole to the execution role. The sts:AssumeRole action allows the execution role to assume itself with a different session name.

   Example:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "sts:AssumeRole",
         "Resource": "arn:aws:iam::111122223333:role/service-role/AmazonSageMaker-ExecutionRole"
       }
     ]
   }

   3. Verify that the execution role's Trust Policy includes the following permissions.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "sagemaker.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       },
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::111122223333:root"
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringLike": {
             "aws:PrincipalArn": "arn:aws:iam::111122223333:role/service-role/AmazonSageMaker-ExecutionRole"
           }
         }
       }
     ]
   }

3. Create a Start notebook lifecycle configuration script similar to the following. This example script retrieves the notebook instance name and then uses the name as the session name. For more information, see Customization of a SageMaker notebook instance using an LCC script.

   
   #Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
   #Permission is hereby granted, free of charge, to any person obtaining a copy of this
   #software and associated documentation files (the "Software"), to deal in the Software
   #without restriction, including without limitation the rights to use, copy, modify,
   #merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
   #permit persons to whom the Software is furnished to do so.
   
   #THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
   #INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
   #PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
   #HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
   #OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
   #SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
   
   #!/bin/bash
   
   set -ex
   
   # Obtain the name of the notebook instance
   nbname=$(jq -r '.ResourceName' /opt/ml/metadata/resource-metadata.json)
   echo "Notebook Name = $nbname"
   
   # Use the AWS Command Line Interface (AWS CLI) to obtain the Amazon Resource Name (ARN) of the IAM execution role
   nbinfo=$(aws sagemaker describe-notebook-instance --notebook-instance-name $nbname)
   nbrole=$(jq -r '.RoleArn' <<< "$nbinfo")
   echo "Notebook Role = $nbrole"
   
   # Obtain the Region of the notebook instance
   nbregion=$(aws configure get region)
   echo "Notebook Region = $nbregion"
   
   # Write Assume Role Provider Settings to a new config file
   echo "Writing new config file"
   cat > /home/ec2-user/.aws/config.new <<EOF1
   [default]
   region=$nbregion
   role_arn = $nbrole
   credential_source = Ec2InstanceMetadata
   role_session_name = $nbname
   sts_regional_endpoints = regional
   EOF1
   
   echo "Moving new config to config file"
   sudo mv /home/ec2-user/.aws/config.new /home/ec2-user/.aws/config
   
   # Secure the "config" file so that it can't be deleted/updated without root user permissions
   sudo chattr +i /home/ec2-user/.aws/config

4. Next, create a SageMaker notebook instance (for example, test-2), and then attach the lifecycle configuration script that you created in the previous step under Additional configuration.
   Note: To create the above notebook instance with root access turned off, see Use Control root access to a SageMaker notebook instance. After you create the Notebook Instance, set the RootAccess field to Disabled. This action prevents the user from deleting or updating the config files.

Identify the notebook instance in CloudTrail

To identify the notebook instance that performed an API action, check the CloudTrail event. Under the userIdentity object, the principalId and arn show the name of the notebook instance.
For example, the following event detail shows that the SageMaker notebook instance test-2 made the API call.

{
  "eventVersion": "1.05",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AAAAAAAAAAAAAAAAAAAA:test-2",
    "arn": "arn:aws:sts::111122223333:assumed-role/AmazonSageMaker-ExecutionRole/test-2",
    "accountId": "111122223333",
    "accessKeyId": "AAAAAAAAAAAAAAAAAAAA",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "AAAAAAAAAAAAAAAAAAAA",
        "arn": "arn:aws:iam::111122223333:role/service-role/AmazonSageMaker-ExecutionRole",
        "accountId": "111122223333",
        "userName": "AmazonSageMaker-ExecutionRole"
      },
      "webIdFederationData": {},
      "attributes": {
        "mfaAuthenticated": "false",
        "creationDate": "2020-09-12T00:45:04Z"
      }
    },
    "invokedBy": "im.amazonaws.com"
  },
  "eventTime": "2020-09-12T00:49:04Z",
  "eventSource": "sagemaker.amazonaws.com",
  "eventName": "CreateEndpoint",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "im.amazonaws.com",
  "userAgent": "im.amazonaws.com",
  "requestParameters": {
    "endpointName": "sagemaker-mxnet-ep",
    "endpointConfigName": "sagemaker-mxnet-epc",
    "tags": []
  },
  "responseElements": {
    "endpointArn": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/sagemaker-mxnet-ep"
  },
  "requestID": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "eventID": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "eventType": "AwsApiCall",
  "recipientAccountId": "111122223333"
}

Related information

How to use SageMaker execution roles