By using AWS re:Post, you agree to the Terms of Use

Insufficient privileges for accessing data in S3 when running a lambda function to create a Personalize dataset import job

1

I am trying to create a lambda function to automate the creation of a dataset import job in Personalize. I followed this guide: https://docs.aws.amazon.com/personalize/latest/dg/granting-personalize-s3-access.html#attaching-s3-policy-to-role

and kept getting the same error saying "Insufficient privileges for accessing data in S3".

Here are the steps I took:

  1. Add AmazonPersonalizeFullAccess to my IAM user
  2. Create a personalizeLambda role with 4 policies:
  • AmazonS3FullAccess
  • CloudWatchLogsFullAccess
  • AmazonPersonalizeFullAccess
  • AWSLambdaBasicExecutionRole

This didn't work with the error above so I added this policy:

  • PersonalizeS3BucketAccessPolicyCustom:

{

"Version": "2012-10-17"
"Id": "PersonalizeS3BucketAccessPolicyCustom",
"Statement": [
    {
        "Sid": "PersonalizeS3BucketAccessPolicy",
        "Effect": "Allow",
        "Action": [
            "s3:*"
        ],
        "Resource": [
            "arn:aws:s3:::<bucket-name>",
            "arn:aws:s3:::<bucket-name>/*"
        ]
    },
    {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": [
            "arn:aws:s3:::<bucket-name>",
            "arn:aws:s3:::<bucket-name>/*"
        ]
    },
    {
        "Effect": "Allow",
        "Action": "lambda:InvokeFunction",
        "Resource": [
            "arn:aws:lambda:<region>:<id>:function:create-personalize-model*",
            "arn:aws:lambda:<region>:<id>:function:create-personalize-dataset-import-job"
        ]
    }
]

}

  1. Create a bucket policy in the S3 bucket that has the dataset files:

{

"Version": "2012-10-17",
"Id": "PersonalizeS3BucketAccessPolicy",
"Statement": [
    {
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::<id>:role/personalizeLambda",
            "Service": "personalize.amazonaws.com"
        },
        "Action": "s3:*",
        "Resource": "arn:aws:s3:::jfna-personalize"
    },
    {
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::<id>:role/personalizeLambda",
            "Service": "personalize.amazonaws.com"
        },
        "Action": "s3:*",
        "Resource": "arn:aws:s3:::jfna-personalize/*"
    }
]

}

I still get the same error no matter how many times I've followed the guide. I would really appreciate it if someone could help figure out what I'm missing or did wrong.

3 Answers
2
Accepted Answer

As noted in another answer it's not totally clear at what point in the chain you're getting the permission error here, but I notice you didn't mention your Personalize execution role.

  1. Your IAM user permissions should be mostly irrelevant (so long as you have access to see what's going on in Personalize and S3 and edit your Lambda).
  2. Your Lambda function will run under an execution role (personalizeLambda per the description) and will need access to whatever Personalize actions it needs to call. specifically, personalize:CreateDatasetImportJob sounds like the main one.
  3. At the point your Lambda calls CreateDatasetImportJob, it will need to pass a Personalize execution role (roleArn in API, role_arn in Python boto3).
    • Personalize needs permission to assume this role (i.e. it should have personalize.amazonaws.com in the trust policy)
    • This role is the one that needs access to S3, because Personalize assumes it to read your data from S3.
    • I believe your Lambda role will also need iam:PassRole permission on the Personalize one, to permit your function to request actions under the Personalize role.

From the description, it sounds like you might be forgetting to set up your Personalize role with S3 access, or just re-using the Lambda role ARN in that slot without adding the extra trust policy to allow Personalize service to use it as well as Lambda... But I could be wrong.

You can find more info about this in the Setting up permissions and Importing bulk records sections of the Personalize Developer Guide.

EXPERT
answered a month ago
  • I agree with everything that Alex_T said. Just one more thought: I think you already did it from your comment above, but it is worth double checking that you added the S3 bucket policy to the bucket you are using to allow Personalize to read data from it ([Giving Amazon Personalize access to Amazon S3 resources] (https://docs.aws.amazon.com/personalize/latest/dg/granting-personalize-s3-access.html))

  • Thank you everyone for your help! I figured that I had all the steps correctly but was using the same IAM role for Lambda and for the role_arn in the Lambda function. As I separated these two, it worked for me. Thank you very much for all your suggestions!

1

Is the error coming from Lambda? The Lambda execution role will need permission to access S3 if it is trying to perform S3 operations. Does this help? https://aws.amazon.com/premiumsupport/knowledge-center/lambda-execution-role-s3-bucket/

answered a month ago
1

For sure your lambda exécution has not enough privilege

Either your user is admin your lambda execution role need to have a policy to redefine the access level of execution

Can you share or have a look at your lambda role policies ?

answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions