Insufficient privileges for accessing data in S3 when running a lambda function to create a Personalize dataset import job

Question

I am trying to create a lambda function to automate the creation of a dataset import job in Personalize. I followed this guide: https://docs.aws.amazon.com/personalize/latest/dg/granting-personalize-s3-access.html#attaching-s3-policy-to-role

and kept getting the same error saying "Insufficient privileges for accessing data in S3".

Here are the steps I took:
1. Add AmazonPersonalizeFullAccess to my IAM user
2. Create a personalizeLambda role with 4 policies: 
- AmazonS3FullAccess
- CloudWatchLogsFullAccess
- AmazonPersonalizeFullAccess
- AWSLambdaBasicExecutionRole
This didn't work with the error above so I added this policy:
- PersonalizeS3BucketAccessPolicyCustom:

{

"Version": "2012-10-17"
    "Id": "PersonalizeS3BucketAccessPolicyCustom",
    "Statement": [
        {
            "Sid": "PersonalizeS3BucketAccessPolicy",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::",
                "arn:aws:s3:::/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": [
                "arn:aws:s3:::",
                "arn:aws:s3:::/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "lambda:InvokeFunction",
            "Resource": [
                "arn:aws:lambda:::function:create-personalize-model*",
                "arn:aws:lambda:::function:create-personalize-dataset-import-job"
            ]
        }
    ]
}

3. Create a bucket policy in the S3 bucket that has the dataset files:

{

"Version": "2012-10-17",
    "Id": "PersonalizeS3BucketAccessPolicy",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam:::role/personalizeLambda",
                "Service": "personalize.amazonaws.com"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::jfna-personalize"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam:::role/personalizeLambda",
                "Service": "personalize.amazonaws.com"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::jfna-personalize/*"
        }
    ]
}

I still get the same error no matter how many times I've followed the guide. I would really appreciate it if someone could help figure out what I'm missing or did wrong.

Accepted Answer

As noted in another answer it's not totally clear at what point in the chain you're getting the permission error here, but I notice you didn't mention your *Personalize execution role*.

1. Your IAM user permissions should be mostly irrelevant (so long as you have access to see what's going on in Personalize and S3 and edit your Lambda).
2. Your Lambda function will run under an execution role (`personalizeLambda` per the description) and will need access to whatever Personalize actions it needs to call. specifically, `personalize:CreateDatasetImportJob` sounds like the main one.
3. At the point your Lambda calls [CreateDatasetImportJob](https://docs.aws.amazon.com/personalize/latest/dg/API_CreateDatasetImportJob.html), it will need to pass a **Personalize execution role** (`roleArn` in API, `role_arn` in Python boto3).
    - Personalize needs permission to assume this role (i.e. it should have `personalize.amazonaws.com` in the trust policy)
    - *This* role is the one that needs access to S3, because Personalize assumes it to read your data from S3.
    - I believe your Lambda role will also need `iam:PassRole` [permission](https://docs.amazonaws.cn/en_us/service-authorization/latest/reference/list_identityandaccessmanagement.html#identityandaccessmanagement-actions-as-permissions) on the Personalize one, to permit your function to request actions under the Personalize role.

From the description, it sounds like you might be forgetting to set up your Personalize role with S3 access, or just re-using the Lambda role ARN in that slot without adding the extra trust policy to allow Personalize service to use it as well as Lambda... But I could be wrong.

You can find more info about this in the [Setting up permissions](https://docs.aws.amazon.com/personalize/latest/dg/aws-personalize-set-up-permissions.html) and [Importing bulk records](https://docs.aws.amazon.com/personalize/latest/dg/bulk-data-import-step.html#bulk-data-import-cli) sections of the Personalize Developer Guide.

Answer

For sure your lambda exécution has not enough privilege

Either your user is admin your lambda execution role need to have a policy to redefine the access level of execution

Can you share or have a look at your lambda role policies ?

Answer

Is the error coming from Lambda? The Lambda execution role will need permission to access S3 if it is trying to perform S3 operations. Does this help? https://aws.amazon.com/premiumsupport/knowledge-center/lambda-execution-role-s3-bucket/

Insufficient privileges for accessing data in S3 when running a lambda function to create a Personalize dataset import job

Relevant content