LakeFormation assumed role cannot access s3 objects during Athena requests ("Permission denied on S3 path" )

0

I have Delta Lake tables (using Symlink text input format) catalogued in Glue, stored in a S3 bucket, with all its resources tagged with LakeFormation Tags (for tag-based governance). The problem is that, although the users can see the database, tables, and metadata within Athena's catalogue, they cannot perform queries against the specific tables because of "Permission denied on S3 path" errors.

LakeFormation has the data location registered for the datalake bucket, with AWSServiceRoleForLakeFormationDataAccess role. And this role has IAM permissions automatically added to the resources:

LakeFormationDataAccessServiceRolePolicy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ]
        }
    ]
}

and

LakeFormationDataAccessPolicyForS3
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LakeFormationDataAccessPermissionsForS3",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::MYBUCKET/*"
            ]
        },
        {
            "Sid": "LakeFormationDataAccessPermissionsForS3ListBucket",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::MYBUCKET"
            ]
        }
    ]
}

I have also tried registering the data location with a role with Admin permissions (Action "" and Resources ""), but even so the same error is thrown.

Looking through the CloudTrail logs, I found that LakeFormation passes custom policies to the role when running AssumeRole:

"policy": "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n         {\n           \"Action\": [\"s3:GetObject\"],\n           \"Effect\": \"Allow\",\n           \"Resource\": [\"arn:aws:s3:::MYBUCKET\",\n           \"Condition\": {\"ForAnyValue:StringLike\":{\"s3:prefix\":[\"MYTABLE/_symlink_format_manifest\",\"MYTABLE/_symlink_format_manifest/*\"]}}\n         },\n         {\n           \"Action\": [\"kms:Decrypt\"],\n           \"Effect\": \"Allow\",\n           \"Resource\": [\"*\"],\n           \"Condition\": {\"StringEquals\":{\"kms:ViaService\":[\"s3.us-east-2.amazonaws.com\"]}}\n         }  ]\n}"
    }

This seems like a malformatted json string that is being passed to the assumed role. Can this be causing the errors I'm having? And does anyone have had this issue before?

PS: I have manually removed ACL control over the S3 bucket and objects. Still same behavior. The error is not shown if I remove the data location, and Athena ignores Lake Formation.

  • Do you mean that you have a table in AWS Lake Formation with the "location" pointing to manifest file(s)?

  • No, it's pointing to a folder with parquet files, and manifests in subfolders.

  • Just saw in the Lake Formation documentation that it does not support queries that use manifests.

  • facing the same issue and my steps are exactly the same as yours and being completely stucked with it...

1개 답변
0

Are your bucket and objects encrypted? If your bucket and objects are encrypted, I would make sure Lakeformation has the proper access to use the KMS Key (KMS Key Policy).

jsonc
답변함 2년 전
  • They are encrypted with AWS managed keys. I've added the specific KMS permissions to the access role I'm using, but still ends up denying the access. I've tried adding full KMS access, still nothing.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인