Glue Role got S3 Access Denied when writing to underlying S3 data in Lake Formation

Question

Hello, I have an ETL workflow and keep getting S3 Access Denied when the Glue Role write a data frame to a destination S3 bucket (registered in Lake Formation).  Here is my current setup. Does Lake Formation vend credentials to the Glue Role for writing permission to S3? How to let the Glue Role write to S3 via Lake Formation?

- Registered a S3 bucket to Lake Formation 
- Grant Data Location to the Glue Role 
- Grant Lake Formation Permissions (ALL) to the Glue Role 
- Attach AWSGlueServiceRole policy to the Glue Role 
- Attach an inline policy below to the Glue Role

```json 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "iam:GetRole",
                "iam:PassRole",
                "lakeformation:GetDataAccess",
                "lakeformation:GrantPermissions"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::amazon-reviews-pds",
                "arn:aws:s3:::amazon-reviews-pds/*"
            ],
            "Effect": "Allow"
        }
    ]
}
```

Accepted Answer

Eventually, I found out that I use incorrect writing method in this case
 
* Correct method: **glueContext.write_dynamic_frame.from_catalog**
* Incorrect method: glueContext.write_dynamic_frame.from_option

Quoted from docs

```
Writes a DynamicFrame using the specified catalog database and table name.
```

[docs](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame-writer.html#aws-glue-api-crawler-pyspark-extensions-dynamic-frame-writer-from_catalog)

Answer

Once you register a location to be managed by LakeFormation, you should not use s3 permissions to handle it since it.  
Now any client API trying to access that table needs to be LakeFormation away in order to be able to use the delegated access permissions, for instance DynamicFrame is able write into such table but not if you write directly to s3 or you use DataFrame.

Glue Role got S3 Access Denied when writing to underlying S3 data in Lake Formation

相關內容