Athena permission denied on S3 path

0

Hi everyone,

I am creating a simple datalake with S3, Glue and Athena through Terraform (Infrastructure As A Code), which is managed by Lake Formation. I grant all the permissions in both IAM and Lake Formation needed for the workflow roles and also Athena users. However, from time to time (I deploy and destroy the infrastructure many times under dev environment) I got an error when I try to query the glue tables in Athena: Permission denied on S3 path: s3://XXX/XX. This query ran against the "XXXX" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 0adbb706-4500-4c3b-8f0d-3174fa4e12b9. The error affects all the tables in the database when it occurs. When I redeploy the same code, the error might not occur at all. I searched on AWS and internet but has not found the reason and a solution. Has anyone encountered this error or known the reason ? Any solution for this problem? By the way, I don't have KMS encryption yet just have S3-SSE turned on at this moment.

Thank you very much in advance.

Feng

asked a year ago472 views
1 Answer
0
Accepted Answer

Hello,

Thanks for sharing the Query ID.

When registering a Lake Formation path, we follow up by granting permissions to the path from "Data Locations" to the role that needs to access LF managed tables with locations that are registered.

Looking at the S3 logs for the provided Athena Query ID, the authentication process is looking similar to when a location is registered with Lake Formation [1]. The process being as follows:

  1. A principal runs a SELECT query in Athena.
  2. Athena analyses the query and checks Lake Formation permissions to see if the principal has been granted access to the table and table columns.
  3. If the principal has access, Athena requests credentials from Lake Formation. If the principal does not have access, Athena issues an access denied error.
  4. Lake Formation issues credentials to Athena to use when reading data from Amazon S3, along with the list of allowed columns.
  5. Athena uses the Lake Formation temporary credentials to query the data from Amazon S3. After the query completes, Athena discards the credentials.

Now, to answer your question - the error for the shared Query ID happened because of a missing s3:ListBucket permission from the Lake Formation end.

Although, currently checking the Role that you are using, I could see that the Role being assumed by Athena (AWSServiceRoleForLakeFormationDataAccess) has permission for the S3 path (s3://xxxx-xxxxx-xxxx-xxx-us-east-1-123456789000/xxxxxx_xxxxx), so you should not be encountering this error at this very moment as I write this.

Once you register your S3 bucket (xxxx-xxxxx-xxxx-xxx-us-east-1-123456789000) in Lake Formation, it updates your Role (AWSServiceRoleForLakeFormationDataAccess) with the missing permissions (s3:ListBucket in your case for the given Query ID) automatically.

That being said, I would recommend that you check that the S3 bucket is getting registered appropriately, when you are doing a deployment through Terraform, so that you do not run into these errors again.


References:

[1] Using Athena to query data registered with AWS Lake Formation - https://docs.aws.amazon.com/athena/latest/ug/security-athena-lake-formation.html

AWS
SUPPORT ENGINEER
Nitin_S
answered a year ago
  • Hi Nitin, Thanks for the quick answer and detailed explanation. As you pointed out, I don't see the error now after a few redeployment trials. I will pay attention to s3: ListBucket permission if the error occurs next time. One thing I don't understand is that since I did not see any error in terraform deployment and the terraform deployment should be very reproducible and stable, why missing permission issue could happen sometimes? It is caused by a bug in terraform, or AWS lake formation or IAM services? Thanks again.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions