Unable to Crawl My Bronze Layer

Question

Hello all,

I am looking for some insight related to an error I'm receiving with a Glue Crawler.
I am working in Lake Formation with AppFlow, S3 and Glue.  Here is where I am so far:
1. I have ingested data into a Landing folder in my S3 bucket.
2. I have created a service role and granted access to my Glue database and All Tables along with my s3 bucket.
3. I have successfully crawled the landing folder to catalog my data.  I have verified that all tables have been created correctly with a 'source_' prefix as specified. 
4. I ran a glue itl job to transform my data from JSON to a Snappy compressed Parquet format in another folder in the same bucket called 'bronze'.
5. In my attempt to crawl the bronze folder in the same bucket, I used the same service role which I used successfully to crawl the landing folder in the same bucket.

I am now running into the following error:
*Insufficient Lake Formation permission(s) on s3://dci-datalake-production/bronze/bounce/ (Database name: dci-database, Table name: bronze_bounce) (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: 7c9d73bc-c7b0-4679-b3ff-1525911e3b7f; Proxy: null). For more information, see Setting up IAM Permissions in the Developer Guide (http://docs.aws.amazon.com/glue/latest/dg/getting-started-access.html).*

Here are the steps I have taken in attempt to resolve the problem:
1. Edit the crawler > Configure Security Settings > Update chosen IAM role.  I was prompted to confirm whether I wanted to update or create a policy specific to the folder in the bucket.  I confirmed, reran the crawler with the same error.
2. Next I revoked all lake formation permissions for the service role and regranted access to the Glue Database and All Tables.  This also did not work.
3. I removed and re registered my s3 bucket in my lake formation.  This also has not worked.
4. Lastly, I granted my service role Database creator permission.

Thank you in advance for any assistance you may be able to give.

Accepted Answer

Check this documentation : 
[] Troubleshoot crawler errors when the crawler is using Lake Formation credentials - Error: Insufficient Lake Formation permission(s) on s3://examplepath - https://docs.aws.amazon.com/glue/latest/dg/error-crawler-config-lf.html#error-location-permissions

To solve the issue you need to grant Lake formation permission for Data Location (the registered S3 path permission)
[] Granting data location permissions (same account) - https://docs.aws.amazon.com/lake-formation/latest/dg/granting-location-permissions-local.html

Crawler need access to read s3 data directly, then update the Catalog. 
So, two Lake Formation permissions required : 
- a. S3 Reregistered Location Grant (need to grant with above step mentioned)
- b. Grant on the database and table to create/update. (you have granted this)

Answer

Hello,

Resolution:
--
You get this error when the following conditions are true:

- The IAM user or role tries to create or alter a Data Catalog resource (database or table) on an Amazon S3 bucket that's registered with Lake Formation.
- The IAM user or role doesn’t have the appropriate data location permissions from Lake Formation.

To resolve this error, you must grant appropriate data location permissions to the IAM user or role that you use to create the database or table. When you're using Athena with Lake Formation, be sure to grant the required S3 permissions to the IAM user or role from Lake Formation in addition to the data access permissions required by the IAM user or role. Data access permissions allow the IAM user or role to read and write data to the underlying Amazon S3 location. However, data location permissions in Lake Formation allow an IAM user or role to create and alter Data Catalog resources that point to the registered Amazon S3 location.

To resolve this error, do the following:

1. Verify that S3 path in Athena is registered with Lake Formation successfully.
2. Grant the required data location permissions to the IAM user or role to access the S3 path.

[+] https://repost.aws/knowledge-center/athena-insufficient-lake-formation-permissions

Unable to Crawl My Bronze Layer

Resolution:

Relevant content