How can I troubleshoot Lake Formation permission errors in AWS Glue?

Short description

To access resources such as, the AWS Glue Data Catalog and Amazon Simple Storage Service (Amazon S3), you must have the correct IAM policies and Lake Formation permissions. When an IAM role that's associated with your Amazon S3 crawler or ETL job doesn't have sufficient Lake Formation permissions, you get an error. You must have the correct Lake Formation permissions to read and write to and from the following resources:

• Database and table in the Data Catalog
• Underlying data in Amazon S3

Resolution

Access issues when creating Data Catalog database

If Data Catalog database creation issues are causing the error, then you get an error message similar to the following one:

Insufficient Lake Formation permission(s): Required Create Database on Catalog

To resolve this error, use the data lake administrator role to access the Lake Formation console. Then, grant the Create database permission to the relevant IAM role.

1. Open the AWS Lake Formation console.
2. In the navigation pane, under Permissions, choose Administrative roles and tasks.
3. Under Database creators, choose Grant.
4. For IAM users and roles, from the dropdown list, choose the IAM role that you want to grand access to.
5. Under Catalog permissions, choose Create database.
6. If you want the IAM role to grant permissions to other roles in your account, then under Grantable permissions, choose Create database.
7. Choose Grant.

Access issues with Data Catalog database

If Data Catalog database issues are causing the error, then you get an error message similar to one of the following one:

Insufficient Lake Formation permission(s) on example_database: Required Create Table
Insufficient Lake Formation permission(s) on example_database: (Database name: example_database)
Insufficient Lake Formation permission(s) on example_table (Database name: example_database, Table Name: example_table)

To resolve the first and third errors, grant the Create table permission for example_database to the IAM role that's associated with the crawler or ETL job.

To resolve the second error, grant the Describe permission for example_database to the IAM role that's associated with the crawler or ETL job.

1. Open the Lake Formation console.
2. In the navigation pane, under Permissions, choose Data lake permissions.
3. Choose Grant.
4. Under Principals, choose IAM users and roles.
5. For IAM users and roles, choose the IAM role that's associated with the crawler.
6. Under LF-Tags or catalog resources, choose Named data catalog resources.
7. For Databases, choose the database that your crawler is writing to.
8. Under Database permissions, choose Create table or Describe based on your use case. Note: The crawler role must have both Describe AND Create Table permissions on the database that it's writing to. For AWS Glue ETL, these permissions are also sufficient, unless you're running an UpdateDatabase or DeleteDatabase API call from the job itself. In these cases, grant Alter or Drop permissions.
9. If you want the IAM role to grant permissions to other roles in your account, then set permissions under Grantable permissions.
10. Choose Grant.

Access issues with Data Catalog table

If Data catalog table issues are causing the error is caused, then you get an error message similar to one of the following one:

Insufficient Lake Formation permission(s) on example_table (Database name: example_database, Table Name: example_table)
Insufficient Lake Formation permission(s): Required Alter on example_table

If you receive this error, then the AWS Glue crawler or ETL job is accessing the table. For the crawler, grant Describe and Alter permissions for example_table to the IAM role that's associated with the crawler.

For ETL jobs that read example_table, grant Describe and Select permissions for example_table to the IAM role that's associated with the job. If the ETL job is updating example_table, then grant Alter permissions for example_table to the IAM role that's associated with the job.

1. Open the Lake Formation console.
2. In the navigation pane, under Permissions, choose Data lake permissions.
3. Choose Grant.
4. Under Principals, choose IAM users and roles.
5. For IAM users and roles, choose the IAM role.
6. Under LF-Tags or catalog resources, choose Named data catalog resources.
7. For Databases, choose the database that your crawler is writing to.
8. For Tables-optional, choose the table that your crawler is accessing.
9. Under Table permissions, choose Select, Describe, or Alter based on your use case.
10. If you want the IAM role to grant permissions to other roles in your account, then set permissions under Grantable permissions.
11. Choose Grant.

Access issues with Lake Formation IAM permissions

If IAM permissions issues for Lake Formation are causing the error, then you get an error message similar to one of the following one:

com.amazonaws.services.lakeformation.model.AccessDeniedException: Service Principal: glue.amazonaws.com is not authorized to perform: lakeformation:GetDataAccess
on resource: s3://sample-bucket/sample-prefix/ because no identity-based policy allows the lakeformation:GetDataAccess action

You get this error when the AWS Glue job role or AWS Glue crawler role doesn't have sufficient IAM permissions. The job or role must have permission to check if Lake Formation can vend temporary credentials to the role and the Amazon S3 location.

1. Open the IAM console.
2. Create an IAM policy for your AWS Glue crawler or AWS Glue job role.
3. Add the permission lakeformation:GetDataAccess as the action for the resource in the policy. Note: The API lakeformation:GetDataAccess must use the wildcard as its resource.
4. Attach the policy to your AWS Glue crawler or AWS Glue job role.

Access issues with Amazon S3 path

If Amazon S3 path issues are causing the error is caused, then the error message looks similar to the following one. The error includes the Amazon S3 path:

Insufficient Lake Formation permission(s) on s3://s3-example-bucket/example-prefix/ (Database name: example-database, Table Name: example-table)<br>
Insufficient Lake Formation permission(s) on s3://s3-example-bucket/example-prefix/

This error occurs when the IAM role that's associated with the crawler or ETL job doesn't have the required permission to access the Amazon S3 path.

To resolve this error, complete the following steps:

1. Open the Lake Formation console.
2. In the navigation pane, under Register and ingest, choose Data lake locations.
3. Verify that the Amazon S3 path or prefix of the path from the error message is a registered location in the Data lake locations list.
4. If the Amazon S3 path or prefix in the error message isn't listed in the Data lake locations list, then choose Register location.
5. For Amazon S3 path, choose Browse, and then choose the correct Amazon S3 path.
6. For IAM role, keep the default selection of AWSServiceRoleForLakeFormationDataAccess. If you use a custom IAM role, then be sure that the relevant requirements are met. Important: When registering an Amazon S3 location, Lake Formation assumes the preceding IAM role. This grants temporary credentials to integrated AWS services that access data in that location. Verify that the IAM role that's associated with the registered Amazon S3 location has the required permissions to read and write to the S3 bucket. This prevents the AccessDeniederror.
7. In the navigation pane, under Permissions, choose Data locations.
8. Choose Grant.
9. For Grant permissions, choose My account.
10. Under IAM users and roles, choose the IAM role that you want to grant access for.
11. Choose Grant.

---

Related information

Managing Lake Formation permissions

Registering an Amazon S3 location