Somebody please make any of the permissions in AWS make sense

Question

Error Category: UNCLASSIFIED_ERROR; Failed Line Number: 245; An error occurred while calling o147.sql. Insufficient LF permission(s) on chorus (Service: Glue, Status Code: 400, Request ID: 0054e9fa-5ddf-4e94-a026-815d5353d15c)

Script Spark Session:

```
 # Build the spark session with the assumed credentials
    spark = (
        SparkSession.builder
        .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
        .config("spark.sql.catalog.glue_catalog", "org.apache.iceberg.spark.SparkCatalog")
        .config("spark.sql.catalog.glue_catalog.warehouse", warehouse)
        .config("spark.sql.catalog.glue_catalog.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog")
        .config("spark.sql.catalog.glue_catalog.glue.id", "123456789101")
        .config("spark.sql.catalog.glue_catalog.client.region", "us-west-1")
        .config("spark.sql.catalog.glue_catalog.client.assume-role.arn", "LFAccessRole")
        .config("spark.sql.catalog.glue_catalog.client.assume-role.region", "us-west-1")
        .config("spark.sql.catalog.glue_catalog.client.assume-role.tags.LFAuthorizedCaller", "true")
        .config("spark.sql.catalog.glue_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
        .config("spark.sql.catalog.glue_catalog.glue.lakeformation-enabled", "true")
        .config("spark.sql.catalog.glue_catalog.client.factory", "org.apache.iceberg.aws.lakeformation.LFAwsClientFactory")
        .config("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider")
        .config("fs.s3a.access.key", credentials['AccessKeyId'])
        .config("fs.s3a.secret.key", credentials['SecretAccessKey'])
        .config("fs.s3a.session.token", credentials['SessionToken'])
        .getOrCreate()
    )
```

LF Expression:

![Role Permissions to Expression](/media/postImages/original/IMvgXXlau6T5GjlhyLeuec9Q)

![LF Expression](/media/postImages/original/IMzCk6ZRnmTe6PMH4GT2kTEQ)

Chorus LF Tags:
![Chorus LF Tags](/media/postImages/original/IMxlYRfAwNRBG5G73bKYyoCw)

Trace:
```
Traceback (most recent call last):
  File "/tmp/glue-job-16265854383127281549/raw_to_cleansed.py", line 235, in main
    cleansed_table_validation(spark, config, reset, warehouse, catalog_name)
  File "/tmp/glue-job-16265854383127281549/raw_to_cleansed.py", line 104, in cleansed_table_validation
    cols_df  = spark.sql(f"DESCRIBE {iceberg_table}")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

How does this make any sense at all? Lake Formation is nearly as complicated and seemingly impossible to use as IAM. AWS really needs to get in touch with invent and simplify on some of these things.

PS - why in the heck is LF spelled out not allowed in a post, what kind of ridiculousness is that?

JP · Answer

Error Category: UNCLASSIFIED_ERROR; Failed Line Number: 245; An error occurred while calling o147.sql. Insufficient Lake Formation permission(s) on the_landing (Service: Glue, Status Code: 400, Request ID: 35994e70-5059-41a9-98e3-5428dd872021)

It is even worse than I thought ... how is this even possible? Given the following:

![Glue Job Role Permissions](/media/postImages/original/IMPWTrb7wZR2KQhKgj5q-YtA)
![LFAccessRole Permissions](/media/postImages/original/IMWKgfZ715SCCgqOAQrC05Pg)

Those are the only two possible roles the ETL job uses. I specifically grant super on ALL of the databases and Tables. I still get that error. How can your system be this messed up?

re:Post Agent · Answer

I understand your frustration with AWS Lake Formation (LF) and IAM permissions. They can indeed be complex and challenging to navigate. Let's break down the issue you're facing and try to make sense of it.

The error message indicates that there are insufficient Lake Formation permissions on the "chorus" resource. This suggests that the role you're using doesn't have the necessary permissions in Lake Formation to perform the operation you're attempting.

In your Spark session configuration, you're using an assumed role ("LFAccessRole") for accessing the Glue catalog. This role needs to have the appropriate permissions in Lake Formation to access and describe the table you're querying.

The LF expression and tags you've shown are part of the Lake Formation permission model. These are used to define fine-grained access control on your data lake resources. The expression seems to be granting permissions based on certain tags, but it's possible that either the role doesn't have the right tags, or the expression isn't granting the specific permission needed for your operation.

To resolve this issue, you might need to:

1. Ensure that the "LFAccessRole" has the necessary Lake Formation permissions on the "chorus" resource. This typically includes at least the DESCRIBE permission.

2. Check if the tags on your role match the conditions in the Lake Formation permission expression.

3. Verify that the Lake Formation tags on the "chorus" resource are correctly set and align with your permission expression.

4. Make sure that the assumed role has the "LakeFormation:GetDataAccess" permission, which is often required for Lake Formation-enabled operations.

As for your frustration with the complexity, it's a valid point. While these systems offer powerful and fine-grained control, they can indeed be challenging to set up and troubleshoot. AWS continues to work on simplifying these services, but balancing flexibility with simplicity is an ongoing challenge.

Regarding the abbreviation restriction, it's likely an overzealous attempt to maintain clarity in posts, but I agree it can be inconvenient in technical discussions where abbreviations are common.

Remember, when working with Lake Formation, it's often helpful to start with broader permissions and then gradually restrict them to find the right balance of security and functionality for your use case.  
**Sources**  
[Troubleshooting AWS Glue Data Quality errors - AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/data-quality-trouble.html)  
[AWS Lake Formation permission errors - Prebid Server Deployment on AWS](https://docs.aws.amazon.com/solutions/latest/prebid-server-deployment-on-aws/aws-lake-formation-permission-errors.html)  
[Crawler errors when the crawler is using Lake Formation permissions - AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/error-crawler-config-lf.html)

Somebody please make any of the permissions in AWS make sense

LFAccessRole IAM role LF-Tag expression

Relevant content