Skip to content

s3 tables maintenance error and Athena PERMISSION_DENIED query error

0

The problems I'm having: s3 tables maintenance fails, cannot access s3 tables in Athena

s3 table maintenance fails: I make my s3 tables in a glue etl job, they show up in my s3 table bucket, but if i click on the table and go to the maintenance section, it says that maintenance failed and gives the following error: Encountered Iceberg validation exception when trying to read table. Ensure that your table is readable, adheres to the Iceberg specification, and contains only S3 paths that begin with your S3 Table alias.

Athena error: My s3 tables appear in Athena, but i get the following when trying to query the table, PERMISSION_DENIED: null (Service: S3, Status Code: 403)

Below is the configuration I've set for my glue etl notebook (using this aws community answer) as well as my method for writing to the table. NOTE: spark.hadoop.fs.defaultFS=s3://s3_bucket_name is included because this is the only way I'm able to read my delta lake tables.

%%configure
{
"--datalake-formats": "delta,iceberg",
"--conf":"""spark.hadoop.fs.defaultFS=s3://s3_bucket_name
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
--conf spark.sql.defaultCatalog=s3tablescatalog
--conf spark.sql.catalog.s3tablescatalog=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.s3tablescatalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
--conf spark.sql.catalog.s3tablescatalog.client.region=Region
--conf spark.sql.catalog.s3tablescatalog.glue.account-id=Account_ID
--conf spark.sql.catalog.s3tablescatalog.glue.id=Account_ID:s3tablescatalog/s3_tables_bucket_name
--conf spark.sql.catalog.s3tablescatalog.warehouse=s3://s3_bucket_name/gold/s3_tables_bucket_name/
"""
}
df = spark.read.format("delta").load("s3://s3_bucket_name/path/to/delta/file/")
... code for setting up table I want to output...
df.writeTo(f"{catalog_name}.{namespace}.{table}").using("iceberg").create()

This results in the tables being registered in my s3 tables bucket with the correct schema.

Permissions: The role i use has the has full permissions for s3, s3tables, and lakeformation. I also have a custom policy (based off access-control-underlying-data and s3tables-catalog-prerequisites).NOTE: i did not see a lakeformation:CreateCatalog that the docs mentioned.

{
            "Sid": "lakeformation",
            "Effect": "Allow",
            "Action": [
                "lakeformation:GetDataAccess",
                "lakeformation:RegisterResource",
                "lakeformation:RegisterResourceWithPrivilegedAccess",
                "lakeformation:GrantPermissions"
            ],
            "Resource": [
                "*"
            ]
        }

my key policy allows for maintenance (using the s3-tables-kms-permissions docs):

{
      "Sid": "Allow S3TablesMaintenance",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "maintenance.s3tables.amazonaws.com",
          "metadata.s3.amazonaws.com"
        ]
      },
      "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
      ],
      "Resource": "*"
    }

Have I misconfigured anything here? what's stopping maintenance and preventing me from querying the tables in Athena?

1 Answer
1
Accepted Answer

Hi Joshua,

Your issue may stem from writing tables directly to a regular S3 bucket instead of an S3 Table Bucket. The error messages indicate:

  1. Maintenance error: "Ensure that your table...contains only S3 paths that begin with your S3 Table alias"
  2. Athena 403 error: Tables aren't in the correct location for S3 Tables to access

Your Configuration

# This is incorrect for S3 Tables:
spark.sql.catalog.s3tablescatalog.warehouse=s3://s3_bucket_name/gold/s3_tables_bucket_name/

You're writing to a regular S3 bucket (s3://s3_bucket_name/), but S3 Tables requires tables to be in an S3 Table Bucket with a special warehouse location format like:

s3://63a8e430-6e0b-46f5-k833abtwr6s8tmtsycedn8s4yc3xhuse1b--table-s3

Source: Creating an Amazon S3 table - Warehouse location format

Solution: Use the Correct S3 Tables Integration

Step 1: Create an S3 Table Bucket (if not already done)

Documentation: Creating a table bucket

# Create table bucket via AWS CLI
aws s3tables create-table-bucket \
  --name my-table-bucket \
  --region us-east-1

Or use the S3 console: S3 → Table buckets → Create table bucket

Step 2: Integrate with AWS Analytics Services

Documentation: Integrating Amazon S3 Tables with AWS analytics services

Prerequisites (from Prerequisites for integration):

  1. Attach AWSLakeFormationDataAdmin policy to your IAM principal
  2. Add these permissions to your IAM role:
    • glue:PassConnection
    • lakeformation:RegisterResource
    • lakeformation:RegisterResourceWithPrivilegedAccess

Enable integration:

  • In S3 console → Table buckets → Check "Enable integration" when creating the bucket
  • This automatically registers with Lake Formation and creates the s3tablescatalog in Glue
  • See integration steps

Step 3: Fix Your Glue ETL Configuration

Remove the custom warehouse location - S3 Tables manages this automatically:

%%configure
{
    "--datalake-formats": "delta,iceberg",
    "--conf": """
    spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
    spark.sql.defaultCatalog=s3tablescatalog
    spark.sql.catalog.s3tablescatalog=org.apache.iceberg.spark.SparkCatalog
    spark.sql.catalog.s3tablescatalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
    spark.sql.catalog.s3tablescatalog.client.region=us-east-1
    spark.sql.catalog.s3tablescatalog.glue.account-id=123456789012
    spark.sql.catalog.s3tablescatalog.glue.id=123456789012:s3tablescatalog/my-table-bucket
    """
    # DO NOT specify warehouse location - S3 Tables manages this
}

Step 4: Create Tables Using S3 Tables API

Documentation: Creating an Amazon S3 table

Option A: Create via Athena (Recommended)

See: Creating a table with Athena

  1. Go to Athena console
  2. Select s3tablescatalog as the data source
  3. Create table with DDL:
CREATE TABLE s3tablescatalog.my_namespace.my_table (
    id bigint,
    name string,
    created_at timestamp
)
LOCATION 's3://my-table-bucket/'  -- S3 Tables manages actual location
TBLPROPERTIES (
    'table_type'='ICEBERG'
);

Option B: Create via AWS CLI

# First create namespace
aws s3tables create-namespace \
  --table-bucket-arn arn:aws:s3tables:us-east-1:123456789012:bucket/my-table-bucket \
  --namespace my_namespace

# Then create table
aws s3tables create-table \
  --table-bucket-arn arn:aws:s3tables:us-east-1:123456789012:bucket/my-table-bucket \
  --namespace my_namespace \
  --name my_table \
  --format ICEBERG

Option C: Write from Spark (after table exists)

# Read your delta table
df = spark.read.format("delta").load("s3://s3_bucket_name/path/to/delta/file/")

# Write to EXISTING S3 table (create table first via Athena/CLI)
df.writeTo("s3tablescatalog.my_namespace.my_table").append()
# or .overwritePartitions() or .replace()

Critical Requirements

1. Lowercase Names Only

From Creating an Amazon S3 table - Important note:

"When creating tables, make sure that you use all lowercase letters in your table names and table definitions. For example, make sure that your column names are all lowercase."

Your column names must be lowercase or you'll get the exact error you're seeing in Athena:

"GENERIC_INTERNAL_ERROR: Get table request failed: com.amazonaws.services.glue.model.ValidationException: Unsupported Federation Resource - Invalid table or column names."

Source: S3 Tables documentation - Case sensitivity requirement

2. Required IAM Permissions

Documentation: Actions, resources, and condition keys for Amazon S3 Tables

Your role needs:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3tables:CreateTable",
                "s3tables:PutTableData",
                "s3tables:GetTableData",
                "s3tables:GetTableMaintenanceConfiguration",
                "s3tables:PutTableMaintenanceConfiguration",
                "lakeformation:GetDataAccess",
                "lakeformation:RegisterResource",
                "lakeformation:RegisterResourceWithPrivilegedAccess",
                "glue:CreateCatalog",
                "glue:GetCatalog",
                "glue:GetTable",
                "glue:GetDatabase"
            ],
            "Resource": "*"
        }
    ]
}

3. KMS Key Policy (if using encryption)

Documentation: Permission requirements for S3 Tables SSE-KMS encryption

Your KMS key policy looks correct, but ensure the key is in the same region as your table bucket. The maintenance service principals you included are correct:

  • maintenance.s3tables.amazonaws.com
  • metadata.s3.amazonaws.com

Source: S3 Tables KMS permissions documentation

Why Your Current Approach Doesn't Work

  1. S3 Tables ≠ Regular S3 Buckets: S3 Tables is a specialized service with its own bucket type
  2. Warehouse location is managed: You can't specify a custom warehouse path - S3 automatically generates it
  3. Integration required: Must register with Lake Formation for Athena access
  4. Special ARN format: Table buckets have ARN format arn:aws:s3tables:region:account:bucket/name

Migration Path

If you have existing Iceberg tables in regular S3:

  1. Create S3 Table Bucket and integrate with analytics services
  2. Create empty tables in S3 Tables with same schema
  3. Use Spark to read from old location and write to S3 Tables:
# Read from old location
df = spark.read.format("iceberg").load("s3://old-bucket/path/to/table")

# Write to S3 Tables (table must exist first)
df.writeTo("s3tablescatalog.namespace.table").append()

Documentation References

AWS
EXPERT

answered 9 months ago

AWS
EXPERT

reviewed 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.