- Newest
- Most votes
- Most comments
Hi Joshua,
Your issue may stem from writing tables directly to a regular S3 bucket instead of an S3 Table Bucket. The error messages indicate:
- Maintenance error: "Ensure that your table...contains only S3 paths that begin with your S3 Table alias"
- Athena 403 error: Tables aren't in the correct location for S3 Tables to access
Your Configuration
# This is incorrect for S3 Tables: spark.sql.catalog.s3tablescatalog.warehouse=s3://s3_bucket_name/gold/s3_tables_bucket_name/
You're writing to a regular S3 bucket (s3://s3_bucket_name/), but S3 Tables requires tables to be in an S3 Table Bucket with a special warehouse location format like:
s3://63a8e430-6e0b-46f5-k833abtwr6s8tmtsycedn8s4yc3xhuse1b--table-s3
Source: Creating an Amazon S3 table - Warehouse location format
Solution: Use the Correct S3 Tables Integration
Step 1: Create an S3 Table Bucket (if not already done)
Documentation: Creating a table bucket
# Create table bucket via AWS CLI aws s3tables create-table-bucket \ --name my-table-bucket \ --region us-east-1
Or use the S3 console: S3 → Table buckets → Create table bucket
Step 2: Integrate with AWS Analytics Services
Documentation: Integrating Amazon S3 Tables with AWS analytics services
Prerequisites (from Prerequisites for integration):
- Attach
AWSLakeFormationDataAdminpolicy to your IAM principal - Add these permissions to your IAM role:
glue:PassConnectionlakeformation:RegisterResourcelakeformation:RegisterResourceWithPrivilegedAccess
Enable integration:
- In S3 console → Table buckets → Check "Enable integration" when creating the bucket
- This automatically registers with Lake Formation and creates the
s3tablescatalogin Glue - See integration steps
Step 3: Fix Your Glue ETL Configuration
Remove the custom warehouse location - S3 Tables manages this automatically:
%%configure { "--datalake-formats": "delta,iceberg", "--conf": """ spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions spark.sql.defaultCatalog=s3tablescatalog spark.sql.catalog.s3tablescatalog=org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.s3tablescatalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog spark.sql.catalog.s3tablescatalog.client.region=us-east-1 spark.sql.catalog.s3tablescatalog.glue.account-id=123456789012 spark.sql.catalog.s3tablescatalog.glue.id=123456789012:s3tablescatalog/my-table-bucket """ # DO NOT specify warehouse location - S3 Tables manages this }
Step 4: Create Tables Using S3 Tables API
Documentation: Creating an Amazon S3 table
Option A: Create via Athena (Recommended)
See: Creating a table with Athena
- Go to Athena console
- Select
s3tablescatalogas the data source - Create table with DDL:
CREATE TABLE s3tablescatalog.my_namespace.my_table ( id bigint, name string, created_at timestamp ) LOCATION 's3://my-table-bucket/' -- S3 Tables manages actual location TBLPROPERTIES ( 'table_type'='ICEBERG' );
Option B: Create via AWS CLI
# First create namespace aws s3tables create-namespace \ --table-bucket-arn arn:aws:s3tables:us-east-1:123456789012:bucket/my-table-bucket \ --namespace my_namespace # Then create table aws s3tables create-table \ --table-bucket-arn arn:aws:s3tables:us-east-1:123456789012:bucket/my-table-bucket \ --namespace my_namespace \ --name my_table \ --format ICEBERG
Option C: Write from Spark (after table exists)
# Read your delta table df = spark.read.format("delta").load("s3://s3_bucket_name/path/to/delta/file/") # Write to EXISTING S3 table (create table first via Athena/CLI) df.writeTo("s3tablescatalog.my_namespace.my_table").append() # or .overwritePartitions() or .replace()
Critical Requirements
1. Lowercase Names Only
From Creating an Amazon S3 table - Important note:
"When creating tables, make sure that you use all lowercase letters in your table names and table definitions. For example, make sure that your column names are all lowercase."
Your column names must be lowercase or you'll get the exact error you're seeing in Athena:
"GENERIC_INTERNAL_ERROR: Get table request failed: com.amazonaws.services.glue.model.ValidationException: Unsupported Federation Resource - Invalid table or column names."
Source: S3 Tables documentation - Case sensitivity requirement
2. Required IAM Permissions
Documentation: Actions, resources, and condition keys for Amazon S3 Tables
Your role needs:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3tables:CreateTable", "s3tables:PutTableData", "s3tables:GetTableData", "s3tables:GetTableMaintenanceConfiguration", "s3tables:PutTableMaintenanceConfiguration", "lakeformation:GetDataAccess", "lakeformation:RegisterResource", "lakeformation:RegisterResourceWithPrivilegedAccess", "glue:CreateCatalog", "glue:GetCatalog", "glue:GetTable", "glue:GetDatabase" ], "Resource": "*" } ] }
3. KMS Key Policy (if using encryption)
Documentation: Permission requirements for S3 Tables SSE-KMS encryption
Your KMS key policy looks correct, but ensure the key is in the same region as your table bucket. The maintenance service principals you included are correct:
maintenance.s3tables.amazonaws.commetadata.s3.amazonaws.com
Source: S3 Tables KMS permissions documentation
Why Your Current Approach Doesn't Work
- S3 Tables ≠ Regular S3 Buckets: S3 Tables is a specialized service with its own bucket type
- Warehouse location is managed: You can't specify a custom warehouse path - S3 automatically generates it
- Integration required: Must register with Lake Formation for Athena access
- Special ARN format: Table buckets have ARN format
arn:aws:s3tables:region:account:bucket/name
Migration Path
If you have existing Iceberg tables in regular S3:
- Create S3 Table Bucket and integrate with analytics services
- Create empty tables in S3 Tables with same schema
- Use Spark to read from old location and write to S3 Tables:
# Read from old location df = spark.read.format("iceberg").load("s3://old-bucket/path/to/table") # Write to S3 Tables (table must exist first) df.writeTo("s3tablescatalog.namespace.table").append()
Documentation References
Relevant content
asked a year ago
asked 2 years ago
asked 3 years ago
