Creating Iceberg table with Pyiceberg

It looks like you're encountering an explicit deny in a resource-based policy when trying to create an Iceberg table using PyIceberg. The error indicates that your ECS task role doesn't have the necessary permissions to perform the s3:PutObject operation on the specified S3 path, despite having general S3 access.

When creating Iceberg tables, you need specific S3 permissions beyond just general access. For Iceberg table creation, you need at least the following S3 permissions:

• s3:PutObject on the bucket/path
• s3:GetObject on the bucket/path
• s3:DeleteObject on the bucket/path

The key part of your error message is "with an explicit deny in a resource-based policy" - this suggests there's a bucket policy or other resource-based policy that's explicitly denying your ECS task role from writing to that specific path, regardless of the IAM permissions attached to your role.

To resolve this issue:

1. Check the bucket policy for any explicit deny statements that might be blocking your ECS task role
2. Verify that there are no VPC endpoint policies restricting access
3. Ensure your KMS key policy allows your ECS task role to use the key for encryption operations
4. Make sure there are no SCPs (Service Control Policies) at the organization level blocking this action

The fact that you can write to the bucket using polars with s3fs but not with PyIceberg suggests that PyIceberg might be using a different API call pattern or accessing slightly different paths that trigger the deny policy.
Sources
Creating Apache Iceberg tables - AWS Lake Formation
CreateMultipartUpload operation: The AWS Access Key Id you provided does not exist in our records. | AWS re:Post