AmazonS3Exception error when running CTAS using Athena engine version 3

0

We are experiencing the following issue, that is blocking us from upgrading to Athena engine version 3. It's important to note that the query runs successfully when using Athena engine version 2. The S3 bucket being used denies uploads of unencrypted objects, in case that could be relevant.

Example query:

CREATE TABLE ctas_1772133c_00c9_440e_934e_c35ac928fdcd WITH (
    format = 'JSON',
    external_location = 's3://athena-query-results-123412341234/tables/ctas_1772133c_00c9_440e_934e_c35ac928fdcd/'
) AS
SELECT name
FROM users
LIMIT 10

Error message received (masked, except for the request/query ids):

Error committing manifest file com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: NAPVPKQ1F1BHQHF8; S3 Extended Request ID: jeovQuVvA/yuDyjNXg+K10z9oJDxxscRsdYO6A+rd53AkT/tq+ZlxDYwfMazypczaCKmuO8rebo=; Proxy: null), S3 Extended Request ID: jeovQuVvA/yuDyjNXg+K10z9oJDxxscRsdYO6A+rd53AkT/tq+ZlxDYwfMazypczaCKmuO8rebo=. You may need to manually clean the data at location 's3://athena-query-results-123412341234/Unsaved/2022/11/28/tables/ctas_1772133c_00c9_440e_934e_c35ac928fdcd' before retrying. Athena will not delete data in your account. This query ran against the "default" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 13a43d53-33d6-4536-98c8-ad116bc12637

This is the S3 bucket policy, for the bucket where data should be saved to:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DenyIncorrectEncryptionHeader",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::athena-query-results-123412341234/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "AES256"
                }
            }
        },
        {
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::athena-query-results-123412341234/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption": "true"
                }
            }
        }
    ]
}
  • Are you trying to overwrite the table into the old S3 location? Given this is an EXTERNAL table, the CREATE TABLE AS should be in a different S3 location compared the source table location.

  • This is always a different and unique ctas_<random_uuid> value for both the table name and the S3 external location, so it never collisions with an existing S3 path.

  • We are also facing same issue. I could not understand why such a core thing not fixed even after these many days. No one wants to write data without encryption.

asked a year ago396 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions