AmazonS3Exception error when running CTAS using Athena engine version 3

0

We are experiencing the following issue, that is blocking us from upgrading to Athena engine version 3. It's important to note that the query runs successfully when using Athena engine version 2. The S3 bucket being used denies uploads of unencrypted objects, in case that could be relevant.

Example query:

CREATE TABLE ctas_1772133c_00c9_440e_934e_c35ac928fdcd WITH (
    format = 'JSON',
    external_location = 's3://athena-query-results-123412341234/tables/ctas_1772133c_00c9_440e_934e_c35ac928fdcd/'
) AS
SELECT name
FROM users
LIMIT 10

Error message received (masked, except for the request/query ids):

Error committing manifest file com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: NAPVPKQ1F1BHQHF8; S3 Extended Request ID: jeovQuVvA/yuDyjNXg+K10z9oJDxxscRsdYO6A+rd53AkT/tq+ZlxDYwfMazypczaCKmuO8rebo=; Proxy: null), S3 Extended Request ID: jeovQuVvA/yuDyjNXg+K10z9oJDxxscRsdYO6A+rd53AkT/tq+ZlxDYwfMazypczaCKmuO8rebo=. You may need to manually clean the data at location 's3://athena-query-results-123412341234/Unsaved/2022/11/28/tables/ctas_1772133c_00c9_440e_934e_c35ac928fdcd' before retrying. Athena will not delete data in your account. This query ran against the "default" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 13a43d53-33d6-4536-98c8-ad116bc12637

This is the S3 bucket policy, for the bucket where data should be saved to:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DenyIncorrectEncryptionHeader",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::athena-query-results-123412341234/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "AES256"
                }
            }
        },
        {
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::athena-query-results-123412341234/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption": "true"
                }
            }
        }
    ]
}
  • Are you trying to overwrite the table into the old S3 location? Given this is an EXTERNAL table, the CREATE TABLE AS should be in a different S3 location compared the source table location.

  • This is always a different and unique ctas_<random_uuid> value for both the table name and the S3 external location, so it never collisions with an existing S3 path.

  • We are also facing same issue. I could not understand why such a core thing not fixed even after these many days. No one wants to write data without encryption.

質問済み 1年前406ビュー
回答なし

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ