access denied on CopyObject

0

I have written a simple script to move some redshift logs to a different location as they are written to the default location. When I test the script by copying the logs into the directory manually it works. However, when redshift dumps the logs into the directory my script fails with this:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

So I know there must be something different between how redshift writes the logs and how I write them but I can't for the life of me figure out what that difference is. Any help?

Here is my code. I've set this up with a role that has full s3 permissions. It's pretty straightforward but I can't figure out why it's failing outside of "it's permissions related" but I can't figure out why or what to change.

import boto3

def lambda_handler(event, context):
    s3 = boto3.resource('s3')

    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']
        log_file = key.split('/')[-1]
        log_type = log_file.split('_')[-2]
        log_env = log_file.split('_')[-3]
        new_key = 'redshift/{}/{}/{}'.format(log_type, log_env, log_file)
        print(key)
        print(new_key)
        
        s3.Object(bucket, new_key).copy_from(CopySource={'Bucket': bucket, 'Key': key})
        s3.Object(bucket, key).delete()
neufpas
質問済み 7年前4817ビュー
2回答
0

I'm completely stuck here. I just gave my bucket full public permissions and it's still failing with Access Denied.

The policy attached to my job grants "s3:*" permissions and I know it works because if I drop a file in the bucket it copies correctly.

This problem only exists with the files created by redshift.

Any help?

neufpas
回答済み 7年前
0

I figured out the problem. It ended up being related to a bad key. It turns out that the key string passed in by the s3 event trigger is urlencoded. These urlencoded strings appear as normal strings when the error message appears in the lambda window but adding debug logging to my script let me see the actual string.

I'm not entirely sure why this was causing an Access Denied error, but adding a call to urlib.parse.unquote fixed the issue.

neufpas
回答済み 7年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ