access denied on CopyObject

0

I have written a simple script to move some redshift logs to a different location as they are written to the default location. When I test the script by copying the logs into the directory manually it works. However, when redshift dumps the logs into the directory my script fails with this:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

So I know there must be something different between how redshift writes the logs and how I write them but I can't for the life of me figure out what that difference is. Any help?

Here is my code. I've set this up with a role that has full s3 permissions. It's pretty straightforward but I can't figure out why it's failing outside of "it's permissions related" but I can't figure out why or what to change.

import boto3

def lambda_handler(event, context):
    s3 = boto3.resource('s3')

    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']
        log_file = key.split('/')[-1]
        log_type = log_file.split('_')[-2]
        log_env = log_file.split('_')[-3]
        new_key = 'redshift/{}/{}/{}'.format(log_type, log_env, log_file)
        print(key)
        print(new_key)
        
        s3.Object(bucket, new_key).copy_from(CopySource={'Bucket': bucket, 'Key': key})
        s3.Object(bucket, key).delete()
neufpas
질문됨 7년 전4818회 조회
2개 답변
0

I'm completely stuck here. I just gave my bucket full public permissions and it's still failing with Access Denied.

The policy attached to my job grants "s3:*" permissions and I know it works because if I drop a file in the bucket it copies correctly.

This problem only exists with the files created by redshift.

Any help?

neufpas
답변함 7년 전
0

I figured out the problem. It ended up being related to a bad key. It turns out that the key string passed in by the s3 event trigger is urlencoded. These urlencoded strings appear as normal strings when the error message appears in the lambda window but adding debug logging to my script let me see the actual string.

I'm not entirely sure why this was causing an Access Denied error, but adding a call to urlib.parse.unquote fixed the issue.

neufpas
답변함 7년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠