access denied on CopyObject

0

I have written a simple script to move some redshift logs to a different location as they are written to the default location. When I test the script by copying the logs into the directory manually it works. However, when redshift dumps the logs into the directory my script fails with this:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

So I know there must be something different between how redshift writes the logs and how I write them but I can't for the life of me figure out what that difference is. Any help?

Here is my code. I've set this up with a role that has full s3 permissions. It's pretty straightforward but I can't figure out why it's failing outside of "it's permissions related" but I can't figure out why or what to change.

import boto3

def lambda_handler(event, context):
    s3 = boto3.resource('s3')

    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']
        log_file = key.split('/')[-1]
        log_type = log_file.split('_')[-2]
        log_env = log_file.split('_')[-3]
        new_key = 'redshift/{}/{}/{}'.format(log_type, log_env, log_file)
        print(key)
        print(new_key)
        
        s3.Object(bucket, new_key).copy_from(CopySource={'Bucket': bucket, 'Key': key})
        s3.Object(bucket, key).delete()
neufpas
已提问 7 年前4815 查看次数
2 回答
0

I'm completely stuck here. I just gave my bucket full public permissions and it's still failing with Access Denied.

The policy attached to my job grants "s3:*" permissions and I know it works because if I drop a file in the bucket it copies correctly.

This problem only exists with the files created by redshift.

Any help?

neufpas
已回答 7 年前
0

I figured out the problem. It ended up being related to a bad key. It turns out that the key string passed in by the s3 event trigger is urlencoded. These urlencoded strings appear as normal strings when the error message appears in the lambda window but adding debug logging to my script let me see the actual string.

I'm not entirely sure why this was causing an Access Denied error, but adding a call to urlib.parse.unquote fixed the issue.

neufpas
已回答 7 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则