access denied on CopyObject

0

I have written a simple script to move some redshift logs to a different location as they are written to the default location. When I test the script by copying the logs into the directory manually it works. However, when redshift dumps the logs into the directory my script fails with this:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

So I know there must be something different between how redshift writes the logs and how I write them but I can't for the life of me figure out what that difference is. Any help?

Here is my code. I've set this up with a role that has full s3 permissions. It's pretty straightforward but I can't figure out why it's failing outside of "it's permissions related" but I can't figure out why or what to change.

import boto3

def lambda_handler(event, context):
    s3 = boto3.resource('s3')

    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']
        log_file = key.split('/')[-1]
        log_type = log_file.split('_')[-2]
        log_env = log_file.split('_')[-3]
        new_key = 'redshift/{}/{}/{}'.format(log_type, log_env, log_file)
        print(key)
        print(new_key)
        
        s3.Object(bucket, new_key).copy_from(CopySource={'Bucket': bucket, 'Key': key})
        s3.Object(bucket, key).delete()
neufpas
preguntada hace 7 años4818 visualizaciones
2 Respuestas
0

I'm completely stuck here. I just gave my bucket full public permissions and it's still failing with Access Denied.

The policy attached to my job grants "s3:*" permissions and I know it works because if I drop a file in the bucket it copies correctly.

This problem only exists with the files created by redshift.

Any help?

neufpas
respondido hace 7 años
0

I figured out the problem. It ended up being related to a bad key. It turns out that the key string passed in by the s3 event trigger is urlencoded. These urlencoded strings appear as normal strings when the error message appears in the lambda window but adding debug logging to my script let me see the actual string.

I'm not entirely sure why this was causing an Access Denied error, but adding a call to urlib.parse.unquote fixed the issue.

neufpas
respondido hace 7 años

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas