Uploading a file to S3 using Python/Boto3 and CodePipeline

Question

Good morning,  
  
This is my first post here in AWS forums and is a question about how to fix an "Access Denied" error that Python 2.7 is giving at the moment of upload a file to S3 bucket.  
  
Explaining a little what I'm doing: I'm executing a single pipeline in CodePipeline to compile and deploy a project, after that, the last step of the pipeline is launching a Lambda Function, this Lambda function download an artifact that is generated by the previous step (CodeBuild) of this Lambda. When download is finished, it extracts the file and upload a file to another bucket different than the one that is using to download the file.  
  
The Lambda function is downloading the file correctly but at the moment of upload it shows an "Access Denied" error. In this post you will find the code used to do the previously explained, the error log that Cloudwatch gives, and the IAM policy that is attached to the lambda function.  
  
What things we already tried:   
  
  * Put all the permissions to all the S3 buckets (Doesn't work)  
  * Put all the permissions to specific S3 bucket (Doesn't work)  
  * Put public writing permissions to S3 bucket (This worked)  
  * Used in python the upload_file and upload_fileobj in Python (Doesn't work)  
  
The code that we are using to do this is the following:

```
from __future__ import print_function
from boto3.session import Session

import json
import urllib
import boto3
import zipfile
import tempfile
import botocore
import traceback

print('Initializing function.')

boto3.set_stream_logger(level=1)

s3 = boto3.client('s3')
codepipeline = boto3.client('codepipeline')

documentationFileName = "swagger.json"

def setup_s3_client(job_data):
    print("Initializing s3")

    key_id = job_data["artifactCredentials"]["accessKeyId"]
    key_secret = job_data["artifactCredentials"]["secretAccessKey"]
    session_token = job_data["artifactCredentials"]["sessionToken"]

    session = Session(aws_access_key_id = key_id,
       aws_secret_access_key = key_secret,
       aws_session_token = session_token)

    print("Created s3 session")

    return session.client("s3", config = botocore.client.Config(signature_version = 's3v4'))

def put_job_success(job, message):
    print('Putting job success')
    print(message)
    codepipeline.put_job_success_result(jobId = job)

def put_job_failure(job, message):
    print('Putting job failure')
    print(message)
    codepipeline.put_job_failure_result(jobId = job, failureDetails = {'message': message, 'type': 'JobFailed'})

def get_documentation(s3, artifacts):
    print("Getting documentation")
    
    doc = artifacts[0]
    objectKey = doc["location"]["s3Location"]["objectKey"]
    bucketName = doc["location"]["s3Location"]["bucketName"]

    with tempfile.NamedTemporaryFile() as tmp_file:
       print("Downloading file form s3")
       s3.download_file(bucketName, objectKey, tmp_file.name)
       with zipfile.ZipFile(tmp_file.name, 'r') as zip:
         print("Printing content on zip")
         zip.printdir()
         print(zip.namelist())
         return zip.read(documentationFileName)

def update_documentation(s3, doc):
    print("Updating documentation")

    bucketName = "project-api-documentation"
    objectKey = "projectEngineApi/api.json"
    fileName = "api.json"

    with tempfile.NamedTemporaryFile() as tmp_file:
       tmp_file.write(doc)
       s3.upload_file(tmp_file.name, bucketName, objectKey)
       tmp_file.close()

def lambda_handler(event, context):
    try:
       print(event)
       job_id = event["CodePipeline.job"]["id"]
       job_data = event["CodePipeline.job"]["data"]
       artifacts = event["CodePipeline.job"]["data"]["inputArtifacts"]
       s3 = setup_s3_client(job_data)
       docs = get_documentation(s3, artifacts)

       if (docs):
         update_documentation(s3, docs)
         put_job_success(job_id, "Doc updated successfully")
       else:
         print("Failure")
         put_job_failure(job_id, "Doc does not exists.")

    except Exception as e:
       print('Function failed')
       print(e)
       traceback.print_exc()
       put_job_failure(job_id, 'Function exception: ' + str(e)) 

    return 'Completed!'
```

The error log in cloudwatch is:

```
15:04:55 START RequestId: 55850db1-cecd-11e7-b4ac-014088afae30 Version: $LATEST
15:04:55 Initializing s3
15:04:55 Created s3 session
15:04:56 Getting documentation
15:04:56 Downloading file form s3
15:04:58 Printing content on zip
15:04:58 File Name Modified Size
15:04:58 swagger.json 2017-11-20 19:34:12 14331
15:04:58 project.jar 2017-11-20 19:36:08 29912075
15:04:58 ['swagger.json', 'project.jar']
15:04:58 Updating documentation
15:04:58 Function failed
15:04:58 Failed to upload /tmp/tmprFknUH to project-api-documentation/projectEngineApi/api.json: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
15:04:58 Putting job failure
15:04:58 Function exception: Failed to upload /tmp/tmprFknUH to project-api-documentation/projectEngineApi/api.json: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
15:04:58 Traceback (most recent call last):
15:04:58 File "/var/task/lambda_function.py", line 84, in lambda_handler
15:04:58 update_documentation(s3, docs)
15:04:58 File "/var/task/lambda_function.py", line 71, in update_documentation
15:04:58 s3.upload_file(tmp_file.name, bucketName, objectKey)
15:04:58 File "/var/runtime/boto3/s3/inject.py", line 110, in upload_file
15:04:58 extra_args=ExtraArgs, callback=Callback)
15:04:58 File "/var/runtime/boto3/s3/transfer.py", line 283, in upload_file
15:04:58 filename, '/'.join([bucket, key]), e))
15:04:58 S3UploadFailedError: Failed to upload /tmp/tmprFknUH to project-api-documentation/projectEngineApi/api.json: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
15:04:59 END RequestId: 55850db1-cecd-11e7-b4ac-014088afae30
15:04:59 REPORT RequestId: 55850db1-cecd-11e7-b4ac-014088afae30	Duration: 3674.95 ms	Billed Duration: 3700 ms Memory Size: 128 MB	Max Memory Used: 94 MB
```

The IAM policy attached to Lambda function is:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "codepipeline:PutApprovalResult",
                "cloudwatch:*",
                "codepipeline:PutJobFailureResult",
                "codepipeline:PutJobSuccessResult",
                "codepipeline:GetJobDetails",
                "logs:CreateLogGroup",
                "logs:PutDestination"
            ],
            "Resource": "*"
        },
        {
            "Sid": "CreatedManually",
            "Effect": "Allow",
            "Action": [
                "lambda:ListVersionsByFunction",
                "lambda:GetFunction",
                "lambda:ListAliases",
                "lambda:InvokeAsync",
                "lambda:GetFunctionConfiguration",
                "lambda:Invoke",
                "logs:PutLogEvents",
                "lambda:UpdateAlias",
                "s3:ListMultipartUploadParts",
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "lambda:ListTags",
                "lambda:PublishVersion",
                "lambda:GetAlias",
                "s3:DeleteObject",
                "lambda:GetPolicy",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:lambda:*:*:function:*",
                "arn:aws:logs:us-east-1:123456789101:log-group:/aws/lambda/*:*:*",
                "arn:aws:s3:::project-api-documentation",
                "arn:aws:s3:::codepipeline-us-east-1-123456789101",
                "arn:aws:s3:::project-api-documentation/*",
                "arn:aws:s3:::codepipeline-us-east-1-123456789101/*"
            ]
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:us-east-1:123456789101:log-group:/aws/lambda/*"
        },
        {
            "Sid": "VisualEditor3",
            "Effect": "Allow",
            "Action": "s3:ListObjects",
            "Resource": [
                "arn:aws:s3:::project-api-documentation",
                "arn:aws:s3:::codepipeline-us-east-1-123456789101",
                "arn:aws:s3:::project-api-documentation/*",
                "arn:aws:s3:::codepipeline-us-east-1-123456789101/*"
            ]
        }
    ]
}
```

Edited by: SebasZiegler on Nov 21, 2017 8:33 AM  
  
Edited by: SebasZiegler on Nov 21, 2017 8:33 AM.  
  
Edited by: SebasZiegler on Nov 21, 2017 10:58 AM

Answer

I managed to fix this, the thing is:  
  
    - I was taking the credentials at the moment of the upload from the event object, those credentials are only to download the artifacts. You cannot use them to upload files to another bucket.  
    - To get the credentials attached to the lambda function you just should use boto3 without creating a new Session, in this case the session is only created with the credentials to get the artifacts.  
  
In that case one solution could be changing the upload_documentation function to:

```
def update_documentation(doc):
    print("Updating documentation")

    bucketName = "project-api-documentation"
    objectKey = "projectEngineApi/api.json"
    fileName = "api.json"

    with tempfile.NamedTemporaryFile() as tmp_file:
       tmp_file.write(doc)
       s3.upload_file(tmp_file.name, bucketName, objectKey)
       tmp_file.close()
```

Uploading a file to S3 using Python/Boto3 and CodePipeline

Relevant content