Skip to content

Lambda from a docker image: Boto3 still rely on aws profile instead of IAM role.

0

I want to deploy a lambda function that call OpenAI, do some processing for text and return result.

def  openaipreprocessing(text):
       # call OpenAI api
      return vector
    
def save_result_to_s3(text, vector):
    s3 = boto3.client("s3")
    file_key = f"results/{datetime.now().isoformat()}.json"
    s3.put_object(
        Bucket=S3_BUCKET_NAME,
        Key=file_key,
        Body=json.dumps({"text": text, "vector": vector})
    )
    return file_key
def lambda_handler(event, context):
    try:
        body = event.get("body", {})
        text = body.get("text")
        if not text:
            return {"statusCode": 400, "body": json.dumps({"error": "Text is required"})}
        vector= openaipreprocessing(text=text)
        file_key = save_result_to_s3(text=text, vector=vector)
        return {
            "statusCode": 200,
            "body": json.dumps({"message": "Vector stored", "file_key": file_key})
        }
    except Exception as e:
        return {"statusCode": 500, "body": json.dumps({"error": str(e)})}

#Create S3 bucket

aws s3api create-bucket --bucket $S3_BUCKET_NAME --region $AWS_REGION  --profile $AWS_PROFILE
aws s3api put-bucket-versioning --bucket $S3_BUCKET_NAME --versioning-configuration Status=Enabled --profile $AWS_PROFILE

#docker file:

FROM public.ecr.aws/lambda/python:3.9
#Install dependencies
COPY requirements.txt /var/task/src/
RUN pip install -r /var/task/src/requirements.txt

# Copy the src directory into the Lambda container
COPY . /var/task/src/

CMD ["src.lambda_function.lambda_handler"]

#I build locally the docker image and run :
docker run --rm -p 9000:8080 openai-service-api
#Testing the lambda function locally fails in:
s3=boto3.client("s3") with the error: The config profile (my-account) could not be found
#I fix this by mounting the local ~/.aws when I run the docker:
docker run --rm -p 9000:8080 ~/.aws:/root/.aws openai-service-api-v
I call the lambda function as following and it is working well:
curl -X POST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{"body": "{\"text\": \"Hello world\"}"}'

#I create the ECR repo and the lambda function and I push lambda using this script:
#Load environment variables
source ./src/.env
#Login to AWS ECR
aws ecr create-repository --repository-name $ECR_REPO_NAME --region $AWS_REGION --profile $AWS_PROFILE aws ecr get-login-password --region $AWS_REGION --profile $AWS_PROFILE| docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com #Build and push Docker image

docker build -t $ECR_REPO_NAME ./src
docker tag $ECR_REPO_NAME:latest $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO_NAME:latest
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO_NAME:latest

#Create a Lambda function in AWS

aws lambda create-function \
  --function-name TextEmbeddingLambda \
  --package-type Image \
  --code ImageUri=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO_NAME:latest \
  --role arn:aws:iam::$AWS_ACCOUNT_ID:role/AWSLambdaBasicExecutionRole \
  --region $AWS_REGION --profile $AWS_PROFILE

#Deploy to AWS Lambda

aws lambda update-function-code --function-name TextEmbeddingLambda \
  --image-uri $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO_NAME:latest --region $AWS_REGION --profile $AWS_PROFILE

#Testing the lambda on the console fails in the same line:
s3 = boto3.client("s3") for the same error: [ERROR] ProfileNotFound: The config profile (my-account) could not be found Traceback (most recent call last)
Removing the saving to S3. Lambda is working fine.
What is the error and why docker does not see the IAM role and still rely on my aws profile on CLI.

2 Answers
1

The error you're encountering is due to a misunderstanding of how IAM roles and AWS credentials work in the context of Lambda functions running in containers. When you run the Docker container locally and mount your ~/.aws directory, you're providing your local AWS credentials, which includes your profile information. However, when the Lambda function runs in AWS, it doesn't have access to these local credentials.

In AWS Lambda, the function automatically assumes the IAM role you specified when creating the function. This role provides the necessary permissions without needing to explicitly configure AWS credentials or profiles within your code.

The issue arises because your code is trying to use a specific AWS profile ("my-account") which doesn't exist in the Lambda environment. To fix this, you should remove any explicit profile references in your code and let the Lambda service handle the credentials using the assigned IAM role.

Here's how you can modify your code to work correctly:

  1. Instead of using a specific profile, create the S3 client without any profile information:

    s3 = boto3.client('s3')
  2. Ensure that the IAM role attached to your Lambda function (AWSLambdaBasicExecutionRole in your case) has the necessary permissions to interact with the S3 bucket. You may need to add specific S3 permissions to this role.

  3. Set the S3_BUCKET_NAME as an environment variable for your Lambda function instead of relying on a configuration file.

By making these changes, your Lambda function will use the permissions granted by its IAM role, and you won't need to worry about AWS profiles or credentials within your code.

Remember to rebuild and redeploy your Docker image after making these changes. This approach will work both in the AWS environment and locally if you use the AWS CLI's assume-role feature to simulate the Lambda execution environment.
Sources
Lambda function cannot access S3 bucket, even after granting IAM role/policy | AWS re:Post
Use CreateFunction with an AWS SDK or CLI - AWS Lambda
Use Invoke with an AWS SDK or CLI - AWS Lambda

answered 2 years ago

EXPERT

reviewed 2 years ago

    1. I don't use a specific profile in boto3.client('s3')
    2. I am sure that the IAM role has the necessary permission to access S3.
    3. S3_BUCKET_NAME is set in an .env file and loaded in config using dotenv, I don't see any problem doing that.
0

You are trying to save a file into S3 and for that your function needs to have permissions. The role that you used when you created the function is: AWSLambdaBasicExecutionRole, which does not include permissions to write to S3. You should create a new role for you function, that contains the basic execution role and permissions to write to S3 and use it for your function.

Further more, your should set the environment variables in the Lambda function. I do not see that in your create-function call. Having a .env file that you source locally, does not create variables in Lambda.

AWS
EXPERT

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.