failed to patch module botocore

0

I am getting the following error in my Lambda logs

failed to patch module botocore

Its a Flask app deployed using Zappa. Essentially following the instructions at

here is the code

import boto3
from aws_xray_sdk.ext.flask.middleware import XRayMiddleware
from aws_xray_sdk.core import patcher, xray_recorder
from aws_xray_sdk.core import patch_all
patch_all()

Modules installed are

aws-xray-sdk==2.11.0
boto3==1.26.69
botocore==1.29.69

It is successfully patching the requests package.

I want to try and resolve this as I believe my downstream calls to services like SSM and S3 are not being tracked in Xray.

Thanks for any help

  • I am facing the same problem, did you find a solution?

asked 2 years ago470 views
2 Answers
0

The patch process is probably (I'm taking a little bit of a guess here) trying to overwrite the botocore and boto3 code in place. That's not going to work as a Lambda function can only write to /tmp. I'm not sure why the requests library is getting patched successfully - unless it is already at the version that you require.

Instead, consider creating a Lambda Layer which contains the appropriate versions of the code that you need and then using that Layer with the Lambda function. This is the best solution if you have multiple functions using the same libraries.

You could also create an archive of the entire function which includes the libraries you need. This is good if you only have a single function.

Ideally, you don't want to patch at runtime as this will increase the cold start time of your function.

profile pictureAWS
EXPERT
answered 2 years ago
  • The patch technique is dynamic in memory, not a file thing. It uses the python "wrapt" module. The actual patch code can be seen at https://github.com/aws/aws-xray-sdk-python/blob/master/aws_xray_sdk/ext/botocore/patch.py

    I am still digging. I am finding others with related issues but nothing that has been able to resolve my core problem so far.

  • The Lambda runtime environment is Firecracker which is a (deliberately) cut-down hypervisor and underlying operating system to allow for rapid start of Lambda functions and also aid in securing the system. This could also explain why patching in memory works on EC2 but doesn't in Lambda. So my suggestion above still stands: Start the execution of the function with the library versions that are required.

  • Its not firecracker. The way you are describing firecracker as a cut down hypervisor resulting in a cut down Python in Lamba that is different to the Python on EC2 or any other environment is off track. Given some modules DO successfully patch that hypothesis is false anyway. It's something about the context and a bit esoteric, but it will make sense when found. It may be the library version as you say, I just don't agree its because of the hypervisor or the inability to modify file system areas. Lambda yes, firecracker no.

  • Without going through the Firecracker code as well as every step of the patch process I wouldn't rule it out - it would be quite easy to get (as you say) an inconsistent result where some patches work and others don't depending on what they (the patching processes) are doing. I've seen other cases where Python libraries work fine on EC2 but don't in Lambda because somewhere they require specific kernel calls or permissions that are not allowed or not present in the Lambda environment.

0

Hi,

Does it help if you add import botocore at the top of your code?

profile picture
EXPERT
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions