AWS Lambda + beautifulSoup python painfully slow

Question

I'm working on web scraping project with Lambda and beautifulsoup,requests libs.

Unfortunately the code is extremely slow. locally the code run 20 seconds, on Lambda its 172 seconds.

I can't work and develop my project like that.

Please suggest me some solutions

Riku_Kobayashi · Accepted Answer

How about increasing the Lambda specs to match the specs of the local development environment?  
The default setting is 128 MB.  
As per the following documentation, Lambda can be configured with up to 10 GB of memory.  
https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-common.html#configuration-memory-console

We recommend that you read the following best practices document once.  
https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html

Antonio_Lagrotteria · Answer

Hi,

Apart increasing memory on lambda, which is something without limitations under free tier as well, you can consider creating the function as a container: https://docs.aws.amazon.com/lambda/latest/dg/images-create.html,as increases portability, supports larger package size (mostly for python numpy and similar libraries).

AWS Lambda + beautifulSoup python painfully slow

Relevant content