Lambda function web scraping with requests.get() is timing out? or getting blocked

Question

I have a lambda function that uses requests.get() to run sentiment analysis on websites, and the function is run through an API call. I recently updated the function though, so that now it's linked to an EFS (where dependencies are stored), and also a VPC due to security. However, now when I test my web extension which will run the lambda function, my lambda function is unable to retrieve text from the website, aka it's failing on the requests.get() line.

I did a bit of research, and I'm thinking that this may be because of the newly connected VPC, which is causing the lambda function to run on subnets that are commonly blocked by most websites. If this is the case, does anyone know a workaround?

Accepted Answer

Hi,

I recommend that you take a look at the [following Knowledge Center article](https://repost.aws/knowledge-center/internet-access-lambda-function) which describes how to grant internet access to an AWS Lambda function that's connected to a VPC, since it may be that the VPC or subnet are not correctly configured and you do not have access to that public website.

Answer

When you connect your function to a VPC, you loose internet access. To access the internet, you will need to deploy a NAT Gateway into the VPC. Note that this will add cost.

To overcome this cost, consider moving the configuration to S3, and run the function not attached to the VPC.

Lambda function web scraping with requests.get() is timing out? or getting blocked

Relevant content