- Newest
- Most votes
- Most comments
Hi,
The timeout you got could indicate that the site you are trying to reach has some firewall configured. There are many different ways to block traffic, such as by IP address/network, user agent (I see you providing a custom), country and many others. You mentioned that code is working from the other sites, but none are based on AWS.
If I recall correctly, I think the default timeout that is set on new lambda functions is 3 seconds. If the request is working for the first two URLs try bumping the timeout of your function to something higher like 5 minutes and rerun it.
I set it to 15 minutes, but still no response for url3 and url4. But in my local machine jupyter notebook or in google colab, it ususaly takes 1 or 2 sec to response.
Try setting the max retries to 5. If that does not work, you'll have to dig into Cloudwatch logs. Perhaps there is some more information in there as to why the function is timing out
The site is blocking the AWS IP. curl https://www.makemytrip.com/ in EC2 is not working.
Relevant content
- asked 2 years ago
- asked 5 years ago
- AWS OFFICIALUpdated 5 months ago
- What's the difference between Lambda function execution role permissions and invocation permissions?AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
That's correct. But if I want to extract cookie of such websites using AWS cloud solution, then what are the possible options?