- Newest
- Most votes
- Most comments
The issue might be caused by cold starts in AWS Lambda, where new execution environments (containers) are created to handle incoming requests, leading to initialization delays and potential timeouts when making outbound calls to external web services. I suggested cold starts as a potential cause because the issue you described seems to happen intermittently, sometimes after only a few invocations, and sometimes after the Lambda function has been running for a while.
It would be good you read this article about Operating Lambda.
Additionally, to mitigate cold starts, you can use AWS Lambda's provisioned concurrency feature, which keeps a pool of initialized execution environments ready to handle incoming requests without the need for cold starts. Alternatively, you can implement techniques like periodic invocations or AWS Lambda Extensions to keep your functions warm and avoid cold starts.
Please note that while this solution is a good starting point for debugging, I cannot guarantee it will resolve your issue.
It is the 3rd party web service returns the 504 error, right? If so, the load on the 3rd party web service is the problem.
I would measure the processing time of the 3rd party's invocation part, check the response from the 3rd party, and contact the 3rd party with the details of the problem.
Hi! Thanks for taking the time time to answer. Sadly I can't blame the 3rd party on this one. It doesn't apear to even it make it out to them. Their logs show no incoming traffic at that point. And when I run the same code locally it runs flawlessly. Only when the lambda is deployed. Thanks though!
How do IPs be fixed to be registered in the list?
Are you running in multiple AZs and can't communicate with the outside world only when running in a specific AZ?
You may be able to find connection problems from VPC Flow Logs.
https://repost.aws/knowledge-center/vpc-flow-logs-and-cloudwatch-logs-insights
Relevant content
- Accepted Answerasked a year ago
- asked a year ago
AWS OFFICIALUpdated 3 years ago- AWS OFFICIALUpdated 5 months ago
- AWS OFFICIALUpdated 3 years ago
Hi! Thanks for taking the time time to answer. I followed the advice in the article. Fired up provisioned concurrency on the lambda (10 instances). Also tried the periodic invocations.
Some more details - the 3rd party endpoint I am hitting results in a few hundred milliseconds both deployed (when it works) and when run locally. I get zero failures running tens of thousands of calls as quickly as I can when running locally - always getting a response in a few hundred milliseconds. When deployed the HTTPClient call fails on timeout (tried everything from 10-100 seconds) about 1/3 of the time. The 3rd party vendor says their logs don't show those ever getting to them.
There is a VPC configured on this. I know it is working because the 3rd party library required a whitelisted IP address. Maybe that is the issue??? I have no idea how to even check.
I am really pulling my hair out on this. Thanks for the help.