API + Lambda has #503 error 20% of the time
Running a step function state machine accessing an AWS HttpApi to run a lambda using a docker container of ~3 GB size. Fails 20% of the time with #503 error, the rest of the time I get the exact results I expect. It's not a container warm up I believe. Logs for API and lambda (non-container and container) don't show any problems. I have maximized the step functions time out, the retries for the specific lambda, the memory for the lambda (10GB). I have also tried to ping the container and wait 30 seconds in the step functions to no avail.
*** Edit: Should have mentioned concurrency for the API is at max (believe it is automatic for HttpApis)
Any advice would be appreciated. I don't want to use ECS fargate due to cost and relative complexity. Thanks!
Although not a direct answer to your question, but have you considered using AppRunner? It runs your docker service and will hibernate if there are no requests saving you money. So very similar to the way lambdas work, but without the complexity if you are using it with docker. I've found it easy to setup and very useful. AWS AppRunner
AWS App Runner is a fully managed service that makes it easy for developers to quickly deploy containerized web applications and APIs, at scale and with no prior infrastructure experience required. Start with your source code or a container image. App Runner builds and deploys the web application automatically, load balances traffic with encryption, scales to meet your traffic needs, and makes it easy for your services to communicate with other AWS services and applications that run in a private Amazon VPC. With App Runner, rather than thinking about servers or scaling, you have more time to focus on your applications.
Glad to help :)
Relevant questions
Lambda Authorizer Context Response Size Limit
Accepted Answerasked 3 months agoHow do I keep the time in sync in docker based lambda functions
Accepted Answerasked 3 months agoDeploy Lambda Function and API Gateway REST
asked 3 months agoHTTP API Integration with lamda function and stage variable not working
asked 10 months agoStep Function to Send Email on Error/Success
asked 4 months agoAPI + Lambda has #503 error 20% of the time
Accepted Answerasked 2 months agoHow to handle failed lambda functions
asked 2 months agohow to trigger a step function from a s3 object notification?
asked a month agoLambda function won't run in parallel via State Machines
asked 4 months agoIntermittend InvalidSignature error when retrieving object from S3 Object Lambda accesspoint in lambda@edge function
asked 2 months ago
Thanks for the mention! I will give it a look-see. I am using caddy in other parts of my workflows (and have used dokku in the past) so this should be a easy setup.