- Newest
- Most votes
- Most comments
The 503 errors you're experiencing during deployments are likely due to the way App Runner handles traffic routing during the transition between old and new instances. This is a common challenge in blue-green deployments, which App Runner uses internally.
To mitigate this issue, you can try the following approaches:
-
Implement a graceful shutdown in your application. When your Node.js Express app receives a SIGTERM signal (which App Runner sends before terminating an instance), it should stop accepting new requests but continue processing existing ones. This can help reduce the number of in-flight requests that get cut off.
-
Increase your health check interval and threshold. By increasing the health check interval to something like 30 seconds and the healthy threshold to 10 requests, you give your new instances more time to warm up and become ready before they start receiving traffic.
-
Implement connection draining in your application. This involves keeping the old instances alive for a short period after new instances are ready, allowing in-flight requests to complete.
-
Consider implementing retry logic in your client applications. This can help handle temporary 503 errors during deployments.
-
If possible, stagger your deployments or perform them during low-traffic periods to minimize the impact on users.
-
Ensure your application starts quickly and is ready to handle requests as soon as possible after launch. Slow startup times can exacerbate issues during deployment.
-
You already implemented increased auto scaling with a minimum size greater than 1. This means that even if one instance is being replaced during a deployment, others are still available to handle incoming requests
Remember, some level of errors during deployments is often unavoidable, especially in high-traffic scenarios. The goal is to minimize their occurrence and impact on your users.
Sources
App runner auto deployment problem with ECR image tag | AWS re:Post
App Mesh connectivity troubleshooting - AWS App Mesh
App Runner Startup Time
Relevant content
asked 5 years ago
- AWS OFFICIALUpdated 10 months ago
I implemented a graceful server shutdown by listening for SIGTERM, but 503 errors continue to occur during deployment. I’ve noticed that the 503 errors stop once SIGTERM is fired on the instances being scaled down. However, several minutes before SIGTERM is fired, the old instances are no longer handling any requests, and this is when the 503 errors occur. I’m wondering if SIGTERM needs to be triggered much earlier so the load balancer stops trying to route requests to them ?