Skip to content

The rollout update of new ECS service version is slow

0

I have deployed a nginx service in ECS. And then uploaded a new version of nginx image into ECR (the image size is only about 22MB).

I tried to upgrade the nginx service to the latest image version via rolling update, and observed it took about 2 minutes for the new version rollout! Is it a normal behavisor?

My rollout update configuration as below. BTW, my service doesn't enable load balancer or service connect. Enter image description here

asked a year ago340 views

2 Answers
0
Accepted Answer

I think even without Load Balancer with ECS service there are few steps, that would play a roll in overall deployment time.

Steps:

  • ECS creates a new deployment, New task set with the new task definition.
  • Launch 1 new task (scale to 2× temporarily allowed by 200%)
  • ENI allocate & attach (awsvpc networking).
  • Pull image (from ECR/Public).
  • Create/start container; task moves to RUNNING.
  • With no health checks, “healthy” = container is RUNNING.
  • Stop the old task
  • ECS sends SIGTERM to the old container and waits (up to the container’s stopTimeout, default ~30s if you didn’t set it).
  • If it doesn’t exit, ECS SIGKILLs it.
  • Deployment complete
  • Service returns to steady state with 1 new task running.

“Ideal” timing (realistic ranges)

  • Assuming the region is healthy, the image is small (22 MB), CPU/mem aren’t highly throttled.
  • ENI allocate/attach: ~5–20s
  • Image pull (22 MB): can be ~15–30s on a true cold path
  • Container start: ~1–3s
  • Service stabilization bookkeeping: ~1–5s
  • Stop old task: typically 5-10s if nginx exits promptly (otherwise up to your stopTimeout, default ~30s)

Best/ideal total: ~45-70 seconds Cold/slower path (no caching, busy account/region): ~70–120 seconds

answered a year ago

0

The 2-minute deployment time for your ECS service update is not unusual, even for a small image like your 22MB nginx container. ECS rolling updates are designed to maintain service availability, which can lead to longer deployment times than you might expect.

During a rolling update, the ECS scheduler follows a specific process:

  1. It starts your new application tasks
  2. Determines if they're ready for traffic
  3. Safely stops old tasks while maintaining application availability

Several factors can affect deployment speed:

  1. Deployment Configuration: Your settings show minimum healthy percent at 100% and maximum percent at 200%. This means ECS will start new tasks before terminating old ones, which is safer but can extend deployment time.

  2. Image Pull Behavior: Even though your image is small (22MB), if the EC2 instance doesn't have it cached locally, it needs to pull it from ECR first.

  3. Task Startup Time: The container needs time to initialize and become ready.

  4. Task Placement: ECS needs to determine where to place new tasks based on available resources.

To speed up your deployments, you could:

  • Adjust your minimum healthy percent to a lower value if your service can tolerate brief unavailability
  • Ensure image caching is enabled on your EC2 instances
  • Use unique image tags instead of reusing the same tag (like "latest")
  • Consider using the "Force new deployment" option when updating your service

Without a load balancer, you're not dealing with health check delays or connection draining, which would otherwise add to deployment time.
Sources
Best Practices - Speeding up deployments - Amazon Elastic Container Service
ECS task definition image tag | AWS re:Post
ECS Deployment shows success but app still runs old code – latest changes not reflecting | AWS re:Post

answered a year ago

    1. "adjust minimum healthy percent " - it is not acceptable since I don't want to break the application high availablity
    2. "cache image in ec2 instance" - I use fargate, so no way to cache image
    3. "unique image tag" - not relevant in my case,
    4. "use force new deployment" - I have tried, nothing improved

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.