ECS tasks running old code after deployment

This issue is related to how Amazon ECS handles container image resolution during deployments. When you deploy a container image on a service, ECS resolves the container image tag to a specific image digest (SHA256), which becomes the authoritative metadata for that image. This digest is cached to maintain consistency within a deployment.

What likely happened in your case is that even though your deployment showed as successful and the new task definition revision was in use, the application was still running the old code because ECS was using the cached image digest rather than pulling the latest image with your new enum value.

Here are some possible reasons and solutions:

1. Image tag resolution: If you're using a static tag like "latest" or a version tag that wasn't updated, ECS might have resolved it to the previous image digest. When ECS starts tasks, it uses the digest it resolved during the initial deployment, not necessarily the most recent image with that tag.

2. Force new deployment: While your CDK deployment completed successfully, it may not have forced ECS to pull the new image. You can use the "force-new-deployment" option when updating your service to ensure it pulls the latest image.

3. Task definition update: Ensure your CDK deployment is creating a new task definition revision with the updated image URI or tag.

4. Unique image tags: Consider using unique tags for each deployment (like git commit hashes or build numbers) instead of reusing the same tag. This makes it clearer which version is running and avoids caching issues.

The manual killing of tasks worked because it forced ECS to start new tasks using the latest resolved image digest. To prevent this issue in the future, make sure your deployment process properly updates the task definition and forces a new deployment, or implement a strategy using unique image tags for each deployment.
Sources
ECS Deployment shows success but app still runs old code – latest changes not reflecting | AWS re:Post
ECS task definition image tag | AWS re:Post

Hello.

Are the ECR image (version) referenced by the problematic ECS task and the newly launched ECS task different?
I think maybe I had launched my ECS task using a version of the task definition that allowed the deployment circuit breaker in ECS to launch successfully.
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-circuit-breaker.html