CloudFormation fails properly re-deploy CloudMap instances

When deleting AWS CloudMap services through CloudFormation, the expectation that CloudFormation would automatically deregister instances before deleting the service is not correct. CloudFormation cannot delete a CloudMap service that still has registered instances.

The error you're seeing indicates that during your CloudFormation stack update or deletion, it's trying to delete a CloudMap service (srv-jxwg46cqhynsn2nx) that still has registered instances. Before CloudMap services can be deleted, all service instances must be explicitly deregistered first.

To resolve this issue, you have a few options:

1. Ensure your application code properly deregisters instances from CloudMap when containers are stopping or being terminated.

2. Add a pre-deletion step in your deployment pipeline that uses the AWS SDK to:

• List all registered instances for the service
• Deregister each instance
• Then proceed with the CloudFormation deployment

3. If you're using ECS with Service Connect, make sure tasks are properly stopped before attempting to delete the CloudMap service.

This issue is similar to other AWS resource deletion dependencies, like trying to delete an ECS cluster that still has active tasks or services. The parent resource cannot be deleted until all child resources are properly cleaned up first.

The intermittent nature of the issue suggests that sometimes your instances are being properly deregistered before deletion, but other times (especially during rapid E2E test cycles) the deregistration may not complete before CloudFormation attempts to delete the service.
Sources
Troubleshoot Amazon ECS cluster deletion issues | AWS re:Post