CloudFormation Fails to Delete ECS Service When Migrating from LaunchType to Capacity Provider Strategy

Question

**Issue Summary**

I’m migrating an ECS service from using LaunchType: EC2 to Capacity Provider in AWS CloudFormation. As per the AWS documentation, when LaunchType is removed, the service should default to the cluster’s [defaultCapacityProviderStrategy](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-service.html#cfn-ecs-service-capacityproviderstrategy).

To ensure the correct behavior, I explicitly set `CapacityProviderStrategy: []`, as suggested [here](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-service.html#cfn-ecs-service-capacityproviderstrategy).

I'm using a CloudFormation template for creating my services. Below is a sample of my template.

```
Parameters:
    ServiceLaunchType:
        Type: String
        Default: ""

Conditions:
    HasLaunchType: !Not [!Equals [!Ref ServiceLaunchType, ""]]

ECSService:
    Type: AWS::ECS::Service
    Properties:
        LaunchType: !If [HasLaunchType, !Ref ServiceLaunchType, !Ref "AWS::NoValue"]
        CapacityProviderStrategy: !If [HasLaunchType, !Ref "AWS::NoValue", []]
```

This is intended to remove the LaunchType property and allow the service to use the default capacity provider configured in the ECS cluster.

**Unexpected CloudFormation Behavior**

After deploying this update, CloudFormation completes the stack update, but it fails to delete the old ECS service, resulting in a DELETE_FAILED status.

The following error appears in the CloudFormation events:
```
Resource handler returned message: "Cannot invoke "com.amazonaws.ecs.service.logger.model.LogContext.updateContextFromException(java.lang.Exception)" because "this.logContext" is null" 
(RequestToken: zzzz-yyyy-zzzz-aaaa-bbbbbbbbbbbb, HandlerErrorCode: InternalFailure)
```

Despite this failure, the new ECS service is created and running successfully.

**Observations:**
* The CloudFormation stack update completes, but has status `Update successful. One or more resources could not be deleted.`
* The old ECS service remains in a hanging state and is not cleaned up automatically. 
* The error suggests an AWS internal issue (LogContext.updateContextFromException), which might indicate a CloudFormation bug.

**Steps Taken:**
* Verified that the ECS cluster has a default capacity provider strategy
```
aws ecs describe-clusters --clusters development-my-cluster --query "clusters[*].defaultCapacityProviderStrategy"
[
    [
        {
            "capacityProvider": "development-my-capacity-provider",
            "weight": 1,
            "base": 0
        }
    ]
]
```
Output confirms the default capacity provider strategy is correctly configured.

* Tried manually deleting the old ECS service. This succeeds.

**Questions**
1. Is this a known CloudFormation issue? The error (LogContext.updateContextFromException) suggests an internal AWS bug rather than a misconfiguration.
2. Why does CloudFormation fail to delete the old ECS service when migrating to Capacity Providers? The new service is created successfully, yet the old one remains in a stuck state.
3. Is there an official workaround for this issue? Should AWS officially document a process for transitioning ECS services from LaunchType to Capacity Providers?

Any insights from the AWS team or community would be greatly appreciated!

re:Post Agent · Answer

This issue appears to be related to a known CloudFormation behavior when transitioning ECS services from LaunchType to Capacity Providers. While not explicitly documented as a bug, it's a situation that can occur during such migrations.

The failure to delete the old ECS service is likely due to the complex nature of ECS service updates, especially when changing fundamental properties like the launch type or capacity provider strategy. CloudFormation may struggle to properly handle this transition, resulting in the DELETE_FAILED status you're experiencing.

The error message you're seeing ("Cannot invoke "com.amazonaws.ecs.service.logger.model.LogContext.updateContextFromException(java.lang.Exception)" because "this.logContext" is null") suggests an internal AWS issue, which could be contributing to the problem.

To work around this issue, you might consider the following approaches:

1. Manually delete the old ECS service after the CloudFormation update completes. Since you mentioned this succeeds, it's a viable temporary solution.

2. Instead of updating the existing service, create a new ECS service with the desired capacity provider strategy, then delete the old service. This two-step process might avoid the CloudFormation deletion issue.

3. Use a custom resource in your CloudFormation template to handle the transition. This could involve using AWS Lambda to manage the service update and deletion process more granularly.

4. If possible, consider using AWS CDK or other infrastructure-as-code tools that might handle this transition more gracefully.

Regarding the transition from LaunchType to Capacity Providers, while AWS documentation does mention the use of defaultCapacityProviderStrategy when LaunchType is omitted, it doesn't provide specific guidance for this migration scenario. It would indeed be beneficial for AWS to provide official documentation or best practices for this transition process.

If this issue persists or significantly impacts your operations, it's recommended to open a support case with AWS. They can provide more tailored assistance and potentially escalate the issue if it's indeed a bug in CloudFormation.

Remember that when working with Auto Scaling groups and capacity providers, it's crucial to carefully consider protection and termination settings to ensure proper cleanup when resources are no longer needed. This can help mitigate issues during stack updates and deletions.
      
**Sources**  
[ECS Autoscaling Group and EC2 Instance not deleted by delete stack CDK typescript | AWS re:Post](https://repost.aws/questions/QUZC5yagX4TemuxDGA1SkiJg/ecs-autoscaling-group-and-ec2-instance-not-deleted-by-delete-stack-cdk-typescript)  
[Troubleshoot CloudFormation stack deployment failure | AWS re:Post](https://repost.aws/knowledge-center/cloudformation-aws-kms-key)

CloudFormation Fails to Delete ECS Service When Migrating from LaunchType to Capacity Provider Strategy

Add your answer

Relevant content