How can I delete a capacity provider that is not associated with a cluster but claims to be associated to one when attempting to delete?

Question

Here is my situation in short
* Capacity provider cannot be deleted because it is said to be associated with a cluster
* The association to the cluster of the capacity provider cannot be confirmed
* The problematic capacity provider is referencing a deleted auto scaling group

And here is the story and details about the situation.
I was exploring features of ECS to run tasks on EC2 instances.
Terraform was used to create the auto scaling group and capacity provider.
The capacity provider seemed to be created but was not associated with the cluster. During my attempt to create an association I was destroying and applying the Terrafom setup and also did some things manually.
Currently I am not able to delete the capacity provider I have created because of the following reason.

> The capacity provider cannot be deleted because it is associated with cluster: my-cluster. Remove the capacity provider from the cluster and try again.

However, according to the management console and AWS CLI results, the capacity provider is not listed as my-cluster's capacity provider.
And the auto scaling group that the capacity provider is referencing does not exist.

I have checked that the capacity provider is not being used by any ECS service and also checked it is not set as the default capacity provider strategy for the cluster as guided in [another post](https://repost.aws/knowledge-center/ecs-capacity-provider-error).
I have also tried to reset the cluster's associated capacity providers using put-cluster-capacity-providers AWS CLI command so that unwanted providers would be disassociated but got the following error.

> An error occurred (ResourceInUseException) when calling the PutClusterCapacityProviders operation: The specified capacity provider is in use and cannot be removed.

Accepted Answer

It turns out a capacity provider can be associated with an ECS cluster while its association is not explicitly shown anywhere.
As I have mentioned in my question the capacity provider was created with Terraform.
In order to make the association with a cluster it seems like you have to declare a aws_ecs_cluster_capacity_providers and specify which capacity providers are associated with a cluster.
At first I did not declare the resource in my Terraform module because it seemed only useful when creating the cluster along with the module.
Because the cluster was not created within the module and the work was done on an existing cluster and when declaring aws_ecs_cluster_capacity_providers seems like you have to specify not only the capacity providers you want to add but the existing ones.
And then I found no other way to make the association so I declared aws_ecs_cluster_capacity_providers temporarily.
I did not add FARGATE and FARGATE_SPOT because it seemed like it was supplied to the cluster by default and cannot be removed anyway.
I think this is where my problem was.

One way to disassociate a capacity provider from a cluster is to use PutClusterCapacityProviders API and specify the capacity providers to associate and others that weren't mentioned in the API call would be disassociated.
I tried this and wouldn't work until I added FARGATE and FARGATE_SPOT.

I attempted to add the capacity provider which was claimed to be associated with the cluster along with FARGATE and FARGATE_SPOT and the API call was processed and then the capacity provider was visible from the management console and from AWS CLI.
Then I was able to delete it.

Answer

Make sure that a capacity provider is not included in any service's capacity provider strategy before deleting it. You can use the UpdateService API to remove a capacity provider from a service's capacity provider strategy. You can also use the forceNewDeployment option to move any tasks that use the Amazon EC2 instance capacity from the provider to the remaining capacity providers.

https://docs.aws.amazon.com/cli/latest/reference/ecs/delete-capacity-provider.html

Following the deletion of an Auto Scaling group, the related capacity provider will enter an INACTIVE state. While this state may remain in your account for a while, you should not depend on INACTIVE capacity providers because their behavior is subject to change.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/delete-capacity-provider-console-v2.html

Capacity providers that are associated with a cluster cannot be directly deleted. To delete such a provider, you first need to remove it from the cluster. This can be done using the PutClusterCapacityProviders API or by deleting the cluster itself​

https://docs.aws.amazon.com/cli/latest/reference/ecs/delete-capacity-provider.html

How can I delete a capacity provider that is not associated with a cluster but claims to be associated to one when attempting to delete?

Relevant content