Use AWS FARGATE_SPOT as fallback

Question

A customer wants to use the default ECS Capacity Providers FARGATE and FARGATE_SPOT but is not satisfied with the default behaviour. Specifically, ECS `desired_count` is not honoured when using several ECS providers and weights. 
 
Example:

Configure an ECS cluster with the following default providers:

`FARGATE_SPOT` base=0 weight=50

`FARGATE` base=0 weight=50

If we run a service that uses the default providers, if the service has a `desired_count=10` and the `FARGATE_SPOT` capacity is not available, ECS will not use the available `FARGATE` capacity to honour desired_count. The service will run with only 5 tasks instead.

Have you encountered this issue with other clients and do you have a good workaround for this?

Accepted Answer

If I understand correctly, this is similar to the behavior that Auto Scaling Groups show when mixing on-demand and spot capacity: the OD portion of the fleet will not exceed the specified weight even if you run out of Spot (this can easily be tested by setting a very low Spot price limit).

A possible workaround would be to have a backup service running 100% in Fargate, initially with desired tasks = 0. Then, with a Lambda function you would monitor that the main service is running all tasks by either 1) a scheduled CloudWatch event, or 2) reacting to an EventBridge event emitted by ECS when a task is stopped. Then you adjust the backup service accordingly, if needed.

Here's some info on how to handle termination notices for Fargate Spot: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-capacity-providers.html#fargate-capacity-providers-termination

I'm more than happy to have someone give a second opinion though, but I think the above could work.

Use AWS FARGATE_SPOT as fallback

Relevant content