How do I handle Spot termination notices in Fargate Spot tasks?

Resolution

Increase the stopTimeout value

The default value for the stopTimeout parameter is 30 seconds. Increase the stopTimeout value to a maximum of 120 seconds in your task definition so that your containers have enough time to exit before they're forcefully stopped. If you exceed this quota, then you receive the "The stopTimeout value must not exceed 120 seconds." error message.

Update your service definition parameters

Update your Amazon ECS service definition parameters. For minimumHealthyPercent, enter the minimum percentage of tasks that must continue to run. For maximumPercent, enter the maximum percentage of tasks that can run during a deployment.

Monitor your Amazon ECS service events to check whether the scheduler finds new capacity when tasks are interrupted. If the scheduler can't find capacity for new tasks, then old tasks end after the stopTimeout.

Update the deregistration delay value

Tasks that run as FARGATE_SPOT use load balancer target groups. These tasks remain registered with the target group until they reach the STOPPED state. Set the target group deregistration delay to less than 2 minutes to accommodate the Spot termination window.

Monitor Spot interruptions and capacity placement failures

To monitor Fargate Spot terminations that are caused by task interruptions or capacity placement failures, use Amazon EventBridge rules.

Complete the following steps:

1. Create an Amazon Simple Notification Service (Amazon SNS) topic.

2. Define the EventBridge rule.

3. Build an event pattern, and choose the following options:
   In the Sample events section, choose AWS events.
   In the Creation method section, for Method, choose Custom patterns (JSON editor).

4. Add one of the following patterns for the issue that you want to monitor.
   Spot interruptions:

   {
     "source": ["aws.ecs"],
     "detail-type": ["ECS Task State Change"],
     "detail": {
       "stopCode": ["SpotInterruption"],
       "clusterArn": ["arn:aws:ecs:REGION:ACCOUNT:cluster/CLUSTER"]
     }
   }

   Capacity placement failures:

   {
     "source": ["aws.ecs"],
     "detail-type": ["ECS Service Action"],
     "detail": {
       "eventName": ["SERVICE_TASK_PLACEMENT_FAILURE"],
       "clusterArn": ["arn:aws:ecs:REGION:ACCOUNT:cluster/CLUSTER"],
       "reason": ["RESOURCE:FARGATE"]
     }
   }

   Note: In the preceding patterns, replace REGION with your AWS Region, ACCOUNT with your AWS account ID, and CLUSTER with your cluster name.

5. Choose Next.

6. Choose the following options to select your target:
   For Target types, choose AWS service.
   For Select a target, choose SNS.
   For Topic, select your SNS topic.

7. Choose Next.

8. (Optional) Add a tag.

9. Review your rule configuration, and then choose Create rule.

Related information

Fargate Spot termination notices