ECS Fargate Spot ignores stopTimeout

SOLVED: This was my mistake, not AWS behavior

After digging deeper into this issue, I discovered that AWS was correctly respecting my stopTimeout: 120 configuration. The early termination was caused by my own container command configuration.

Root Cause: timeout Command Kill-After Logic

My container was using this command, since ECS does not support setting max execution time:

timeout -k 10s 3600 python ./main.py

The -k 10s parameter was the culprit. Here's what actually happened:

1. AWS sent SIGTERM to my container during spot interruption (correctly)
2. timeout process received SIGTERM and forwarded it to my Python script
3. timeout immediately started its own 10-second kill timer due to -k 10s
4. After 10 seconds, timeout sent SIGKILL to my Python script
5. Process terminated with exit code 137

The Technical Details

The GNU timeout command's signal handler doesn't distinguish between internal timeouts and external signals. When it receives any signal (including external SIGTERM from ECS), it triggers the kill-after logic if the -k parameter is specified.

From the timeout source code:

static void cleanup (int sig) {
  if (0 < monitored_pid) {
    if (kill_after) {  // My -k 10s parameter
      settimeout (kill_after, false);  // Starts 10s kill timer!
    }
    send_sig (monitored_pid, sig);  // Forwards signal to child
  }
}

Solution

I fixed this by updating my container command to:

timeout -k 120s 3600 python ./main.py

This allows my application the full 120 seconds for graceful shutdown, matching my ECS stopTimeout configuration.

Key Takeaway

AWS Fargate correctly respects stopTimeout during spot interruptions. If you're seeing early termination, check your application's signal handling and any wrapper commands that might be interfering with the graceful shutdown process.