Instance in shutting down state for 9 days

Question

I have an instance managed by Amazon EKS. The node associated with this instance is in the 'NotReady' state
i-030cfce4f5d6008a1   NotReady      9d     v1.32.8-eks-e386d34

The instance in the AWS ec2 list is in the shutting-down state.

I tries to stop the instance using kubernetes delete node command, but that does not work. I tried to remove this from the AWS console, and ec2-command line, but I cannot do that since I am not authroized to remove this instance, because of k8 policy. I checked pods running on this node

Non-terminated Pods:          (2 in total)
  Namespace                   Name                      CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                      ------------  ----------  ---------------  -------------  ---
  amazon-cloudwatch           cloudwatch-agent-vv95r    250m (14%)    500m (28%)  128Mi (4%)       512Mi (16%)    2d1h
  amazon-cloudwatch           fluent-bit-cn45v          50m (2%)      500m (28%)  25Mi (0%)        250Mi (8%)     2d1h

and they both are "pending"

cloudwatch-agent-vv95r                                            0/1     Pending   0              2d1h
fluent-bit-cn45v                                                  0/1     Pending   0              2d1h

Not sure what todo next, how to get of this catch 22 situation

re:Post Agent · Answer

When an EC2 instance is stuck in the "shutting-down" state for an extended period (like 9 days in your case), it indicates an issue with the termination process. Since this instance is managed by Amazon EKS and you're facing authorization issues due to Kubernetes policies, you'll need to take a different approach.

For instances stuck in the "shutting-down" state that are part of an EKS cluster, here's what you can try:

1. First, try to remove the machine's finalizer if it's an EKS Anywhere cluster node. This might help release the node from its stuck state.

2. Try using the AWS CLI to force terminate the instance:
```
aws ec2 terminate-instances --instance-ids i-030cfce4f5d6008a1
```

3. If that doesn't work, try force stopping the instance first:
```
aws ec2 stop-instances --instance-ids i-030cfce4f5d6008a1 --force
```

4. If you're still unable to terminate the instance due to permission issues, you may need to:
- Contact your AWS administrator who has the necessary permissions to override the Kubernetes policies
- If you have AWS Support, create a technical support case explaining that the instance has been stuck in "shutting-down" state for 9 days

The good news is that you're not being charged for EC2 compute usage while the instance is in the "shutting-down" state. However, you may still be charged for any attached EBS volumes.

For the pending pods, they're likely in that state because they're assigned to a node that's no longer functional. Once you resolve the instance termination issue, you should be able to reschedule those pods on healthy nodes.  
**Sources**  
[Instance stuck in "shutting down" state | AWS re:Post](https://repost.aws/questions/QUoS_D8QokTCebTu40TFYVtQ/instance-stuck-in-shutting-down-state)  
[Troubleshoot nodes for Amazon EKS Anywhere clusters | AWS re:Post](https://repost.aws/knowledge-center/eks-nodes-anywhere-clusters)  
[Troubleshoot Amazon EC2 instance termination issues - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesShuttingDown.html)  
[Troubleshoot Amazon EC2 instance stop issues - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesStopping.html)  
[Understand why GPU and metal EC2 instances take a long time to stop | AWS re:Post](https://repost.aws/knowledge-center/ec2-gpu-metal-instance-stop-time)

Taz · Answer

Hey,

Hope you're keeping well.

In EKS managed node groups, EC2 instances are owned by the underlying Auto Scaling group, so direct termination from the EC2 console is usually blocked. A stuck instance in `shutting-down` state often means the ASG is trying to drain and replace it but is blocked by IAM or Kubernetes finalizers. Check the node group in the EKS console and try scaling it down to force the ASG to remove the instance. If that fails, go to the Auto Scaling group in the EC2 console, detach the instance, then terminate it from EC2 (requires appropriate IAM permissions). You may also need to remove any Kubernetes finalizers from the Node object with `kubectl edit node ` so the deletion can complete.

Thanks and regards,  
Taz

Instance in shutting down state for 9 days

Add your answer

Relevant content