AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
EC2 instance unresponsive with high CPU and OOM event
Hi there.
We recently had an issue where our t3.medium EC2 nodes became unresponsive and we were unable to SSH into it.
CPU utilization was sustained at approximately 75–100% for several hours, and the EC2 instance status check failed. Compute Optimizer also reports the instance as under-provisioned.
During investigation, we found an Out of Memory (OOM) event where Linux killed an aws process. We also identified a monitoring script scheduled via cron every minute. The script uses kubectl to check EKS pods and sends metrics to CloudWatch.
We found that a single execution can take more than 3 minutes, resulting in multiple executions overlapping. We suspect these overlapping processes contributed to the high CPU/memory usage and eventual instability.
Could you please advise whether this is a reasonable root-cause conclusion and whether there are any additional AWS metrics or logs we should check to confirm it?
Thank you.
- Newest
- Most votes
- Most comments
You nailed the diagnosis. Overlapping cron jobs on an under-provisioned instance is exactly the pattern AWS documents for this failure mode (the EC2 Knowledge Center article on unresponsive Linux instances calls out checking for scheduled job overlaps as a root-cause step, and OOM kills when memory exhausts with no swap).
Two durable fixes worth considering: CloudWatch Container Insights with the EKS add-on replaces custom kubectl polling entirely. It's a DaemonSet that collects pod/cluster metrics natively without cron scheduling, so you can't get runaway script overlaps. Setup is ~30 min (kubectl apply the add-on), and cost is usually $20–$50/month for small clusters (CloudWatch Logs ingestion, pay per GB).
Second: Compute Optimizer flagged the t3.medium as under-provisioned before this incident hit. Acting on those recommendations proactively (before they become OOM crashes) is part of the Well-Architected Reliability pillar's design-to-prevent-failure practice. If you haven't opted in to Compute Optimizer yet, it's free and takes <5 min.
I do AWS reliability and observability advisory work. If you want a second set of eyes on the monitoring architecture or a Well-Architected review scoped to this specific incident (what alarm thresholds would have caught it early, how to safely cut over to Container Insights without losing visibility), happy to help.
answered 15 days ago
Your root-cause conclusion appears reasonable. When an instance experiences high resource usage, it can become unresponsive, and if out-of-memory issues result in kernel panic errors, the operating system might initiate a shutdown. The combination of sustained high CPU utilization (75-100%), an OOM event that killed a process, and overlapping executions of your monitoring script strongly suggests that resource exhaustion was the primary cause of the unresponsiveness and status check failure.
To further confirm your analysis, consider reviewing the following:
-
Check your instance's system logs and console output for additional OOM messages or kernel panic indicators. These logs can help you identify the exact processes that were terminated and the timing of memory exhaustion events.
-
Review any CloudWatch alarms, EventBridge rules, or AWS Systems Manager automation you may have configured. If you have automations set up to stop unresponsive instances, one of these might have triggered during the high resource usage period.
-
Examine CloudWatch metrics for memory utilization patterns during the incident timeframe to correlate with the CPU spikes and identify when memory pressure began.
-
Analyze the timing and frequency of your cron job executions against the resource usage patterns to confirm the overlap hypothesis.
Given that Compute Optimizer has identified the instance as under-provisioned and you're experiencing overlapping script executions, you should consider either optimizing your monitoring script to complete faster, adjusting the execution frequency to prevent overlaps, or upgrading to a larger instance type with more CPU and memory resources to handle the workload.
Sources
Understand why EC2 instance stopped | AWS re:Post
Troubleshoot Amazon EC2 Linux instances with failed status checks - Amazon Elastic Compute Cloud
Relevant content
asked 3 years ago
- AWS OFFICIALUpdated 7 months ago
This is insightful. Thanks for your response