AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
Why Do My t3.medium EC2s Max Out CPU/Memory on Boot with Agents Installed?
My ec2 instances are failing to start properly, or when they do start, they are taking too long to boot. When they do come up I ssh in and see that CPU and memory are running at 100%. I use t3.medium instances and I’m loading the SSM agent, a Cloudwatch agent, and a vulnerability scanning agent, all of them required by my company. What can I do to resolve this?
- Topics
- Compute
- Tags
- Amazon EC2
- Language
- English
asked a year ago379 views
- Newest
- Most votes
- Most comments
Hi. Thanks for the question! I've worked through this issue before with burstable instance types, and these are the things I've tried to get past this.
You can configure Agent Resource Limits by creating systemd service limits in order to make sure that one agent load is not impacting the rest during the load. Here's what that would look like:
bash
For CloudWatch Agent
sudo mkdir -p /etc/systemd/system/amazon-cloudwatch-agent.service.d/ cat > /etc/systemd/amason-cloudwatch-agent.servie.d/limits.conf << eof [Service] CPUQuota=20% MemoryLimit=256M EOF sudo systemctl daemon-reload
If this doesn't solve the issue, you may consider using a larger instance size in the same family, i.e. t3.large vs t3.medium, for startup, and then move to the smaller instance size after launch.
Also, you can monitor and alert on resource usage with CloudWatch alarms. To set that up from the command line, you can use the following:
bash
aws cloudwatch put-metric-alarm
-- alarm-name "High-CPU-During-Startup"
-- alarm-description "Alert when CPU > 90% for 5 minutes"
-- metric-name CPUUtilization
-- namespace AWS/EC2
-- statistic Average
-- period 300
-- threshold 90
-- comparison-operator GreaterThanThreshold
-- evaluation periods 1
Try the above steps and see what is most effective for you. If you need any more help, please reach out!
Based on your description, you're experiencing high CPU and memory utilization on your t3.medium instances during boot, which is causing slow startup times or boot failures. This is likely related to the combination of instance type and the agents you're running. Here are some potential causes and solutions:
-
Burstable Instance Limitations: T3 instances are burstable performance instances with a limited CPU credit balance. When multiple resource-intensive agents start simultaneously during boot, they can quickly deplete your CPU credits, causing throttling and performance issues.
-
SSM Agent Issues: There have been reported cases where the SSM agent can consume 100% CPU, especially during startup or when performing operations. This can severely impact smaller instance types.
-
Resource Contention: Multiple agents starting simultaneously can create resource contention, particularly on instances with limited resources like t3.medium (2 vCPUs, 4GB RAM).
Solutions to consider:
-
Upgrade Instance Type: Consider upgrading to a larger instance type with more CPU and memory resources. You could move to a t3.large for more memory, or switch to a non-burstable instance type like m5 or c5 series that have dedicated CPU resources.
-
Stagger Agent Startup: Modify your instance initialization scripts to start agents sequentially rather than simultaneously, giving each agent time to initialize before starting the next one.
-
Enable Unlimited Mode: If you must stay with t3 instances, enable "unlimited" mode which allows the instance to burst beyond its baseline even when CPU credits are depleted (though at additional cost).
-
Optimize Agent Configurations: Review and optimize the configuration of each agent to reduce resource consumption. For example, adjust polling intervals or reduce the scope of monitoring.
-
Monitor CPU Credits: Set up CloudWatch alarms to monitor CPU credit balance and utilization to help identify when you're approaching resource limits.
-
Check for Agent Issues: Review logs for the SSM agent and other agents to identify if any specific agent is causing excessive resource consumption. The SSM agent logs can be particularly helpful in identifying issues.
If you must maintain these agents on your instances, upgrading to a larger instance type is likely the most straightforward solution to ensure reliable performance.
Sources
AWS EC2 T2 Medium Performance Issue | AWS re:Post
Troubleshoot high CPU utilization | AWS re:Post
Why is ssm-agent-worker using 100% of CPU? | AWS re:Post
Relevant content
- AWS OFFICIALUpdated 2 years ago
