High EC2 CPU Utilization without recent changes in the code
Hi community, I need your help on my investigation to understand why my EC2 instances began to behave like that. Since last Tuesday the CPU utilization of the two instances I had in the frontend has very high. It used to be around 30% most of the time, but since Tuesday it varied a lot and has stayed around the 80% zone for a long time. Today it fell to 30% again. All this happened without any big change in the code of my application. And even after adding a third instance back in Tuesday, the utilization stayed the same. Could it be some issue at the AWS servers? What could I do to investigate the further? All the three instances are t3.large with 8Gb of storage using Amazon Linux 2023.
- Newest
- Most votes
- Most comments
If CPU utilisation continues to be high, the simplest thing you can do to see what's happening is to log on to the operating system on the instance and use its internal tools, like top in Linux and Task Manager in Windows to find out which processes the CPUs are busy with.
What you're seeing almost certainly is not related to anything AWS is doing. They don't run software on your EC2 instance.
I'll actually suggest accessing the instances, and see running processes, the users, PID, memory and CPU ultilization of all running processes. This is done on linux/ mac using ... ps aux
Then pick any process that may be a culprit and run... ps -p <PID> -o command where -p points to the process id (PID) and -o, the output format.
Following up on this, as I said on another comment the high load happened due to a high traffic coming from Facebook ASN. So what I did to stop that was to create a rule on WAF to block traffic coming from those IPs, next step is to refine the configuration and block only unwanted traffic. Thank you for your help!
Relevant content
- asked 2 years ago
Hey guys, thank you for your prompt answer. As I said, the CPU utilization has returned to the 30% region just a few minutes before I decided to post here. I ran the
topcommand and gotnodeas the application that is using CPU the most, which is what I expected. But right now there is nothing uncommon, so I can't really get to a conclusion with that information. Though there is something else I wanted to add that might help. I created an internet monitor on CloudWatch these days and it seems my application has been getting a lot of traffic from FACEBOOK (AS32934). Does that ring a bell?Yes, traffic from Facebook's Autonomous System (AS32934) could indeed be a significant factor in the increased CPU utilization of your EC2 instances