We have a CentOS Linux server in AWS that runs monitoring software. This server has run exceptionally well for a few years until a few weeks ago where it started experiencing high loads even when sitting idle. The previous time this occurred, we troubleshot it, upgraded the software and the OS, and felt like it was potentially fixed. With a high load yet low CPU, low memory, and zero swap used, the assumption was maybe some disk I/O issue in the server farm that resolved itself. The problem resurfaced late yesterday and after digging more this evening, we noticed the CPU steal time ('st' in the top command) is absurdly high; CPU steal time is where the hypervisor or other servers in the shared virtual environment are causing this system to wait on CPU time. If the system is simply rebooted, the problem remains. The only way we have resolved it both times is if the instance is stopped, we wait a few mins, and then start it again. Is it possible the instance is switched to a different server farm without a resource hog when we perform the extended shutdown? Does this mean AWS has over-allocated resources? Are there any other ways to prevent this from happening again beyond paying for a dedicated server? Any help would be appreciated!
top - 21:47:37 up 10 days, 1:30, 2 users, load average: 18.77, 11.27, 11.48
Tasks: 155 total, 5 running, 150 sleeping, 0 stopped, 0 zombie
%Cpu(s): 18.3 us, 4.1 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 77.6 st
KiB Mem : 3878444 total, 1144048 free, 910772 used, 1823624 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 2483948 avail Mem
top - 21:47:46 up 10 days, 1:30, 2 users, load average: 17.20, 11.18, 11.45
Tasks: 157 total, 9 running, 148 sleeping, 0 stopped, 0 zombie
%Cpu(s): 19.0 us, 4.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 76.7 st
KiB Mem : 3878444 total, 1120580 free, 934224 used, 1823640 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 2460496 avail Mem
AWS OFFICIALUpdated a year ago
What instance type is this?