How do I troubleshoot high memory usage issues on my EC2 Linux instance?

Short description

When EC2 Linux instances experience high memory utilization, it can lead to system performance degradation or freezes. These performance issues typically occur because of memory leaks, resource-intensive processes, or insufficient memory allocation. To resolve these issues, first use Linux system tools to identify the root cause of the issue. Then, follow the troubleshooting steps for the cause of the issue.

Resolution

Identify the root cause of the issue

To identify the cause of the issue, identify memory usage patterns for your EC2 instance. To check current memory usage, run the following command:

free -h

Example output:

 total        used        free      shared  buff/cache   availableMem:           15Gi       7.2Gi       2.1Gi       1.2Gi       5.7Gi       
 6.3GiSwap:           8Gi       6.5Gi       1.5Gi

To find memory-intensive processes on your EC2 Linux instance, run the top command:

top

Example output:

    top - 14:23:45 up 15 days, 3:42, 1 user, load average: 2.15, 1.98, 1.75
Tasks: 256 total,   2 running, 254 sleeping,   0 stopped,   0 zombie
%Cpu(s): 24.8 us, 12.3 sy,  0.0 ni, 62.1 id,  0.8 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  32768.0 total,   2453.2 free,  24156.4 used,   6158.4 buff/cache
MiB Swap:   4096.0 total,   3012.5 free,   1083.5 used.   2537.2 avail Mem 

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  1234 mysql     20   0   18.2g  15.1g   5.1g S  45.0  47.2   220:31 mysqld
  5678 tomcat    20   0   12.8g  11.2g   124m S  85.2  70.1   445:22 java

To find processes that consume the most memory in your instance, run the following command:

ps aux --sort=-%mem | head -n 5

Example output:

USER       PID  %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMANDmysql     1234  45.0 47.2 18.2g 15.1g ?        Ssl  Jan10 220:31 mysqldjava      5678  85.2 70.1 12.8g 11.2g ?        Ssl  Jan12 445:22 javaapache    9012  25.3 23.8  4.2g  3.8g ?        Ss   Jan12 178:44 httpd

High swap activity can significantly slow down your system. For more information, see Enable instance store swap volume for M1 and C1 EC2 instances.

To check for swap activity, run the following command:

vmstat 1 5

Example output:

    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0 6650880  2150012 189408 5678024 45  67   234   567 300 450 65 15 10 10  0

Troubleshoot your issue based on your memory usage patterns

Your instance has a high swap usage rate

Complete the following steps:

1. To reduce swappiness, run the following command:

   sudo sysctl vm.swappiness=10

2. To clear swap, run the following command:

   # Find processes using most swap
   $ smem -s swap -r
   
   # For specific process, check its swap usage
   $ grep VmSwap /proc/[PID]/status
   
   # Restart specific high-swap-usage processes during low-traffic periods
   $ systemctl restart [service-name]
   
   # Use Extreme caution when running this command - can cause OOM kills if physical memory is insufficient:
   $ sudo swapoff -a && sudo swapon -a

   Important: If the physical memory of the instance isn't sufficient for the swap command, then you might get out-of-memory (OOM) errors.

3. If your instance needs more space, then run the following command:

   sudo dd if=/dev/zero of=/swapfile bs=1G count=8

There are system-wide memory issues

Complete the following steps:

1. To check for OOM errors, run the following commands:

   sudo grep -i 'out of memory' /var/log/syslog
   sudo dmesg | grep -i 'out of memory'
   sudo journalctl -k | grep -i 'oom'
   sudo grep -i 'killed process' /var/log/messages

2. To monitor memory usage for your instance, run the following command:

   watch -n 1 free -m

3. To check for available memory pages, run the following command:

   cat /proc/buddyinfo

For information, see Amazon EC2 instance types.

There are memory leaks in your Java applications

Complete the following steps:

1. To adjust heap size, run the following command:

   java -Xms2g -Xmx4g -jar application.jar

   Note: Replace 2g and 4g with the minimum and maximum heap sizes for your application.
2. To turn on garbage collection (GC) logging, run the following command:

   java -Xlog:gc*=debug:file=gc.log -jar application.jar

   Note: Replace gc.log with your log file name and path.
3. To use the G1 garbage collector to better manage your instance memory, run the following command:

   java -XX:+UseG1GC -jar application.jar

   Note: Replace -XX:+UseG1GC with your preferred garbage collection algorithm.

There are issues with your database servers

Complete the following steps:

1. To monitor specific database server processes, run the following command:

   ps -eo pid,ppid,cmd,%mem,%cpu,rss,vsz | grep mysql

2. To check the InnoDB buffer pool usage, run the following command:

   mysql -e "SHOW ENGINE INNODB STATUS\G" | grep "Buffer pool"

3. To analyze the memory use for each connection, run the following command:

   mysql -e "SHOW VARIABLES LIKE '%buffer%';"

For information on how to optimize database server performance, see Best practices for working with MySQL.

Related information

Collect metrics, logs, and traces with the CloudWatch agent