How can I monitor my Lightsail resource usage from the command line?

Short description

There are several factors, including high resource utilization, that affect services running on Lightsail instances. Using common tools you can monitor resources such as CPU, memory, disk I/O, and network usage on Lightsail instances in real time from the command line.

Analyzing and monitoring Lightsail instance resource usage from the command line provides real-time system insight. This also allows for a granular view of which processes are using the most CPU, memory, disk I/O, or network.

The following are resources you can use to analyze and monitor your Lightsail instance's resources from the command line:

• The atop tool
• The htop tool
• The iostat, vmstat, and mpstat commands

Resolution

The atop tool

The atop tool reports the activity of all processes, even if those processes have finished during the specified interval. This tool logs all relevant system information continuously. Because of this continuous logging, if issues reoccur, you have historical data to analyze. The atop tool is included in official repositories for most Linux distributions. Install the atop tool by running the following commands:

Red Hat-based systems

$ sudo yum install atop

Debian-based systems

$ sudo apt install atop

The atop tool logs all activity in 600-second intervals by default. To change the atop tool's configuration so that activities are logged in 60-second intervals, run the following command:

Red Hat-based systems

$sudo sed 's/600/60/' /etc/atop/atop.daily -i

Debian-based systems

$ sudo sed 's/600/60/' /etc/default/atop -i

After running the preceding command, atop logs all activities within 60-second internals. The data is stored in log files in /var/log/atop. These files are named in the following format " atop_ccyymmdd". For example, "atop_20210902" is the recording for September 2, 2021.

To access the log file, run the command atop -r atologfilepath. The command and log file are shown in the following example:

atop -r /var/log/atop/atop_20210902 

ATOP - ip-172-20-139-91                2021/09/02  17:03:44                ----------------                 3h33m7s elapsed
PRC |  sys    6.51s  |  user   7.85s  |  #proc    103  |  #tslpi    81 |  #tslpu     0  |  #zombie    0  |  #exit      0  |
CPU |  sys     0%  |  user      3%  |  irq       0%  |  idle    197% |  wait      0%  |  ipc notavail  |  curscal   ?%  |
cpu |  sys     0%  |  user      1%  |  irq       0%  |  idle     98% |  cpu000 w  0%  |  ipc notavail  |  curscal   ?%  |
cpu |  sys     0%  |  user      1%  |  irq       0%  |  idle     98% |  cpu001 w  0%  |  ipc notavail  |  curscal   ?%  |

In the preceding output example, the first recorded snapshot was at 2021/09/02 17:03:44. To move forward to the next snapshot, press the ‘ t’ key on the keyboard. To move back to the previous snapshot, press the "T" (uppercase "t") key.

The following are commonly used options that you can launch atop with:

Sort by memory

$ atop -m

Sort by disk details

$ atop -d

Sort by network details

$ atop -n

The htop tool

The htop tool is an interactive and real-time process monitoring application for Linux that shows your usage per CPU or core. This tool also provides a meaningful text graph of your memory and swap usage.

Run the following commands to install htop:

Red Hat-based systems

$ sudo yum install htop

Debian and Ubuntu-based systems

$ sudo apt install htop

Use the following command to run htop on your system:

$ htop

The htop tool lists all running processes on the system and includes information about how much CPU and memory each process is using. The command used to start the process is also included.

The following are definitions for each column in the output:

• PID: A process ID number for the process.
• BUSER: The process owner.
• PR: The process’s priority. The lower the number, the higher the priority.
• NI: The nice value of the process, which affects its priority.
• VIRT: How much virtual memory the process is using.
• RES: How much physical RAM the process is using, measured in kilobytes.
• SHR: How much shared memory the process is using.
• S: The current status of the process (zombied, sleeping, running, uninterruptedly sleeping, or traced).
• CPU: The percentage of the processor time used by the process.
• MEM: The percentage of physical RAM used by the process.
• TIME+: How much processor time the process has used.
• COMMAND: The name of the command that started the process.

The iostat, vmstat, and mpstat commands

The iostat, vmstat, and mpstat commands are part of the sysstat bundle. Use the following commands to install this set of tools in your Lightsail instance:

Red Hat-based systems, such as Amazon Linux, CentOS, or RHEL

$ sudo yum install sysstat

Debian and Ubuntu

$ sudo apt-get install sysstat

Use the following commands to start and enable sysstat for continuous data collection:

$ sudo systemctl start sysstat

$ sudo systemctl enable sysstat

'iostat' command

The iostat command collects and displays system input and output storage device statistics. This tool is often used to trace storage device performance issues including devices and disks. The following are common iostat options:

• iostat: Get report and statistics.
• iostat -x: Show more detailed statistics information.
• iostat -c: Show only the CPU statistic.
• iostat -d: Display only the device report.
• iostat -xd: Show extended I/O statistics for the device only.

The following is example iostat command output:

$ iostat
Linux 4.14.246-187.474.amzn2.x86_64     10/22/2021     _x86_64_    (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.34    0.00    1.36    1.06    1.67   90.58

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
xvda             41.64      1496.90       890.16     502148     298612

'vmstat' command

The vmstat (virtual memory statistics) command collects and displays information about system memory, processes, interrupts, paging, and block I/O. The vmstat command allows you to specify a sampling interval that permits observing system activity in near-real time. The following is an example of the vmstat command and output:

$ vmstat 2 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 0 0 1409320 2088 485404 0 0 568 724 232 393 4 2 93 1 1
0 0 0 1409332 2088 485404 0 0 0 67 146 261 0 0 100 0 0
0 0 0 1408916 2088 485404 0 0 0 0 157 276 0 0 100 0 0
0 0 0 1408980 2088 485408 0 0 0 0 161 259 0 0 100 0 0
0 0 0 1408984 2088 485408 0 0 0 0 105 190 0 0 100 0 0

The following are commonly used options you can use with the vmstat command:

The -a option displays the active and inactive memory of the system:

$ vmstat -a
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free inact active si so bi bo in cs us sy id wa st
2 0 0 1402016 311232 218864 0 0 60 77 74 129 0 0 99 0 0

The -d option reports disk statistics:

$ vmstat -d
disk- ------------reads------------ ------------writes----------- -----IO------
total merged sectors ms total merged sectors ms cur sec
nvme0n1 7728 25 433681 4544 3133 93 555950 4196 0 1

'mpstat' command The mpstat command collects and displays performance statistics for all logical processors in the system. Running the mpstat command with no options generates a single report that contains the performance statistics for all logical processors since boot time.

$ mpstat 
Linux 4.18.0-305.el8.x86_64 (ip-xxx-xx-x-xx.ec2.internal)     10/27/2021     _x86_64_    (2 CPU)

04:23:26 PM  CPU    %usr   %nice    %sys  %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
04:23:26 PM  all    8.42    0.13    2.43    1.49     0.18    0.05    0.22    0.00    0.00   87.08

The first line in the preceding example is a set of column labels. The second line is the value for each column:

• %usr: % CPU usage at the user level.
• %nice: % CPU usage for user processes labeled "nice".
• %sys: % CPU usage at the system (Linux kernel) level.
• %iowait: % CPU usage idling waiting on a disk read/write.
• %irq: % CPU usage handling hardware interrupts.
• %soft: % CPU usage handing software interrupts.
• %steal: % CPU usage being forced to wait for a hypervisor handling other virtual processors.
• %guest: % CPU usage spent running a virtual processor.
• idle: %CPU usage on idle time (no processes, and not waiting on a disk read/write).

Using the preceding command line tools, you can identify the processes with high resource usage. And, you can use this information to choose instances with sufficient resources for your workload.

---