How can I configure and use SAR to monitor performance metrics on my Amazon EC2 Linux instance?

Short description

Use the SAR monitoring tool to collect performance data for CPU, memory, and I/O usage, and configure SAR to collect historical metrics.

Note: Although SAR is part of the sysstat package, the tool might not be available by default on newer Linux distributions. For more information, see How do I configure the ATOP Monitoring and SAR monitoring tools for my EC2 instance running Amazon Linux, RHEL, CentOS, or Ubuntu?

Resolution

View and edit the configuration file

The configuration file contains several default values that control SAR activity. To review or edit these values, open the /etc/sysconfig/sysstat configuration file.

• HISTORY=28 - This parameter determines the number of days that SAR keeps performance log files. The default value is 28 days.
• COMPRESSAFTER=31 - This parameter determines the number of days that SAR waits before it compresses files. The default value is 31 days.
  Note: If you set the configuration file to keep log files longer than 28 days, then SAR keeps each month's files in a separate directory.
• SA_DIR=/var/log/sa - This parameter determines the location where SAR saves log files.
• ZIP="bzip2" - This parameter determines the default zip file that contains the compressed log files. If you keep files for more than 31 days, then make sure that you turn on compression.

The system activity data collector (sadc) operation performs data collection. For more information, see SADC(8) on the ArchLinux website. Collected data is formatted as text and contained in the sar## files in the /var/log/sa/ directory.

The sadc operation consists of the following components:

• sa1 - Collect and store binary data in the system activity daily data file.
• sa2 - Write a daily report to the /var/log/sa directory.

The /var/log/sa/ directory contains the following two types of files:

• sa## - system activity binary data files.
• sar## - system activity text files.

Note: The hashtag characters in the sa and sar file names represent the day of the month. For more information, see the Files on the ArchLinux website.

You can also collect data other than disk statistics, such as partition and file system statistics.

To change the data that the sadc operation collects, complete the following steps:

1. To use the vi text editor to access the /etc/sysconfig/sysstat file, run the following command:

   vi /etc/sysconfig/sysstat

2. To change SADC_OPTIONS from -S Disk to -S XALL, run the following command:

   SADC_OPTIONS=" -S XALL"

Set data collection intervals

Amazon Linux 1 and Amazon Linux 2

To view the scripts that the SAR utility runs in crontab to generate data, run the following command:

cat /etc/cron.d/sysstat

Because cron runs every 10 minutes by default, SAR also generates data every 10 minutes. To allow SAR to generate data for shorter or longer intervals, edit this time value. The following example command shows a crontab file where the data is set to generate every 10 minutes:

*/10 * * * * root /usr/lib64/sa/sa1 1 1

It's a best practice to change cron to run once each minute or after every five minutes. When you change the time value to a shorter interval, you might see cases with observable spikes.

To change the interval, complete the following steps:

1. To access the /etc/cron.d/sysstat file, run the following command:

   vim /etc/cron.d/sysstat

2. To change the amount of time from 10 minutes to 5 minutes, run the following command:

   @reboot /usr/lib64/sa/sa1 --boot
   */5 * * * * root /usr/lib64/sa/sa1 1 1

   Note: In the preceding example command, the @reboot line resets the counter when the OS reboots.

Amazon Linux 2023

To edit the unit file and create the drop-in file /etc/systemd/system/sysstat-collect.timer.d/override.conf, run the following command:

# systemctl edit sysstat-collect.timer

To add the entries, run the following command:

### Editing /etc/systemd/system/sysstat-collect.timer.d/override.conf
### Anything between here and the comment below will become the new contents of the file

[Unit]
Description=Run system activity accounting tool every 1 minute

[Timer]
OnCalendar=
OnCalendar=*:00/1

### Lines below this comment will be discarded

In the preceding command, the blank "OnCalendar=" line removes the earlier setting and then creates a new one.

To reload the change, run the following systemctl daemon-reload command:

# systemctl daemon-reload

To check the status of the timer, run the following command:

# systemctl status sysstat-collect.timer 
sysstat-collect.timer - Run system activity accounting tool every 1 minute
     Loaded: loaded (/usr/lib/systemd/system/sysstat-collect.timer; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/sysstat-collect.timer.d
             └─override.conf
     Active: active (running) since Thu 2023-07-20 08:49:55 UTC; 18s ago
      Until: Thu 2023-07-20 08:49:55 UTC; 18s ago
    Trigger: n/a
   Triggers: ● sysstat-collect.service

Generate performance reports

CPU usage report

To generate a CPU usage data report, use the sar command. The following example command generates a usage data report five times every two seconds:

sar 2 5 -P 0

Note: In the preceding command, -P indicates an individual CPU or core. Type a number to indicate the CPU or core that you want to generate statistics. For example, 1, 2, or 3. Or, use the command -P ALL to show statics for all CPUs.

Memory statistics report

To generate a memory statistics report, use the -r option. The following example command generates memory statistics five times every two seconds:

sar -r 2 5

Block device statistics report

To generate a block device statistics report, use the -d option. The following example command generates block device statistics five times every two seconds. In the following example, the -p option shows the device names as they appear in /dev:

sar -d -p 2 5

Network statistics report

To generate a network statistics report, use the -n option. The following example command generates statistics for specific entities:

sar -n TCP 2 5
sar -n UDP 2 5
sar -n ALL 2 5

Note: The preceding command can generate statistics for TCP, UDP, network file system (NFS), or ALL.

Read the most recent log file

To extract details from the most recent log file in /var/log, run sar commands. The following command uses the -f option to read the complete file:

sar -f /var/log/sa

Extract log details for a specific day

Log files are saved every day. To extract details for a specific day, then use the -f option for the relevant sa file on that date. The following command shows all the data from the log file that contain data generated on day 12 of the month:

sar -f /var/log/sa/sa12

Read specific statistics from a log file

Memory statistics

To read all memory statistics, use the -r option. The following example command shows all memory statistics from the log file generated on day 12 of the month:

sar -f /var/log/sa/sa12 -r

I/O activity

To show all combined I/O activity, use the -b option. The following example command shows all I/O activity from the log file generated on day 12 of the month:

sar -f /var/log/sa/sa12 -b

Block device statistics

To show block device statistics, use the -dp option. The following example command shows block device statistics from the log file generated on day 12 of the month:

sar -f /var/log/sa/sa12 -dp

Network statistics

To show network statistics, use the -n option. The following example command shows all network statistics from the log file generated on day 12 of the month:

sar -f /var/log/sa/sa12 -n ALL

CPU statistics

To show network statistics CPU statistics, use the -P option. The following example command shows CPU statistics for CPU 0:

sar -f /var/log/sa/sa12 -P 0

View statistics for a specific time period

To select the start and end times, use the -s and -e options. You must enter the time in the 24 hour format. The default end time is 18:00. The following example command shows block device statistics from the log file generated on day 30 of the month between 2:00 a.m. and 5:00 p.m.:

sar -f /var/log/sa/sa30 -dp -s 02:00:00 -e 05:00:00

Related information

sar(1) -Linux man page on the Linux website