How can I configure and use SAR to monitor performance metrics on my EC2 Linux instance?

Short description

The SAR monitoring tool allows you to collect performance data as needed for CPU, memory, and I/O usage. You can configure SAR to collect historical metrics.

Resolution

Install SAR

SAR, which is part of the sysstat package, might not be available by default on newer Linux distributions. For detailed installation instructions, see How do I configure the ATOP and SAR monitoring tools for my EC2 instance running Amazon Linux, RHEL, CentOS, or Ubuntu?

Configuration overview

You can view and edit the configuration file, /etc/sysconfig/sysstat, after installation. The configuration file contains the following default values:

• HISTORY=28 - This parameter determines how long SAR keeps performance log files. The default value is 28 days.
• COMPRESSAFTER=31 - This parameter determines after how many days SAR compresses files. The default value is 31 days.
  Note: If you set the configuration file to keep log files longer than 28 days, then SAR keeps each month's files in a separate directory.
• SA_DIR=/var/log/sa - SAR saves log files to this location.
• ZIP="bzip2" - This is the default zip file containing compressed log files. Make sure that you enable compression if you keep files for more than 31 days. Compressed files occupy the least possible space on the disk.

The sadc (system activity data collector) operation performs data collection. Collected data is written in text format and contained in the sar## files in the /var/log/sa/ directory.

The sadc operation is classified into two components:

1. sa1 - Collect and store binary data in the system activity daily data file.
2. sa2 - Write a daily report to the /var/log/sa directory.

The /var/log/sa/ directory contains the following two sets of files:

• sa# - system activity binary data files.
• sar## - system activity report files.

Note: ## represents the day of the month.

You can collect data other than disk statistics, such as partition and file system statistics*.* To change what data is collected, do the following:

1.    Access the /etc/sysconfig/sysstat file using a text editor, such as the vi editor:

vi /etc/sysconfig/sysstat

2.    Change SADC_OPTIONS from -S Disk to -S XALL:

SADC_OPTIONS=" -S XALL"

Set data collection intervals

Run the following command to view the scripts that the SAR utility currently runs in crontab to generate data:

cat /etc/cron.d/sysstat

By default, SAR generates data every 10 minutes. This is due to cron, which runs every 10 minutes. You can edit this time to allow SAR to generate data for shorter or longer intervals. The following is an example of the crontab file showing that the data is set to generate every 10 minutes:

*/10 * * * * root /usr/lib64/sa/sa1 1 1

It's a best practice to change cron to run on a 5 or 1 minute interval. By changing to a shorter interval, you can catch cases where spikes are observed or when every minute monitoring is required.

To change the interval:

1.    Access the /etc/cron.d/sysstat file:

vim /etc/cron.d/sysstat

Change the run time from 10 to 5:

@reboot /usr/lib64/sa/sa1 --boot
*/5 * * * * root /usr/lib64/sa/sa1 1 1

Note: In the previous example, the @reboot line resets the counter when the OS reboots. This is optional, but is a best practice.

Example commands to generate performance reports as needed

CPU usage report

Use the sar command to generate a CPU usage data report. The following example command generates a usage data report 5 times every 2 seconds. In the following command, -P indicates an individual CPU or core. Indicate which CPU or core you want to generate statistics for using 0, 1, 2, 3, and so on. Or, you can use the command -P ALL to display statics for all CPUs.

sar 2 5 -P 0

Memory statistics report

Use the -r option to generate a memory statistics report. The following example command generates memory statistics 5 times every 2 seconds:

sar -r 2 5

Block device statistics report

Use the -d option to generate a block device statistics report. The following example command generates block device statistics 5 times every 2 seconds. In the following example, the -p option displays the device names as they appear in /dev:

sar -d -p 2 5

Network statistics report

Use the -n option to generate a network statistics report. You can generate statistics for specific entities, such as TCP, UDP, NFS, etc. or ALL as shown in the following examples:

sar -n TCP 2 5

sar -n UDP 2 5

sar -n ALL 2 5

Read the most recent log file

Running sar commands extracts details from the most recent log file in /var/log. To read the complete file, use the -f option, as shown in the following example:

sar -f /var/log/sa

Extract log details for a specific day

The log file is saved by day. If you want to extract details for a specific day, use the -f option for the relevant sa file on that date. For example, the following command displays all the data from the log file containing data from the 12th day of the current month:

sar -f /var/log/sa/sa12

Read specific statistics from a log file

Memory statistics

Use the -r option to read all memory statistics. The following example command displays all memory statistics from the log file generated on the 12th day of the month:

sar -f /var/log/sa/sa12 -r

I/O activity

Use the -b option to display all combined I/O activity. The following example command displays all I/O activity from the log file generated on the 12th day of the month.

sar -f /var/log/sa/sa12 -b

Block device statistics

Use the -dp option to display block device statistics. The following example command displays block device statistics from the log file generated on the 12th day of the month.

sar -f /var/log/sa/sa12 -dp

Network statistics

Use the -n option to display network statistics. The following example command displays all network statistics from the log file generated on the 12th day of the month:

sar -f /var/log/sa/sa12 -n ALL

CPU statistics

Use the -P option to display CPU statistics. The following example command displays CPU statistics for CPU 0.

sar -f /var/log/sa/sa12 -P 0

View statistics for a specific time period

Use the -s and -e options to designate start and end times. The times entered must be in the 24 hour format. The default end time is 18:00. The following example command displays block device statistics from the log file generated on the 30th day of the month between 2:00 and 5:00.

sar -f /var/log/sa/sa30 -dp -s 02:00:00 -e 05:00:00

Related information

sar(1) -Linux man page