Collect data with sar

Table of Contents

man sar - The sar(system activity report) command writes to standard output the contents of selected cumulative activity counters in the operating system.

Configuration files #

The main configuration file is /etc/sysconfig/sysstat. On Debian, it’s /etc/default/sysstat. It contains information about the location of the log files generated by sar, log compression to used, how long to keep log files, etc:

[root@rockylinux ~]# cat /etc/sysconfig/sysstat
# sysstat-11.7.3 configuration file.

# How long to keep log files (in days).
# If value is greater than 28, then use sadc's option -D to prevent older
# data files from being overwritten. See sadc(8) and sysstat(5) manual pages.
HISTORY=28

# Compress (using xz, gzip or bzip2) sa and sar files older than (in days):
COMPRESSAFTER=31

# Parameters for the system activity data collector (see sadc manual page)
# which are used for the generation of log files.
SADC_OPTIONS=" -S DISK"

# Directory where sa and sar files are saved. The directory must exist.
SA_DIR=/var/log/sa

# Compression program to use.
ZIP="xz"

# By default sa2 script generates yesterday's summary, since the cron job
# usually runs right after midnight. If you want sa2 to generate the summary
# of the same day (for example when cron job runs at 23:53) set this variable.
#YESTERDAY=no

# By default sa2 script generates reports files (the so called sarDD files).
# Set this variable to false to disable reports generation.
#REPORTS=false

Differences On Debian and Redhat/Centos Distributions #

As of this writing, Debian based distros and Redhat Versions less than 8, keeps a cronjob that runs every 10 mins be default. It’s in /etc/cron.d/sysstat:

root@debian:/var/log/sysstat# cat /etc/cron.d/sysstat 
# The first element of the path is a directory where the debian-sa1
# script is located
PATH=/usr/lib/sysstat:/usr/sbin:/usr/sbin:/usr/bin:/sbin:/bin

# Activity reports every 10 minutes everyday
5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1

# Additional run at 23:59 to rotate the statistics file
59 23 * * * root command -v debian-sa1 > /dev/null && debian-sa1 60 2

It’s not enabled by default on Debian. Change false to true in /etc/default/sysstat, and restart the sysstat service.

Redhat 8 #

I’m using Rocky Linux. Instead of a cronjob, we have a SystemD Timer:

[root@rockylinux sa]# find / -name *sysstat* | grep collect.timer
/etc/systemd/system/sysstat.service.wants/sysstat-collect.timer
/usr/lib/systemd/system/sysstat-collect.timer

The content:

[root@rockylinux sa]# cat /usr/lib/systemd/system/sysstat-collect.timer
# /usr/lib/systemd/system/sysstat-collect.timer
# (C) 2014 Tomasz Torcz <tomek@pipebreaker.pl>
#
# sysstat-11.7.3 systemd unit file:
#        Activates activity collector every 10 minutes

[Unit]
Description=Run system activity accounting tool every 10 minutes

[Timer]
OnCalendar=*:00/10

[Install]
WantedBy=sysstat.service

sa logs #

The logs are located in /var/log/sa/:

[root@rockylinux sa]# ls -l
total 164
-rw-r--r--. 1 root root 32720 May 16 19:30 sa16
-rw-r--r--. 1 root root 24436 May 17 19:40 sa17
-rw-r--r--. 1 root root 65800 May 18 15:40 sa18
-rw-r--r--. 1 root root 29940 May 19 14:10 sa19
-rw-r--r--. 1 root root  6576 May 22 11:50 sa22

The number after sa is the date of the current month. The contents are not in plain text:

[root@rockylinux sa]# file -i sa*
sa16: application/octet-stream; charset=binary
sa17: application/octet-stream; charset=binary
sa18: application/octet-stream; charset=binary
sa19: application/octet-stream; charset=binary
sa22: application/octet-stream; charset=binary

The binary for the current day is updated every 10 mins. Starting at boot time. To open an sa file, use sar -f:

[root@rockylinux sa]# sar -f sa22
Linux 4.18.0-372.9.1.el8.x86_64 (rockylinux) 	05/22/2022 	_x86_64_	(1 CPU)

11:15:57     LINUX RESTART	(1 CPU)

11:20:21 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
11:30:21 AM     all      0.74      0.00      3.83      0.03      0.00     95.40
11:40:21 AM     all      0.86      0.00      3.51      0.01      0.00     95.63
Average:        all      0.80      0.00      3.67      0.02      0.00     95.51

11:46:32 AM  LINUX RESTART	(1 CPU)

My server was rebooted twice. You can just run sar, and the last generated file will be read.

Common Usage/Examples #

sar can display the usage of CPU, Memory, Swap, loadavg, Disk, I/O, and Network Packets. All command options listed below can be used alonside sar -f to look for specific stats in an sa file. Most sar outputs contains keywords such as rtps, and bread/s. It’s challenging to remember everything at first. Just consult its man page.

Display the CPU usage every second for 3 times: sar 1 3
Display CPU stats for all processors: sar -P all 1 3
Get Memory usage: sar -r 2 5
Swap usage: sar -S 2 5
Paging usage: sar -B 2 10
Paging vs Swapping
Monitor Load Average: sar -q 2 10
Stats about I/O: sar -b 1 10
I/O for each Block Device(-p for pretty print): sar -d -p 2 10(with -p you get to see lvm and disks with actual name instead of minor and major numbers)
Monitor the network: sar -n DEV 1 10 or sar -n NFS 2 5(consult the man page for more options)

There’s also two third party tools to convert sa files into a graph or html