Why do I see "audit: backlog limit exceeded" errors in my EC2 Linux instance's screenshot and system logs, and what can I do to avoid this?

Short description

The audit backlog buffer in a Linux system is a kernel-level socket buffer queue that the operating system uses to maintain or log audit events. When a new audit event occurs, the system logs the event and then adds it to the audit backlog buffer queue.

The backlog_limit parameter value is the number of audit backlog buffers. By default, the parameter is set to 320, as the following example shows:

# auditctl -s
enabled 1
failure 1
pid 2264
rate_limit 0
backlog_limit 320
lost 0
backlog 0

The following errors on the instance are caused when audit events are logged beyond the default number:

audit: audit_backlog=321 > audit_backlog_limit=320 

audit: audit_lost=44393 audit_rate_limit=0 audit_backlog_limit=320 

audit: backlog limit exceeded

-or-

audit_printk_skb: 153 callbacks suppressed

audit_printk_skb: 114 callbacks suppressed

An audit buffer queue at or over capacity might also cause the instance to freeze or remain in an unresponsive state.

To avoid backlog limit exceeded errors, increase the backlog_limit parameter value. Large servers have a larger number of audit logs activated. Increase buffer space to help avoid error messages.

Note: The increase of the audit buffer consumes more of the instance's memory. The value that you set for the backlog_limit parameter depends on the total memory of the instance. If the system has enough memory, you can double the existing backlog_limit parameter value.

The following example is a calculation of the memory required for the audit backlog. Use this calculation to determine how much you can increase the size of the backlog queue without causing memory stress on your instance.

One audit buffer = 8970 Bytes
Default number of audit buffers (backlog_limit parameter) = 320
320 * 8970 = 2870400 Bytes, or 2.7 MiB

The MAX_AUDIT_MESSAGE_LENGTH parameter defines the size of the audit buffer. For more information, see MAX_AUDIT_MESSAGE_LENGTH in the Linux audit library on the GitHub website.

Note: If your instance is inaccessible and you see backlog limit exceeded messages in the system log, first stop and start the instance. Then, complete the following steps to change the audit buffer value.

Resolution

Note: In this example, the backlog_limit parameter value is changed to 8192 buffers. 8192 buffers equals 70 MiB of memory based on the previous calculation. You can use any value based on your memory calculation.

1. Use SSH to connect to the instance.

2. Verify the current audit buffer size.

   Note: The backlog_limit parameter is listed as -b. For more information, see auditctl(8) on the die.net website.

   Amazon Linux 1 and other operating systems that don't have systemd

   $ sudo cat /etc/audit/audit.rules
   # This file contains the auditctl rules that are loaded
   # whenever the audit daemon is started via the initscripts.
   # The rules are simply the parameters that would be passed
   # to auditctl.
   
   # First rule - delete all
   -D
   
   # Increase the buffers to survive stress events.
   # Make this bigger for busy systems
   -b 320
   
   # Disable system call auditing.
   # Remove the following line if you need the auditing.
   -a never,task
   
   # Feel free to add below this line. See auditctl man page

   Amazon Linux 2 and other operating systems that use systemd

   $ sudo cat /etc/audit/audit.rules
   # This file is automatically generated from /etc/audit/rules.d
   -D
   -b 320
   -f 1

3. To access the audit.rules file, use an editor such as vi:

   Amazon Linux 1 and other operating systems that don't use systemd

   $ sudo vi /etc/audit/audit.rules

   Amazon Linux 2 and other operating systems that use systemd

   $ sudo vi /etc/audit/rules.d/audit.rules

4. Edit the -b parameter to a larger value. The following example changes the -b value to 8192.

   $ sudo cat /etc/audit/audit.rules
   # This file contains the auditctl rules that are loaded
   # whenever the audit daemon is started via the initscripts.
   # The rules are simply the parameters that would be passed
   # to auditctl.
   
   # First rule - delete all
   -D
   
   # Increase the buffers to survive stress events.
   # Make this bigger for busy systems
   -b 8192
   
   # Disable system call auditing.
   # Remove the following line if you need the auditing.
   -a never,task
   
   # Feel free to add below this line. See auditctl man page
   
   $ sudo auditctl -s
   enabled 1
   failure 1
   pid 2264
   rate_limit 0
   backlog_limit 320
   lost 0
   backlog 0

   Restart the auditd service. The new backlog_limit value takes effect. The value also updates in auditctl -s, as following example shows:

   # sudo service auditd stop
   Stopping auditd:                                           [  OK  ]
   # sudo service auditd start
   Starting auditd:                                           [  OK  ]
   # auditctl -s
   enabled 1
   failure 1
   pid 26823
   rate_limit 0
   backlog_limit 8192
   lost 0
   backlog 0

   Note: If your instance is inaccessible and backlog limit exceeded messages appear in the system log, first stop and start the instance. Then, complete the previous steps to change the audit buffer value.