How do I resolve SSM Agent document worker timeout errors?

Short description

Systems Manager commands might fail on target Amazon Elastic Compute Cloud (Amazon EC2) instances when you use the following capabilities of AWS Systems Manager:

• Run Command
• Association
• Automation
• Session Manager

Error messages that resemble the following example message indicate that a Systems Manager command failed:

"document process failed unexpectedly: document worker timed out; check 12345678-abcd-1234-efgh-123456789012 log for crash reason."

Note: In the previous example error message, the placeholder value 12345678-abcd-1234-efgh-123456789012 represents the actual ssm-document-worker or ssm-session-worker ID number.

To resolve this error, troubleshoot the following issues:

• Insufficient resource issues
• Out of memory errors
• Insufficient disk space on instances
• "Too many open files" errors

Resolution

Prerequisite: Update SSM Agent to the latest version.

Note: You can subscribe to amazon-ssm-agent/RELEASE NOTES on the GitHub website to receive notifications about SSM Agent updates.

To identify the issue that cause your error, view your SSM Agent logs. Then complete the troubleshooting steps in following the sections based on the issue you identify.

Resolve insufficient resource issues

A session with a target instance might crash when the session experiences resource limitations, such as when the instance exceeds available memory or disk space. Resource limitations prevent the SSM Agent from functioning correctly. To resolve this issue, verify that your instances have sufficient memory and disk space available to manage the workload running on the target instances.

Resolve out of memory errors

If processes on your instance use all available memory, and a program or operating system can't allocate space, then the instance can't load additional programs. Then, processes stop responding, and the operating system terminates low-priority processes.

To resolve this issue, see Troubleshoot an unreachable instance. For Linux OOM issues, see Out of memory: end process.

Resolve insufficient disk space on your instance

Insufficient disk space errors occur on Linux instances when you try to write data or save files but the instance lacks available storage. To resolve this issue, take the following actions:

• Identify files that use significant disk space on the partition.
• Delete unnecessary files or move data to free up space on the partition.
• Verify that the working directories have sufficient space, including the Systems Manager orchestration folders in /var/lib/amazon/ssm.
• Increase the volume size. For instructions, see How do I increase the size of my EBS volume if I receive an error that there's no space left on my file system?

Resolve "too many open files" errors

When Linux instances stop processing Systems Manager Run Command because SSM Agent worker processes fail, you receive a "too many open files" error. When the instance opens too many files, SSM Agent can't start command processors and reports this error in the SSM Agent logs. This error occurs in the following scenarios:

• The SSM Agent process reached the maximum number of open files allowed for the operating system user account.
• The total number of open files across the system reached the system-wide limit of maximum open files.
• The system reached the kernel's inotify watch limit, which monitors file system changes.

To resolve this issue, complete the following steps:

1. Identify the PID of the SSM Agent process:

   sudo ps -C amazon-ssm-agent -o pid=

   Note: For commands in the following steps, replace PID with the process ID that this command returns.

2. Check the file descriptor limits for the SSM Agent process. The first number represents the soft limit (the current enforced limit), and the second number represents the hard limit (the maximum value the soft limit can reach):

   sudo cat /proc/PID/limits |grep "Max open files"

3. Check the total number of open files for the SSM Agent process:

   sudo lsof -p PID |wc -l

4. Compare the total number of open files (from step 3) with the hard limit (from step 2). If the total number of open files is close to the hard limit, then the system can't open new files.

To resolve this issue, increase the hard limit value in the SSM Agent service configuration file, and then restart SSM Agent.

When you increase the hard limit, consider your instance's workload requirements, including the number of concurrent users, network connections, and file operations. Linux distributions set default hard limits to prevent accidental resource exhaustion.

To update your hard limit, complete the following actions, based on your operating system:

For Amazon Linux 1, Ubuntu 14.04, and Ubuntu 16.04 with .deb package, run the following command:

echo "limit nofile example-hard-limit" >> /etc/init/amazon-ssm-agent.override

For Amazon Linux 2, RHEL 7.x, and RHEL 8.x/, run the following command:

sudo systemctl edit amazon-ssm-agent

Then, when the editor opens, add the following code to your file:

[Service]
LimitNOFILE=example-hard-limit

For Ubuntu 22.04 LTS, 20.10 STR & 20.04, and 18.04, run the following command:

sudo systemctl edit snap.amazon-ssm-agent.amazon-ssm-agent

Then, when the editor opens, add the following code to your file:

[Service]
LimitNOFILE=example-hard-limit

Note: In the previous commands, replace example-hard-limit with your updated value.

Be sure to stress test your new hard limit. Monitor your new hard limit and adjust it, if the new hard limit causes other issues.

Check the worker logs

To identify the root cause of document worker timeout errors, review the SSM Agent logs and also the OS-level logs on your instance.

Check the OS-level logs for error messages, crash reports, or kernel issues related to SSM Agent at the following locations, based on your OS:

• For Windows:
  C:\Windows\System32\winevt\Logs
• For Ubuntu and Debian:
  /var/log/syslog
• For Amazon Linux, CentOS, and RHEL:
  /var/log/messages
• For Suse:
  /var/log/messages

If you need to enable debug logging for more detailed troubleshooting, see Allowing SSM Agent debug logging.

Note: SSM Agent debug logging generates large amounts of log data that might affect system storage. After you finish troubleshooting, it's a best practice to turn off debug logging.

Related information

Troubleshooting SSM Agent