Why do I receive the “Plugin with name Standard_Stream not found” error when I use Session Manager to connect to my Amazon EC2 instance?

Short description

If the AWS Systems Manager Agent (SSM Agent) can't create the required files to establish a session, then you receive the following error message:

"Your session has been terminated for the following reasons: Plugin with name Standard_Stream not found. Step name: Standard_Stream"

This issue typically occurs if there's insufficient storage space on your instance or if you have too many files open simultaneously.

To identify the cause of the issue, check your systems logs for specific error messages. Then, take the following troubleshooting actions based on the errors that you find.

Resolution

Troubleshoot "No space left on device" errors

You must have enough space on the root partition for the SSM Agent to create the temporary data required to start a session. If you receive the "No space left on device" error message, then you must increase the available space on the root file system. First, remove unused files from the root partition. If there's still not enough space, then use Elastic Volumes to increase your Amazon Elastic Block Store (Amazon EBS) volume. Or, use one of the following methods to extend the root file system at the operating system (OS) level.

Use SSH or the EC2 Serial Console to extend the root file system

Complete the following steps:

1. Use SSH or the EC2 Serial Console to connect to your instance.
   Note: To use the EC2 Serial Console, you must configure access to the EC2 Serial Console. For more information about requirements, see Prerequisites for the EC2 Serial Console.

2. To check the available space on the root partition, run the following command:

   df -Th

   Example output:

   $ df -Th
   
   Filesystem Type Size Used Avail Use% Mounted on
   
   devtmpfs devtmpfs 4.0M 0 4.0M 0% /dev
   
   tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm
   
   tmpfs tmpfs 1.6G 440K 1.6G 1% /run
   
   /dev/nvme0n1p1 xfs 8.0G 2.0G 6.0G 25% /
   
   tmpfs tmpfs 3.9G 0 3.9G 0% /tmp
   
   /dev/nvme0n1p128 vfat 10M 1.3M 8.7M 13% /boot/efi
   
   tmpfs tmpfs 782M 0 782M 0% /run/user/1000

3. To view details such as the name and file system types of the block devices and the root partition, run the following lsblk command:

   lsblk -f

   Example output:

   $ lsblk -f
   NAME          FSTYPE LABEL           UUID                                 MOUNTPOINT
   nvme0n1
   ├─nvme0n1p1   xfs    /               abcd123-abcd-1234-abcd-abcdef1234 /
   └─nvme0n1p128

4. To extend a partition, run the following command:

   sudo growpart /dev/nvme0n1 1

   Note: Replace nvme0n1 with the partition name.

5. To verify that you extended the partition rerun the lsblk command. In the output, make sure that the partition size is the same as the volume size.

6. To expand the file system, run one of the following commands based on your file system type.
   ext4 volumes:

   sudo resize2fs /dev/nvme0n1p1

   Note: Replace nvme0n1p1 with the partition name.
   XFS volumes:

   sudo xfs_growfs -d /

Use a rescue instance to extend the root file system

If you can't use SSH to connect to an unreachable instance, then create a rescue instance in the same Availability Zone as the unreachable instance. For instructions, see Troubleshoot the "No space left on device" error on How do I troubleshoot an EC2 Linux instance that fails a status check because of resource over-usage? After you mount the root volume to the rescue instance, extend the file system.

Troubleshoot "Too many open files" errors

If you exceed the maximum inotify resources, then the SSM Agent can't create the new file descriptors required to establish a session. This issue occurs if you have too many files or file descriptors open simultaneously, or the kernel's inotify subsystem exceeded its maximum instance or watch quota. For more information, see inotify on the man7 website.

To troubleshoot this issue, take the following actions.

Reboot or restart your instance

To restart all processes and release the inotify resources in use, reboot or stop and start the instance.

Note: When you stop and start an instance, the instance's public IP address changes. It's a best practice to use an Elastic IP address to route external traffic to your instance instead of a public IP address. If you use Amazon Route 53, then you might need to update the Route 53 DNS records when the public IP address changes.

Before you stop and start your instance, take the following actions:

• If your instance uses an instance store, then save the volume data on the instance store to persistent storage. For example, you can save the data to Amazon EBS volumes or Amazon Simple Storage Service (Amazon S3) buckets.
  Important: Amazon EC2 deletes instance store data when you stop the instance.
• Create a snapshot of your Amazon EBS volume. If you experience issues, then you can use the snapshot to restore your instance.
• Temporarily remove the instance from its Amazon EC2 Auto Scaling group so that you don't accidentally terminate the instance when you stop it.
  Note: EC2 Auto Scaling might terminate stopped instances in an Auto Scaling group based on your scale-in protection settings. Instances that you launch with Amazon EMR, AWS CloudFormation, or AWS Elastic Beanstalk might be in an Auto Scaling group.
• Set the instance shutdown behavior to Stop to make sure that the instances don't terminate when you stop them.

Increase your inotify quota

If you still encounter issues after a reboot or restart, then complete the following steps to increase the inotify quota on your instance:

1. Run the following commands to check the inotify quota:

   cat /proc/sys/fs/inotify/max_user_watches 
   cat /proc/sys/fs/inotify/max_user_instances

   Note: By default, max_user_watches is 8192 and max_user_instances is 128.
2. To temporarily increase the maximum quota values, run the following command:

   sudo sysctl fs.inotify.max_user_watches=newwatchesquota
   sudo sysctl fs.inotify.max_user_instances=newinstancesquota

   Note: Replace newwatchesquota with the new quota for max_user_watches and newinstancesquota with the new quota for max_user_instances. The preceding command updates the quotas until you next reboot the instance. It's a best practice to test updated values through temporary changes first.
3. To make the quota updates permanent, add the following parameters to the /etc/sysctl.conf file:

   echo "fs.inotify.max_user_watches = newwatchesquota" >> /etc/sysctl.d/20-fs-inotify.conf 
   echo "fs.inotify.max_user_instances = newinstancesquota" >> /etc/sysctl.d/20-fs-inotify.conf

   Note: Replace newwatchesquota with the new quota for max_user_watches and newinstancesquota with the new quota for max_user_instances.
4. Reboot the instance to apply changes.

Note: It's a best practice to monitor your system performance after you modify your instance to validate that the updated quotas align with your system requirements.

Related information

Why can't I use Session Manager to connect to my Amazon EC2 instance?

How do I extend my Linux file system after I increase my Amazon EBS volume on my Amazon EC2 instance?