Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

SSM agent gets removed at runtime

0

Hi everyone, we are having an issue where the ssm-agent gets removed by snap at run time, so we lose connection to that system

The system: Ubuntu 22.04, snap details(from "snap list" command): amazon-ssm-agent 3.1.634.0 5102 - aws** classic

The logs where we see it doing that Nov 17 08:53:54 enbyim-2 snapd[796]: services.go:1152: RemoveSnapServices - disabling snap.amazon-ssm-agent.amazon-ssm-agent.service, with the rest of the log pasted below:

Nov 17 08:53:54 enbyim-2 systemd[1]: snap-amazon\x2dssm\x2dagent-5102.mount: Deactivated successfully.
Nov 17 08:53:54 enbyim-2 systemd[1]: Configuration file /etc/systemd/system/greengrass.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Nov 17 08:53:54 enbyim-2 systemd[1]: /lib/systemd/system/snapd.service:23: Unknown key name 'RestartMode' in section 'Service', ignoring.
Nov 17 08:53:54 enbyim-2 systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Nov 17 08:53:54 enbyim-2 systemd[1]: Reloading.
Nov 17 08:53:54 enbyim-2 snapd[796]: services.go:1152: RemoveSnapServices - disabling snap.amazon-ssm-agent.amazon-ssm-agent.service
Nov 17 08:54:05 enbyim-2 geoclue[1808]: Service not used for 60 seconds. Shutting down..
Nov 17 08:54:02 enbyim-2 systemd-timesyncd[651]: Timed out waiting for reply from 185.125.190.56:123 (ntp.ubuntu.com).
Nov 17 08:53:55 enbyim-2 systemd[1]: Configuration file /etc/systemd/system/greengrass.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Nov 17 08:53:55 enbyim-2 systemd[1]: /lib/systemd/system/snapd.service:23: Unknown key name 'RestartMode' in section 'Service', ignoring.
Nov 17 08:53:54 enbyim-2 systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Nov 17 08:53:54 enbyim-2 systemd[1]: Reloading.
Nov 17 08:53:54 enbyim-2 systemd[1]: snap-amazon\x2dssm\x2dagent-5102.mount: Deactivated successfully.
Nov 17 08:53:54 enbyim-2 systemd[1]: Configuration file /etc/systemd/system/greengrass.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Nov 17 08:53:54 enbyim-2 systemd[1]: /lib/systemd/system/snapd.service:23: Unknown key name 'RestartMode' in section 'Service', ignoring.
Nov 17 08:53:54 enbyim-2 systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Nov 17 08:53:54 enbyim-2 systemd[1]: Reloading.
Nov 17 08:53:54 enbyim-2 snapd[796]: services.go:1152: RemoveSnapServices - disabling snap.amazon-ssm-agent.amazon-ssm-agent.service
Nov 17 08:53:54 enbyim-2 systemd[1]: snap.amazon-ssm-agent.amazon-ssm-agent.service: Consumed 1.799s CPU time.
Nov 17 08:53:54 enbyim-2 systemd[1]: Stopped Service for snap application amazon-ssm-agent.amazon-ssm-agent.
Nov 17 08:53:54 enbyim-2 systemd[1]: snap.amazon-ssm-agent.amazon-ssm-agent.service: Deactivated successfully.
Nov 17 08:53:54 enbyim-2 amazon-ssm-agent.amazon-ssm-agent[959]: 2025-11-17 08:53:54 INFO [amazon-ssm-agent] Bye.
Nov 17 08:53:52 enbyim-2 systemd-timesyncd[651]: Timed out waiting for reply from 185.125.190.57:123 (ntp.ubuntu.com).
Nov 17 08:53:47 enbyim-2 systemd[1]: Started Time & Date Service.
Nov 17 08:53:47 enbyim-2 dbus-daemon[740]: [system] Successfully activated service 'org.freedesktop.timedate1'
Nov 17 08:53:47 enbyim-2 amazon-ssm-agent.amazon-ssm-agent[959]: 2025-11-17 08:53:47 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Receiving stop signal, stop worker monitor
Nov 17 08:53:47 enbyim-2 amazon-ssm-agent.amazon-ssm-agent[959]: 2025-11-17 08:53:47 INFO [amazon-ssm-agent] Stopping Core Agent
Nov 17 08:53:47 enbyim-2 amazon-ssm-agent.amazon-ssm-agent[959]: 2025-11-17 08:53:47 INFO [amazon-ssm-agent] amazon-ssm-agent got signal:terminated value:0x7939f5d53340
Nov 17 08:53:47 enbyim-2 systemd[1]: Stopping Service for snap application amazon-ssm-agent.amazon-ssm-agent...
Nov 17 08:53:47 enbyim-2 systemd[1]: Starting Time & Date Service...
Nov 17 08:53:47 enbyim-2 dbus-daemon[740]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.26' (uid=0 pid=796 comm="/usr/lib/snapd/snapd " label="unconfined")
Nov 17 08:53:47 enbyim-2 snapd[796]: stateengine.go:161: state ensure error: Get "https://api.snapcraft.io/api/v1/snaps/sections": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Nov 17 08:53:41 enbyim-2 systemd-timesyncd[651]: Timed out waiting for reply from 91.189.91.157:123 (ntp.ubuntu.com).
Nov 17 08:53:36 enbyim-2 systemd[1]: systemd-localed.service: Deactivated successfully.
Nov 17 08:53:36 enbyim-2 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Nov 17 08:53:32 enbyim-2 systemd[1]: systemd-timedated.service: Deactivated successfully.
Nov 17 08:53:30 enbyim-2 systemd[1]: systemd-fsckd.service: Deactivated successfully.
Nov 17 08:53:28 enbyim-2 pulseaudio[1363]: GetManagedObjects() failed: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Nov 17 08:53:28 enbyim-2 dbus-daemon[740]: [system] Failed to activate service 'org.bluez': timed out (service_start_timeout=25000ms)
Nov 17 08:53:23 enbyim-2 systemd[1]: Startup finished in 9.162s (firmware) + 3.356s (loader) + 3.166s (kernel) + 23.924s (userspace) = 39.610s.
Nov 17 08:53:23 enbyim-2 systemd[1]: Finished Record Runlevel Change in UTMP.
Nov 17 08:53:23 enbyim-2 systemd[1]: systemd-update-utmp-runlevel.service: Deactivated successfully.
Nov 17 08:53:23 enbyim-2 systemd[1]: Created slice Slice /system/getty.
Nov 17 08:53:23 enbyim-2 systemd[1]: Finished Set console scheme.
Nov 17 08:53:23 enbyim-2 systemd[1]: Starting Record Runlevel Change in UTMP...
Nov 17 08:53:23 enbyim-2 systemd[1]: Starting Set console scheme...
Nov 17 08:53:23 enbyim-2 systemd[1]: Reached target Graphical Interface.
Nov 17 08:53:23 enbyim-2 systemd[1]: Reached target Multi-User System.
Nov 17 08:53:23 enbyim-2 systemd[1]: Finished Hold until boot process finishes up.
Nov 17 08:53:23 enbyim-2 systemd[1]: Received SIGRTMIN+21 from PID 520 (plymouthd).

More log details from snap:

snap changes 
ID  Status  Spawn    Ready    Summary 
48  Done   today at 08:53 CST    today at 08:53 CST   Remove "amazon-ssm-agent" snap

It doesn't recognize the ssm agent either:

snap list amazon-ssm-agent
error: no matching snaps installed

Strangely a reboot seems to bring it back online. Sadly that's not a convenient option for our system since we don't have access to it when the ssm agent is down. Any ideas on why it gets removed like this and solutions to them perhaps? Note that we only allow the ssm agent through our firewall. So you will see additional noise in the logs

Topics
Management & GovernanceCompute
Tags
AWS Systems ManagerLinux ProvisioningIAM Policies
Language
English

asked 10 months ago253 views

1 Answer
0

The issue you're experiencing with the SSM Agent being removed by snap at runtime is causing your EC2 instance to disconnect from Session Manager. Based on your logs, it appears that snapd is actively disabling and removing the amazon-ssm-agent service, which explains why you lose connection to the system.

This could be happening for several reasons:

  1. Network connectivity issues: The logs show timeout errors when trying to connect to ntp.ubuntu.com and api.snapcraft.io. This suggests your instance might be experiencing network connectivity problems, which could be causing snap to remove the SSM Agent package.

  2. Snap package management issues: The log entry "Remove amazon-ssm-agent snap" indicates that snap is intentionally removing the package, possibly due to dependency issues or conflicts.

  3. Firewall restrictions: Since you mentioned only allowing SSM Agent through your firewall, this could be preventing the snap service from properly maintaining the SSM Agent package. Snap may need to communicate with its repositories to verify package integrity.

Here are some potential solutions:

  1. Install the SSM Agent using the deb package instead of snap: This would avoid the snap management system altogether. You can download and install the deb package directly.

  2. Configure snap to retain the SSM Agent: You might be able to prevent snap from removing the package by marking it as essential or using snap's hold functionality.

  3. Adjust firewall rules: Ensure that snap can communicate with its required endpoints (api.snapcraft.io) to prevent it from entering a failure state that leads to package removal.

  4. Set up automatic SSM Agent restart: Create a systemd service that monitors the SSM Agent and restarts it if it stops running.

  5. Implement a scheduled task: Set up a cron job to periodically check if the SSM Agent is running and reinstall it if necessary.

Since a reboot brings the agent back online, this suggests that the issue is with the runtime environment rather than the installation itself. The SSM Agent is likely being reinstalled during the boot process.
Sources
Instance is not connected to session manager | AWS re:Post
Cant connect to private ec2 instance using ssm and bastion? | AWS re:Post
Ping status: Connection lost | AWS re:Post
Troubleshooting SSM Agent - AWS Systems Manager

answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content