About 10 ubuntu ec2 instances created at the same time started to keep outputting ssm agent errors from 2023-08-31.
Looking at amazon-ssm-agent.log, it appears that after the ssm-agent was auto-updated, it started to keep outputting ERROR about ssm:UpdateInstanceInformation having no authority.
Before the auto-update, I was getting a similar message of no authority as well, but it was INFO.
I have not enabled ssm from the management console, do I need to enable ssm or assign ssm:UpdateInstanceInformation to the EC2 IAM role to suppress this error?
These instances were created separately for two different AWS accounts, so I don't think this is an account-specific issue.
amazon-ssm-agent.log
2023-05-31 07:40:06 INFO [ssm-agent-worker] [StartupProcessor] Executing startup processor tasks
2023-05-31 07:40:06 INFO [ssm-agent-worker] [StartupProcessor] Write to serial port: Amazon SSM Agent v3.1.1927.0 is running
2023-05-31 07:40:06 INFO [ssm-agent-worker] [StartupProcessor] Write to serial port: OsProductName: Ubuntu
2023-05-31 07:40:06 INFO [ssm-agent-worker] [StartupProcessor] Write to serial port: OsVersion: 20.04
2023-05-31 07:40:06 INFO [ssm-agent-worker] Entering SSM Agent hibernate - AccessDeniedException: User: arn:aws:sts::___:assumed-role/___/i-___ is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:ap-northeast-1:___:instance/i-___ because no identity-based policy allows the ssm:UpdateInstanceInformation action
status code: 400, request id: 5e4f8873-1acb-46b5-91cb-ff4edfee5eff
2023-08-31 00:26:07 INFO [amazon-ssm-agent] amazon-ssm-agent got signal:terminated value:0x7f9c1fb2da60
2023-08-31 00:26:07 INFO [amazon-ssm-agent] Stopping Core Agent
2023-08-31 00:26:07 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Receiving stop signal, stop worker monitor
2023-08-31 00:26:08 INFO [ssm-agent-worker] Received termination message from core agent {"SchemaVersion":1,"Topic":"TerminateWorkerRequest","Payload":null}
2023-08-31 00:26:08 INFO [ssm-agent-worker] Stopping ssm agent worker
2023-08-31 00:26:08 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Received worker termination result, &{SchemaVersion:1 Topic:GetWorkerHealthResult Payload:[123 34 83 99 104 101 109 97 86 101 114 115 105 111 110 34 58 49 44 34 78 97 109 101 34 58 34 115 115 109 45 97 103 101 110 116 45 119 111 114 107 101 114 34 44 34 87 111 114 107 101 114 84 121 112 101 34 58 34 76 111 110 103 82 117 110 110 105 110 103 34 44 34 80 105 100 34 58 51 49 48 51 54 55 51 44 34 73 115 84 101 114 109 105 110 97 116 105 110 103 34 58 116 114 117 101 125]}
2023-08-31 00:26:14 INFO [amazon-ssm-agent] Bye.
2023-08-31 00:26:18 WARN Error adding the directory '/etc/amazon/ssm' to watcher: no such file or directory
2023-08-31 00:26:18 INFO Proxy environment variables:
2023-08-31 00:26:18 INFO http_proxy:
2023-08-31 00:26:18 INFO no_proxy:
2023-08-31 00:26:18 INFO https_proxy:
2023-08-31 00:26:18 INFO Checking if agent identity type OnPrem can be assumed
2023-08-31 00:26:18 INFO Checking if agent identity type EC2 can be assumed
2023-08-31 00:26:18 INFO Agent will take identity from EC2
2023-08-31 00:26:18 INFO [amazon-ssm-agent] using named pipe channel for IPC
2023-08-31 00:26:18 INFO [amazon-ssm-agent] using named pipe channel for IPC
2023-08-31 00:26:18 INFO [amazon-ssm-agent] using named pipe channel for IPC
2023-08-31 00:26:18 INFO [amazon-ssm-agent] amazon-ssm-agent - v3.2.1377.0
2023-08-31 00:26:18 INFO [amazon-ssm-agent] OS: linux, Arch: amd64
2023-08-31 00:26:18 INFO [amazon-ssm-agent] Starting Core Agent
2023-08-31 00:26:18 INFO [amazon-ssm-agent] Registrar detected. Attempting registration
2023-08-31 00:26:18 INFO [Registrar] Starting registrar module
2023-08-31 00:26:18 INFO [EC2Identity] Checking disk for registration info
2023-08-31 00:26:18 INFO [EC2Identity] No registration info found for ec2 instance, attempting registration
2023-08-31 00:26:18 INFO [EC2Identity] Generating registration keypair
2023-08-31 00:26:18 INFO [EC2Identity] Checking write access before registering
2023-08-31 00:26:18 INFO [EC2Identity] Registering EC2 instance with Systems Manager
2023-08-31 00:26:18 INFO [EC2Identity] EC2 registration was successful.
2023-08-31 00:26:18 INFO [amazon-ssm-agent] Registration attempted. Resuming core agent startup.
2023-08-31 00:26:18 INFO [CredentialRefresher] credentialRefresher has started
2023-08-31 00:26:18 INFO [CredentialRefresher] Starting credentials refresher loop
2023-08-31 00:26:18 WARN EC2RoleProvider Failed to connect to Systems Manager with instance profile role credentials. Err: retrieved credentials failed to report to ssm. RequestId: 4d4fefc0-2142-47e8-a847-8efaf34ec3aa Error: AccessDeniedException: User: arn:aws:sts::___:assumed-role/___/i-___ is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:ap-northeast-1:___:instance/i-___ because no identity-based policy allows the ssm:UpdateInstanceInformation action
status code: 400, request id: 4d4fefc0-2142-47e8-a847-8efaf34ec3aa
2023-08-31 00:26:19 ERROR EC2RoleProvider Failed to connect to Systems Manager with SSM role credentials. error calling RequestManagedInstanceRoleToken: AccessDeniedException: Systems Manager's instance management role is not configured for account: ___
status code: 400, request id: 6ffd3041-b2c1-40e0-a71b-0ffc031fa454
2023-08-31 00:26:19 ERROR [CredentialRefresher] Retrieve credentials produced error: no valid credentials could be retrieved for ec2 identity
2023-08-31 00:26:19 INFO [CredentialRefresher] Sleeping for 1s before retrying retrieve credentials