Unanswered Questions tagged with Amazon EC2

Content language: English

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

What causes cpu overload and then server down after log of "Stopped, Starting Snap Daemon" in EC2?

I was running a spring web server in ubuntu on aws ec2. The server unexpectedly encountered a status check error during off-duty hours and forced a restart. From the monitoring tool, it seems that the server did not work normally after a few minutes after the CPU increased rapidly. After the forced restart, it worked normally, and when I checked the system log, it was as follows. ``` Oct 6 18:32:45 ip-12-0-10-30 snapd[62550]: github.com/snapcore/snapd/overlord/ifacestate/udevmonitor.(*Monitor).Run.func1(0xc00004cfc8, 0x55568c85dad2) Oct 6 18:32:45 ip-12-0-10-30 snapd[62550]: #011/build/snapd/parts/snapd-deb/build/overlord/ifacestate/udevmonitor/udevmon.go:147 +0x329 Oct 6 18:32:45 ip-12-0-10-30 snapd[62550]: gopkg.in/tomb%2ev2.(*Tomb).run(0xc000152c60, 0xc000076b40) Oct 6 18:32:46 ip-12-0-10-30 systemd[1]: snapd.service: Failed with result 'watchdog'. Oct 6 18:32:46 ip-12-0-10-30 snapd[62550]: #011/build/snapd/parts/snapd-deb/build/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2d Oct 6 18:32:47 ip-12-0-10-30 snapd[62550]: created by gopkg.in/tomb%2ev2.(*Tomb).Go Oct 6 18:32:47 ip-12-0-10-30 snapd[62550]: #011/build/snapd/parts/snapd-deb/build/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xc9 Oct 6 18:32:48 ip-12-0-10-30 snapd[62550]: rax 0xca Oct 6 18:32:49 ip-12-0-10-30 snapd[62550]: rbx 0x55568def63a0 Oct 6 18:32:49 ip-12-0-10-30 snapd[62550]: rcx 0x55568c42a793 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: rdx 0x0 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: rdi 0x55568def64e8 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: rsi 0x80 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: rbp 0x7ffc63c1fd38 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: rsp 0x7ffc63c1fcf0 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: r8 0x0 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: r9 0x0 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: r10 0x0 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: r11 0x286 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: r12 0xff Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: r13 0x0 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: r14 0x55568d089af2 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: r15 0x0 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: rip 0x55568c42a791 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: rflags 0x286 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: cs 0x33 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: fs 0x0 Oct 6 18:32:50 ip-12-0-10-30 snapd[62550]: gs 0x0 Oct 6 18:32:50 ip-12-0-10-30 systemd[1]: snapd.service: Scheduled restart job, restart counter is at 3. Oct 6 18:32:51 ip-12-0-10-30 snapd[98728]: AppArmor status: apparmor is enabled and all features are available Oct 6 18:32:51 ip-12-0-10-30 systemd[1]: Stopped Snap Daemon. Oct 6 18:32:52 ip-12-0-10-30 systemd[1]: Starting Snap Daemon... Oct 6 18:32:52 ip-12-0-10-30 systemd[1]: snapd.service: start operation timed out. Terminating. Oct 6 18:32:53 ip-12-0-10-30 systemd[1]: snapd.service: Failed with result 'timeout'. ``` skip ``` Oct 6 19:01:27 ip-12-0-10-30 systemd[1]: Stopped Snap Daemon. Oct 6 19:02:54 ip-12-0-10-30 systemd[1]: Starting Snap Daemon... Oct 6 19:05:14 ip-12-0-10-30 systemd[1]: snapd.service: start operation timed out. Terminating. Oct 6 19:07:10 ip-12-0-10-30 systemd[1]: snapd.service: Failed with result 'timeout'. Oct 6 19:08:57 ip-12-0-10-30 systemd[1]: Failed to start Snap Daemon. Oct 6 19:14:41 ip-12-0-10-30 systemd[1]: snapd.service: Scheduled restart job, restart counter is at 20. Oct 6 19:16:14 ip-12-0-10-30 systemd[1]: Stopped Snap Daemon. Oct 6 19:18:46 ip-12-0-10-30 systemd[1]: Starting Snap Daemon... Oct 6 19:22:44 ip-12-0-10-30 CRON[99169]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 6 19:31:10 ip-12-0-10-30 systemd-networkd[425]: ens5: Could not set DHCPv4 address: Connection timed out Oct 6 19:33:19 ip-12-0-10-30 systemd-networkd[425]: ens5: Failed Oct 6 20:41:15 ip-12-0-10-30 CRON[99214]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Oct 6 20:50:55 ip-12-0-10-30 systemd-timesyncd[352]: Network configuration changed, trying to establish connection. ``` ![ec2 cpu monitoring graph](/media/postImages/original/IMX8ujANjPTkOit8MHrBQeUg) The graph above shows a sharp increase in cpu. At 17:40, the cpu usage increased, and from 17:44, it increased sharply. It peaked at 93% usage at 17:55 and the CPU usage dropped at 18:55. The status check failed from 20:25, and the server went down around 20:30. My guess is snapd.service: Watchdog timeout (limit 5min)! It seems that snapd started and stopped after that. What is the cause of this and what can be done to prevent it?
0
answers
0
votes
13
views
asked a month ago

Unable to login via console or ssh

I see the following in the logs, the sshd service is down and cannot hit enter on console to do anything [ 0.178251] RETBleed: WARNING: Spectre v2 mitigation leaves CPU vulnerable to RETBleed attacks, data leaks possible! [ 0.957547] ena 0000:00:05.0: LLQ is not supported Fallback to host mode policy. /dev/nvme0n1p1: clean, 170121/3276800 files, 1791759/13106939 blocks [ 1.498829] systemd[1]: [/lib/systemd/system/atd.service:7] Executable path is not absolute, ignoring: -find /var/spool/cron/atjobs -type f -name "=*" -not -newercc /run/systemd -delete [ 2.585036] cloud-init[195]: Cloud-init v. 0.7.9 running 'init-local' at Wed, 26 Oct 2022 16:24:28 +0000. Up 2.53 seconds. [FAILED] Failed to start Entropy Daemon based on the HAVEGE algorithm. See 'systemctl status haveged.service' for details. [ OK ] Started Raise network interfaces. Starting Initial cloud-init job (metadata service crawler)... [ OK ] Reached target Network. [ 5.700778] cloud-init[536]: Cloud-init v. 0.7.9 running 'init' at Wed, 26 Oct 2022 16:24:31 +0000. Up 5.20 seconds. [ 5.707092] cloud-init[536]: ci-info: +++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++ [ 5.716709] cloud-init[536]: ci-info: +--------+------+---------------+---------------+-------+-------------------+ [ 5.724038] cloud-init[536]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | [ 5.816066] cloud-init[536]: ci-info: +--------+------+---------------+---------------+-------+-------------------+ [ OK ] Started Initial cloud-init job (metadata service crawler). [ 5.833254] cloud-init[536]: ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . | [ 5.916718] cloud-init[536]: ci-info: | lo: | True | . | . | d | . | [ OK ] Reached target System Initialization. [ 5.923178] cloud-init[536]: ci-info: | eth0: | True | 172.31.88.127 | 255.255.240.0 | . | 12:fa:90:47:c4:ca | [ OK ] Started Clean PHP session files every 30 mins. [ 6.020583] cloud-init[536]: ci-info: | eth0: | True | . | . | d | 12:fa:90:47:c4:ca | [ OK ] Listening on UUID daemon activation socket. [ 6.022681] cloud-init[536]: ci-info: +--------+------+---------------+---------------+-------+-------------------+ [ 6.205833] cloud-init[536]: ci-info: +++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++ [ OK ] Started Daily Cleanup of Temporary Directories. [ 6.220241] cloud-init[536]: ci-info: +-------+-------------+-------------+---------------+-----------+-------+ [ OK ] Listening on D-Bus System Message Bus Socket. [ 6.222215] cloud-init[536]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ 6.325642] cloud-init[536]: ci-info: +-------+-------------+-------------+---------------+-----------+-------+ [ 6.421173] cloud-init[536]: ci-info: | 0 | 0.0.0.0 | 172.31.80.1 | 0.0.0.0 | eth0 | UG | Starting vsftpd FTP server... [ OK ] Started D-Bus System Message Bus. [ 6.423014] cloud-init[536]: ci-info: | 1 | 172.31.80.0 | 0.0.0.0 | 255.255.240.0 | eth0 | U | [ 6.520804] cloud-init[536]: ci-info: +-------+-------------+-------------+---------------+-----------+-------+ Starting The Apache HTTP Server... [ OK ] Started irqbalance daemon. [ OK ] Started Deferred execution scheduler. Starting chrony, an NTP client/server... [ OK ] Started Regular background program processing daemon. Starting Login Service... [ OK ] Started Daily apt download activities. [ OK ] Started Daily apt upgrade and clean activities. [ OK ] Reached target Timers. Starting System Logging Service... [ OK ] Started Unattended Upgrades Shutdown. [ OK ] Reached target Cloud-config availability. [ OK ] Reached target Network is Online. Starting LSB: bitnami init script... Starting Apply the settings specified in cloud-config... Starting Permit User Sessions... [ 6.952653] bitnami[600]: resize2fs 1.43.4 (31-Jan-2017) [ 7.018179] bitnami[600]: open: No such file or directory while opening /dev/xvda Starting OpenBSD Secure Shell server... [ OK ] Started System Logging Service. [ OK ] Started vsftpd FTP server. [ OK ] Started The Apache HTTP Server. [ OK ] Started chrony, an NTP client/server. [ OK ] Started Permit User Sessions. [FAILED] Failed to start OpenBSD Secure Shell server. See 'systemctl status ssh.service' for details. [ OK ] Started Login Service. [ OK ] Stopped OpenBSD Secure Shell server. Starting OpenBSD Secure Shell server... [FAILED] Failed to start OpenBSD Secure Shell server. See 'systemctl status ssh.service' for details. [ 7.739533] cloud-init[601]: Cloud-init v. 0.7.9 running 'modules:config' at Wed, 26 Oct 2022 16:24:33 +0000. Up 7.52 seconds. [ OK ] Started Apply the settings specified in cloud-config. [ OK ] Stopped OpenBSD Secure Shell server. Starting OpenBSD Secure Shell server... [FAILED] Failed to start OpenBSD Secure Shell server. See 'systemctl status ssh.service' for details. [ OK ] Stopped OpenBSD Secure Shell server. Starting OpenBSD Secure Shell server... [FAILED] Failed to start OpenBSD Secure Shell server. See 'systemctl status ssh.service' for details. [ OK ] Stopped OpenBSD Secure Shell server. Starting OpenBSD Secure Shell server... [FAILED] Failed to start OpenBSD Secure Shell server. See 'systemctl status ssh.service' for details. [ OK ] Stopped OpenBSD Secure Shell server. [FAILED] Failed to start OpenBSD Secure Shell server. See 'systemctl status ssh.service' for details. [ 10.186723] bitnami[600]: 650000+0 records in [ 10.189086] bitnami[600]: 650000+0 records out [ 10.191201] bitnami[600]: 665600000 bytes (666 MB, 635 MiB) copied, 3.19479 s, 208 MB/s [ 11.261631] bitnami[600]: Setting up swapspace version 1, size = 634.8 MiB (665595904 bytes) [ 11.265199] bitnami[600]: no label, UUID=04885367-8022-432b-99f3-3c6e5b617dbf [ 13.635358] bitnami[600]: ## 2022-10-26 16:24:39+00:00 ## INFO ## 80 has been blocked [ 15.642776] bitnami[600]: ## 2022-10-26 16:24:41+00:00 ## INFO ## 443 has been blocked [ 18.717920] bitnami[600]: /opt/bitnami/mysql/scripts/ctl.sh : mysql started at port 3306 [ 26.287353] bitnami[600]: /opt/bitnami/php/scripts/ctl.sh : php-fpm started [ 31.622222] bitnami[600]: [Wed Oct 26 16:24:57.506786 2022] [core:trace3] [pid 1145] core.c(3355): Setting LogLevel for all modules to trace8 [ 31.663500] bitnami[600]: Syntax OK [ 31.754163] bitnami[600]: [Wed Oct 26 16:24:57.638831 2022] [core:trace3] [pid 1147] core.c(3355): Setting LogLevel for all modules to trace8 [ 31.768089] bitnami[600]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:80 [ 31.785974] bitnami[600]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80 [ 31.790871] bitnami[600]: no listening sockets available, shutting down [ 31.794003] bitnami[600]: AH00015: Unable to open logs [ 31.885824] bitnami[600]: /opt/bitnami/apache2/scripts/ctl.sh : httpd could not be started [ 40.254998] bitnami[600]: Starting gonit daemon [FAILED] Failed to start LSB: bitnami init script. See 'systemctl status bitnami.service' for details. [ OK ] Reached target Multi-User System. Starting Execute cloud user/final scripts... [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ 40.814772] cloud-init[1263]: Cloud-init v. 0.7.9 running 'modules:final' at Wed, 26 Oct 2022 16:25:06 +0000. Up 40.71 seconds. [ 40.819181] cloud-init[1263]: Cloud-init v. 0.7.9 finished at Wed, 26 Oct 2022 16:25:06 +0000. Datasource DataSourceEc2. Up 40.80 seconds [ OK ] Started Execute cloud user/final scripts. [ OK ] Reached target Cloud-init target.
0
answers
0
votes
24
views
asked a month ago

AWS DNS resolver replies with empty answer for query from docker container

Hello, I am investigating an issue with a docker container, run on a AWS Batch managed - EC2 environment, which is consistently failing to resolve "eu-central-1.wasabisys.com". When I demonstratively run a simple alpine container, I am not able to resolve the DNS record for "`eu-central-1.wasabisys.com`" whereas the record for "`us-central-1.wasabisys.com`" is working as expected. ```# docker run -it alpine / # nslookup eu-central-1.wasabisys.com. Server: 172.31.0.2 Address: 172.31.0.2:53 Non-authoritative answer: Non-authoritative answer: ``` I am successfully able to resolve the record on the host as well as with dig on the container, after installing it's package. This is a tcpdump captured on the host. In the beginning of the following snippet the resolution was done using nslookup (it contained no answers) and in the second try the same resolution was done using dig (it contained answers) 172.31.44.35 = Host IP 172.17.0.2 = Container IP 172.31.0.2 = AWS Resolver (automatically assinged) ```[root@ip-172-31-44-35 ec2-user]# tcpdump -nnevvvi any port 53 tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 15:53:09.235581 P 02:42:ac:11:00:02 ethertype IPv4 (0x0800), length 88: (tos 0x0, ttl 255, id 36324, offset 0, flags [DF], proto UDP (17), length 72) 172.17.0.2.60435 > 172.31.0.2.53: [bad udp cksum 0x587a -> 0x190b!] 14781+ A? eu-central-1.wasabisys.com. (44) 15:53:09.235600 In 02:42:ac:11:00:02 ethertype IPv4 (0x0800), length 88: (tos 0x0, ttl 255, id 36324, offset 0, flags [DF], proto UDP (17), length 72) 172.17.0.2.60435 > 172.31.0.2.53: [bad udp cksum 0x587a -> 0x190b!] 14781+ A? eu-central-1.wasabisys.com. (44) 15:53:09.235612 Out 0a:8c:33:66:0d:1e ethertype IPv4 (0x0800), length 88: (tos 0x0, ttl 254, id 36324, offset 0, flags [DF], proto UDP (17), length 72) 172.31.44.35.60435 > 172.31.0.2.53: [bad udp cksum 0x84a9 -> 0xecdb!] 14781+ A? eu-central-1.wasabisys.com. (44) 15:53:09.236215 In 0a:11:09:8b:8f:49 ethertype IPv4 (0x0800), length 88: (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 72) 172.31.0.2.53 > 172.31.44.35.60435: [udp sum ok] 14781| q: A? eu-central-1.wasabisys.com. 0/0/0 (44) 15:53:09.236220 Out 02:42:de:e4:72:cc ethertype IPv4 (0x0800), length 88: (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 72) 172.31.0.2.53 > 172.17.0.2.60435: [udp sum ok] 14781| q: A? eu-central-1.wasabisys.com. 0/0/0 (44) 15:53:09.236221 Out 02:42:de:e4:72:cc ethertype IPv4 (0x0800), length 88: (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 72) 172.31.0.2.53 > 172.17.0.2.60435: [udp sum ok] 14781| q: A? eu-central-1.wasabisys.com. 0/0/0 (44) [root@ip-172-31-44-35 ec2-user]# tcpdump -nnevvvi any port 53 tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 15:53:29.219638 P 02:42:ac:11:00:02 ethertype IPv4 (0x0800), length 111: (tos 0x0, ttl 255, id 9196, offset 0, flags [none], proto UDP (17), length 95) 172.17.0.2.50529 > 172.31.0.2.53: [bad udp cksum 0x5891 -> 0x120c!] 12676+ [1au] A? eu-central-1.wasabisys.com. ar: . OPT UDPsize=4096 (67) 15:53:29.219638 In 02:42:ac:11:00:02 ethertype IPv4 (0x0800), length 111: (tos 0x0, ttl 255, id 9196, offset 0, flags [none], proto UDP (17), length 95) 172.17.0.2.50529 > 172.31.0.2.53: [bad udp cksum 0x5891 -> 0x120c!] 12676+ [1au] A? eu-central-1.wasabisys.com. ar: . OPT UDPsize=4096 (67) 15:53:29.219665 Out 0a:8c:33:66:0d:1e ethertype IPv4 (0x0800), length 111: (tos 0x0, ttl 254, id 9196, offset 0, flags [none], proto UDP (17), length 95) 172.31.44.35.50529 > 172.31.0.2.53: [bad udp cksum 0x84c0 -> 0xe5dc!] 12676+ [1au] A? eu-central-1.wasabisys.com. ar: . OPT UDPsize=4096 (67) 15:53:29.220724 In 0a:11:09:8b:8f:49 ethertype IPv4 (0x0800), length 467: (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 451) 172.31.0.2.53 > 172.31.44.35.50529: [udp sum ok] 12676 q: A? eu-central-1.wasabisys.com. 23/0/1 eu-central-1.wasabisys.com. [20s] A 130.117.252.29, eu-central-1.wasabisys.com. [20s] A 130.117.252.26, eu-central-1.wasabisys.com. [20s] A 130.117.252.13, eu-central-1.wasabisys.com. [20s] A 130.117.252.18, eu-central-1.wasabisys.com. [20s] A 130.117.252.35, eu-central-1.wasabisys.com. [20s] A 130.117.252.16, eu-central-1.wasabisys.com. [20s] A 130.117.252.27, eu-central-1.wasabisys.com. [20s] A 130.117.252.20, eu-central-1.wasabisys.com. [20s] A 130.117.252.28, eu-central-1.wasabisys.com. [20s] A 130.117.252.23, eu-central-1.wasabisys.com. [20s] A 130.117.252.24, eu-central-1.wasabisys.com. [20s] A 130.117.252.12, eu-central-1.wasabisys.com. [20s] A 130.117.252.17, eu-central-1.wasabisys.com. [20s] A 130.117.252.31, eu-central-1.wasabisys.com. [20s] A 130.117.252.25, eu-central-1.wasabisys.com. [20s] A 130.117.252.21, eu-central-1.wasabisys.com. [20s] A 130.117.252.10, eu-central-1.wasabisys.com. [20s] A 130.117.252.11, eu-central-1.wasabisys.com. [20s] A 130.117.252.22, eu-central-1.wasabisys.com. [20s] A 130.117.252.34, eu-central-1.wasabisys.com. [20s] A 130.117.252.33, eu-central-1.wasabisys.com. [20s] A 130.117.252.19, eu-central-1.wasabisys.com. [20s] A 130.117.252.32 ar: . OPT UDPsize=4096 (423) 15:53:29.220734 Out 02:42:de:e4:72:cc ethertype IPv4 (0x0800), length 467: (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 451) 172.31.0.2.53 > 172.17.0.2.50529: [udp sum ok] 12676 q: A? eu-central-1.wasabisys.com. 23/0/1 eu-central-1.wasabisys.com. [20s] A 130.117.252.29, eu-central-1.wasabisys.com. [20s] A 130.117.252.26, eu-central-1.wasabisys.com. [20s] A 130.117.252.13, eu-central-1.wasabisys.com. [20s] A 130.117.252.18, eu-central-1.wasabisys.com. [20s] A 130.117.252.35, eu-central-1.wasabisys.com. [20s] A 130.117.252.16, eu-central-1.wasabisys.com. [20s] A 130.117.252.27, eu-central-1.wasabisys.com. [20s] A 130.117.252.20, eu-central-1.wasabisys.com. [20s] A 130.117.252.28, eu-central-1.wasabisys.com. [20s] A 130.117.252.23, eu-central-1.wasabisys.com. [20s] A 130.117.252.24, eu-central-1.wasabisys.com. [20s] A 130.117.252.12, eu-central-1.wasabisys.com. [20s] A 130.117.252.17, eu-central-1.wasabisys.com. [20s] A 130.117.252.31, eu-central-1.wasabisys.com. [20s] A 130.117.252.25, eu-central-1.wasabisys.com. [20s] A 130.117.252.21, eu-central-1.wasabisys.com. [20s] A 130.117.252.10, eu-central-1.wasabisys.com. [20s] A 130.117.252.11, eu-central-1.wasabisys.com. [20s] A 130.117.252.22, eu-central-1.wasabisys.com. [20s] A 130.117.252.34, eu-central-1.wasabisys.com. [20s] A 130.117.252.33, eu-central-1.wasabisys.com. [20s] A 130.117.252.19, eu-central-1.wasabisys.com. [20s] A 130.117.252.32 ar: . OPT UDPsize=4096 (423) 15:53:29.220737 Out 02:42:de:e4:72:cc ethertype IPv4 (0x0800), length 467: (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 451) 172.31.0.2.53 > 172.17.0.2.50529: [udp sum ok] 12676 q: A? eu-central-1.wasabisys.com. 23/0/1 eu-central-1.wasabisys.com. [20s] A 130.117.252.29, eu-central-1.wasabisys.com. [20s] A 130.117.252.26, eu-central-1.wasabisys.com. [20s] A 130.117.252.13, eu-central-1.wasabisys.com. [20s] A 130.117.252.18, eu-central-1.wasabisys.com. [20s] A 130.117.252.35, eu-central-1.wasabisys.com. [20s] A 130.117.252.16, eu-central-1.wasabisys.com. [20s] A 130.117.252.27, eu-central-1.wasabisys.com. [20s] A 130.117.252.20, eu-central-1.wasabisys.com. [20s] A 130.117.252.28, eu-central-1.wasabisys.com. [20s] A 130.117.252.23, eu-central-1.wasabisys.com. [20s] A 130.117.252.24, eu-central-1.wasabisys.com. [20s] A 130.117.252.12, eu-central-1.wasabisys.com. [20s] A 130.117.252.17, eu-central-1.wasabisys.com. [20s] A 130.117.252.31, eu-central-1.wasabisys.com. [20s] A 130.117.252.25, eu-central-1.wasabisys.com. [20s] A 130.117.252.21, eu-central-1.wasabisys.com. [20s] A 130.117.252.10, eu-central-1.wasabisys.com. [20s] A 130.117.252.11, eu-central-1.wasabisys.com. [20s] A 130.117.252.22, eu-central-1.wasabisys.com. [20s] A 130.117.252.34, eu-central-1.wasabisys.com. [20s] A 130.117.252.33, eu-central-1.wasabisys.com. [20s] A 130.117.252.19, eu-central-1.wasabisys.com. [20s] A 130.117.252.32 ar: . OPT UDPsize=4096 (423) ``` Has anybody any clue what is going on? Why is the AWS resolver not replying correctly? Thanks a lot for you help!
0
answers
0
votes
27
views
KniFFeL
asked a month ago

Why are my EC2 instances not reporting their compliance status to SSM Patch Manager?

In SSM Patch Manager, under Compliance Reporting, our Amazon Linux 2 EC2 instances appear but in the 'Compliance status' column say 'Never reported'. The instances appear in Fleet Manager with 'SSM Agent ping status' of 'Online', and I can connect to the instances remotely using SSM `start-session`. I've checked all the troubleshooting steps in the docs at [Troubleshooting SSM Agent](https://docs.aws.amazon.com/systems-manager/latest/userguide/troubleshooting-ssm-agent.html), [this article about SSM logs](https://aws.amazon.com/premiumsupport/knowledge-center/ssm-agent-logs/) and [Troubleshooting Patch Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/patch-manager-troubleshooting.html#patch-manager-troubleshooting-contact-support), and everything appears to be set up properly (the instance role has the right permissions, the named servers are reachable, and the instances can reach public S3 buckets via the internet, we're not using a VPC endpoint). I've also tried restarting the SSM Agent. In the SSM Agent logs on the instance, I'm seeing: ``` 2022-10-25 00:36:48 INFO [ssm-agent-worker] [StartupProcessor] Write to serial port: Amazon SSM Agent v3.1.1732.0 is running ... 2022-10-25 01:15:00 INFO [ssm-agent-worker] [HealthCheck] HealthCheck reporting agent health. 2022-10-25 01:16:48 INFO [ssm-agent-worker] [MessageService] [MessageHandler] started idempotency deletion thread 2022-10-25 01:16:48 WARN [ssm-agent-worker] [MessageService] [MessageHandler] [Idempotency] encountered error open /var/lib/amazon/ssm/i-XXXXXXXXXXXXXXXXX/idempotency: no such file or directory while listing directories in /var/lib/amazon/ssm/i-XXXXXXXXXXXXXXXXX/idempotency 2022-10-25 01:16:48 INFO [ssm-agent-worker] [MessageService] [MessageHandler] ended idempotency deletion thread 2022-10-25 01:16:50 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread started 2022-10-25 01:16:50 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread done 2022-10-25 01:17:05 INFO [ssm-agent-worker] [MessageService] [Association] Schedule manager refreshed with 0 associations, 0 new associations associated 2022-10-25 01:20:00 INFO [ssm-agent-worker] [HealthCheck] HealthCheck reporting agent health. ``` Any clues why the instances aren't reporting their compliance status to Patch Manager? What additional steps can I use to troubleshoot this?
0
answers
0
votes
30
views
asked a month ago