EC2 Instance Failed Status Checks

Question

One of my instances reported a failed status check and became unreachable. After rebooting it work again but after sometimes it will become unreachable again with the status check shows failed status check.
This is what the system log show
```
[    3.831448] platform eisa.0: Probing EISA bus 0
[    3.838396] platform eisa.0: EISA: Cannot allocate resource for mainboard
[    3.850447] platform eisa.0: Cannot allocate resource for EISA slot 1
[    3.863830] platform eisa.0: Cannot allocate resource for EISA slot 2
[    3.875461] platform eisa.0: Cannot allocate resource for EISA slot 3
[    3.892850] platform eisa.0: Cannot allocate resource for EISA slot 4
[    3.904830] platform eisa.0: Cannot allocate resource for EISA slot 5
[    3.912690] platform eisa.0: Cannot allocate resource for EISA slot 6
[    3.922446] platform eisa.0: Cannot allocate resource for EISA slot 7
[    3.929242] platform eisa.0: Cannot allocate resource for EISA slot 8
[    3.939895] platform eisa.0: EISA: Detected 0 cards
[    3.949482] intel_pstate: CPU model not supported
[    3.956557] ledtrig-cpu: registered to indicate activity on CPUs
[    3.974916] drop_monitor: Initializing network drop monitor service
[    3.982754] NET: Registered PF_INET6 protocol family
[    3.992002] Segment Routing with IPv6
[    3.997043] In-situ OAM (IOAM) with IPv6
[    4.005629] NET: Registered PF_PACKET protocol family
[    4.015445] Key type dns_resolver registered
[    4.024832] No MBM correction factor available
[    4.033589] IPI shorthand broadcast: enabled
[    4.039481] sched_clock: Marking stable (2720996247, 1318442093)->(4804311581, -764873241)
[    4.052297] registered taskstats version 1
[    4.059374] Loading compiled-in X.509 certificates
[    4.067793] Loaded X.509 cert 'Build time autogenerated kernel key: d06b9f2c3f179b9d49bb62177eec69dc191199fc'
[    4.082365] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969'
[    4.098475] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19'
[    4.122852] blacklist: Loading compiled-in revocation X.509 certificates
[    4.133185] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing: 61482aa2830d0ab2ad5af10b7250da9033ddcef0'
[    4.152445] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2017): 242ade75ac4a15e50d50c84b0d45ff3eae707a03'
[    4.166714] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (ESM 2018): 365188c1d374d6b07c3c8f240f8ef722433d6a8b'
[    4.188911] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2019): c0746fd6c5da3ae827864651ad66ae47fe24b3e8'
[    4.208542] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2021 v1): a8d54bbb3825cfb94fa13c9f8a594a195c107b8d'
[    4.228155] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2021 v2): 4cf046892d6fd3c9a5b03f98d845f90851dc6a8c'
[    4.244462] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2021 v3): 100437bb6de6e469b581e61cd66bce3ef4ed53af'
[    4.259197] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (Ubuntu Core 2019): c1d57b8f6b743f23ee41f4f7ee292f06eecadfb9'
[    4.283182] zswap: loaded using pool lzo/zbud
[    4.297209] Key type .fscrypt registered
[    4.308845] Key type fscrypt-provisioning registered
[    4.317945] Key type encrypted registered
[    4.323208] AppArmor: AppArmor sha1 policy hashing enabled
[    4.329744] ima: No TPM chip found, activating TPM-bypass!
[    4.336721] Loading compiled-in module X.509 certificates
[    4.354879] Loaded X.509 cert 'Build time autogenerated kernel key: d06b9f2c3f179b9d49bb62177eec69dc191199fc'
[    4.371678] ima: Allocated hash algorithm: sha1
[    4.377887] ima: No architecture policies found
[    4.387791] evm: Initialising EVM extended attributes:
[    4.394065] evm: security.selinux
[    4.399238] evm: security.SMACK64
[    4.403489] evm: security.SMACK64EXEC
[    4.407694] evm: security.SMACK64TRANSMUTE
[    4.413556] evm: security.SMACK64MMAP
[    4.418542] evm: security.apparmor
[    4.423329] evm: security.ima
[    4.427476] evm: security.capability
[    4.432837] evm: HMAC attrs: 0x1
[    4.437703] PM:   Magic number: 11:612:635
[    4.443643] PM:   hash matches drivers/base/power/main.c:599
[    4.452537] RAS: Correctable Errors collector initialized.
[    4.477100] md: Waiting for all devices to be available before autodetect
[    4.494717] md: If you don't use raid, use raid=noautodetect
[    4.503357] md: Autodetecting RAID arrays.
[    4.509632] md: autorun ...
[    4.514637] md: ... autorun DONE.
[    4.531110] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[    4.548425] VFS: Mounted root (ext4 filesystem) readonly on device 202:1.
[    4.558791] devtmpfs: mounted
[    4.569667] Freeing unused decrypted memory: 2036K
[    4.577314] Freeing unused kernel image (initmem) memory: 3248K
[    4.590315] Write protecting the kernel read-only data: 30720k
[    4.598809] Freeing unused kernel image (text/rodata gap) memory: 2036K
[    4.613076] Freeing unused kernel image (rodata/data gap) memory: 1412K
[    4.676837] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    4.694866] x86/mm: Checking user space page tables
[    4.756038] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    4.763963] Run /sbin/init as init process
[    4.990389] systemd[1]: Inserted module 'autofs4'
[    5.019889] systemd[1]: systemd 245.4-4ubuntu3.20 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
[    5.062760] systemd[1]: Detected virtualization xen.
[    5.068446] systemd[1]: Detected architecture x86-64.

Welcome to [1mUbuntu 20.04.4 LTS[0m!

[    5.126454] systemd[1]: Set hostname to .
[    5.459401] systemd[1]: Created slice system-modprobe.slice.
[[0;32m  OK  [0m] Created slice [0;1;39msystem-modprobe.slice[0m.
[    5.477537] systemd[1]: Created slice system-postgresql.slice.
[[0;32m  OK  [0m] Created slice [0;1;39msystem-postgresql.slice[0m.
[    5.490672] systemd[1]: Created slice system-serial\x2dgetty.slice.
[[0;32m  OK  [0m] Created slice [0;1;39msystem-serial\x2dgetty.slice[0m.
[    5.504128] systemd[1]: Created slice User and Session Slice.
[[0;32m  OK  [0m] Created slice [0;1;39mUser and Session Slice[0m.
[    5.516713] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[[0;32m  OK  [0m] Started [0;1;39mForward Password R…uests to Wall Directory Watch[0m.
[    5.536948] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[[0;32m  OK  [0m] Set up automount [0;1;39mArbitrary…s File System Automount Point[0m.
[    5.790201] systemd[1]: Reached target User and Group Name Lookups.
[[0;32m  OK  [0m] Reached target [0;1;39mUser and Group Name Lookups[0m.
[    5.805886] systemd[1]: Reached target Slices.
[[0;32m  OK  [0m] Reached target [0;1;39mSlices[0m.
[    5.816370] systemd[1]: Reached target Mounting snaps.
[[0;32m  OK  [0m] Reached target [0;1;39mMounting snaps[0m.
[    5.826899] systemd[1]: Reached target Swap.
[[0;32m  OK  [0m] Reached target [0;1;39mSwap[0m.
[    5.839474] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[[0;32m  OK  [0m] Listening on [0;1;39mDevice-mapper event daemon FIFOs[0m.
[    5.854561] systemd[1]: Listening on LVM2 poll daemon socket.
[[0;32m  OK  [0m] Listening on [0;1;39mLVM2 poll daemon socket[0m.
[    5.865912] systemd[1]: Listening on multipathd control socket.
[[0;32m  OK  [0m] Listening on [0;1;39mmultipathd control socket[0m.
[    5.878651] systemd[1]: Listening on Syslog Socket.
[[0;32m  OK  [0m] Listening on [0;1;39mSyslog Socket[0m.
[    5.893720] systemd[1]: Listening on fsck to fsckd communication Socket.
[[0;32m  OK  [0m] Listening on [0;1;39mfsck to fsckd communication Socket[0m.
[    5.911351] systemd[1]: Listening on initctl Compatibility Named Pipe.
[[0;32m  OK  [0m] Listening on [0;1;39minitctl Compatibility Named Pipe[0m.
[    5.928617] systemd[1]: Listening on Journal Audit Socket.
[[0;32m  OK  [0m] Listening on [0;1;39mJournal Audit Socket[0m.
[    5.942465] systemd[1]: Listening on Journal Socket (/dev/log).
[[0;32m  OK  [0m] Listening on [0;1;39mJournal Socket (/dev/log)[0m.
[    5.957885] systemd[1]: Listening on Journal Socket.
[[0;32m  OK  [0m] Listening on [0;1;39mJournal Socket[0m.
[    5.969451] systemd[1]: Listening on Network Service Netlink Socket.
[[0;32m  OK  [0m] Listening on [0;1;39mNetwork Service Netlink Socket[0m.
[    5.983947] systemd[1]: Listening on udev Control Socket.
[[0;32m  OK  [0m] Listening on [0;1;39mudev Control Socket[0m.
[    5.997231] systemd[1]: Listening on udev Kernel Socket.
[[0;32m  OK  [0m] Listening on [0;1;39mudev Kernel Socket[0m.
[    6.014107] systemd[1]: Mounting Huge Pages File System...
         Mounting [0;1;39mHuge Pages File System[0m...
[    6.027263] systemd[1]: Mounting POSIX Message Queue File System...
         Mounting [0;1;39mPOSIX Message Queue File System[0m...
[    6.044462] systemd[1]: Mounting Kernel Debug File System...
         Mounting [0;1;39mKernel Debug File System[0m...
[    6.058729] systemd[1]: Mounting Kernel Trace File System...
         Mounting [0;1;39mKernel Trace File System[0m...
[    6.075755] systemd[1]: Starting Journal Service...
         Starting [0;1;39mJournal Service[0m...
[    6.088407] systemd[1]: Starting Set the console keyboard layout...
         Starting [0;1;39mSet the console keyboard layout[0m...
[    6.106878] systemd[1]: Starting Create list of static device nodes for the current kernel...
         Starting [0;1;39mCreate list of st…odes for the current kernel[0m...
[    6.127974] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
         Starting [0;1;39mMonitoring of LVM…meventd or progress polling[0m...
[    6.156994] systemd[1]: Starting Load Kernel Module chromeos_pstore...
         Starting [0;1;39mLoad Kernel Module chromeos_pstore[0m...
[    6.180201] systemd[1]: Starting Load Kernel Module drm...
         Starting [0;1;39mLoad Kernel Module drm[0m...
[    6.200164] systemd[1]: Starting Load Kernel Module efi_pstore...
         Starting [0;1;39mLoad Kernel Module efi_pstore[0m...
[    6.219232] systemd[1]: Starting Load Kernel Module pstore_blk...
         Starting [0;1;39mLoad Kernel Module pstore_blk[0m...
[    6.246793] systemd[1]: Starting Load Kernel Module pstore_zone...
         Starting [0;1;39mLoad Kernel Module pstore_zone[0m...
[    6.269986] systemd[1]: Starting Load Kernel Module ramoops...
         Starting [0;1;39mLoad Kernel Module ramoops[0m...
[    6.285813] systemd[1]: Condition check resulted in OpenVSwitch configuration for cleanup being skipped.
[    6.302561] systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped.
[    6.315961] systemd[1]: Starting File System Check on Root Device...
         Starting [0;1;39mFile System Check on Root Device[0m...
[    6.332078] systemd[1]: Starting Load Kernel Modules...
         Starting [0;1;39mLoad Kernel Modules[0m...
[    6.346384] systemd[1]: Starting udev Coldplug all Devices...
         Starting [0;1;39mudev Coldplug all Devices[0m...
[    6.362134] IPMI message handler: version 39.2
[    6.363020] systemd[1]: Starting Uncomplicated firewall...
         Starting [0;1;39mUncomplicated firewall[0m...[    6.384173] ipmi device interface

[    6.392481] systemd[1]: Started Journal Service.
[[0;32m  OK  [0m] Started [0;1;39mJournal Service[0m.
[    6.408095] bpfilter: Loaded bpfilter_umh pid 203
[    6.409836] Started bpfilter
[[0;32m  OK  [0m] Mounted [0;1;39mHuge Pages File System[0m.
[[0;32m  OK  [0m] Mounted [0;1;39mPOSIX Message Queue File System[0m.
[[0;32m  OK  [0m] Mounted [0;1;39mKernel Debug File System[0m.
[[0;32m  OK  [0m] Mounted [0;1;39mKernel Trace File System[0m.
[[0;32m  OK  [0m] Finished [0;1;39mSet the console keyboard layout[0m.
[[0;32m  OK  [0m] Finished [0;1;39mCreate list of st… nodes for the current kernel[0m.
[[0;32m  OK  [0m] Finished [0;1;39mLoad Kernel Module chromeos_pstore[0m.
[[0;32m  OK  [0m] Finished [0;1;39mLoad Kernel Module drm[0m.
[[0;32m  OK  [0m] Finished [0;1;39mLoad Kernel Module efi_pstore[0m.
[[0;32m  OK  [0m] Finished [0;1;39mLoad Kernel Module pstore_blk[0m.
[[0;32m  OK  [0m] Finished [0;1;39mLoad Kernel Module pstore_zone[0m.
[[0;32m  OK  [0m] Finished [0;1;39mLoad Kernel Module ramoops[0m.
[[0;32m  OK  [0m] Finished [0;1;39mFile System Check on Root Device[0m.
[[0;32m  OK  [0m] Finished [0;1;39mLoad Kernel Modules[0m.
         Mounting [0;1;39mFUSE Control File System[0m...
         Mounting [0;1;39mKernel Configuration File System[0m...
[[0;32m  OK  [0m] Started [0;1;39mFile System Check Daemon to report status[0m.
         Starting [0;1;39mRemount Root and Kernel File Systems[0m...
         Starting [0;1;39mApply Kernel Variables[0m...
[[0;32m  OK  [0m] Mounted [0;1;39mFUSE Control File System[0m.[    6.606629] EXT4-fs (xvda1): re-mounted. Opts: discard. Quota mode: none.

[[0;32m  OK  [0m] Mounted [0;1;39mKernel Configuration File System[0m.
[[0;32m  OK  [0m] Finished [0;1;39mudev Coldplug all Devices[0m.
[[0;32m  OK  [0m] Finished [0;1;39mRemount Root and Kernel File Systems[0m.
         Starting [0;1;39mFlush Journal to Persistent Storage[0m...
         Starting [0;1;39mLoad/Save Random Seed[0m...
         Starting [0;1;39mCreate System Users[0m...
         Starting [0;1;39mudev Wait for Complete Device Initialization[0m...
[[0;32m  OK  [0m] Finished [0;1;39mApply Kernel Variables[0m.
[[0;32m  OK  [0m] Finished [0;1;39mLoad/Save Random Seed[0m.
[[0;32m  OK  [0m] Finished [0;1;39mCreate System Users[0m.
         Starting [0;1;39mCreate Static Device Nodes in /dev[0m...
[[0;32m  OK  [0m] Finished [0;1;39mCreate Static Device Nodes in /dev[0m.
         Starting [0;1;39mudev Kernel Device Manager[0m...
[[0;32m  OK  [0m] Finished [0;1;39mUncomplicated firewall[0m.
[[0;32m  OK  [0m] Started [0;1;39mudev Kernel Device Manager[0m.
[[0;32m  OK  [0m] Started [0;1;39mDispatch Password …ts to Console Directory Watch[0m.
[[0;32m  OK  [0m] Reached target [0;1;39mLocal Encrypted Volumes[0m.
[[0;32m  OK  [0m] Finished [0;1;39mFlush Journal to Persistent Storage[0m.
[[0;32m  OK  [0m] Found device [0;1;39m/dev/ttyS0[0m.
[[0;32m  OK  [0m] Listening on [0;1;39mLoad/Save RF …itch Status /dev/rfkill Watch[0m.
```
Any ideas?
Thanks

Accepted Answer

The system log you provided indicates that there may be an issue with resource allocation for the EISA bus and slots on your instance. The lines [ 3.838396] platform eisa.0: EISA: Cannot allocate resource for mainboard and [ 3.850447] platform eisa.0: Cannot allocate resource for EISA slot 1 suggest that the system is unable to allocate the necessary resources for the EISA bus and its associated slots.

> EISA (Extended Industry Standard Architecture) is an older bus standard that allows expansion cards to be connected to the motherboard. It's possible that your instance is encountering difficulties in allocating resources for this bus, which could lead to failed status checks and the instance becoming unreachable.

Here are a few steps you can take to troubleshoot the issue:
* Stop and start the instance: Since rebooting the instance temporarily resolves the issue, stopping and starting it might help. This action can allocate new underlying resources and potentially resolve any resource allocation conflicts.
* Check the instance type: Verify that the instance type you are using is compatible with the EISA bus and its associated devices. If the instance type does not support EISA, it may be causing conflicts and failed status checks.
* Update instance firmware and drivers: If your instance has outdated firmware or drivers, it could lead to compatibility issues with the EISA bus. Update the firmware and drivers to the latest versions provided by your instance provider. (This is not in your Control and AWS Should do it from the backend).
* Take an AMI Of the Instance and try to launch a copy of the same with different Configuration. 
* Contact AWS Support: If the issue persists and you are unable to resolve it, reach out to AWS Support for further assistance. They can help diagnose the problem and provide guidance specific to your AWS environment.
* Terminate and Launch a new server if you can afford it (Be careful if it is any Critical Server).

EC2 Instance Failed Status Checks

Relevant content