Cannot connect to AWS Linux instance

Question

A number of times over the past week or so I have been unable to connect to my AWS Linux Instance. Nothing is accessible, http, mail, ssh etc.  
Checking the System Log from the AWS EC2 dashboard shows the login prompt as usual.  
I have checked out on the net and it would appear the dhcp is dropping the ip address and not renewing. The instance is working fine but nothing can access it.  
  
Below is a copy of the messages log showing the time the dhcp stopped and me having to reboot to gain access.  
  
Sep  4 23:58:49 cpc-world dhclient\[2146]: DHCPREQUEST on eth0 to 172.31.0.1 port 67 (xid=0x24e9fa70)  
Sep  4 23:58:49 cpc-world dhclient\[2146]: DHCPACK from 172.31.0.1 (xid=0x24e9fa70)  
Sep  4 23:58:49 cpc-world dhclient\[2146]: bound to 172.31.0.242 -- renewal in 1782 seconds.  
Sep  4 23:58:49 cpc-world ec2net: \[get_meta] Trying to get http://169.254.169.254/latest/meta-data/network/interfaces/macs/02:42:27:40:4f:d9/local-ipv4s  
Sep  4 23:58:49 cpc-world ec2net: \[rewrite_aliases] Rewriting aliases of eth0  
Sep  5 00:00:15 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 120050ms.  
Sep  5 00:02:15 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 125640ms.  
Sep  5 00:04:20 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 118350ms.  
Sep  5 00:06:19 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 109350ms.  
Sep  5 00:08:08 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 108180ms.  
Sep  5 00:09:57 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 130940ms.  
Sep  5 00:12:08 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 116760ms.  
Sep  5 00:14:04 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 128530ms.  
Sep  5 00:16:13 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 128870ms.  
Sep  5 00:18:22 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 129670ms.  
Sep  5 00:20:32 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 131290ms.  
Sep  5 00:22:43 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 116830ms.  
Sep  5 00:24:40 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 111860ms.  
Sep  5 00:26:32 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 127300ms.  
Sep  5 00:28:31 cpc-world dhclient\[2146]: DHCPREQUEST on eth0 to 172.31.0.1 port 67 (xid=0x24e9fa70)  
Sep  5 00:28:31 cpc-world dhclient\[2146]: DHCPACK from 172.31.0.1 (xid=0x24e9fa70)  
Sep  5 00:28:31 cpc-world dhclient\[2146]: bound to 172.31.0.242 -- renewal in 1784 seconds.  
Sep  5 00:28:31 cpc-world ec2net: \[get_meta] Trying to get http://169.254.169.254/latest/meta-data/network/interfaces/macs/02:42:27:40:4f:d9/local-ipv4s  
Sep  5 00:28:31 cpc-world ec2net: \[rewrite_aliases] Rewriting aliases of eth0  
Sep  5 00:28:39 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 130760ms.  
Sep  5 00:30:50 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 116880ms.  
Sep  5 00:32:47 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 122170ms.  
Sep  5 00:34:49 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 126950ms.  
Sep  5 00:36:56 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 130550ms.  
Sep  5 00:39:07 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 127530ms.  
Sep  5 00:41:15 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 111150ms.  
  
Sep  5 00:43:06 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 124060ms.  
Sep  5 03:30:53 cpc-world dhclient\[2244]: XMT: Solicit on eth0, interval 128010ms.  
Sep  5 07:30:24 cpc-world kernel: imklog 5.8.10, log source = /proc/kmsg started.  
  
Sep  5 07:30:24 cpc-world rsyslogd: \[origin software="rsyslogd" swVersion="5.8.10" x-pid="2311" x-info="http://www.rsyslog.com"] start  
Sep  5 07:30:24 cpc-world kernel: \[    0.000000] Linux version 4.14.62-65.117.amzn1.x86_64 (mockbuild@gobi-build-60009) (gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC)) #1 SMP Fri Aug 10 $  
Sep  5 07:30:24 cpc-world kernel: \[    0.000000] Command line: root=LABEL=/ console=tty1 console=ttyS0 selinux=0 LANG=en_US.UTF-8 KEYTABLE=us  
Sep  5 07:30:24 cpc-world kernel: \[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'  
Sep  5 07:30:24 cpc-world kernel: \[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'  
Sep  5 07:30:24 cpc-world kernel: \[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'  
Sep  5 07:30:24 cpc-world kernel: \[    0.000000] x86/fpu: xstate_offset\[2]:  576, xstate_sizes\[2]:  256  
Sep  5 07:30:24 cpc-world kernel: \[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.  
Sep  5 07:30:24 cpc-world kernel: \[    0.000000] e820: BIOS-provided physical RAM map:  
Sep  5 07:30:24 cpc-world kernel: \[    0.000000] BIOS-e820: \[mem 0x0000000000000000-0x000000000009dfff] usable  
Sep  5 07:30:24 cpc-world kernel: \[    0.000000] BIOS-e820: \[mem 0x000000000009e000-0x000000000009ffff] reserved  
Sep  5 07:30:24 cpc-world kernel: \[    0.000000] BIOS-e820: \[mem 0x00000000000e0000-0x00000000000fffff] reserved  
Sep  5 07:30:24 cpc-world kernel: \[    0.000000] BIOS-e820: \[mem 0x0000000000100000-0x000000007fffffff] usable

Any ideas on how to resolve the issue?  
  
Cheers  
  
Peter

AWS-User-3892169 · Answer

Hello Peter,  
  
I have checked the instance's Console Output and the DHCP messages don't look unusual. However, in the same log I have found the line below that looks much more concerning.  
  
_mysqld invoked oom-killer ..._  
  
As you may know, the oom-killer message above informs about the kernel killing mysqld process due to the system being short of memory. In this case I would recommend you to check and monitor what processes take the most of your system's memory to manage this situation. You may need to tune the MySQL if this is the culprit. Else you could also consider modifying your EC2 instance's size to use one with more memory.  
  
If you wish to monitor memory usage on your instance from CloudWatch you could also install CloudWatch Monitoring Scripts \[1].   
  
Links:  
\[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html  
  
Regards,  
awstomas

peterwj · Answer

Tomas,  
  
Thanks for the response.  
  
I will look into it.  
  
I did notice in the maillog that dovecot need more memory so that looks like the culprit.  
  
I'll try a tune a bit but I suspect and increase in instance size will be the better option.  
  
Many thanks  
  
Peter

Cannot connect to AWS Linux instance

Add your answer

Relevant content