How do I troubleshoot connecting to my EC2 Linux instance using a secondary IP address?

Short description

To connect to your instance through SSH using a secondary IP address, make sure that your instance meets the following prerequisites:

• If you launched your instances in a private subnet, then connect through SSH using the secondary private IP address. Make sure to choose the secondary IPv4 address from the IPv4 CIDR block range of the subnet for the network interface.
• If you launched your instances in a public subnet, then allocate an Elastic IP address to the secondary private IP address attached to your instance.
• Make sure that the operating system on your EC2 instance recognizes the secondary private IP address.
  For Amazon Linux, see Configure the operating system on your instance to recognize secondary private IPv4 addresses.
  For Ubuntu, see How can I make my secondary network interface work in my Ubuntu EC2 instance?
  For other Linux distributions, see the network configuration documentation for your distribution.

If your instance meets these prerequisites, then do the following to troubleshoot connecting through SSH:

1. Connect through SSH with verbose messaging on to identify the error.
2. Review the system logs to look for errors.

Resolution

Note: Review the general connection prerequisites before beginning.

Connect through SSH with verbose messaging on to identify the error

For detailed instructions, see How do I troubleshoot problems connecting to my Amazon EC2 Linux instance using SSH?

Review the system logs to look for errors

If the preceding steps don't resolve the issue, then review the instance's system logs. There are two methods for accessing the logs on your instance:

Method 1: Use the EC2 Serial Console

If you enabled EC2 Serial Console for Linux, then you can use it to troubleshoot supported Nitro-based instance types. The serial console helps you troubleshoot boot issues, network configuration, and SSH configuration issues. The serial console connects to your instance without the need for a working network connection. You can access the serial console using the Amazon EC2 console or the AWS Command Line Interface (AWS CLI).

Before using the serial console, you must grant access to it at the account level and create AWS Identity and Access Management (IAM) policies granting access to your IAM users. Also, every instance using the serial console must include at least one password-based user. If your instance is unreachable and you haven’t configured access to the serial console, follow the instructions in Method 2. For information on configuring the EC2 Serial Console for Linux, see Configure access to the EC2 Serial Console.

Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.

Method 2: Access the logs using a rescue instance

Warning: Before starting this procedure, be aware that:

• Stopping the instance erases any data on instance store volumes. Be sure that you backup any data on the instance store volume that you want to keep. For more information, see Determine the root device type of your instance.
• Stopping and then starting the instance changes the public IP address of your instance. It's a best practice to use an Elastic IP address instead of a public IP address when routing external traffic to your instance.

1.    Open the Amazon EC2 console.

2.    Choose Instances from the navigation pane, and select instance that you're trying to connect to.

3.    Choose Instance State, Stop Instance, and then select Stop. Make a note of the Instance ID.

Note: If you're not using the New EC2 Experience, then select the instance that you're trying to connect to. Then choose Actions, Instance State, Stop, Stop.

4.    Detach the root Amazon Elastic Block Store (Amazon EBS) volume from the stopped instance. Make a note of the device name of the root EBS volume. The device name is required when you reattach the volume after troubleshooting.

5.    Launch a new EC2 instance in the same Availability Zone as the original instance. The new instance becomes your "rescue" instance.

Note: It's a best practice to use an Amazon Linux 2 instance as the rescue instance. Using an Amazon Linux 2 instance prevents the rescue instance from booting up from the attached EBS volume because the UUID or name of the EBS volume are the same.

6.    After the rescue instance launches, choose Volumes from the navigation pane, and then select the detached root volume of the impaired instance.

Note: If the impaired instance's root volume has Marketplace codes and the rescue instance isn't Amazon Linux, then stop the rescue instance before attaching the root EBS volume. An instance might have Marketplace codes if you launched the instance from an official RHEL or CentOS AMI, for example.

7.    Choose Actions, Attach Volume.

8.    Select Instances in the navigation pane, and then select the rescue instance.

9.    Choose Instance state, Start instance.

Note: If you're not using the New EC2 Experience console, then select the instance that you're trying to connect to, then choose Actions, Instance State, Start.

11.    Connect to the rescue instance through SSH.

12.    Run the following command to verify that the EBS volume successfully attached to the rescue instance. In the following command, the volume attached as /dev/sdf.

$ lsblk

The following is an example of the command output:

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   20G  0 disk 
└─xvda1 202:1    0   20G  0 part /
xvdf    202:80   0  100G  0 disk

13.    Use the following commands to create a mount point directory, and then mount the attached volume to the rescue instance. In the following example, the mount point directory is /test.

$ sudo su
$ mkdir /test
$ mount /dev/xvdf1 /test
$ df -h
$ cd /test

14.    Locate errors in the system logs and authentication-related logs that are timestamped with the times that you attempted access.

Amazon Linux, RHEL, CentOS

$ sudo cat /test/var/log/messages

Amazon Linux, RHEL, CentOS (Authentication-related issues)

$ sudo cat /test/var/log/secure

Ubuntu, Debian (System Logs)

$ sudo cat /test/var/log/syslog

Ubuntu, Debian (Authentication-related issues)

$ sudo cat /test/var/log/auth.log

15.    After reviewing the configurations and addressing any errors, unmount and detach the EBS root volume from the rescue instance.

$ umount /test

16.    Attach the volume to the original instance. The device name is /dev/xvda.

---

Related information

Connect to your Linux instance using an SSH client