How do I access the EC2 Serial Console of an unreachable or inaccessible Amazon EC2 Linux instance?

Resolution

Prerequisite: Make sure that you adhere to the serial console prerequisites.

First, check whether you can connect to the EC2 Serial Console. If you can connect, then the only configuration that's missing is the OS user password. Proceed to Set the password for the root user or OS user. If you can't connect, then you must use a rescue EC2 instance to update the configuration.

Note: To update the configuration with a rescue instance, you must stop and start the original instance.

Configure your instance for a stop and start

Note: When you stop and start an instance, the instance's public IP address changes. It's a best practice to use an Elastic IP address to route external traffic to your instance instead of a public IP address. If you use Amazon Route 53, then you might need to update the Route 53 DNS records when the public IP address changes. A stop and start is different from an instance reboot. For more information, see How EC2 instance stop and start works.

Before you stop and start your instance, take the following actions:

• If your instance uses an instance store, then save the volume data on the instance store to persistent storage. For example, you can save the data to Amazon Elastic Block Store (Amazon EBS) volumes or Amazon Simple Storage Service (Amazon S3) buckets.
  Important: Amazon EC2 deletes instance store data when you stop the instance.
• Create a snapshot of your Amazon EBS volume. If you experience issues, then you can use the snapshot to restore your instance.
• Temporarily remove the instance from its Amazon EC2 Auto Scaling group so that you don't accidentally terminate the instance when you stop it.
  Note: EC2 Auto Scaling might terminate stopped instances in an Auto Scaling group based on your scale-in protection settings. Instances that you launch with Amazon EMR, AWS CloudFormation, or AWS Elastic Beanstalk might be in an Auto Scaling group.
• Set the instance shutdown behavior to Stop to make sure that the instance doesn't terminate when you stop it.

Access the instance's root volume

Complete the following steps:

1. Get the volume ID and device name for the unreachable instance's root volume.

2. Stop the unreachable instance.

3. Launch a rescue instance from an Amazon Machine Image (AMI) with the same Linux OS version in the same Availability Zone.

4. Detach the root volume from the unreachable instance and attach it to the rescue instance as a secondary volume. Note the volume device name.

5. Use SSH to connect to the rescue instance.

6. To change to the root user, run the following command:

   sudo su

7. To identify the block device name and partition, run the following command:

   lsblk

   Example output:

   [root ~]$ lsblk
   NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
   xvda    202:0    0    8G  0 disk
   └─xvda1 202:1    0    8G  0 part /
   xvdf    202:80   0  101G  0 disk
   └─xvdf1 202:81   0  101G  0 part

   Note: The example output shows a XEN instance with blockfront drivers. Both /dev/xvda and /dev/xvdf are partitioned volumes.
   If your volume is partitioned, then run the following command to mount the partition instead of the volume:

   mount -o nouuid /dev/xvdf1 /mnt

   If you use an instance that's built on the AWS Nitro System, then the volume device name looks similar to the /dev/nvme[0-26]n1 format. If your Nitro instance uses an non-volatile memory express (NVMe) driver, then run the following command to mount the partition at the /mnt directory:

   mount -o nouuid  /dev/nvme1n1p1 /mnt

   Note: Replace /dev/nvme1n1p1 with the root partition of the unreachable instance's volume.

8. To create a chroot environment in the /mnt directory, run the following command:

   for i in dev proc sys run; do mount -o bind /$i /mnt/$i; done; chroot /mnt

   In the preceding command, the /dev, /proc, /sys, and /run directories are bind-mounted from the original root file system. With this configuration, processes that run inside the chroot environment can access these system directories.

Set the password for the root user or OS user

Note: When you set the password, the password is visible in plaintext.

Set the OS or root user password.

To give root access to the SSH daemon (sshd) in Linux, run the following command:

sed -i 's/#\?PermitRootLogin .*/PermitRootLogin yes/' /etc/ssh/sshd_config

Pass kernel parameters to the bootloader GRUB

To send all console output to the serial port, you must pass the console=tty0 and console=ttyS0,115200 kernel parameters to the bootloader GRUB.

Note: The tty0 and ttyS0 contain a zero and not the letter O. For more information about kernel parameters, see Linux Serial Console on the Linux kernel website.

To add console=tty0 and console=ttyS0,115200 to the GRUB_CMDLINE_LINUX_DEFAULT line in the /etc/default/grub file, complete the following steps:

1. To open the /etc/default/grub file with a text editor in chroot, run one of the following commands:

   nano /etc/default/grub

   -or-

   vi /etc/default/grub

2. Locate the line that starts with GRUB_CMDLINE_LINUX_DEFAULT.
   Example text:

   GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

3. Add console=tty0 and console=ttyS0,115200 to the line. Enter a space to separate the kernel parameters from the current parameters.
   Example updated text:

   GRUB_CMDLINE_LINUX_DEFAULT="quiet splash console=tty0 console=ttyS0,115200"

4. Save the changes, and then exit the text editor.

5. To update the GRUB configuration, run the following command based on your Linux distribution and configuration.
   Red Hat Enterprise Linux (RHEL), CentOS, Amazon Linux, or Rocky Linux on BIOS-based machines:

   grub2-mkconfig -o /boot/grub2/grub.cfg

   RHEL, CentOS, Amazon Linux, or Rocky Linux on UEFI-based machines:

   sudo grub2-mkconfig -o /boot/efi/EFI/distro/grub.cfg

   Note: Replace distro with your distribution, such as rhel or amzn.
   Ubuntu and Debian:

   update-grub

   Note: The preceding commands regenerate the GRUB configuration file with the new kernel parameters.

For more information, see Using EC2 Serial Console to access the GRUB menu and recover from boot failures.

Stop the rescue instance, and then start the original instance

Complete the following steps:

1. To exit from chroot and unmount /dev, /run, /proc, and /sys, run the following command:

   exit; umount -fl /mnt/{dev,proc,run,sys,}

2. To unmount /mnt, run the following command:

   umount /mnt

3. Stop the rescue instance.

4. Detach the root volume from the rescue instance.

5. Attach the root volume to the original instance as the /dev/sda1 root volume.

6. Start the original instance.

7. Use the browser-based client to connect to your instance's serial console.

8. Log in as the root user with the password that you set.