How can I configure access to the EC2 Serial Console of an unreachable or inaccessible Linux instance?

Short description

To configure access to the serial console, do the following:

1.    Access the instance's root volume.

2.    Set the password for the root user or any other OS user.

3.    Check and update the GRUB settings for the Serial Console.

Note: You can skip step 3 if the EC2 Serial Console is working properly on the affected instance and you just need to set the password for your OS user.

Prerequisites

To use the serial console, make sure that you met the prerequisites, except for Set an OS user password. Setting a password is discussed in the following resolution.

Resolution

Access the instance's root volume using a rescue instance

Create a temporary rescue instance, and then remount your Amazon Elastic Block Store (Amazon EBS) volume on the rescue instance. From the rescue instance, you can check and modify the GRUB settings for the serial console. You can also set the password for the root user or any other OS user.

Important: Don't perform this procedure on an instance store-backed instance. Because the recovery procedure requires a stop and start of your instance, any data on that instance is lost. For more information, see Determine the root device type of your instance.

1.    Create an EBS snapshot of the root volume. For more information, see Create Amazon EBS snapshots.

2.    Open the Amazon EC2 console.

Note: Be sure that you're in the correct Region.

3.    Select Instances from the navigation pane, and then choose the impaired instance.

4.    Choose Instance State, Stop instance, and then select Stop.

5.    In the Storage tab, under Block devices, select the Volume ID for /dev/sda1 or /dev/xvda.

Note: The root device differs by AMI, but /dev/xvda or /dev/sda1 are reserved for the root device. For example, Amazon Linux 1 and 2 use /dev/xvda. Other distributions, such as Ubuntu 16, 18, CentOS 7, and RHEL 7.5, use /dev/sda1.

6.    Choose Actions, Detach Volume, and then select Yes, Detach. Note the Availability Zone.

Note: You can tag the EBS volume before detaching it to help identify it in later steps.

7.    Launch a rescue EC2 instance in the same Availability Zone.

Note: Depending on the product code, you might be required to launch an EC2 instance of the same OS type. For example, if the impaired EC2 instance is a paid RHEL AMI, you must launch an AMI with the same product code. For more information, see Get the product code for your instance.

If the original instance is running SELinux (RHEL, CentOS 7 or 8, for example), launch the rescue instance from an AMI that uses SELinux. If you select an AMI running a different OS, such as Amazon Linux 2, any modified file on the original instance has broken SELinux labels.

8.    After the rescue instance launches, choose Volumes from the navigation pane, and then choose the detached root volume of the impaired instance.

9.    Choose Actions, Attach Volume.

10.    Choose the rescue instance ID ( id-xxxxx), and then set an unused device. In this example, /dev/sdf.

11.     Use SSH to connect to the rescue instance.

12.    Run the lsblk command to view your available disk devices:

lsblk

The following is an example of the output:

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0     0   15G  0 disk
└─xvda1 202:1     0   15G  0 part /
xvdf    202:0     0   15G  0 disk
    └─xvdf1 202:1 0   15G  0 part

Note: Nitro-based instances expose EBS volumes as NVMe block devices. The output generated by the lsblk command on Nitro-based instances shows the disk names as nvme[0-26]n1. For more information, see Amazon EBS and NVMe on Linux instances. The following is an example of the lsblk command output on a Nitro-based instance:

NAME           MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1        259:0    0    8G  0 disk 
└─nvme0n1p1    259:1    0    8G  0 part /
└─nvme0n1p128  259:2    0    1M  0 part 
nvme1n1        259:3    0  100G  0 disk 
└─nvme1n1p1    259:4    0  100G  0 part /

13.    Run the following command to become root:

sudo -i

14.    Mount the root partition of the mounted volume to /mnt. In the preceding example, /dev/xvdf1 or /dev/nvme2n1p2 is the root partition of the mounted volume. For more information, see Make an Amazon EBS volume available for use on Linux.

Note: In the following example, replace /dev/xvdf1 with the correct root partition for your volume.

mount -o nouuid /dev/xvdf1 /mnt

Note: If /mnt doesn't exist on your configuration, create a mount directory, and then mount the root partition of the mounted volume to this new directory.

mkdir /mnt
mount -o nouuid /dev/xvdf1 /mnt

You can now access the data of the impaired instance through the mount directory.

15.    Mount /dev, /run, /proc, and /sys of the rescue instance to the same paths as the newly mounted volume:

for m in dev proc run sys; do mount -o bind {,/mnt}/$m; done

Call the chroot function to change into the mount directory.

Note: If you have a separate /boot and /etc partitions, mount them to /mnt/boot and /mnt/etc before running the following command.

chroot /mnt

Set the password for the root user or any other OS user.

Use the passwd command to set the password for your OS user. In the following example, the user is root:

passwd root

Check and update the GRUB settings for the serial console.

Note: You can skip this step if the EC2 Serial console is working properly on the affected instance and you just need to set the password for your OS user.

The supported serial console port for Linux is ttyS0. If the screen remains black without providing any output when connecting to the EC2 Serial Console, you should make sure that the console entry is properly configured in the GRUB settings. The examples provided below are taken from the AWS Marketplace AMIs for the different distributions in which the serial console is working properly:

GRUB2 for Amazon Linux 2, RHEL , and CentOS 7

1.    Verify that the console entry for the ttyS0 is properly set in the GRUB_CMDLINE_LINUX_DEFAULT line of the /etc/default/grub file:

Amazon Linux 2

GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 nvme_core.io_timeout=4294967295 rd.emergency=poweroff rd.shell=0"

RHEL 7

GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau crashkernel=auto"

CentOS 7

GRUB_CMDLINE_LINUX="console=tty0 crashkernel=auto net.ifnames=0 console=ttyS0"

2.    If the console entry for the ttyS0 is not set, add it in the GRUB_CMDLINE_LINUX_DEFAULT line. Then update GRUB to regenerate the /boot/grub2/grub.cfg file:

grub2-mkconfig -o /boot/grub2/grub.cfg

GRUB1 (Legacy GRUB) for Red Hat 6 and Amazon Linux 1

1.    Verify that the console entry for the ttyS0 is properly set in the kernel line of the /boot/grub/grub.conf file:

Amazon Linux 1

kernel /boot/vmlinuz-4.14.252-131.483.amzn1.x86_64 root=LABEL=/ console=tty1 console=ttyS0 selinux=0 nvme_core.io_timeout=4294967295

Red Hat 6

kernel /boot/vmlinuz-2.6.32-573.el6.x86_64 console=ttyS0 console=ttyS0,115200n8 ro root=UUID=0e6b1614-7bbe-4d6e-bc78-a5556a123ba8 rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 xen_blkfront.sda_is_xvda=1 console=tty0 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM

2.    If the console entry for the ttyS0 is not set, add it in the /boot/grub/grub.conf file according to the preceding examples.

GRUB2 for Ubuntu 16.04, 18.04 and 20.04

1.    Verify that the console entry for the ttyS0 is properly set in the GRUB_CMDLINE_LINUX_DEFAULT line of the /etc/default/grub.d/50-cloudimg-settings.cfg file:

GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295"

2.    If the console entry console=ttyS0 is not present**,** add it in the GRUB_CMDLINE_LINUX_DEFAULT line. Then update the GRUB configuration using the following command:

update-grub

**GRUB2 for RHEL 8, CentOS 8, and Amazon Linux 2023
**

1.    Run the grubby --default-kernel command to see the current default kernel:

grubby --default-kernel

2.    Run the grubby --info=ALL command to see all the available kernels, their indexes, and their args:

grubby --info=ALL

3.    Verify that the console entry for the ttyS0 is properly set in the args line for the default kernel outlined in the step 1.

RHEL 8

index=0
kernel="/boot/vmlinuz-4.18.0-305.el8.x86_64"
args="ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=auto $tuned_params"
root="UUID=d35fe619-1d06-4ace-9fe3-169baad3e421"
initrd="/boot/initramfs-4.18.0-305.el8.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (4.18.0-305.el8.x86_64) 8.4 (Ootpa)"
id="0c75beb2b6ca4d78b335e92f0002b619-4.18.0-305.el8.x86_64"

CentOS 8

index=2
kernel="/boot/vmlinuz-4.18.0-193.19.1.el8_2.x86_64"
args="ro console=ttyS0,115200n8 no_timer_check net.ifnames=0 nvme_core.io_timeout=4294967295 nvme_core.max_retries=10 crashkernel=auto $tuned_params"
root="UUID=b437cbaa-8fe5-49e4-8537-0895c219037a"
initrd="/boot/initramfs-4.18.0-193.19.1.el8_2.x86_64.img $tuned_initrd"
title="CentOS Linux (4.18.0-193.19.1.el8_2.x86_64) 8 (Core)"
id="dc49529e359897df0b9664481b009b1f-4.18.0-193.19.1.el8_2.x86_64"

Amazon Linux 2023

[root@ip-172-31-92-173 ~]# grubby --info DEFAULTindex=0
kernel="/boot/vmlinuz-6.1.15-28.43.amzn2023.x86_64"
Acessing serial console on Amazon Linux 2023
root="UUID=7efef47b-a4f8-4b90-9504-8196067a31b6"
initrd="/boot/initramfs-6.1.15-28.43.amzn2023.x86_64.img"
title="Amazon Linux (6.1.15-28.43.amzn2023.x86_64) 2023"
id="02c860b8daad4861b87f9b3603834c8f-6.1.15-28.43.amzn2023.x86_64"

4.    If the console entry for the ttyS0 isn't set, then use the following grubby command to append it to the args of the default kernel:

grubby --args "console=tty0 console=ttyS0,115200n8" --update-kernel DEFAULT

Unmount and detach the root volume from the rescue instance, and then attach the volume to the impaired instance

1.    Exit from chroot, and unmount /dev, /run, /proc, and /sys:

exit
umount /mnt/{dev,proc,run,sys,}

2.    From the Amazon EC2 console, choose Instances, and then choose the rescue instance.

3.    Choose Instance State, Stop instance, and then select Yes, Stop.

4.    Detach the root volume id-xxxxx (the volume from the impaired instance) from the rescue instance.

5.    Attach the root volume you detached in step 4 to the impaired instance as the root volume (/dev/sda1), and then start the instance.

Note: The root device differs by AMI. The names /dev/xvda or /dev/sda1 are reserved for the root device. For example, Amazon Linux 1 and 2 use /dev/xvda. Other distributions, such as Ubuntu 16, 18, CentOS 7, and RHEL 7.5, use /dev/sda1.

Now you can access the OS of the impaired instance through the EC2 Serial Console using the password defined in the previous step for the root or any other OS user.

Related information

Troubleshoot your Linux instance using GRUB