How do I access the EC2 Serial Console of an unreachable or inaccessible Linux instance?

Short description

This article is applicable to the following Linux operating systems (OS):

• Red Hat 7
• Amazon Linux 2
• CentOS 7
• Rocky Linux
• Ubuntu 16
• Debian

Prerequisites: To use the serial console, make sure that all prerequisites are met. Note that the set an OS user password step must be completed after you access the instance's root volume.

Resolution

To configure access to the serial console, the following steps must be completed.

Note: If your EC2 Serial console works properly and you only need to set the password for your OS user, then complete only that resolution step. To verify that the EC2 Serial Console works, see Connect to the EC2 Serial Console.

Access the instance's root volume

To access the instance's root volume, launch a temporary instance. Next, remount your Amazon Elastic Block Store (Amazon EBS) volume on the rescue instance. Then, from the rescue instance, check and modify the GRUB settings for the serial console. You can also set the password for the root user or OS user and allow root access for the SSH daemon.

Important: When you set up a temporary instance, note the following:

• If you stop and restart the instance, then data on instance store volumes is erased. Back up any data on the instance store volume that you want to keep. For more information, see Root volumes for your Amazon EC2 instances.
• If you stop and restart the instance, then the public IP address of your instance changes. It's a best practice to use an Elastic IP address instead of a public IP address when you route external traffic to your instance.
• Create a snapshot of your root volume before you make any changes.

Attach the root disk to a rescue instance

Complete the following steps:

1. Create a new key pair or use an existing key pair.

2. Get the volume ID and device name for the original instance's root volume.

3. Stop the original instance.

4. Launch a temporary instance from an Amazon Machine Image (AMI) with the same Linux OS version in the same Availability Zone.

5. Detach the root volume from the original instance and attach it to the temporary instance as a secondary volume. Note the volume device name.

6. Connect to the temporary instance with your SSH key pair.

7. To change to the root user, run the following command:

   [ec2-user ~]$ sudo su

8. To identify the block device name and partition, run the following command from the temporary instance:

   [root ~]$ lsblk
   NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
   xvda    202:0    0    8G  0 disk
   └─xvda1 202:1    0    8G  0 part /
   xvdf    202:80   0  101G  0 disk
   └─xvdf1 202:81   0  101G  0 part

   The previous example uses a XEN instance with blockfront drivers. Both volume device name /dev/xvda and /dev/xvdf are partitioned volumes, and /dev/xvdg is not. If your volume is partitioned, then run the following command to mount the partition (/dev/xvdf1) instead of the raw device (/dev/xvdf):

   [root ~]$ mount -o nouuid /dev/xvdf1 /mnt

   If you use a Nitro based instance, then the volume device name looks similar to /dev/nvme[0-26]n1. If your instance is built on Nitro with NVMe, then mount the partition at the /mnt directory. Use the device name that you identified earlier to run the following lsblk command:

   [root ~]$ mount -o nouuid  /dev/nvme1n1p1 /mnt

   For more information, see Device names for volumes on Amazon EC2 instances.

9. To create a chroot environment in the /mnt directory, run the following command:

   [root ~]$ for i in dev proc sys run; do mount -o bind /$i /mnt/$i; done; chroot /mnt

   In the previous example, the /dev, /proc, /sys, and /run directories are bind-mounted from the original root file system. This setup allows processes that run inside the chroot environment to access these system directories.

Set the password for the root user or any other OS user

In chroot, use the passwd command to set the password for your OS user and allow root access in the SSH configuration.

In the following example, the user is root. You can set a root passwd with the following command:

\[root ~\]$ passwd root

To set allow root access for the SSH daemon (sshd) in Linux change and change #PermitRootLogin yes to PermitRootLogin yes, run the following code:

\[root ~\]$ sed -i 's/#\\?PermitRootLogin .\*/PermitRootLogin yes/' /etc/ssh/sshd\_config

Pass kernel parameters to the bootloader GRUB

To send all console output to the serial port, the following kernel parameters console=tty0 console=ttyS0,115200 must be passed to the bootloader GRUB.

Note: The tty0 and ttyS0 contain a zero and not the letter O.

The following list shows the selected options for this example:

• console= is a kernel parameter that specifies the preferred console device for kernel messages.
• tty0 refers to the first virtual console (text mode) on the system.
• ttyS0 refers to the first serial port.
• 115200 is the baud rate (bits per second) for the serial port. This is a common baud rate used for serial communication.
• n8 specifies the character size and parity settings for the serial port. n stands for no parity. 8 shows that the character size is 8 bits.
• The tty0 parameter activates the kernel to send console messages to the first virtual console (tty0). That is typically the primary text-mode console on Linux systems.
• The ttyS0 parameter configures the kernel to send console messages to the first serial port (ttyS0). Messages are sent at a baud rate of 115200 with no parity and 8 bits per character.

Check and update the GRUB2 settings for the Serial Console

Add the kernel parameters console=tty0 and console=ttyS0,115200 to the GRUB_CMDLINE_LINUX_DEFAULT line in the /etc/default/grub file

Complete the following steps:

1. Open the /etc/default/grub file with a text editor in chroot. For example, to use the nano or vi editor, run one of the following commands:

   [root ~]$ nano /etc/default/grub
   [root ~]$ vi /etc/default/grub

2. Locate the line that starts with GRUB_CMDLINE_LINUX_DEFAULT. The line looks similar to the following:

   GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

3. Add the kernel parameters console=tty0 and console=ttyS0,115200 to this line. Enter a space to separate these kernel parameters from the current parameters. The updated line looks similar to the following:

   GRUB_CMDLINE_LINUX_DEFAULT="quiet splash console=tty0 console=ttyS0,115200"

4. Save the changes and exit the text editor.

5. To update the GRUB configuration, run the following command:

   For Red Hat 7+, CentOS 7+, Amazon Linux 2+, and Rocky Linux

   On BIOS-based machines:

   [root ~]$ grub2-mkconfig -o /boot/grub2/grub.cfg

   On UEFI-based machines:

   [root ~]$ grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

   For Ubuntu and Debian

   The following command regenerates the GRUB configuration file with the new kernel parameters.

   [root ~]$ update-grub

6. To exit from chroot, and unmount /dev, /run, /proc, and /sys, run the following command:

   [root ~]$ exit; umount -fl /mnt/{dev,proc,run,sys,}

7. From the Amazon EC2 console, choose Instances, and then choose the rescue instance.

8. Choose Instance state, Stop instance, and then select Yes, stop.

9. Detach the root volume id-##### (the volume from the impaired instance) from the rescue instance.

10. Attach the root volume to the impaired instance as the root volume (/dev/sda1), and then start the instance.

11. Use the browser-based client to connect to your EC2 instance's serial console.