How can I send user-data output to the console logs on an EC2 instance running RHEL 7 or RHEL 8?

Short description

To troubleshoot issues on your EC2 instance bootstrap without having to access the instance through SSH, you can add code to your user-data bash script that redirects all the output both to the /var/log/user-data.log and to /dev/console. When the code is executed, you can see your user-data invocation logs in your console.

Note: This resolution is for RHEL 7 and RHEL 8 only. For information on Amazon Linux and Amazon Linux 2, see How can I send user-data output to the console logs on an EC2 instance running Amazon Linux or Amazon Linux 2?

Resolution

1.    Open the Amazon EC2 console.

2.    Launch a new instance and SSH into it.

3.    Edit the GRUB_CMDLINE_LINUX line in /etc/default/grub and change "console=ttyS0,115200n8 console=tty0" to "console=tty1 console=ttyS0".

RHEL 7:

# cat /etc/default/grub
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0 net.ifnames=0 rd.blacklist=nouveau crashkernel=auto"
GRUB_DISABLE_RECOVERY="true"

RHEL 8:

# cat /etc/default/grub
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=auto"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

4.    Recreate the /boot/grub2/grub.cfg file:

RHEL 7:

# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-1062.1.2.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1062.1.2.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-026767dbe06a4910a5ce3bd0def903c0
Found initrd image: /boot/initramfs-0-rescue-026767dbe06a4910a5ce3bd0def903c0.img
done

RHEL 8:

# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
done

5.   Stop the instance.

6.    Create a new AMI from the modified instance.

7.    Launch a new instance from the new AMI.

8.    Enter the following command to redirect the user-data output console:

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
  yum -y update
  echo "Hello from user-data!"

The following is the line that redirects the user-data output:

exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

The following is sample console output:

RHEL 7:

  subscription-manager.x86_64 0:1.24.26-3.el7_8
  subscription-manager-rhsm.x86_64 0:1.24.26-3.el7_8
  subscription-manager-rhsm-certificates.x86_64 0:1.24.26-3.el7_8
  sudo.x86_64 0:1.8.23-9.el7
  systemd.x86_64 0:219-73.el7_8.8
  systemd-libs.x86_64 0:219-73.el7_8.8
  systemd-sysv.x86_64 0:219-73.el7_8.8
  teamd.x86_64 0:1.29-1.el7
  tuned.noarch 0:2.11.0-8.el7
  tzdata.noarch 0:2020a-1.el7
  util-linux.x86_64 0:2.23.2-63.el7
  yum.noarch 0:3.4.3-167.el7
  yum-utils.noarch 0:1.1.31-54.el7_8

Replaced:
  iwl7265-firmware.noarch 0:22.0.7.0-72.el7

Complete!
+ echo 'Hello from user-data!'
Hello from user-data!

RHEL 8:

Installed:
  grub2-tools-efi-1:2.02-82.el8_2.1.x86_64

  kernel-4.18.0-193.13.2.el8_2.x86_64

  kernel-core-4.18.0-193.13.2.el8_2.x86_64

  kernel-modules-4.18.0-193.13.2.el8_2.x86_64

  linux-firmware-20191202-97.gite8a0f4c9.el8.noarch

Complete!
+
echo 'Hello from user-data!'
Hello from user-data!

Related information

Running commands on your Linux instance at launch