How can I send user-data output to the console logs on an EC2 instance that’s running RHEL?

Short description

To troubleshoot issues with your EC2 instance bootstrap, modify your user-data bash script to redirect all output. Redirect the output to both /var/log/user-data.log and /dev/console. When the script runs, you can view the user-data invocation logs directly in your console.

Note: The following resolution is only for RHEL 7, RHEL 8, and RHEL 9. For information on Amazon Linux, Amazon Linux 2 and Amazon Linux 2023, see How can I send user-data output to the console logs on an EC2 instance for Amazon Linux 1, Amazon Linux 2, or Amazon Linux 2023?

Resolution

To send user-data output to the console logs on an EC2 instance that's running RHEL, complete the following steps:

1. Open the Amazon EC2 console.

2. Connect to an existing instance, or launch a new instance. Then, use SSH to connect to the instance.

3. Edit the GRUB_CMDLINE_LINUX line in /etc/default/grub and change "console=ttyS0,115200n8 console=tty0" to "console=tty1 console=ttyS0":
   RHEL 7

   # cat /etc/default/grub  
   GRUB_TIMEOUT=1
   GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
   GRUB_DEFAULT=saved
   GRUB_DISABLE_SUBMENU=true
   GRUB_TERMINAL_OUTPUT="console"
   GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0 net.ifnames=0 rd.blacklist=nouveau crashkernel=auto"
   GRUB_DISABLE_RECOVERY="true"

   RHEL 8

   # cat /etc/default/grub  
   GRUB_TIMEOUT=1
   GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
   GRUB_DEFAULT=saved
   GRUB_DISABLE_SUBMENU=true
   GRUB_TERMINAL_OUTPUT="console"
   GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=auto"
   GRUB_DISABLE_RECOVERY="true"
   GRUB_ENABLE_BLSCFG=true

   RHEL 9

   # cat /etc/default/grub
   GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295"
   GRUB_TIMEOUT=0
   GRUB_ENABLE_BLSCFG=true
   GRUB_DEFAULT=saved

4. To recreate the /boot/grub2/grub.cfg file, run the following script:
   RHEL 7

   # grub2-mkconfig -o /boot/grub2/grub.cfgGenerating grub configuration file ...
   Found linux image: /boot/vmlinuz-3.10.0-1062.1.2.el7.x86_64
   Found initrd image: /boot/initramfs-3.10.0-1062.1.2.el7.x86_64.img
   Found linux image: /boot/vmlinuz-0-rescue-026767dbe06a4910a5ce3bd0def903c0
   Found initrd image: /boot/initramfs-0-rescue-026767dbe06a4910a5ce3bd0def903c0.img
   done

   RHEL 8

   # grub2-mkconfig -o /boot/grub2/grub.cfg
   Generating grub configuration file ...
   done

   RHEL 9

   # grub2-mkconfig -o /boot/grub2/grub.cfg
   Generating grub configuration file ...
   Adding boot menu entry for UEFI Firmware Settings ...
   done

5. Stop the instance.

6. (Optional) To run the user data every time the instance reboots, configure your user data script and cloud-init directives with a MIME multi-part file.
   Note: By default, user data scripts and cloud-init directives run only during the first boot cycle of an EC2 instance launch.

7. To redirect the user-data output to the console, run the following command to edit the user data field:

   Content-Type: multipart/mixed; boundary="//"
   MIME-Version: 1.0
   
   --//
   Content-Type: text/cloud-config; charset="us-ascii"
   MIME-Version: 1.0
   Content-Transfer-Encoding: 7bit
   Content-Disposition: attachment; filename="cloud-config.txt"
   
   #cloud-config
   cloud_final_modules:
   - [scripts-user, always]
   
   --//
   Content-Type: text/x-shellscript; charset="us-ascii"
   MIME-Version: 1.0
   Content-Transfer-Encoding: 7bit
   Content-Disposition: attachment; filename="userdata.txt"
   
   #!/bin/bash -xe
   exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
   cat /etc/redhat-release
   echo "Hello from user-data!"
   
   --//--

   Note: The script to redirect user-data output begins with the line #!/bin/bash -xe. The preceding script is cloud-init configuration data that has the shell script run every time the instance starts up. In the preceding command, the following line redirects the user-data output:

   exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

8. Start the instance, and then view the console log output. You receive a console output that's similar to the following example:
   RHEL 7

   <13>May 21 03:11:44 user-data: + cat /etc/redhat-release
   <13>May 21 03:11:44 user-data: Red Hat Enterprise Linux Server release 7.9 (Maipo)
   <13>May 21 03:11:44 user-data: + echo 'Hello from user-data!'
   <13>May 21 03:11:44 user-data: Hello from user-data!

   RHEL 8

   <13>May 21 03:11:21 user-data: + cat /etc/redhat-release
   <13>May 21 03:11:21 user-data: Red Hat Enterprise Linux release 8.6 (Ootpa)
   <13>May 21 03:11:21 user-data: + echo 'Hello from user-data!'
   <13>May 21 03:11:21 user-data: Hello from user-data!

   RHEL 9

   <13>May 21 03:12:34 user-data: + cat /etc/redhat-release
   <13>May 21 03:12:34 user-data: Red Hat Enterprise Linux release 9.3 (Plow)
   <13>May 21 03:12:34 user-data: + echo 'Hello from user-data!'
   <13>May 21 03:12:34 user-data: Hello from user-data!

Important: Your user-data is visible in the console. Don't include confidential information in the data that you send.

Related information

Run commands on your Linux instance at launch