Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Is it possible to manually create a bootable UEFI RHEL 8 AMI?

0

We have a requirement to produce AMI built to CIS hardened standards from a Red Hat Enterprise Linux 8 OS including custom partitioning scheme with mount points of /home, /var, /var/log, etc. Our requirement dictates the AMI can only have a single root EBS backed device. Multiple EBS devices are not satisfactory in this situation. Also note, we are running the process via Packer in automated pipelines.

Our first approach was attempting to modify a running EC2 RHEL 8 instance's root EBS device using the AWS CLI, but with the introduction of the 6 hour cool-down period from the time an EBS device is created to when it can be modified, we had to abandon this approach.

When spinning up a new EC2 instance, the root partition will chew up all configured disk space. We have also tried shrinking the root volume partition to make room for additional partitions, but abandoned that approach.

The closest we have made it is to use an auxiliary RHEL8 server and snapshot it's root EBS device. Then, using Packer, spin up a new 'source' EC2 instance that starts with the source's root EBS device, as well as a blank EBS device (we'll call myroot).

At the beginning of the Packer build, the source devices look like the following:

[root@ip-172-31-67-219 ec2-user]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1     259:0    0   10G  0 disk
├─nvme0n1p1 259:1    0  200M  0 part /boot/efi
├─nvme0n1p2 259:2    0  512M  0 part /boot
└─nvme0n1p3 259:3    0  9.3G  0 part /
nvme1n1     259:4    0   10G  0 disk

Note, source OS device is named /dev/nvme0n1, and the empty myroot device is /dev/nvme1n1. During the build, we reach out and create a new EBS device from the auxiliary server's snapshot, and attach it to the source server. At this point, we have the following configuration:

[root@ip-172-31-67-219 ec2-user]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1     259:0    0   10G  0 disk
├─nvme0n1p1 259:1    0  200M  0 part /boot/efi
├─nvme0n1p2 259:2    0  512M  0 part /boot
└─nvme0n1p3 259:3    0  9.3G  0 part /
nvme1n1     259:4    0   10G  0 disk
nvme2n1     259:5    0   10G  0 disk
├─nvme2n1p1 259:6    0  200M  0 part
├─nvme2n1p2 259:7    0  512M  0 part
└─nvme2n1p3 259:8    0  9.3G  0 part

Note the new /dev/nvme2n1 device as the auxiliary server's non-running OS drive.

Our intent is to now use the aux server's OS files to make a running OS out of our blank myroot device.

We can do this by partitioning and copying OS files to new temporary mount points on myroot device. The result of the partitioning and file movement process shows this configuration:

[root@ip-172-31-67-219 ec2-user]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1     259:0    0   10G  0 disk
├─nvme0n1p1 259:1    0  200M  0 part
├─nvme0n1p2 259:2    0  512M  0 part /boot
└─nvme0n1p3 259:3    0  9.3G  0 part /
nvme1n1     259:4    0   10G  0 disk
├─nvme1n1p1 259:9    0  191M  0 part /boot/efi
├─nvme1n1p2 259:10   0  488M  0 part /newboot
├─nvme1n1p3 259:11   0  4.3G  0 part /mnt/myroot/root
├─nvme1n1p4 259:12   0    1G  0 part /mnt/myroot/var
├─nvme1n1p5 259:13   0    1G  0 part /mnt/myroot/varlog
├─nvme1n1p6 259:14   0    1G  0 part /mnt/myroot/varlogaudit
└─nvme1n1p7 259:15   0    2G  0 part /mnt/myroot/home

Note: we remove the temporary /dev/nvme2n1 disk prior to creating and modifying the /boot and /boot/efi partitions. We have also modified myroot's /etc/fstab file with the appropriate partition identifiers in preparation for creating a new AMI off myroot device.

[root@ip-172-31-67-219 ec2-user]# cat /mnt/myroot/root/etc/fstab
UUID=DCE5-01EE  /boot/efi    vfat    defaults,uid=0,gid=0,umask=077,shortname=winnt 0    2
UUID=2fe05fef-dab6-4ced-be2d-f7a9bd613269  /boot        xfs     defaults                                       0    0
UUID=2ca0c201-48ae-482b-850c-d26dc26b9de0  /            xfs     defaults                                       0    0
UUID=3830ef77-5f8d-4a73-bec8-c7f7d4085bba  /var            xfs  defaults,nofail,nodev                          0    0
UUID=85b31b3b-f420-4adb-a221-204d79718626  /var/log        xfs  defaults,nofail,nodev,nosuid                   0    0
UUID=26531d55-fa36-47f2-9778-5f035fb6fac7  /var/log/audit  xfs  defaults,nofail,nodev,nosuid                   0    0
UUID=3506a1fd-a1b9-4748-8c07-a978eca2bc7f  /home           xfs  defaults,nofail,nodev,nosuid                   0    0

When an AMI is created from myroot EBS device, we run into problems with launching an EC2 instance. EC2 status checks for reachability are not satisfied and fail after about 5 minutes, and viewing system boot logs in the console has brought us to a GRUB loader screen, but it doesn't seem to find the root RHEL8 OS on myroot device.

I feel I am close to getting the instance to launch, but I'm missing something in either the /boot or /boot/efi configuration points.

Has anyone successfully manipulated these directories and grub loader configurations and launched an EC2 instance?

  • Systemgeek
    a year ago

    I know its been a while since this was posted. I wish I had seen this a year ago. I have successfully managed to use packer to create a RockyLinux8 AMI that boots and we use in our production ENV. My Rocky 8 AMI is not EFI but does use LVM. It was because of LVM that I went down this path to begin with. I can easily strip out our corporate configs and have been considering posting the rest on GitHub.

    I am now working on a RockyLinux 9 AMI and this one uses EFI and LVM.

    Let me know if you are still looking for answers.

    Oh and DO NOT use rsync when you can build out the second EBS very easily.

  • adam gantt
    a year ago

    Hi there, this project went on hold due to our inability to solve the challenge, but it is still outstanding work for us to resolve. I'd be happy to coordinate a GitHub repo share for anything you may feel comfortable sharing. You can find me on GitHub with the username 'gantta'.

Topics
ComputeStorage
Tags
Amazon EC2LinuxLinux ProvisioningAmazon Elastic Block Store
Language
English
asked 3 years ago1.6K views
1 Answer
1

I've done exactly this, you're basically doing the same as I did (slice up a second disk into all your mountpoints and rsync the first disk across to the second).

You need to specifically pay attention to the character devices in /dev, making sure the ownership, perms, major & minor numbers are carried across. And do an install -Ddm 000755 on the bind devices on the new disks. Also remember to re-make your GRUB config at the end.

I also found it useful to disable FIPS mode and SELinux just while I got it working, these can be re-enabled later.

I found this very helpful indeed https://github.com/plus3it/amigen8 and looking at how it was done in here got me over a lot of hurdles.

EXPERT
answered 3 years ago
  • adam gantt
    3 years ago

    @RWC Appreciate the pointers, and thanks for pointing me to the amigen8 repo. Looks like I have some homework ahead of me tomorrow.

  • Steve_M EXPERT
    3 years ago

    You're welcome. Much as I would like to be able to post my code here (or put it on a public repo in github) I believe my employer will consider it to be their IP, which is why I have to stick to vague, general advice. The amigen8 repo is great though - clone it and run it yourself and you'll find it works. Then pick through the code and see what it's doing that you've missed.

    Another couple of things - don't worry about a separate boot device, just fold it into the root filesystem. It will still work, and grub2-install & grub2-mkconfig will take care of pointing things at the right place. Also your new root disk can be any size you like, doesn't need to match the original.

  • adam gantt
    3 years ago

    I understand being under the corporate IP; same here so I can't publish what I have thus far. While we were able to glean a few nuggets from the amigen8 repo, they built theirs to a CentOS system using LVM volumes. We are stuck using RHEL 8 system without the ability use LVM.

    Unfortunately the Grub2 loader configuration and installation process they use does not work for secure UEFI devices, which I've ran into past testing of my solution. So still stuck at trying to get the /boot and /boot/efi mounts configured correctly as the last step in order to rip off an AMI and get an instance.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content