We have a requirement to produce AMI built to CIS hardened standards from a Red Hat Enterprise Linux 8 OS including custom partitioning scheme with mount points of /home, /var, /var/log, etc. Our requirement dictates the AMI can only have a single root EBS backed device. Multiple EBS devices are not satisfactory in this situation. Also note, we are running the process via Packer in automated pipelines.
Our first approach was attempting to modify a running EC2 RHEL 8 instance's root EBS device using the AWS CLI, but with the introduction of the 6 hour cool-down period from the time an EBS device is created to when it can be modified, we had to abandon this approach.
When spinning up a new EC2 instance, the root partition will chew up all configured disk space. We have also tried shrinking the root volume partition to make room for additional partitions, but abandoned that approach.
The closest we have made it is to use an auxiliary RHEL8 server and snapshot it's root EBS device. Then, using Packer, spin up a new 'source' EC2 instance that starts with the source's root EBS device, as well as a blank EBS device (we'll call myroot).
At the beginning of the Packer build, the source devices look like the following:
[root@ip-172-31-67-219 ec2-user]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 10G 0 disk
├─nvme0n1p1 259:1 0 200M 0 part /boot/efi
├─nvme0n1p2 259:2 0 512M 0 part /boot
└─nvme0n1p3 259:3 0 9.3G 0 part /
nvme1n1 259:4 0 10G 0 disk
Note, source OS device is named /dev/nvme0n1, and the empty myroot device is /dev/nvme1n1. During the build, we reach out and create a new EBS device from the auxiliary server's snapshot, and attach it to the source server. At this point, we have the following configuration:
[root@ip-172-31-67-219 ec2-user]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 10G 0 disk
├─nvme0n1p1 259:1 0 200M 0 part /boot/efi
├─nvme0n1p2 259:2 0 512M 0 part /boot
└─nvme0n1p3 259:3 0 9.3G 0 part /
nvme1n1 259:4 0 10G 0 disk
nvme2n1 259:5 0 10G 0 disk
├─nvme2n1p1 259:6 0 200M 0 part
├─nvme2n1p2 259:7 0 512M 0 part
└─nvme2n1p3 259:8 0 9.3G 0 part
Note the new /dev/nvme2n1 device as the auxiliary server's non-running OS drive.
Our intent is to now use the aux server's OS files to make a running OS out of our blank myroot device.
We can do this by partitioning and copying OS files to new temporary mount points on myroot device. The result of the partitioning and file movement process shows this configuration:
[root@ip-172-31-67-219 ec2-user]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 10G 0 disk
├─nvme0n1p1 259:1 0 200M 0 part
├─nvme0n1p2 259:2 0 512M 0 part /boot
└─nvme0n1p3 259:3 0 9.3G 0 part /
nvme1n1 259:4 0 10G 0 disk
├─nvme1n1p1 259:9 0 191M 0 part /boot/efi
├─nvme1n1p2 259:10 0 488M 0 part /newboot
├─nvme1n1p3 259:11 0 4.3G 0 part /mnt/myroot/root
├─nvme1n1p4 259:12 0 1G 0 part /mnt/myroot/var
├─nvme1n1p5 259:13 0 1G 0 part /mnt/myroot/varlog
├─nvme1n1p6 259:14 0 1G 0 part /mnt/myroot/varlogaudit
└─nvme1n1p7 259:15 0 2G 0 part /mnt/myroot/home
Note: we remove the temporary /dev/nvme2n1 disk prior to creating and modifying the /boot and /boot/efi partitions. We have also modified myroot's /etc/fstab file with the appropriate partition identifiers in preparation for creating a new AMI off myroot device.
[root@ip-172-31-67-219 ec2-user]# cat /mnt/myroot/root/etc/fstab
UUID=DCE5-01EE /boot/efi vfat defaults,uid=0,gid=0,umask=077,shortname=winnt 0 2
UUID=2fe05fef-dab6-4ced-be2d-f7a9bd613269 /boot xfs defaults 0 0
UUID=2ca0c201-48ae-482b-850c-d26dc26b9de0 / xfs defaults 0 0
UUID=3830ef77-5f8d-4a73-bec8-c7f7d4085bba /var xfs defaults,nofail,nodev 0 0
UUID=85b31b3b-f420-4adb-a221-204d79718626 /var/log xfs defaults,nofail,nodev,nosuid 0 0
UUID=26531d55-fa36-47f2-9778-5f035fb6fac7 /var/log/audit xfs defaults,nofail,nodev,nosuid 0 0
UUID=3506a1fd-a1b9-4748-8c07-a978eca2bc7f /home xfs defaults,nofail,nodev,nosuid 0 0
When an AMI is created from myroot EBS device, we run into problems with launching an EC2 instance. EC2 status checks for reachability are not satisfied and fail after about 5 minutes, and viewing system boot logs in the console has brought us to a GRUB loader screen, but it doesn't seem to find the root RHEL8 OS on myroot device.
I feel I am close to getting the instance to launch, but I'm missing something in either the /boot or /boot/efi configuration points.
Has anyone successfully manipulated these directories and grub loader configurations and launched an EC2 instance?
asked 9 months ago
AWS OFFICIALUpdated 7 months ago
@RWC Appreciate the pointers, and thanks for pointing me to the amigen8 repo. Looks like I have some homework ahead of me tomorrow.
You're welcome. Much as I would like to be able to post my code here (or put it on a public repo in github) I believe my employer will consider it to be their IP, which is why I have to stick to vague, general advice. The amigen8 repo is great though - clone it and run it yourself and you'll find it works. Then pick through the code and see what it's doing that you've missed.
Another couple of things - don't worry about a separate boot device, just fold it into the root filesystem. It will still work, and grub2-install & grub2-mkconfig will take care of pointing things at the right place. Also your new root disk can be any size you like, doesn't need to match the original.
I understand being under the corporate IP; same here so I can't publish what I have thus far. While we were able to glean a few nuggets from the amigen8 repo, they built theirs to a CentOS system using LVM volumes. We are stuck using RHEL 8 system without the ability use LVM.
Unfortunately the Grub2 loader configuration and installation process they use does not work for secure UEFI devices, which I've ran into past testing of my solution. So still stuck at trying to get the /boot and /boot/efi mounts configured correctly as the last step in order to rip off an AMI and get an instance.