Optimizing Memory after CentOS Upgrades in Lightsail

When we upgraded AWS Lightsail instances from CentOS 7 to AlmaLinux 8, free RAM was around 200MB lower than what we got doing similar upgrades in other platforms. Those instances were created with the Lightsail CentOS 7 template (now discontinued) and upgraded using the AlmaLinux ELevate procedure. For server instances with 1 GB RAM, 200MB less memory (20%) could potentially impact performance and applications due to increased swap usage.

This was the typical memory available after upgrading those Lightsail instances:

# free
               total        used        free      shared  buff/cache 
available
Mem:         760308      430864       58052         268      271392 
215872
Swap:       1048572       48128     1000444

We contacted AWS support, but they don't provide assistance with issues on operating systems upgrades other than Amazon Linux. The solution turned out to be relatively simple. It is posted here in case it's useful to other users.

We first looked at kernel memory reservations reported in 'dmesg'

# dmesg | grep -i "reserving"
[    0.000000] ACPI: Reserving FACP table memory at [mem 0x3e3eff80-0x3e3efff3]
[    0.000000] ACPI: Reserving DSDT table memory at [mem 0x3e3ee3a0-0x3e3ef488]
[    0.000000] ACPI: Reserving FACS table memory at [mem 0x3e3eff40-0x3e3eff7f]
[    0.000000] ACPI: Reserving SSDT table memory at [mem 0x3e3ef6c0-0x3e3eff39]
[    0.000000] ACPI: Reserving APIC table memory at [mem 0x3e3ef5d0-0x3e3ef645]
[    0.000000] ACPI: Reserving SRAT table memory at [mem 0x3e3ef530-0x3e3ef5cf]
[    0.000000] ACPI: Reserving SLIT table memory at [mem 0x3e3ef4c0-0x3e3ef52b]
[    0.000000] ACPI: Reserving WAET table memory at [mem 0x3e3ef490-0x3e3ef4b7]
[    0.000000] ACPI: Reserving HPET table memory at [mem 0xc9000-0xc9037]
[    0.000000] ACPI: Reserving SSDT table memory at [mem 0xc9040-0xc90ba]
[    0.000000] Reserving 192MB of memory at 560MB for crashkernel (System RAM: 995MB)

The last line is the culprit. It states 192MB were reserved for crashkernel, which shouldn't be necessary in small instances operating normally. crashkernel is usually configured in grub boot args.

#  grubby --default-kernel
/boot/vmlinuz-4.18.0-553.8.1.el8_10.x86_64

# grubby --info /boot/vmlinuz-4.18.0-553.8.1.el8_10.x86_64
index=0
kernel="/boot/vmlinuz-4.18.0-553.8.1.el8_10.x86_64"
args="ro console=tty0 crashkernel=auto net.ifnames=0 console=ttyS0 
$tuned_params"
root="UUID=9cff3d69-3769-4ad9-8460-9c54050583f9"
initrd="/boot/initramfs-4.18.0-553.8.1.el8_10.x86_64.img $tuned_initrd"
title="AlmaLinux (4.18.0-553.8.1.el8_10.x86_64) 8.10 (Cerulean Leopard)"
id="3fbdb8b24a17e67dc83d346877013e3e-4.18.0-553.8.1.el8_10.x86_64"

That crashkernel=auto probably came from the Lightsail CentOS 7 template. It doesn't show up in upgrades we have done in other platforms. That option can be removed from boot args using grubby.

# grubby --remove-args="crashkernel=auto" --update-kernel=ALL

# grubby --info /boot/vmlinuz-4.18.0-553.8.1.el8_10.x86_64
index=0
kernel="/boot/vmlinuz-4.18.0-553.8.1.el8_10.x86_64"
args="ro console=tty0 net.ifnames=0 console=ttyS0 $tuned_params"
root="UUID=9cff3d69-3769-4ad9-8460-9c54050583f9"
initrd="/boot/initramfs-4.18.0-553.8.1.el8_10.x86_64.img $tuned_initrd"
title="AlmaLinux (4.18.0-553.8.1.el8_10.x86_64) 8.10 (Cerulean Leopard)"
id="3fbdb8b24a17e67dc83d346877013e3e-4.18.0-553.8.1.el8_10.x86_64"

After that change and a reboot, the instance reports an additional 192MB of available memory.

# free
               total        used        free      shared  buff/cache 
available
Mem:         956920      397252      108940         364      450728 
419648
Swap:       1048572       17664     1030908