LVM Troubleshooting

 Info: 

A bare grub> prompt means the boot loader started but couldn't find its menu file (grub.cfg). So it doesn't know what to boot. The kernel and your data are almost certainly fine. These two servers most likely had GRUB packages damaged by the original bad run, much like lvm2 was. You can start them by hand from this prompt and then fix the menu.

1. Start the server by hand from grub> (vCenter console, one server at a time):

ls

This lists the disks, for example (hd0) (hd0,msdos1) (hd0,msdos2). The first partition is normally /boot, so:

set root=(hd0,msdos1)
ls /

You should see files starting with vmlinuz- and initramfs-. If you don't, try (hd0,msdos2) or (hd0,gpt2).

Then type these lines. Pressing TAB after vmlinuz- and initramfs- fills in the name, so pick the newest 553.184.1 version:

linux /vmlinuz-4.18.0-553.184.1.el8_10.x86_64 root=/dev/mapper/rhel-root ro rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap
initrd /initramfs-4.18.0-553.184.1.el8_10.x86_64.img
boot

If it can't find rhel-root, the server's disk group has a different name. In that case, revert that server to its snapshot instead.

2. Once it's up, fix GRUB:

ssh rbb-ref-a-vlim04
sudo -i
[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
rpm -q grub2-pc grub2-pc-modules grub2-tools
ls -l /boot/grub2/grub.cfg /boot/loader/entries/

• If any grub2 package is "not installed":

  dnf install -y grub2-pc grub2-pc-modules grub2-tools --enablerepo=os-base
  

• If it said BIOS (most likely), rebuild the menu and reinstall the boot code:

  grub2-mkconfig -o /boot/grub2/grub.cfggrub2-install /dev/sda
  

  Use lsblk first to confirm the disk is sda.
• If it said UEFI, stop and send me the output. The files are in different places, and grub2-install must not be run.

3. Test:

grubby --default-kernel
reboot

It should boot on its own to the 553.184.1 kernel. Check with uname -r.

Then do the same on vlim05.

4. Check the rest before any other reboots:

ansible 'all:!rbb-ref-b-vadmin' -i inventories/ref_rbb/hosts_ref_rbb.yml --ask-vault-pass -u dbv6admin -b -m shell \
  -a "rpm -q grub2-pc grub2-tools; ls -l /boot/grub2/grub.cfg"

Every server should show both packages installed and a grub.cfg that isn't 0 bytes.

If the manual boot doesn't work, revert to the snapshot. That's the safe fallback, and we can fix GRUB before re-patching.