LVM Update

 Info:

These results explain the vlim02 boot hang, and they show 7 more servers would hang the same way after patching.

What it shows:

• 7 servers have no lvm2: vstat, vlip02, vadmin, vlip03, vlim04, vlim05 and vlip06. The original bad patch run removed it. They still boot today only because their current boot image was built before lvm2 was removed. The next kernel update would build a boot image without LVM, and that's exactly what happened to vlim02.
• 4 servers are OK: ansible, vlim02, vlim03 and vlim06 all have lvm2.
• The 22 doesn't settle it. That count only checks the kernel each server is running now, not the one it will boot next.

1. Install lvm2 on the 7 servers. os-base was switched off at the end of the last run, so this command turns it on just for this install:

cd /etc/ansible/tricycle_ansible_master
ansible 'rbb-ref-a-vstat,rbb-ref-a-vlip02,rbb-ref-a-vadmin,rbb-ref-a-vlip03,rbb-ref-a-vlim04,rbb-ref-a-vlim05,rbb-ref-a-vlip06' \
  -i inventories/ref_rbb/hosts_ref_rbb.yml --ask-vault-pass -u dbv6admin -b \
  -m dnf -a "name=lvm2 state=present enablerepo=os-base"

Expect CHANGED on all 7.

2. Rebuild the boot images on those 7. This is safe, and it covers any server that already has a new kernel waiting from the bad run. It takes about a minute per server.

ansible 'rbb-ref-a-vstat,rbb-ref-a-vlip02,rbb-ref-a-vadmin,rbb-ref-a-vlip03,rbb-ref-a-vlim04,rbb-ref-a-vlim05,rbb-ref-a-vlip06' \
  -i inventories/ref_rbb/hosts_ref_rbb.yml --ask-vault-pass -u dbv6admin -b \
  -m command -a "dracut -f --regenerate-all"

3. Check the kernel each server will boot next, on all servers:

ansible 'all:!rbb-ref-b-vadmin' -i inventories/ref_rbb/hosts_ref_rbb.yml --ask-vault-pass -u dbv6admin -b -m shell \
  -a "rpm -q lvm2; k=\$(grubby --default-kernel | sed 's|.*vmlinuz-||'); echo \$k; lsinitrd /boot/initramfs-\$k.img | grep -c lvm; ls /etc/lvm/*.rpmsave 2>/dev/null"

Every server should show lvm2 installed and a count above 0. If any server lists an lvm.conf.rpmsave, send it to me before going on.

4. Take snapshots of the servers in vCenter.

5. Patch: wizard Step 4, typing all:!rbb-ref-b-vadmin, then Steps 5 to 13.