INFO:
Status Update
- I completed the follow-up patching round on the A side. All reachable A servers, including the Ansible master, are now on kernel 4.18.0-553.185.1 with the updated packages from the November Nessus findings. Every server was rebooted and the login banner shows the new kernel.
- Before patching, the pre-checks found 6 A servers missing two GRUB boot loader packages. The boot files themselves were intact, so I installed the matching packages from our repo and all servers then passed.
- On vadmin I switched off the unused databridgev6 repo because it points to an address that does not answer (10.212.176.62). This needs a follow-up with the team.
- The post-patch checks and the MOTD update passed on both sides.
- With the patching script, a complete patch run for one side now takes about 20 minutes.
- The Nessus scans are running now and I will check the results in the morning.
Next Steps
- Review the Nessus results in the morning and compare them with the November findings.
- Clean up the old repo folders and artefact files on both Ansible masters to free disk space.
- Confirm with the team whether the 10.212.176.62 address on vadmin is expected.
- The pyOpenSSL finding separately.
Clean-up for tomorrow
- On A:
/var/www/html/rhel-8-10-nov2026-prev(the old repo)/var/www/html/baseos_redhat8-20261108T220631.tar.gz(the old artefact)/var/www/html/rbb-rhel8-repo-20261117.tar.gz(the new artefact, which is no longer needed now that the repo is built)/var/www/html/rhel-8-10-oct2026, after checking it withdu -sh
- On B:
/var/www/html/rbb-rhel8-repo-20261117.tar.gz, if you haven’t deleted it yet.
=========================================================
Neither problem is as bad as it looked.
Problem 1: GRUB. The boot files are fine; only the packages are missing
All 10 servers boot the older BIOS way and use the disk sda. On the 6 flagged servers, the GRUB package version (2.02-170.el8_10.3) is the same as on the healthy servers, so nothing is mismatched.
The important line is i386-pc files: 284 on every server. Those are the files GRUB actually loads at boot, and they're all present. So these servers boot fine today. What's missing is only the two packages that own and update those files. Without them, a future GRUB update could leave the boot files out of date, which is the grub> prompt risk.
The fix: install the two missing packages from A's repo, at exactly the version already on the servers. A's repo has them. Installing them doesn't touch the boot files or the disk, so you don't need grub2-install this time. Ask your senior if they want it run anyway.
Problem 2: vadmin's repo points to the wrong address
Address for databridgev6 |
Works? | |
|---|---|---|
| A master | http://10.212.176.61/... (A master itself) |
✅ The folder databridgev6_yum_repo-1.1.0 exists |
| vadmin | http://10.212.176.62/... |
❌ Nothing answers there |
vadmin is asking a different server, .62, for that repo. That's why the master's own check passed and vadmin failed. vadmin doesn't need this repo for OS patching, so switch it off on vadmin, the same as on B. Mention the .62 address to your senior, because someone may have set it on purpose.
Fix commands (as dbv6admin, from /etc/ansible/tricycle_ansible_master)
1. Switch off databridgev6 on vadmin only
ansible rbb-ref-a-vadmin -i inventories/ref_rbb/hosts_ref_rbb.yml --ask-vault-pass -u dbv6admin -b -m shell -a 'for f in $(grep -l "^\[databridgev6\]" /etc/yum.repos.d/*.repo); do sed -i "/^\[databridgev6\]/,/^\[/ s/^enabled.*/enabled = 0/" $f; done; grep -h -A8 "^\[databridgev6\]" /etc/yum.repos.d/*.repo | grep enabled'
It should print enabled = 0.
2. Install the GRUB packages on vlim02 first, as a test
ansible rbb-ref-a-vlim02 -i inventories/ref_rbb/hosts_ref_rbb.yml --ask-vault-pass -u dbv6admin -b -m shell -a 'dnf -y --repofrompath=wizpatch,http://10.212.176.61/os-patch --setopt=wizpatch.gpgcheck=0 --disablerepo=databridgev6 install grub2-pc-2.02-170.el8_10.3 grub2-pc-modules-2.02-170.el8_10.3 && rpm -q grub2-pc grub2-pc-modules'
The last lines should list both packages at 2.02-170.el8_10.3.
3. If step 2 worked, run the same install on the other 5
ansible 'rbb-ref-a-vadmin:rbb-ref-a-vlip02:rbb-ref-a-vlip03:rbb-ref-a-vlip06:rbb-ref-a-vstat' -i inventories/ref_rbb/hosts_ref_rbb.yml --ask-vault-pass -u dbv6admin -b -m shell -a 'dnf -y --repofrompath=wizpatch,http://10.212.176.61/os-patch --setopt=wizpatch.gpgcheck=0 --disablerepo=databridgev6 install grub2-pc-2.02-170.el8_10.3 grub2-pc-modules-2.02-170.el8_10.3 && rpm -q grub2-pc grub2-pc-modules'
4. Rerun Step 3 in the wizard
All 10 servers should now show OK. vadmin should show a real number of packages to upgrade, about 11, instead of 0. Then you're clear for Step 4: vlim02 first, then the rest, then the master last, then reboot the master.
For the next wizard version, I'll make Step 3 check each server's own repo settings, not just the master's, so the vadmin case is caught and offered automatically.
=========================================================
Let me check one thing in the wizard code first, then I'll give you the diagnosis.
Two separate problems showed up. Neither has changed anything yet, because Step 3 only reads.
Problem 1: GRUB packages missing on 6 servers (vadmin, vlim02, vlip02, vlip03, vlip06, vstat)
GRUB is the boot loader, the program that starts the server before Linux loads. On older-style "BIOS" servers it needs the grub2-pc and grub2-pc-modules packages. Without them, a server can stop at a grub> prompt after a reboot instead of starting. That's the failure this check was added for. The A master and vlim03, vlim04 and vlim05 have the packages. On B, every server had them.
Before fixing anything, confirm it's real. This check only reads:
cd /etc/ansible/tricycle_ansible_master
ansible 'rbb-ref-a-*:!rbb-ref-a-vlim06' -i inventories/ref_rbb/hosts_ref_rbb.yml --ask-vault-pass -u dbv6admin -b -o -m shell -a '[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS; rpm -qa "grub2*" | sort | tr "\n" " "; echo; echo i386-pc files: $(ls /boot/grub2/i386-pc 2>/dev/null | wc -l); lsblk -dno NAME,TYPE | grep disk'
For each server, this shows:
- whether it boots BIOS or UEFI
- which grub2 packages it has
- whether the GRUB module files are still in
/boot - its disk name, which the fix will need
Also check that A's repo has the packages to install:
ls /var/www/html/rhel-8-10-nov2026 | grep -E '^grub2-(pc|common|tools)'
Problem 2: vadmin can't reach the databridgev6 repo
vadmin also shows "Packages to upgrade: 0". That's because dnf stopped at the broken repo before it could count anything, not because vadmin is up to date.
The message says "rerun Step 3", but that won't help here. Step 3 only looks at the master's own repo list, and on the A master databridgev6 is fine. The broken setting exists only on vadmin. That's a gap in the wizard, and I'll fix it in the next version. For now, look at it with this read-only check:
ansible rbb-ref-a-vadmin -i inventories/ref_rbb/hosts_ref_rbb.yml --ask-vault-pass -u dbv6admin -b -m shell -a 'grep -h -A8 databridgev6 /etc/yum.repos.d/*.repo'
grep -h -A8 databridgev6 /etc/yum.repos.d/*.repo
ls -d /var/www/html/databridgev6*
These show:
- the address vadmin uses for that repo
- the master's own setting for it
- which databridgev6 folders the master actually has
What you can do now
vlim03, vlim04, vlim05 and the A master passed, so Step 4 would patch only those. My advice is to wait. Send me the output of the checks above, and I'll give you the exact fix commands. The GRUB fix writes to the boot disk, so it's worth getting right and running past your senior. After the fixes, a rerun of Step 3 should pass all 10 servers, and you can patch them in one go.
Comments
Post a Comment