Skip to main content

Nov 17

 INFO:

Only one package is missing, and it's my mistake. The command I gave you to check installed packages (in the short runbook) left out python3-unbound. So I wrongly told you no server had it. In fact every B server has python3-unbound, and it must be the same version as unbound-libs.

The good news: that's the only problem. Kernel, vim, libtiff, rhc, LVM and GRUB all passed on every server.

Fix:

  1. Download one more file from Red Hat (RHSA-2026:70754, RHEL 8 x86_64):

    python3-unbound-1.16.2-5.14.el8_10.4.x86_64.rpm
    
  2. Copy it to the B master:

    scp python3-unbound-1.16.2-5.14.el8_10.4.x86_64.rpm root@<rbb-ref-b-ansible>:/root/releases/2026.11/extra-rpms/
    
  3. Step 2 again:

    • answer y to "Use the existing folder"
    • answer y to "Copy them into the repo?"
    • answer y to the commands and to "Carry on"

    os-patch already points at rhel-8-10-nov2026, so nothing changes there.

  4. Step 3 again. Every B server except vlim06 should show OK.

vlim03 has an older python3-unbound (5.el8 instead of 5.9.el8_10), which is fine. The same new file updates it too.

I'll also fix the check command in both runbooks so python3-unbound is included next time:

The short runbook's check command now includes python3-unbound. The full runbook already had it, so only the short one needed fixing.

Once you've added the file and re-run Steps 2 and 3, send me the Step 3 summary if any server still shows a problem.

=======================================

Status Update

  • Started the follow-up patch round to fix the remaining Critical findings from the November post-patching Nessus scan, beginning with the B side.
  • Findings being fixed this round: kernel (RHSA-2026:98099), unbound (RHSA-2026:70754), vim (RHSA-2026:98119), libtiff (RHSA-2026:69095), rhc (RHSA-2026:89741) and compat-openssl10 (RHSA-2026:80510).
  • Downloaded the fixed RPMs from Red Hat (kernel 4.18.0-553.185.1.el8_10 and the matching packages), checked which ones each B server has installed, and built a new November repo (rhel-8-10-nov2026) on the B Ansible Master. It's based on the repo that patched the B servers cleanly last week, plus the new packages. The servers now install from it.
  • Freed about 3.6 GB on the B Ansible Master by removing the original November download files, which are no longer needed. The disk was at 93%.
  • The pre-patch server check found one gap. Every B server has python3-unbound installed, and it needs the same new version as unbound-libs. I'm adding that one package before patching. All other checks passed, including kernel, vim, LVM and GRUB.
  • The databridgev6 repo on the B servers points at version 1.1.0, which isn't on the B Ansible Master yet. I've switched it off for this round so it doesn't block patching.
  • The OS patching wizard I'm building in my training course keeps getting a little smarter with each round. This time it learned to step around repos that aren't there, and to remember which servers are already done. Every round it finds a new way to surprise me, and every round I teach it not to.

Next Steps

  • Add python3-unbound to the repo, then re-run the server check on all B servers.
  • Take VMware snapshots, then patch one B server first, check it, then patch the rest and the B Ansible Master last. Every server reboots for the new kernel.
  • Run the post-patch checks and confirm every server is on kernel 4.18.0-553.185.1.
  • Rescan the B servers with Nessus and confirm the six Critical findings are cleared.
  • Copy databridgev6_yum_repo-1.1.0 from the A Ansible Master to the B Ansible Master, then switch the databridgev6 repo back on.
  • Repeat the round on the A side using the same packages.
  • pyOpenSSL is still open and needs a separate fix. Agree the approach with the senior engineer, because Ansible depends on Python 3.6.
  • vlim06 is still unreachable on both sides and needs checking.



==========================================================

Yes, that works, and it's a reasonable workaround for this round. The OS patch doesn't need the databridgev6 repo. Turn it off on all B servers, not only the master: every B server has the same setting, so the Step 3 dry run and the patch run would fail on it as well.

Turn it off (from /etc/ansible/tricycle_ansible_master):

ansible 'rbb-ref-b-*' -i inventories/ref_rbb/hosts_ref_rbb.yml --ask-vault-pass -u dbv6admin -b \
  -m ini_file -a "path=/etc/yum.repos.d/databridgev6.repo section=databridgev6 option=enabled value=0"

Check it:

ansible 'rbb-ref-b-*' -i inventories/ref_rbb/hosts_ref_rbb.yml --ask-vault-pass -u dbv6admin -b -o \
  -m shell -a "grep enabled /etc/yum.repos.d/databridgev6.repo"

Every server should show enabled = 0. vlim06 will be unreachable, as usual.

Then run Step 2 again. Answer y to "Use the existing folder and skip extracting?" and y to the rest. yum makecache should now pass.

Turn it back on later, once databridgev6_yum_repo-1.1.0 exists on the B master. It's the same command with value=1:

ansible 'rbb-ref-b-*' -i inventories/ref_rbb/hosts_ref_rbb.yml --ask-vault-pass -u dbv6admin -b \
  -m ini_file -a "path=/etc/yum.repos.d/databridgev6.repo section=databridgev6 option=enabled value=1"

Two things to know:

  • Running site.yml switches it back on. The MOTD step (Step 13) runs site.yml, and that's what pointed it at 1.1.0 and switched it on in the first place. If Step 13 fails with the same 404, run the "turn it off" command again, or leave Step 13 until the 1.1.0 repo is in place.
  • Note it in your Jira next steps: the B master needs databridgev6_yum_repo-1.1.0 (copy it from the A master), and the repo needs switching back on afterwards. Until then, nothing can be installed from that repo on the B servers.




===============================

Your setup is ready: the 1.5.1 script is in place, there's 5.1 GB free (Step 2 needs about 2 GB), and the release folder has the tar file plus extra-rpms. Every setting shows "not set yet", so the wizard is starting fresh. That's fine, because the questions below fill them all in.

1. Answer the settings. You're at question 9 now. Type rbb-ref-b-* and press Enter. Then press A to answer the rest in order:

# Setting Answer
1 Release version 2026.11
2 OS repo artefact (auto) rbb-rhel8-repo-20261117.tar.gz
3 Repo date (auto) nov2026
4 Mech-ID dbv6admin
5 Repo group wheel
6 Ansible directory Enter (/etc/ansible/tricycle_ansible_master)
7 Environment pick ref_rbb
8 Environment type (auto) Ref/Lab
9 Servers to patch rbb-ref-b-*
10 Patch repo file pick rhel_patches.repo
11 Expected kernel 4.18.0-553.185.1.el8_10.x86_64
12 Patch repo URL http://10.212.176.62/os-patch

Check that the settings screen shows the repo folder /var/www/html/rhel-8-10-nov2026, then press B.

2. Step 1. It lists the release folder. Nothing to answer.

3. Step 2. It moves the tar file to /var/www/html and unpacks it into rhel-8-10-nov2026.

  • Answer y to "Copy them into the repo?" (the 16 extra RPMs).
  • Answer y to each command.
  • Repo report: the newest kernel should show 553.185.1. Kernel, vim and unbound must not be listed as incomplete. The usual old leftovers (glusterfs, samba, sssd and so on) are fine, so answer y to "Carry on".
  • Answer y to "Point os-patch at rhel-8-10-nov2026". Write down the roll-back command it prints.

4. Step 3.

  1. Confirm the visual checks.
  2. Answer y to the server check, and press Enter to take rbb-ref-b-*.
  3. Every B server should show OK. vlim06 will show NO ANSWER and be left out.

5. Take VMware snapshots of all the B servers, including rbb-ref-b-ansible.

6. Step 4, three times. Press Enter each time:

  1. Test server first, for example vlim02. Then run Step 11, and check its kernel: ssh rbb-ref-b-vlim02 uname -r should show 553.185.1.
  2. The rest.
  3. This Ansible Master. It reboots. Log back in, start the wizard, and answer y to mark the run finished.

7. Steps 5 to 13. Step 7 checks for kernel 553.185.1 on every server.

8. Rescan the B servers with Nessus. These should be gone: kernel, unbound, vim, libtiff, rhc and compat-openssl10. pyOpenSSL is a separate fix, so it will still show.

If anything stops or looks wrong, send me a screenshot of that step.



==================================================================

Leave the extra-rpms folder as it is. Don't tar it. Step 2 looks for loose .rpm files in /root/releases/2026.11/extra-rpms/ and copies them into the repo. Your release folder is now set up exactly right: the 1.7 GB tar file plus the extra-rpms folder next to it.

About the version: the file I sent you as os_patch_wizard_1.5.1.sh does contain 1.5.1. I checked it when I made it. If your copy says 1.5, an older file was probably copied under the new name. Here's the right one again:

I've attached the correct file again. I've confirmed it contains SCRIPT_VERSION="1.5.1".

Copy it to rbb-ref-b-ansible over the one in /root, then check it:

chmod 700 /root/os_patch_wizard_1.5.1.sh
grep SCRIPT_VERSION= /root/os_patch_wizard_1.5.1.sh

It should print readonly SCRIPT_VERSION="1.5.1". Your saved settings and progress are kept in /root/.os_patch_wizard, so replacing the file doesn't lose anything.

Also run df -h / before Step 2. It needs about 2 GB free to unpack the repo.

Then start it:

  1. Press C, then 9, and enter rbb-ref-b-*. Press B to go back.
  2. Step 1: it should pick rbb-rhel8-repo-20261117.tar.gz and show Repo folder: /var/www/html/rhel-8-10-nov2026.
  3. Step 2: answer y to copy the extra RPMs, check the repo report, and answer y to point os-patch at the new repo.


==================================================

Yes, remove that folder. You don't have enough space at the moment:

  • /, /root and /var/www/html are all on the same disk, which has only 3.3 GB free (93% full).
  • This round needs about 3.9 GB: about 1.75 GB for the new tar file, about 1.9 GB when Step 2 unpacks it, and about 0.2 GB for the extra RPMs.
  • Docker uses the same disk, so it must not fill up.

1. See what's taking the space:

du -sh /root/releases/baseos_redhat8-20261108T220631 /var/www/html/* 2>/dev/null | sort -h

2. Free the space. All of these are safe to remove: they're copies of the original November download, which has since been replaced by the repo that works.

rm -rf /root/releases/baseos_redhat8-20261108T220631          # about 1.9 GB, the unpacked original download
rm -f  /var/www/html/baseos_redhat8-20261108T220631.tar.gz    # about 1.75 GB, the original tar file (still has the -29 openssh files)
rmdir  /var/www/html/baseos_redhat8-20261108T220631 2>/dev/null   # empty leftover folder
df -h /

Aim for at least 6 GB free.

Don't delete these yet:

  • rhel-8-10-rhel-8-10-oct2026: it's the repo the servers use now. It's also your source for the new tar file, and your way back if something goes wrong.
  • os-patch: the link the servers install from.

os-patch.old-20261113-0538 is the old February repo. You can delete it if the du output shows it's large. You won't need it again.

3. Pack this month's artefact:

cd /var/www/html/rhel-8-10-rhel-8-10-oct2026
tar -czf /root/releases/2026.11/rbb-rhel8-repo-20261117.tar.gz *.rpm
ls -lh /root/releases/2026.11/

4. Check the wizard version:

grep SCRIPT_VERSION= /root/os_patch_wizard_1.5.1.sh

It should say 1.5.1.

5. Start the wizard:

/root/os_patch_wizard_1.5.1.sh
  1. Press C, then 9, enter rbb-ref-b-*, then press B to go back.
  2. Step 1 is the first step to run. It finds rbb-rhel8-repo-20261117.tar.gz by itself and sets the repo to rhel-8-10-nov2026.
  3. Step 2: answer y to copy the extra RPMs, check the repo report, and answer y to point os-patch at the new repo.
  4. Step 3, then snapshots, then Step 4 pressing Enter three times, then Step 11, then Steps 5 to 13.

Once the rescan is clean, you can delete rhel-8-10-rhel-8-10-oct2026 to get its space back.

======================================================

Still to download (10), at exactly these versions:

PackageFile to download
kernel-corekernel-core-4.18.0-553.185.1.el8_10.x86_64.rpm
kernel-moduleskernel-modules-4.18.0-553.185.1.el8_10.x86_64.rpm
kernel-headerskernel-headers-4.18.0-553.185.1.el8_10.x86_64.rpm
kernel-toolskernel-tools-4.18.0-553.185.1.el8_10.x86_64.rpm
kernel-tools-libskernel-tools-libs-4.18.0-553.185.1.el8_10.x86_64.rpm
python3-perfpython3-perf-4.18.0-553.185.1.el8_10.x86_64.rpm
unbound-libsunbound-libs-1.16.2-5.14.el8_10.4.x86_64.rpm
vim-minimalvim-minimal-8.0.1763-34.el8_10.x86_64.rpm
vim-enhancedvim-enhanced-8.0.1763-34.el8_10.x86_64.rpm
vim-filesystemvim-filesystem-8.0.1763-34.el8_10.noarch.rpm (note: noarch)

Comments

Popular posts from this blog

New Command 3

 INFO: These results are almost identical to the last run, which tells us the servers were still checking against an old copy of the repo list. Two things caused that. The B master is still running the old script. The message reads "dnf reports a conflict. Run: dnf update --assumeno", which is the 1.4 wording. Version 1.4.1 would also print dnf: lines underneath. The servers reused yesterday's repo list. When dnf reads a repo, it keeps a copy of the package list and reuses it for up to 48 hours. So the B servers checked against the list from before you added the 8 AppStream RPMs, and couldn't see them. That's why vlim03 still asks for perl and gcc. I've fixed the second one in the script: version 1.4.2 makes the dry run read the repo fresh every time. I've attached the script, the README (Word and PDF) and the pseudocode, all updated. Next steps: Copy os_patch_wizard.sh v1.4.2 to rbb-ref-b-ansible , replacing the old one. Check it with: grep...

New A Run

 INFO: You're in good shape at 5.8 GB free. The tar file can go straight from where it is. dbv6admin can already read /var/www/html/rbb-rhel8-repo-20261117.tar.gz , so it doesn't need copying to /tmp on B first. Only extra-rpms (160 MB) has to go through /tmp , because dbv6admin can't get into /root . On B, as root 1. Put extra-rpms where dbv6admin can reach it, and take the tar file's fingerprint cp -r /root/releases/2026.11/extra-rpms /tmp/ chown -R dbv6admin:dbv6admin /tmp/extra-rpms cp /root/os_patch_wizard_1.5.2.sh /tmp/ chown dbv6admin:dbv6admin /tmp/os_patch_wizard_1.5.2.sh sha256sum /var/www/html/rbb-rhel8-repo-20261117.tar.gz exit Write down the code sha256sum prints. You'll compare it on A. Skip the two wizard lines if A already has 1.5.2. To check, run grep SCRIPT_VERSION= /root/os_patch_wizard*.sh on A. On B, as dbv6admin 2. Send everything to A's /tmp scp /var/www/html/rbb-rhel8-repo-20261117.tar.gz dbv6admin@<A-master>:/tmp/ scp -r /tmp/...