Skip to main content

Switch MOP

 INFO:



Method of Procedure (MOP)

Upgrade Dell PowerSwitch S4148F-ON (VLT pairs) from OS10 10.5.6.x to OS10 10.6.1.3P1

Item

Details

Purpose

Fix the current Critical and High security findings on the CALEA ToR switches by moving to the Dell OS10 release that contains the fixes.

Devices

Dell PowerSwitch S4148F-ON ToR switches, deployed in VLT pairs.

Current version

OS10 10.5.6.x on all switches (confirm on each switch in Step A2).

Upgrade path

10.5.6.x → 10.6.1.3P1 in ONE step. Dell supports this direct upgrade for VLT pairs, so no in-between version is needed.

Target version

OS10 10.6.1.3P1. Do NOT use plain 10.6.1.3: Dell pulled it because of a functional issue.

Expected impact

Each switch reboots once. Because the pair is VLT, traffic keeps flowing through the other switch. Short blips are possible, so book a maintenance window.

Time needed

Prep (no outage): about 45 min per pair. Maintenance window: about 60 to 90 min per pair.

Back-out

Switch back to the old version that is still saved on the switch (Section D). About 15 min per switch.

Prepared by / Date

______________________  /  ____________

Plain-English summary: Every OS10 switch has two "slots" for the operating system: the one it is running now (active) and a spare (standby). I load the new version into the spare slot ahead of time while the switch keeps running normally. During the window I tell each switch to start from the spare slot and reboot it, one switch at a time. The old version stays in the other slot, so going back is quick.

A. Preparation (before the window, no outage)

A1. Download the software

  1. Sign in at dell.com with the account that holds the switch entitlement, then go to My Account → Software Licenses and Subscriptions (this is Dell Digital Locker).

  2. Open the entitlement (order number) for the switches, then go to the Available Downloads tab. Set the page size to 75 per page so the list is easier to scroll.

  3. Download the OS10 10.6.1.3P1 switch installer and its Release Notes. The file name looks like PKGS_OS10-Enterprise-10.6.1.3P1…bookworm-installer-x86_64.bin. Also note its SHA-256 checksum from the download page.

Wrong file: OS10_Virtualization_10.6.1.3P1….zip ("GNS3 Bundle") is the virtual lab switch. It will NOT install on a real S4148F-ON.

  1. Download the ONIE Firmware Updater release notes for the S4100-ON series (see A3).

  2. Put the files on an SCP/SFTP server that the switch management interfaces can reach. Write down the file names exactly.

If the download is not listed: the entitlement or support contract may have expired, or the files may sit under a different order number. Open a Dell support case and ask for access to 10.6.1.3P1.

A2. Confirm every switch is on 10.5.6.x

On each switch, run the command below and write the exact version in the tracking table.

show version

Switch name

VLT role (Primary / Secondary)

Current version (10.5.6.__)

Upgraded to 10.6.1.3P1 (✓ / time)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Stop if: any switch shows a version that is NOT 10.5.6.x. The one-step path in this MOP only applies to 10.5.6.x. Any other version needs extra upgrades first.

Good to know: 10.5.6.x runs on Debian 10 ("buster") and 10.6.x runs on Debian 12 ("bookworm"). It is still a normal OS10 upgrade, but the underlying Linux changes, which is one more reason to test on the lab switch first.

A3. Check the ONIE firmware (one-time check)

For the S4100-ON family (which includes the S4148F-ON), Dell's upgrade order is: ONIE update → ONIE firmware update → SSD update → OS10 upgrade. Read the S4100-ON ONIE Firmware Updater release notes. If they say the switches need a newer firmware updater, do that first in its own step, test it on the lab switch, and note that it adds one extra reboot per switch.

A4. Prove it on the lab switch first

  1. Follow Section B and Section C on the test switch, using the same image file.

  2. Confirm that the switch boots, the config loads, the interfaces come up, OME still sees the switch, and SNMP/syslog still work.

  3. Practice the back-out (Section D) once, so it is familiar and its timing is known.

  4. Note: from 10.6.1.3 onward, show license status shows "Virtual License Applied". This is expected and not an error.

B. Pre-stage the new version (can be done days ahead, no outage)

Do this on BOTH switches in the pair. It loads the new version into the spare slot only. Nothing changes on the running switch until I reboot it in Section C. This is what makes the window short.

  1. Save and back up the config to the SCP server (use a different file name for each switch):

write memory

copy running-configuration scp://<user>:<password>@<server>/<path>/<switchname>-pre-upgrade.txt

  1. Download the image onto the switch, then watch the progress until the transfer shows success:

image download sftp://<user>:<password>@<server>/<path>/<10.6.1.3P1-file>.bin

show image status

If SCP/SFTP will not connect: a common fix is allowing the older ssh-rsa host key on the file server (HostkeyAlgorithms +ssh-rsa). You can also use http:// or a USB stick (usb://).

  1. Make sure the file is not damaged: compare its checksum with the SHA-256 value from Dell's download page. They must match exactly.

system "sha256sum /var/os10-image/<10.6.1.3P1-file>.bin"

  1. Install the image into the spare (standby) slot. This usually takes about 30 minutes. Keep running the status command until it shows "Completed: Success":

image install image://<10.6.1.3P1-file>.bin

show image status

  1. Confirm that the standby slot now holds 10.6.1.3P1 and that the active slot still holds 10.5.6.x:

show boot detail

Do not run "boot system standby" or "reload" yet. Those happen inside the maintenance window.

C. Maintenance window: upgrade the pair one switch at a time

C1. Pre-checks (both switches)

  1. Find out which switch is Primary and which is Secondary, and confirm that the VLT link (VLTi, usually Port-channel 1000) is Up:

show vlt all

  1. Confirm that the backup (heartbeat) link shows Peer Heartbeat Status = Up:

show vlt all backup-link

  1. Save a "before" snapshot so I can compare afterwards. Paste the output into a text file:

show running-configuration | no-more

show interface status | no-more

show port-channel summary

show lldp neighbors

  1. Check that there are no active alarms in OME, and tell the NOC and the CALEA team that work is starting.

C2. Upgrade the SECONDARY switch first

  1. Tell the switch to start from the new version next time, and confirm that Next-Boot points to standby (10.6.1.3P1):

boot system standby

show boot

  1. Reboot the switch. The other switch carries all the traffic while this one restarts (about 5 to 10 minutes):

reload

  1. Log back in and confirm the new version and that VLT is back Up with its peer:

show version

show vlt all

show vlt all backup-link

  1. Compare the interfaces and port-channels against the "before" snapshot. Wait about 5 minutes and confirm that everything is stable.

Rule: while the two switches run different versions, make NO configuration changes on either switch.

C3. Upgrade the PRIMARY switch

  1. Repeat C2 steps 1 to 4 on the Primary switch. During its reboot, the already-upgraded switch carries the traffic.

C4. Final checks (both switches)

  1. Both switches show 10.6.1.3P1, VLT is Up, the heartbeat is Up, and all expected ports and port-channels are Up.

  2. Servers and uplinks can reach their gateways, and the CALEA / lawful-intercept team confirms that their sessions and mirrors work.

  3. OME shows both switches as healthy, with the new version in inventory.

  4. Save the config and take an "after" backup:

write memory

copy running-configuration scp://<user>:<password>@<server>/<path>/<switchname>-post-upgrade.txt

  1. Ask the CSO / security team to re-scan the switches to confirm that the findings are closed.

D. Back-out plan (if something goes wrong)

The old 10.5.6.x version is still saved in the other slot, so going back is the same steps in reverse. Back out only the switch that has the problem.

  1. Point the switch back to the old slot, confirm the change, and reboot:

boot system standby

show boot

reload

  1. Confirm that 10.5.6.x is running and VLT is Up. If the config does not look right, compare it with the "pre-upgrade" backup file and open a Dell support case before making changes.

  2. Stop the upgrade on the second switch and record what happened for Dell support.

Back-out triggers: the switch does not boot, VLT does not come back Up within 15 minutes, important ports or port-channels stay down, or the CALEA team reports that intercept is broken.

E. After the upgrade: turn off risky services

Telnet, the REST API and SNMP v1/v2c are currently enabled. Do this only after BOTH switches are upgraded, and try it on the lab switch first.

configure terminal

no ip telnet server enable

end

write memory

  • Telnet: turn it off. Use SSH only. Dell's April 2026 security update (10.6.0.8) fixed a well-known Telnet flaw (CVE-2026-24061, scored 9.8 out of 10) that attackers use to get full control. Turning Telnet off removes that risk no matter which version is running.

  • REST API and SNMP v1/v2c: before turning these off, check how OME and any monitoring tools talk to the switches. If OME uses SNMP v2c, move it to SNMP v3 first, then remove the old community strings. If nothing uses the REST API, disable it ("no rest api restconf").

  • Limit who can log in: only allow SSH, SNMP and the API from the management/jump-host network, using a control-plane access list.

F. Note on Dell OME

OME is useful here for inventory, health and confirming versions before and after. Dell offers OS10 as an OME-friendly update package (DUP) only for some models (E3200-ON, N3200-ON and S3248T-ON), and the S4148F-ON is not on that list. So this MOP does the upgrade from the switch command line, and I use OME to check the results.

Before production: run every command exactly as written on the lab switch and check it against the 10.6.1.3P1 Release Notes. Command output can look slightly different between releases.

Comments

Popular posts from this blog

Nov 17

 INFO: Only one package is missing, and it's my mistake. The command I gave you to check installed packages (in the short runbook) left out python3-unbound . So I wrongly told you no server had it. In fact every B server has python3-unbound , and it must be the same version as unbound-libs . The good news: that's the only problem. Kernel, vim, libtiff, rhc, LVM and GRUB all passed on every server. Fix: Download one more file from Red Hat (RHSA-2026:70754, RHEL 8 x86_64): python3-unbound-1.16.2-5.14.el8_10.4.x86_64.rpm Copy it to the B master: scp python3-unbound-1.16.2-5.14.el8_10.4.x86_64.rpm root@<rbb-ref-b-ansible>:/root/releases/2026.11/extra-rpms/ Step 2 again: answer y to "Use the existing folder" answer y to "Copy them into the repo?" answer y to the commands and to "Carry on" os-patch already points at rhel-8-10-nov2026 , so nothing changes there. Step 3 again. Every B server except vlim06 should show OK . v...

New Command 3

 INFO: These results are almost identical to the last run, which tells us the servers were still checking against an old copy of the repo list. Two things caused that. The B master is still running the old script. The message reads "dnf reports a conflict. Run: dnf update --assumeno", which is the 1.4 wording. Version 1.4.1 would also print dnf: lines underneath. The servers reused yesterday's repo list. When dnf reads a repo, it keeps a copy of the package list and reuses it for up to 48 hours. So the B servers checked against the list from before you added the 8 AppStream RPMs, and couldn't see them. That's why vlim03 still asks for perl and gcc. I've fixed the second one in the script: version 1.4.2 makes the dry run read the repo fresh every time. I've attached the script, the README (Word and PDF) and the pseudocode, all updated. Next steps: Copy os_patch_wizard.sh v1.4.2 to rbb-ref-b-ansible , replacing the old one. Check it with: grep...

New A Run

 INFO: You're in good shape at 5.8 GB free. The tar file can go straight from where it is. dbv6admin can already read /var/www/html/rbb-rhel8-repo-20261117.tar.gz , so it doesn't need copying to /tmp on B first. Only extra-rpms (160 MB) has to go through /tmp , because dbv6admin can't get into /root . On B, as root 1. Put extra-rpms where dbv6admin can reach it, and take the tar file's fingerprint cp -r /root/releases/2026.11/extra-rpms /tmp/ chown -R dbv6admin:dbv6admin /tmp/extra-rpms cp /root/os_patch_wizard_1.5.2.sh /tmp/ chown dbv6admin:dbv6admin /tmp/os_patch_wizard_1.5.2.sh sha256sum /var/www/html/rbb-rhel8-repo-20261117.tar.gz exit Write down the code sha256sum prints. You'll compare it on A. Skip the two wizard lines if A already has 1.5.2. To check, run grep SCRIPT_VERSION= /root/os_patch_wizard*.sh on A. On B, as dbv6admin 2. Send everything to A's /tmp scp /var/www/html/rbb-rhel8-repo-20261117.tar.gz dbv6admin@<A-master>:/tmp/ scp -r /tmp/...