From 73129a0564945f40b6a2a38cdd01736573a73037 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 4 Oct 2026 07:45:05 -0700 Subject: [PATCH] ops(esh-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20 (one reboot, Prime on site); ESH cluster uniform on 9 --- .../esh-pve-cluster-pve9-upgrade-plan.md | 23 +++- persistent-memory.md | 6 +- playbooks/esh-pve-pve9-prereboot.yaml | 78 ++++++++++++++ playbooks/esh-pve-pve9-upgrade.yaml | 102 ++++++++++++++++++ servers/esh-pve/README.md | 2 +- 5 files changed, 206 insertions(+), 5 deletions(-) create mode 100644 playbooks/esh-pve-pve9-prereboot.yaml create mode 100644 playbooks/esh-pve-pve9-upgrade.yaml diff --git a/docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md b/docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md index 79770ce..34d8d84 100644 --- a/docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md +++ b/docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md @@ -1,4 +1,4 @@ -# esh-pve-cluster: Proxmox VE 8 → 9 upgrade plan (PLAN ONLY — not executed) +# esh-pve-cluster: Proxmox VE 8 → 9 upgrade plan — DONE (esh-nas-pve 2026-10-03, esh-pve 2026-10-04) Requested by Prime via Miranda, 2026-10-02 (thread `01M3YX5QGQ5V8BQXYM46E5T3PB`). **Do not execute without a further green light from Prime.** Facts below were read live on 2026-10-02 ~1930 PT, @@ -184,3 +184,24 @@ There is **no supported in-place downgrade** from PVE 9 / Debian 13 back to 8 / - pve8to9 --full: 0 FAIL, 3 WARN (2-node cluster, running guests, dkms). - Nightly vzdump 03:30 OK (100, 101, 111); 108 (subnet router) excluded by design; 110 esh-ml1 not backed up and not one-off'd: rebuildable from `stacks/embed-rerank`, and embed/rerank are load-shared with nh3-ml1 since 2026-09-26. + +## esh-pve: UPGRADED to PVE 9.2.21 / kernel 7.0.14-20, 2026-10-04 0733-0745 (Prime on site; ONE reboot, no PVE 8 test, Prime's call) + +- `playbooks/esh-pve-pve9-upgrade.yaml`: guards (pve8to9 0 FAIL; prep in; nvidia built for the newest 6.8; VG 16 G free + + dm-snapshot in initramfs), root snapshot `pve/root-pre-pve9` 15 G, trixie sources (`ceph.list` + `pve-enterprise.list` + moved to `/root/pve9-prep-20261004/`), plan read (833 installs; removals only t64 libs + python3-distutils/future/lib2to3; + `proxmox-headers-7.0` came with it), detached dist-upgrade 0734-0740, EXIT 0. +- `playbooks/esh-pve-pve9-prereboot.yaml`: all 8 OK — **nvidia 580.178.04 DKMS built for 7.0.14-20** ("Building module(s) + … done"; the nh3-pve trap did not recur because `proxmox-default-headers` was in), pins + dm-snapshot in the 7.0 initramfs, + shim + GRUB rewritten in both EFI/proxmox and EFI/BOOT ("Installation finished. No error reported."), every bridge port + exists, enterprise repo `Enabled: false`. pve8to9 0 FAIL. +- Reboot 0740:54. ESH watcher (from nh3-dev): esh-ml1 dark 0741:12, esh-docker-vm 0741:26, esh-pve + esh-scale 0741:36; + **esh-pve + mesh back 0742:39** (the MS-01 boots in about a minute); esh-ml1 + esh-docker-vm back ~0743:10. + **ESH impact: mesh ~1 min; DNS / Home Assistant / Zigbee ~2 min.** +- Post-check PASSED 0745: kernel 7.0.14-20; names held; vmbr0 = enp3s0f0np0 + enp3s0f1np1 and **the networking journal is clean** + (the "bridge port enp87s0 does not exist" error is gone); no failed units; chrony, corosync active; **cluster quorate, both + nodes now 9.2.21**; nvidia module loaded, esh-ml1 sees the RTX 2000E (11.8 GiB used), TEI :8001/:8013 200; gateway + embeddings 6/6 (dim 1024); esh-docker-vm 18/18 Up (same set as before) with **mosquitto and zigbee2mqtt up by themselves** + (z2m "MQTT publish" from 0743:17; restart counts 0), AdGuard resolving, HA :8123 200, Homepage 200; esh-vm-db postgres + + mongod active; esh-matter listening :5580; esh-scale on the mesh. pve8to9 0 FAIL, 2 WARN (2-node, dkms). +- Open: drop `pve/root-pre-pve9` on/after 2026-10-05 (infra-hermes, with pfi-pve's). Never `zpool upgrade` the ESH pools. diff --git a/persistent-memory.md b/persistent-memory.md index a44276d..5913273 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-10-04 ~0720 PT (pfi-pve on PVE 9; albok-service 0.1.6 live after Prime's push go; esh-nas-pve + nh3-pve on PVE 9; QNAP LCD fixed, awaiting Prime's look; NH3 Mac mini asleep, fixed on site today.)_ +_Last updated: 2026-10-04 ~0750 PT (pfi-pve + esh-pve on PVE 9; albok-service 0.1.6 live after Prime's push go; esh-nas-pve + nh3-pve on PVE 9; QNAP LCD fixed, awaiting Prime's look; NH3 Mac mini asleep, fixed on site today.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -131,7 +131,7 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca - **cira-tunnel-watchdog** on pfi-tacticalrmm has been live since 1232 and drops :4433 tunnels silent >180 s. Watch `journalctl -t cira-tunnel-watchdog`. - **RustDesk server (ana-docker) updated 1.1.14 → 1.1.16 at 1424** (same key; ports 21115-21119 reachable via rustdesk.phasefinal.com). Private key vaulted first as `ana-docker/rustdesk/id_ed25519`. "Change ID" is refused by the OSS server BY DESIGN (Pro feature; `rendezvous_server.rs` answers TCP RegisterPk with NOT_SUPPORT). Workaround, read at client source and untested: stop RustDesk, delete `enc_id` from RustDesk.toml, set a plain `id`, restart. → `servers/ana-docker/README.md` - **Booth vs 10.100.10.166 (Prime, ~1420): that client cannot reach the Booth.** The Booth was up and answered from 4 hosts at 1417; .166 is on nh3-dev's own subnet. Prime interrupted my check, so it is OPEN and resumes only on his word. Lead for later: .166 is Prime's Wi-Fi MacBook (`Mac`, 36:c7:30:13:aa:28, nh3-userland). So it is Wi-Fi → wired VM on the same /24; check the SSID's client-isolation / proxy-ARP settings first. -- **esh-nas-pve: DONE, PVE 9.2.21 / kernel 7.0.14-20 since 1604** (Prime on site; ESH DNS never dropped; the /boot-LV-full trap was hit and fixed). Keep `nvme/ROOT@pre-pve9` about a week (until ~2026-10-10), then destroy it. `openipmi.service` fails at boot (no IPMI on the QNAP); harmless, maskable. **`pve` (esh-pve) is still on 8.4.20; separate window** (DNS VM, CT 108 mesh, esh-ml1 NVIDIA → `proxmox-default-headers` + a /boot check first). → `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md` +- **esh-nas-pve: DONE, PVE 9.2.21 / kernel 7.0.14-20 since 1604** (Prime on site; ESH DNS never dropped; the /boot-LV-full trap was hit and fixed). Keep `nvme/ROOT@pre-pve9` about a week (until ~2026-10-10), then destroy it. `openipmi.service` fails at boot (no IPMI on the QNAP); harmless, maskable. **esh-pve: DONE 2026-10-04 0742, PVE 9.2.21 / 7.0.14-20** (Prime on site, one reboot; ESH mesh ~1 min, DNS/HA/Zigbee ~2 min). Prep found + fixed a stale vmbr0 `bridge-ports enp87s0` (dead since the 09-24 GPU renumbered the igc ports). nvidia DKMS built for 7.0. **The ESH cluster is uniform 9.2.21.** Open: `lvremove pve/root-pre-pve9` on/after 10-05 (infra-hermes). → `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md` - **NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, Wi-Fi, 10.100.10.121) asleep since ~1407; MeshCentral + RustDesk can't reach it.** Wake-on-LAN from nh3-dev (same subnet) did NOT wake it (Wi-Fi had disassociated). Prime fixes it on site 2026-10-04: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally plus Ethernet. - **albok-service 0.1.6 LIVE (2026-10-04 0545; review fold: a locked keyword-index build); 0.1.5 at 2039 and 0.1.4 at 1611, each after Prime's "push it".** Each vh/albok push is a per-release go from Prime, not standing authorization. 0.1.5 = search 4.5 s → ~0.25 s. Deploy BETWEEN Nemi walks: `flock -n ~/.local/state/albok-nemi/run.lock true` (the 0.1.4 redeploy cut the 1601 walk; a catch-up walk 1612-1625 was clean). → `stacks/albok-service/README.md` - **QNAP front-panel LCD on esh-nas-pve fixed (1614), AWAITING Prime's visual check.** Upstream DigitalFunk/QnapLCD-Proxmox opened on a TrueNAS-only screen (`cli`) and died on Proxmox; it also hid vmbr0's IP. Our menu (host/IP, PVE version, uptime, IPs, pools) + a restarting unit: `services/qnap-lcd/`, `playbooks/qnap-lcd.yaml`; the originals are kept beside. @@ -142,7 +142,7 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca - the post-deletion Worldtree gate (infra-hermes runs it as a regression check); - the PVE subscription popup patched on nh3-pve, nh3-pve-2, pfi-pve and esh-pve; - nh3-pve PVE 9 + its post-fixes; nh3-dev's VM 102 snapshot deleted. -- **ESH PVE 8→9 plan:** written, NOT executed (needs Prime's green light). `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`. The tank blocker is cleared. +- **ESH PVE 8→9: DONE** (esh-nas-pve 10-03, esh-pve 10-04). `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`. ### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01) diff --git a/playbooks/esh-pve-pve9-prereboot.yaml b/playbooks/esh-pve-pve9-prereboot.yaml new file mode 100644 index 0000000..5033408 --- /dev/null +++ b/playbooks/esh-pve-pve9-prereboot.yaml @@ -0,0 +1,78 @@ +# esh-pve: after the PVE 9 dist-upgrade, BEFORE the reboot. One fix-up step, the rest is read-only. +# scripts/elway infra-ops@10.0.250.35 --playbook playbooks/esh-pve-pve9-prereboot.yaml +# Every verify here is something that, if wrong, makes the reboot an ESH outage only someone at the console can end +# (esh-pve carries esh-scale, ESH's only mesh path, and has no AMT). Template: playbooks/pfi-pve-pve9-prereboot.yaml. +steps: + - name: the upgrade ships pve-enterprise.sources enabled; turn it off (no subscription) + sudo: true + shell: | + f=/etc/apt/sources.list.d/pve-enterprise.sources + if grep -q '^Enabled:' $f; then sed -i 's/^Enabled:.*/Enabled: false/' $f; else echo 'Enabled: false' >> $f; fi + cat $f + when: "test -f /etc/apt/sources.list.d/pve-enterprise.sources && ! grep -q '^Enabled: *false' /etc/apt/sources.list.d/pve-enterprise.sources" + +verify: + - name: the detached dist-upgrade finished with EXIT 0 + sudo: true + shell: tail -1 /root/pve9-upgrade.log | grep -qx 'EXIT 0' + changed_when: "false" + - name: dpkg is clean and nothing is left for apt -f + sudo: true + shell: | + test -z "$(dpkg --audit 2>&1)" + test "$(apt-get -s -f install 2>/dev/null | grep -cE '^(Inst|Conf) ')" -eq 0 + changed_when: "false" + - name: PVE 9 userland + sudo: true + shell: pveversion | grep -E '^pve-manager/9\.' + changed_when: "false" + - name: nvidia DKMS module is built for the new kernel (esh-ml1's GPU; the nh3-pve trap) + sudo: true + shell: | + new=$(ls /boot/vmlinuz-* | sed 's#/boot/vmlinuz-##' | sort -V | tail -1) + /usr/sbin/dkms status | cat + /usr/sbin/dkms status | grep -q "nvidia/.*, $new, x86_64: installed" + changed_when: "false" + - name: newest kernel has an initramfs carrying the NIC pins and dm-snapshot (root has an LVM snapshot) + sudo: true + shell: | + set -e + new=$(ls /boot/vmlinuz-* | sed 's#/boot/vmlinuz-##' | sort -V | tail -1) + echo "newest kernel: $new" + lsinitramfs /boot/initrd.img-$new > /tmp/pve9-initrd.lst + grep -q 10-pin-enp3s0f0np0.link /tmp/pve9-initrd.lst + grep -q 10-pin-enp88s0.link /tmp/pve9-initrd.lst + grep -q 'dm-snapshot.ko' /tmp/pve9-initrd.lst + grep -q "vmlinuz-$new" /boot/grub/grub.cfg + changed_when: "false" + - name: ESP rewritten by the trixie GRUB (shim path); removable copy matches; firmware boots proxmox shim first + sudo: true + shell: | + set -e + ls -la --time-style=long-iso /boot/efi/EFI/proxmox/ /boot/efi/EFI/BOOT/ + test "$(md5sum < /boot/efi/EFI/proxmox/shimx64.efi)" = "$(md5sum < /boot/efi/EFI/BOOT/BOOTx64.efi)" + test "$(md5sum < /boot/efi/EFI/proxmox/grubx64.efi)" = "$(md5sum < /boot/efi/EFI/BOOT/grubx64.efi)" + # The binary carries no greppable version string; the trixie grub-efi-amd64 postinst's own report is the proof. + grep -q 'Installation finished. No error reported.' /root/pve9-upgrade.log + first=$(efibootmgr | sed -n 's/^BootOrder: \([0-9A-F]*\).*/\1/p') + efibootmgr -v | grep -E "^Boot${first}\*? " | grep -qi 'EFI\\proxmox\\shimx64.efi' + changed_when: "false" + - name: networking will come back (static vmbr0, interfaces parse, every bridge port exists, sysctl.conf has nothing to lose) + sudo: true + shell: | + set -e + dpkg-query -W -f='${Status}' ifupdown2 | grep -q 'install ok installed' + for p in $(sed -n -E 's/^\s*bridge-ports\s+//p' /etc/network/interfaces); do test -e /sys/class/net/$p || { echo "missing $p"; exit 1; }; done + # ifupdown2 3.3 (PVE 9) flags PVE's own `bridge-fd 0` as out of range and `-s` exits 1 on that warning alone. + # nh3-pve and esh-nas-pve show the same warning and booted with vmbr0 fine (STP off, forward_delay 0), so + # only a warning OTHER than that one fails this check. + out=$(ifreload -a -s 2>&1) || true + echo "$out" + # (an `if`, not a bare `! pipeline`: set -e ignores a negated pipeline, so that form could never fail) + if echo "$out" | grep -v 'bridge-fd: value of out range "0"' | grep -q .; then echo "unexpected ifupdown2 output"; exit 1; fi + test -z "$(grep -vE '^\s*(#|$)' /etc/sysctl.conf 2>/dev/null)" + changed_when: "false" + - name: pve8to9 --full (post-upgrade checks) + sudo: true + shell: pve8to9 --full 2>&1 | grep -E '^(FAIL|WARN):|TOTAL|PASSED|SKIPPED|WARNINGS|FAILURES' | cat + changed_when: "false" diff --git a/playbooks/esh-pve-pve9-upgrade.yaml b/playbooks/esh-pve-pve9-upgrade.yaml new file mode 100644 index 0000000..d5da18d --- /dev/null +++ b/playbooks/esh-pve-pve9-upgrade.yaml @@ -0,0 +1,102 @@ +# esh-pve (`pve`): PVE 8.4 → 9 upgrade, the package half (Prime go 2026-10-04, on site). Does NOT reboot. +# scripts/elway infra-ops@10.0.250.35 --playbook playbooks/esh-pve-pve9-upgrade.yaml +# Prereq: playbooks/esh-pve-pve9-prep.yaml. NO PVE 8 reboot test here: Prime chose one reboot (2026-10-04, on site), +# so the prep's next-boot changes (bridge-ports fix, NIC pins, kernel, microcode) land together with PVE 9. +# Template: playbooks/pfi-pve-pve9-upgrade.yaml. Plan: docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md. +# The dist-upgrade runs DETACHED (systemd-run → /root/pve9-upgrade.log), so a dropped ssh cannot kill it mid-dpkg. +# Poll: sudo tail /root/pve9-upgrade.log ; systemctl status pve9-dist-upgrade +# Then: playbooks/esh-pve-pve9-prereboot.yaml, then the reboot. +# Rollback: `lvconvert --merge pve/root-pre-pve9` and reboot (the merge runs at the next activation of pve/root); +# from a rescue boot if 9 does not boot. The snapshot covers / and /boot (one LV), not the ESP or guest storage. +vars: + stage_dir: /root/pve9-prep-20261004 + +steps: + - name: GUARD — pve8to9 --full reports no FAIL + sudo: true + shell: | + out=$(pve8to9 --full 2>&1) + echo "$out" | grep -E '^(FAIL|WARN):' | cat + ! echo "$out" | grep -q '^FAIL:' + changed_when: "false" + + - name: GUARD — prep is in (systemd-boot out, NIC pins, nvidia built for the newest 6.8) + sudo: true + shell: | + set -e + # an `if`, not `! pipeline`: set -e ignores a negated pipeline, so that line could never fail + if dpkg-query -W -f='${Status}' systemd-boot 2>/dev/null | grep -q 'install ok installed'; then echo "systemd-boot still installed"; exit 1; fi + test -f /etc/systemd/network/10-pin-enp3s0f0np0.link + dpkg-query -W -f='${Status}' proxmox-default-headers | grep -q 'install ok installed' + /usr/sbin/dkms status | grep -q "nvidia/.*, $(ls /boot/vmlinuz-* | sed 's#/boot/vmlinuz-##' | sort -V | tail -1), x86_64: installed" + changed_when: "false" + + - name: GUARD — the VG has room for the root snapshot and dm-snapshot is in the running kernel's initramfs + sudo: true + shell: | + set -e + free=$(vgs --noheadings --units g --nosuffix -o vg_free pve | tr -d ' ' | cut -d. -f1) + echo "pve VG free: ${free}G" + test "$free" -ge 15 + lsinitramfs /boot/initrd.img-$(uname -r) | grep -q 'dm-snapshot.ko' + changed_when: "false" + + - name: root snapshot for rollback (pve/root-pre-pve9, 15G COW) + sudo: true + shell: lvcreate -s -L 15G -n root-pre-pve9 pve/root + when: "! lvs pve/root-pre-pve9 >/dev/null 2>&1" + + - name: switch apt sources bookworm → trixie (Debian lines rewritten; PVE repo moved to deb822 proxmox.sources) + sudo: true + shell: | + set -e + cp -p /etc/apt/sources.list {{ stage_dir }}/sources.list.bookworm + cat > /etc/apt/sources.list <<'EOF' + deb http://ftp.us.debian.org/debian trixie main contrib non-free-firmware + deb http://ftp.us.debian.org/debian trixie-updates main contrib non-free-firmware + deb http://security.debian.org/debian-security trixie-security main contrib non-free-firmware + EOF + cat > /etc/apt/sources.list.d/proxmox.sources <<'EOF' + Types: deb + URIs: http://download.proxmox.com/debian/pve + Suites: trixie + Components: pve-no-subscription + Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg + EOF + for f in /etc/apt/sources.list.d/pve-enterprise.list /etc/apt/sources.list.d/ceph.list; do + if [ -e "$f" ]; then mv "$f" {{ stage_dir }}/; fi + done + cat /etc/apt/sources.list; ls /etc/apt/sources.list.d/ + when: "grep -q bookworm /etc/apt/sources.list" + + - name: apt update on trixie (must be clean) + sudo: true + shell: | + ! apt-get update -q 2>&1 | grep -E '^(E|Err):' + changed_when: "false" + + - name: GUARD — read the plan; refuse if it would remove anything this host cannot boot or network without + sudo: true + shell: | + apt-get -s dist-upgrade > {{ stage_dir }}/dist-upgrade-plan.txt 2>&1 + grep -cE '^Inst ' {{ stage_dir }}/dist-upgrade-plan.txt | sed 's/^/ installs\/upgrades: /' + grep -E '^Remv ' {{ stage_dir }}/dist-upgrade-plan.txt | sed 's/^/ /' || echo " no removals" + ! grep -E '^Remv (proxmox-ve|pve-manager|grub-efi-amd64|grub-efi-amd64-bin|grub-efi-amd64-signed|shim-signed|ifupdown2|lvm2|openssh-server|qemu-server|pve-container|dkms|proxmox-default-headers) ' {{ stage_dir }}/dist-upgrade-plan.txt + changed_when: "false" + + - name: start the dist-upgrade DETACHED (log /root/pve9-upgrade.log) + sudo: true + shell: | + systemd-run --unit=pve9-dist-upgrade --setenv=DEBIAN_FRONTEND=noninteractive \ + /bin/sh -c 'apt-get dist-upgrade -y -q -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold > /root/pve9-upgrade.log 2>&1; echo "EXIT $?" >> /root/pve9-upgrade.log' + when: "! test -f /root/pve9-upgrade.log" + +verify: + - name: root snapshot exists + sudo: true + shell: lvs pve/root-pre-pve9 + changed_when: "false" + - name: sources are trixie + sudo: true + shell: "! grep -rq bookworm /etc/apt/sources.list /etc/apt/sources.list.d/" + changed_when: "false" diff --git a/servers/esh-pve/README.md b/servers/esh-pve/README.md index f1a8df0..4c7aa97 100644 --- a/servers/esh-pve/README.md +++ b/servers/esh-pve/README.md @@ -13,7 +13,7 @@ Proxmox VE hypervisor for the ESH home lab (`pve.esteban.net`). Non-PFI scope. - **CPU:** 13th Gen Intel Core i9-13900H - **RAM:** 62.5 GB -- **Kernel:** `6.8.12-42-pve`, PVE 8.4.20 (as of 2026-09-24) +- **PVE / kernel:** 9.2.21 / `7.0.14-20-pve` since 2026-10-04 0742 (upgraded from 8.4.20; `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`). Boot: shim → GRUB 2.12 (`\EFI\proxmox\shimx64.efi`, removable copy kept in step). NIC names pinned by MAC (`/etc/systemd/network/10-pin-*.link`); vmbr0 ports = `enp3s0f0np0 enp3s0f1np1` (the stale `enp87s0` entry, dead since the GPU went in 2026-09-24, removed 2026-10-04) - **GPU:** NVIDIA RTX 2000E Ada, 16 GB (`01:00.0`) — driven by the **host** (580.178.04, DKMS), used by the esh-ml1 LXC. See below. - **Storage:** local `pve-root` at 94G (21% used) + two large NFS mounts from `10.0.50.50`: - `/mnt/pve/esh-nas` — 92 TB available on `/mnt/pvestore`