ops(pfi-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20; reboot test + post-check recorded; pre-reboot checks corrected for PVE 9
This commit is contained in:
@@ -86,8 +86,10 @@ sent 08:32, her ack at 08:33:29).** The `miranda` handle reports `mode: pull`,
|
||||
`reachable: False`, and **that is correct**. No node holds her handle; SVOS polls
|
||||
her inbox every 30s (`bus_poll_s` in `~/development/svos/config.yaml`) and relays
|
||||
it into her conversation. ⚠ **So this channel depends on `svos.service` (:8770)
|
||||
and `hermes-gateway` on nh3-dev.** If an incident takes either of them down,
|
||||
Miranda cannot hear you, and even a routine svos restart is about 100s of
|
||||
and `hermes-gateway` on nh3-dev, and on the LiteLLM gateway on ana-docker**
|
||||
(`10.250.50.70:4000` is the model `base_url` in both `~/.hermes/config.yaml` and the svos config; found during
|
||||
the 2026-10-04 pfi-pve upgrade, where every pfi-pve reboot took the gateway out for ~20 min). If an incident takes
|
||||
any of them down, Miranda cannot hear you, and even a routine svos restart is about 100s of
|
||||
deafness (about 70s of that is vault reads). Reach Prime another way as well (the
|
||||
Booth board, or say it in this session).
|
||||
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# pfi-pve (Anaheim): Proxmox VE 8 → 9 upgrade
|
||||
|
||||
**STATUS: DONE 2026-10-04 0715 — PVE 9.2.21 / kernel 7.0.14-20, post-check passed. Open: drop the root snapshot
|
||||
on/after 2026-10-05 (see Log).**
|
||||
|
||||
Requested by Prime 2026-10-04 ~0550 ("drive the ana pve upgrade to proxmox 9"; "proceed" ~0600).
|
||||
Siblings: `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md` (esh-nas-pve, done 2026-10-03) and
|
||||
nh3-pve (done 2026-10-03, `playbooks/nh3-pve-pve9-postfix.yaml`). Official guide:
|
||||
@@ -99,3 +102,41 @@ execute ping 10.250.250.30 # iDRAC
|
||||
`grubx64.efi` rewritten 06:08 (md5 a4a8d8d1… → d9d0a989…, BOOTX64.EFI identical), `/boot/grub/x86_64-efi` refreshed,
|
||||
BootOrder 0003 (proxmox) first, four NIC pins in the -43 and -8 initramfs. pve8to9 --full: 0 FAIL, 1 WARN (running
|
||||
guests), NOTICE local-lvm autoactivation (local storage, left as is). No guest touched.
|
||||
- 0611:51 **PVE 8 reboot test** issued (FortiGate-side ping watcher running, independent of ana-scale). ana-docker
|
||||
stopped answering 0614:20; ana-scale + ana-nas 0617:39; host DOWN 0617:58; **host UP 0620:14** (R750xs POST ~2 min);
|
||||
mesh back 0621:23; ana-docker 0624:22; all 14 guests running by ~0635. **Passed:** booted 6.8.12-43 via the new
|
||||
GRUB, NIC names held, `.31` by DHCP, microcode 0xd000410, pools healthy, no failed units on the host.
|
||||
Services: gitea/vaultwarden 200, PBS 200, ana-nas NFS active (ana-docker `/mnt/backup` remounted), Worldtree
|
||||
demo/personal 200 0635, pinned 0636. ana-docker 44/45 Up (missing only `buildx_buildkit_worldtree-builder0`, an
|
||||
on-demand buildx builder with no restart policy). ⚠ **LiteLLM took until 0634:51** (container up 0627:56, then
|
||||
~6.5 min in `prisma` toolchain prep + `migrate deploy` under boot-storm load, load avg 11): the gateway, and with
|
||||
it Miranda/Hermes, was out ~20 min, not ~10. Pre-existing, not ours: pfi-pteradactyl `apache2` fails every boot
|
||||
(nginx owns :80) + `systemd-journal-flush` failed.
|
||||
- 0637 upgrade playbook: (a first run went to a mistyped `10.250.50.31` and died at elway's sudo probe, nothing
|
||||
executed). 0638 real run: pve8to9 0 FAIL; root snapshot `pve/root-pre-pve9` 15G; trixie sources; plan read =
|
||||
770 installs, 53 removals (t64 library renames + python3-distutils/future/lib2to3; isc-dhcp-client, srvadmin
|
||||
stay); detached dist-upgrade started 0638 (`pve9-dist-upgrade.service`). Targets: pve-manager 9.2.21, kernel
|
||||
7.0.14-20, GRUB 2.12-9+pmx2, ZFS 2.4.4 userland (⚠ no `zpool upgrade`), ceph-common 19.2.6, lxcfs 7.0.
|
||||
sshd restarted mid-upgrade (~0640, connection reset for a few seconds), as expected.
|
||||
- 0647 dist-upgrade EXIT 0 (770 packages; GRUB 2.12-9+pmx2 postinst: "Installing for x86_64-efi platform …
|
||||
Installation finished. No error reported."). Pre-reboot check: two of my checks were wrong for PVE 9 and were
|
||||
fixed, not waived: (1) the GRUB binary carries no greppable version string, so the postinst's own report is the
|
||||
proof; (2) ifupdown2 3.3 flags PVE's own `bridge-fd 0` as out of range and `ifreload -a -s` exits 1 on that
|
||||
warning alone. Positive control: nh3-pve and esh-nas-pve (same ifupdown2 3.3.0-1+pmx12, same line) show the same
|
||||
warning and run vmbr0 fine (STP off, forward_delay 0). The check now fails on any OTHER ifupdown2 output (tested
|
||||
both ways). Re-run: 7/7 OK. Enterprise repo set `Enabled: false`. Snapshot COW 40% (≈6 G) after the upgrade.
|
||||
- 0648:56 **PVE 9 reboot** issued. ana-docker stopped answering 0650:32; mesh down 0653:51; host DOWN 0654:10;
|
||||
**host UP 0656:45**; mesh back 0657:52; all 14 guests running 0704 (boot storm: host load 37, ospool HDD
|
||||
read-IOPS-bound with a cold ARC); dockerd on ana-docker active 0708; LiteLLM 0711:25; Worldtree ×3 0712:55;
|
||||
OpenWebUI healthy 0715.
|
||||
- **Post-check PASSED (0715):** pve-manager 9.2.21, kernel 7.0.14-20, booted via GRUB 2.12; NIC names held, `.31`
|
||||
by DHCP, vmbr0 STP off; NASPool + ospool ONLINE (⚠ ZFS 2.4.4 now — do NOT `zpool upgrade` until 9 is kept);
|
||||
no failed host units; chrony synced; all 5 storages active incl. the pbs-ana datastore; pve8to9 0 FAIL / 0 WARN
|
||||
(NOTICE: local-lvm autoactivation, local storage, left). Gateway embed (dim 1024) + `gen-small` chat OK;
|
||||
gitea/vaultwarden 200; PBS 200; TacticalRMM 200; pfi-postgres :5432, rest-server-ana :8000 open; ana-nas NFS
|
||||
active; ana-docker 44/45 Up (again only the on-demand `buildx_buildkit_worldtree-builder0`); nag patch re-applied.
|
||||
- **Outage actually paid, per reboot:** NH3/ESH↔Anaheim ~4 min; host ~2.5 min; the LiteLLM gateway (and with it
|
||||
Miranda/Hermes/SVOS) **~20-21 min** each time (0614-0635, 0650-0711), because ana-docker's dockerd and LiteLLM's
|
||||
prisma start are slow under the 14-guest boot storm on HDD-backed ospool.
|
||||
- Open: drop `pve/root-pre-pve9` (`lvremove -y pve/root-pre-pve9`) after ~24 h of clean running, i.e. on/after
|
||||
2026-10-05; sooner if `lvs` Data% passes 80 (a full thick snapshot only invalidates itself, origin unharmed).
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-10-04 ~0546 PT (albok-service 0.1.6 live after Prime's push go; esh-nas-pve + nh3-pve on PVE 9; QNAP LCD fixed, awaiting Prime's look; NH3 Mac mini asleep, fixed on site today.)_
|
||||
_Last updated: 2026-10-04 ~0720 PT (pfi-pve on PVE 9; albok-service 0.1.6 live after Prime's push go; esh-nas-pve + nh3-pve on PVE 9; QNAP LCD fixed, awaiting Prime's look; NH3 Mac mini asleep, fixed on site today.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -119,6 +119,7 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca
|
||||
|
||||
### 2026-10-03/04: live now (as of 2026-10-04 ~0546 PT)
|
||||
|
||||
- **pfi-pve (Anaheim): DONE, PVE 9.2.21 / kernel 7.0.14-20 since 0656 2026-10-04** (Prime: "drive it", "proceed"). Prep fixed a real boot trap: the box boots UEFI but had `grub-pc`, so the ESP GRUB was the 2023 install's; now `grub-efi-amd64`. NIC pins, microcode, a PVE 8 reboot test (passed), then trixie. All 14 guests + services verified 0715. ⚠ Each pfi-pve reboot = ~4 min of NH3/ESH↔ANA (ana-scale) and **~20 min of LiteLLM, which also silences Miranda**. Break-glass = FortiGate WAN SSH `infra-ops@38.120.12.42` (verified). ⚠ ifupdown2 3.3 `ifreload -a -s` exits 1 on PVE's own `bridge-fd 0` — cosmetic (same on nh3-pve/esh-nas-pve). **Open: `lvremove -y pve/root-pre-pve9` on/after 2026-10-05** (asked infra-hermes); no `zpool upgrade`. → `docs/runbooks/pfi-pve-pve9-upgrade-plan.md`
|
||||
- **nh3-pve: DONE. ON PVE 9.2.21 / kernel 7.0.14-20 (booted 1402).** Post-check: every guest is back (the stopped 104/108 are as before), ZFS healthy, DNS, post office, albok, svos/hermes, mesh and AMT all OK. ⚠ **nh3-ml1 (CT 109) is DOWN:** no `proxmox-headers-7.0` was installed, so the nvidia 580.178.04 DKMS module was never built for 7.0, `nvidia-persistenced` failed, and CT 109 has no /dev/nvidia*. **FIXED 1409 (Prime go, `playbooks/nh3-pve-pve9-postfix.yaml`):** installed `proxmox-default-headers` + headers-7.0.14-20, nvidia DKMS built for 7.0 (580.178.04 compiles fine), nh3-ml1 back (TEI :8001/:8013 healthy, embed dim 1024); intel-microcode installed (non-free-firmware added; takes effect next boot); `pve-enterprise.sources` off; pve8to9 now 0 FAIL / 1 WARN (dkms, benign). VM 102 snapshot `pre-rootgrow-20261002` DELETED (Prime go). No `pre-pve9` snapshot had been taken. **POST-CHECK DONE: nh3-pve clean.** ⚠ Lesson: on a PVE host with DKMS, check that `proxmox-default-headers` is installed BEFORE a major upgrade, or the new kernel boots with no module.
|
||||
- **nh3-pve-2** (MS-03, NH3) is live at 10.100.250.62 on nh3-mgmt:
|
||||
- X710 `nic3` on USW port 25;
|
||||
|
||||
@@ -45,7 +45,8 @@ verify:
|
||||
set -e
|
||||
ls -la --time-style=long-iso /boot/efi/EFI/proxmox/ /boot/efi/EFI/BOOT/
|
||||
test "$(md5sum < /boot/efi/EFI/proxmox/grubx64.efi)" = "$(md5sum < /boot/efi/EFI/BOOT/BOOTX64.EFI)"
|
||||
strings /boot/efi/EFI/proxmox/grubx64.efi | grep -m1 -E '^2\.1[2-9]' || strings /boot/efi/EFI/proxmox/grubx64.efi | grep -m1 -i 'grub 2\.1'
|
||||
# The binary carries no greppable version string; the trixie grub-efi-amd64 postinst's own report is the proof.
|
||||
grep -q 'Installation finished. No error reported.' /root/pve9-upgrade.log
|
||||
first=$(efibootmgr | sed -n 's/^BootOrder: \([0-9A-F]*\).*/\1/p')
|
||||
efibootmgr -v | grep -E "^Boot${first}\*? " | grep -qi 'EFI\\proxmox\\grubx64.efi'
|
||||
changed_when: "false"
|
||||
@@ -55,7 +56,13 @@ verify:
|
||||
set -e
|
||||
dpkg-query -W -f='${Status}' isc-dhcp-client | grep -q 'install ok installed' || dpkg-query -W -f='${Status}' dhcpcd-base | grep -q 'install ok installed'
|
||||
dpkg-query -W -f='${Status}' ifupdown2 | grep -q 'install ok installed'
|
||||
ifreload -a -s
|
||||
# ifupdown2 3.3 (PVE 9) flags PVE's own `bridge-fd 0` as out of range and `-s` exits 1 on that warning alone.
|
||||
# nh3-pve and esh-nas-pve show the same warning and booted with vmbr0 fine (STP off, forward_delay 0), so
|
||||
# only a warning OTHER than that one fails this check.
|
||||
out=$(ifreload -a -s 2>&1) || true
|
||||
echo "$out"
|
||||
# (an `if`, not a bare `! pipeline`: set -e ignores a negated pipeline, so that form could never fail)
|
||||
if echo "$out" | grep -v 'bridge-fd: value of out range "0"' | grep -q .; then echo "unexpected ifupdown2 output"; exit 1; fi
|
||||
test -z "$(grep -vE '^\s*(#|$)' /etc/sysctl.conf 2>/dev/null)"
|
||||
changed_when: "false"
|
||||
- name: pve8to9 --full (post-upgrade checks)
|
||||
|
||||
@@ -16,7 +16,11 @@ Proxmox VE hypervisor for the Anaheim colo (`pve.phasefinal.com`). Hosts every V
|
||||
- **Chassis:** Dell PowerEdge R750xs (2U rackmount)
|
||||
- **CPU:** Intel Xeon Silver 4310 @ 2.10 GHz
|
||||
- **RAM:** 188 GB
|
||||
- **Kernel:** `6.8.12-8-pve` (Proxmox 8.x)
|
||||
- **PVE / kernel:** 9.2.21 / `7.0.14-20-pve` since 2026-10-04 (upgraded from 8.3.5; `docs/runbooks/pfi-pve-pve9-upgrade-plan.md`)
|
||||
- **Boot:** UEFI → `\EFI\proxmox\grubx64.efi` on the BOSS card; `grub-efi-amd64` (was wrongly `grub-pc` until 2026-10-04). NIC names pinned by MAC (`/etc/systemd/network/10-pin-*.link`). Mgmt IP is DHCP on `vmbr0.250` (FortiGate lease)
|
||||
- **iDRAC9:** Enterprise licence (virtual console), fw 6.10.30.00; creds `secret get pfi-pve/idrac-infra-ops` (user `infra-ops`)
|
||||
- ⚠ **PSU1 has no input power** (repeatedly since 2026-04-23, latest 2026-09-11); running on PSU2 alone
|
||||
- ⚠ **Rebooting this host takes all of Anaheim off the fleet for ~4 min** (CT 114 ana-scale is the only NH3/ESH↔ANA path) **and the LiteLLM gateway off for ~20 min** (boot storm). Out-of-band way in: FortiGate WAN SSH `infra-ops@38.120.12.42` (see the upgrade plan § Break-glass)
|
||||
- **Storage:** local `pve-root` at 65G (49% used) + `pve` LVM pool for VM disks
|
||||
|
||||
## What it runs
|
||||
|
||||
Reference in New Issue
Block a user