ops: retire VM 101 (DC) and VM 104 (nh3-laser) from running + backups; infra-ops login on pbs-ana/pbs-nh3; nh3-pve-2 follow-ups verified done

2026-10-04 16:37:15 -07:00
parent d0c1bc628a
commit 0e3ab447af
9 changed files with 13 additions and 13 deletions
@@ -119,15 +119,12 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca

 ### 2026-10-03/04: live now (as of 2026-10-04 ~0546 PT)

+- **VM 101 PFI-ANA-DC (pfi-pve) and VM 104 nh3-laser (nh3-pve) RETIRED from running (Prime 2026-10-04 1634: "we aren't running the DC or the remote laser")**: 101 shut down cleanly, both `onboot 0`, both excluded from their nightly vzdump (last backups kept). That also ends the PVE 9 Secure Boot/VirtIO backup warnings. ⚠ If either is ever started again: enroll the Microsoft 2023 certs first (`qm enroll-efi-keys`, BitLocker check).
+- **infra-ops login on pbs-ana + pbs-nh3** (Prime 2026-10-04): uid 850, fleet key, NOPASSWD + `log_output`, bootstrapped via guest agent; `servers/pbs-*/ssh-target` now infra-ops.
 - **PBS 3 → 4: DONE 2026-10-04 1144** (Prime via Miranda): pbs-nh3 then pbs-ana, both PBS 4.2.7 / Debian 13.7. VM backups first (pbs-ana had none anywhere), disk snapshots `pre-pbs4`, read-only datastores, detached dist-upgrades. Proven: v3→v4 sync, PVE 9→PBS 4 write (ct/113), v4→v4 sync. ⚠ My first prep killed apt mid-dpkg on both (qemu-guest-agent restarted itself under the in-band apt); repaired in ~5 min, PBS never stopped serving. **Open:** sfsrv-ana (PVE 8) 04:00 run on 10-05 is the one untested pair; drop `pre-pbs4` snapshots after it; vzdumps until ~10-11. → `docs/runbooks/pbs-pbs4-upgrade-plan.md`
 - **pfi-pve (Anaheim): DONE, PVE 9.2.21 / kernel 7.0.14-20 since 0656 2026-10-04** (Prime: "drive it", "proceed"). Prep fixed a real boot trap: the box boots UEFI but had `grub-pc`, so the ESP GRUB was the 2023 install's; now `grub-efi-amd64`. NIC pins, microcode, a PVE 8 reboot test (passed), then trixie. All 14 guests + services verified 0715. ⚠ Each pfi-pve reboot = ~4 min of NH3/ESH↔ANA (ana-scale) and **~20 min of LiteLLM, which also silences Miranda**. Break-glass = FortiGate WAN SSH `infra-ops@38.120.12.42` (verified). ⚠ ifupdown2 3.3 `ifreload -a -s` exits 1 on PVE's own `bridge-fd 0` — cosmetic (same on nh3-pve/esh-nas-pve). **Open: `lvremove -y pve/root-pre-pve9` on/after 2026-10-05** (asked infra-hermes); no `zpool upgrade`. → `docs/runbooks/pfi-pve-pve9-upgrade-plan.md`
 - **nh3-pve: DONE. ON PVE 9.2.21 / kernel 7.0.14-20 (booted 1402).** Post-check: every guest is back (the stopped 104/108 are as before), ZFS healthy, DNS, post office, albok, svos/hermes, mesh and AMT all OK. ⚠ **nh3-ml1 (CT 109) is DOWN:** no `proxmox-headers-7.0` was installed, so the nvidia 580.178.04 DKMS module was never built for 7.0, `nvidia-persistenced` failed, and CT 109 has no /dev/nvidia*. **FIXED 1409 (Prime go, `playbooks/nh3-pve-pve9-postfix.yaml`):** installed `proxmox-default-headers` + headers-7.0.14-20, nvidia DKMS built for 7.0 (580.178.04 compiles fine), nh3-ml1 back (TEI :8001/:8013 healthy, embed dim 1024); intel-microcode installed (non-free-firmware added; takes effect next boot); `pve-enterprise.sources` off; pve8to9 now 0 FAIL / 1 WARN (dkms, benign). VM 102 snapshot `pre-rootgrow-20261002` DELETED (Prime go). No `pre-pve9` snapshot had been taken. **POST-CHECK DONE: nh3-pve clean.** ⚠ Lesson: on a PVE host with DKMS, check that `proxmox-default-headers` is installed BEFORE a major upgrade, or the new kernel boots with no module.
- **nh3-pve-2** (MS-03, NH3) is live at 10.100.250.62 on nh3-mgmt:
-  - X710 `nic3` on USW port 25;
-  - AMT `nic0` with `auto` + `/etc/sysctl.d/90-amt-port.conf`;
-  - vmstore 913 GiB;
-  - subscription popup patched.
-  Open and NOT approved yet (Prime's call): disable the ceph enterprise apt source, apt upgrade, and one reboot to prove `.62` and nic0 survive. → `servers/nh3-pve-2/README.md`
+- **nh3-pve-2** (MS-03, NH3, 10.100.250.62): **follow-ups DONE 2026-10-04** (Prime rebooted 0724; .62, nic0/AMT, CIRA connected, enterprise repos off, up to date). No guests; purpose TBD. → `servers/nh3-pve-2/README.md`
 - **esh-pve-2** (MS-03, ESH) is unplugged by Prime on purpose. vmstore + AMT phone-home are done. The permanent address is deferred by Prime ("address can wait"; `servers/esh-pve-2/README.md`). Run `playbooks/pve-nag-patch.yaml` on it when it is back.
 - **cira-tunnel-watchdog** on pfi-tacticalrmm has been live since 1232 and drops :4433 tunnels silent >180 s. Watch `journalctl -t cira-tunnel-watchdog`.
 - **RustDesk server (ana-docker) updated 1.1.14 → 1.1.16 at 1424** (same key; ports 21115-21119 reachable via rustdesk.phasefinal.com). Private key vaulted first as `ana-docker/rustdesk/id_ed25519`. "Change ID" is refused by the OSS server BY DESIGN (Pro feature; `rendezvous_server.rs` answers TCP RegisterPk with NOT_SUPPORT). Workaround, read at client source and untested: stop RustDesk, delete `enc_id` from RustDesk.toml, set a plain `id`, restart. → `servers/ana-docker/README.md`