ops: retire VM 101 (DC) and VM 104 (nh3-laser) from running + backups; infra-ops login on pbs-ana/pbs-nh3; nh3-pve-2 follow-ups verified done

2026-10-04 16:37:15 -07:00
parent d0c1bc628a
commit 0e3ab447af
9 changed files with 13 additions and 13 deletions
@@ -85,3 +85,6 @@ on 4 can pull from a pbs-ana still on 3 in between.
  first real run is the 04:00 job on 2026-10-05; tomorrow's 08:00 backup-freshness check covers `pbs/sfsrv-pve`.
 - Open: drop both `pre-pbs4` snapshots on/after 2026-10-05 after that 04:00 run lands; keep the two one-off vzdumps
  until ~2026-10-11.
 - 1636 (Prime: "give yourself a login on the backup servers"): **infra-ops bootstrapped on both** (uid/gid 850, fleet key,
  NOPASSWD sudo with `log_output`, same content as `playbooks/bootstrap-infra-ops-user.yaml`, applied through the guest
  agent). `ssh infra-ops@10.250.50.90` / `@10.100.50.90` + `sudo -n` verified; `servers/pbs-*/ssh-target` repointed.
@@ -119,15 +119,12 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca
 ### 2026-10-03/04: live now (as of 2026-10-04 ~0546 PT)
 - **VM 101 PFI-ANA-DC (pfi-pve) and VM 104 nh3-laser (nh3-pve) RETIRED from running (Prime 2026-10-04 1634: "we aren't running the DC or the remote laser")**: 101 shut down cleanly, both `onboot 0`, both excluded from their nightly vzdump (last backups kept). That also ends the PVE 9 Secure Boot/VirtIO backup warnings. ⚠ If either is ever started again: enroll the Microsoft 2023 certs first (`qm enroll-efi-keys`, BitLocker check).
 - **infra-ops login on pbs-ana + pbs-nh3** (Prime 2026-10-04): uid 850, fleet key, NOPASSWD + `log_output`, bootstrapped via guest agent; `servers/pbs-*/ssh-target` now infra-ops.
 - **PBS 3 → 4: DONE 2026-10-04 1144** (Prime via Miranda): pbs-nh3 then pbs-ana, both PBS 4.2.7 / Debian 13.7. VM backups first (pbs-ana had none anywhere), disk snapshots `pre-pbs4`, read-only datastores, detached dist-upgrades. Proven: v3→v4 sync, PVE 9→PBS 4 write (ct/113), v4→v4 sync. ⚠ My first prep killed apt mid-dpkg on both (qemu-guest-agent restarted itself under the in-band apt); repaired in ~5 min, PBS never stopped serving. **Open:** sfsrv-ana (PVE 8) 04:00 run on 10-05 is the one untested pair; drop `pre-pbs4` snapshots after it; vzdumps until ~10-11. → `docs/runbooks/pbs-pbs4-upgrade-plan.md`
 - **pfi-pve (Anaheim): DONE, PVE 9.2.21 / kernel 7.0.14-20 since 0656 2026-10-04** (Prime: "drive it", "proceed"). Prep fixed a real boot trap: the box boots UEFI but had `grub-pc`, so the ESP GRUB was the 2023 install's; now `grub-efi-amd64`. NIC pins, microcode, a PVE 8 reboot test (passed), then trixie. All 14 guests + services verified 0715. ⚠ Each pfi-pve reboot = ~4 min of NH3/ESH↔ANA (ana-scale) and **~20 min of LiteLLM, which also silences Miranda**. Break-glass = FortiGate WAN SSH `infra-ops@38.120.12.42` (verified). ⚠ ifupdown2 3.3 `ifreload -a -s` exits 1 on PVE's own `bridge-fd 0` — cosmetic (same on nh3-pve/esh-nas-pve). **Open: `lvremove -y pve/root-pre-pve9` on/after 2026-10-05** (asked infra-hermes); no `zpool upgrade`. → `docs/runbooks/pfi-pve-pve9-upgrade-plan.md`
 - **nh3-pve: DONE. ON PVE 9.2.21 / kernel 7.0.14-20 (booted 1402).** Post-check: every guest is back (the stopped 104/108 are as before), ZFS healthy, DNS, post office, albok, svos/hermes, mesh and AMT all OK. ⚠ **nh3-ml1 (CT 109) is DOWN:** no `proxmox-headers-7.0` was installed, so the nvidia 580.178.04 DKMS module was never built for 7.0, `nvidia-persistenced` failed, and CT 109 has no /dev/nvidia*. **FIXED 1409 (Prime go, `playbooks/nh3-pve-pve9-postfix.yaml`):** installed `proxmox-default-headers` + headers-7.0.14-20, nvidia DKMS built for 7.0 (580.178.04 compiles fine), nh3-ml1 back (TEI :8001/:8013 healthy, embed dim 1024); intel-microcode installed (non-free-firmware added; takes effect next boot); `pve-enterprise.sources` off; pve8to9 now 0 FAIL / 1 WARN (dkms, benign). VM 102 snapshot `pre-rootgrow-20261002` DELETED (Prime go). No `pre-pve9` snapshot had been taken. **POST-CHECK DONE: nh3-pve clean.** ⚠ Lesson: on a PVE host with DKMS, check that `proxmox-default-headers` is installed BEFORE a major upgrade, or the new kernel boots with no module.
- **nh3-pve-2** (MS-03, NH3) is live at 10.100.250.62 on nh3-mgmt:
+- **nh3-pve-2** (MS-03, NH3, 10.100.250.62): **follow-ups DONE 2026-10-04** (Prime rebooted 0724; .62, nic0/AMT, CIRA connected, enterprise repos off, up to date). No guests; purpose TBD. → `servers/nh3-pve-2/README.md`
  - X710 `nic3` on USW port 25;
  - AMT `nic0` with `auto` + `/etc/sysctl.d/90-amt-port.conf`;
  - vmstore 913 GiB;
  - subscription popup patched.
  Open and NOT approved yet (Prime's call): disable the ceph enterprise apt source, apt upgrade, and one reboot to prove `.62` and nic0 survive. → `servers/nh3-pve-2/README.md`
 - **esh-pve-2** (MS-03, ESH) is unplugged by Prime on purpose. vmstore + AMT phone-home are done. The permanent address is deferred by Prime ("address can wait"; `servers/esh-pve-2/README.md`). Run `playbooks/pve-nag-patch.yaml` on it when it is back.
 - **cira-tunnel-watchdog** on pfi-tacticalrmm has been live since 1232 and drops :4433 tunnels silent >180 s. Watch `journalctl -t cira-tunnel-watchdog`.
 - **RustDesk server (ana-docker) updated 1.1.14 → 1.1.16 at 1424** (same key; ports 21115-21119 reachable via rustdesk.phasefinal.com). Private key vaulted first as `ana-docker/rustdesk/id_ed25519`. "Change ID" is refused by the OSS server BY DESIGN (Pro feature; `rendezvous_server.rs` answers TCP RegisterPk with NOT_SUPPORT). Workaround, read at client source and untested: stop RustDesk, delete `enc_id` from RustDesk.toml, set a plain `id`, restart. → `servers/ana-docker/README.md`
@@ -6,7 +6,7 @@ possibly a dev environment for security-software work. Standalone node, not clus
 - **Host:** `10.100.250.62` on nh3-mgmt (`nh3-pve-2.nh3.internal`, `nh3-pve-2.phasefinal.com` in /etc/hosts). Static
  in `/etc/network/interfaces`, plus a UDM reservation on `38:05:25:3b:a0:a8`. `ssh infra-ops@10.100.250.62`
  (fleet key, NOPASSWD sudo with `log_output`).
- PVE 9.2.21, installed 2026-10-03 on site to the right disk (the 256 GB Toshiba).
+- PVE 9.2.21, installed 2026-10-03 on site to the right disk (the 256 GB Toshiba). **Follow-ups DONE 2026-10-04:** Prime rebooted it at 0724 onto 7.0.14-20; verified 1635: `.62` on vmbr0, `nic0` up at 1 Gb with `arp_ignore=8` + IPv6 off, MeshCentral shows `nh3-pve-2-amt` connected (CIRA, power on), ceph + pve enterprise sources `Enabled: false`, 0 upgradable, no failed units. No guests yet.
 | NIC | MAC | Chip | Use |
 |---|---|---|---|
@@ -27,7 +27,7 @@ after a power loss** is the column that matters in a recovery: a guest listed as
 | 100 | nh3-docker (`10.100.50.40`) | VM | 1 | up |
 | 101 | nh3-extdev (`10.100.50.42`) | VM | 1 | up |
 | 102 | nh3-dev (`10.100.10.50`) | VM | 1 | up |
-| 104 | nh3-laser | VM | — | **stays off — on-demand** (Prime, 2026-09-24) |
+| 104 | nh3-laser | VM | — | **stays off — on-demand** (Prime, 2026-09-24); `onboot 0` + excluded from vzdump 2026-10-04 (Prime: not in use). Before any start: enroll MS 2023 Secure Boot certs (`qm enroll-efi-keys 104`) |
 | 105 | pbs-nh3 (`10.100.50.90`) | VM | 1 | up (onboot added 2026-09-24; it had none and stayed down after the outage) |
 | 108 | opnsense-lab | VM | 0 | stays off (lab) |
 | 103 | nh3-wg | CT | 1 | up |
@@ -1,6 +1,6 @@
 # pbs-ana
-> **2026-10-04: PBS 4.2.7 on Debian 13.7 (kernel 6.12.111)**, upgraded from 3.4.8 (`docs/runbooks/pbs-pbs4-upgrade-plan.md`). PBS-on-plain-Debian install (no `proxmox-backup` meta package, Debian kernel), so `pbs3to4` FAILs one check by design. infra-ops has no login here: root ops go through the hypervisor's guest agent (`qm guest exec`).
+> **2026-10-04: PBS 4.2.7 on Debian 13.7 (kernel 6.12.111)**, upgraded from 3.4.8 (`docs/runbooks/pbs-pbs4-upgrade-plan.md`). PBS-on-plain-Debian install (no `proxmox-backup` meta package, Debian kernel), so `pbs3to4` FAILs one check by design. **`ssh infra-ops@10.250.50.90`** (uid 850, fleet key, NOPASSWD sudo with `log_output`) since 2026-10-04 (Prime); the hypervisor's guest agent (`qm guest exec`) is the fallback if the network is down.
 Proxmox Backup Server — **fleet primary**. VM on pfi-pve with an
 NFS-mounted datastore on the Ana NAS. All 5 hypervisors (pfi-pve,
@@ -1 +1 @@
-lkraven@10.250.50.90
+infra-ops@10.250.50.90
@@ -1,6 +1,6 @@
 # pbs-nh3
-> **2026-10-04: PBS 4.2.7 on Debian 13.7 (kernel 6.12.111)**, upgraded from 3.4.8 (`docs/runbooks/pbs-pbs4-upgrade-plan.md`). PBS-on-plain-Debian install (no `proxmox-backup` meta package, Debian kernel), so `pbs3to4` FAILs one check by design. infra-ops has no login here: root ops go through the hypervisor's guest agent (`qm guest exec`).
+> **2026-10-04: PBS 4.2.7 on Debian 13.7 (kernel 6.12.111)**, upgraded from 3.4.8 (`docs/runbooks/pbs-pbs4-upgrade-plan.md`). PBS-on-plain-Debian install (no `proxmox-backup` meta package, Debian kernel), so `pbs3to4` FAILs one check by design. **`ssh infra-ops@10.100.50.90`** (uid 850, fleet key, NOPASSWD sudo with `log_output`) since 2026-10-04 (Prime); the hypervisor's guest agent (`qm guest exec`) is the fallback if the network is down.
 Proxmox Backup Server — **DR mirror**. VM on nh3-pve with an
 NFS-mounted datastore on the NH3 Synology. Pulls nightly from pbs-ana
@@ -1 +1 @@
-lkraven@10.100.50.90
+infra-ops@10.100.50.90
@@ -30,7 +30,7 @@ live here. Full enumeration as of 2026-05-14 (via `qm list` on the
 hypervisor):
 - **VM 100** — `pbs-ana` (Proxmox Backup Server; 10.250.50.90) — `servers/pbs-ana/`
- **VM 101** — `PFI-ANA-DC` — Active Directory domain controller; **not yet in `servers/` inventory**
+- **VM 101** — `PFI-ANA-DC` — Active Directory domain controller; **not in use: shut down, `onboot 0`, excluded from vzdump since 2026-10-04 (Prime)**
 - **VM 102** — `ana-docker` (PFI-ANA-Docker) — `servers/ana-docker/`
 - **VM 103** — `PFI-SlaveBot` — **not yet in `servers/` inventory** (purpose TBD)
 - **VM 104** — `PFI-Mongo` — MongoDB host; **not yet in `servers/` inventory**