From 0e3ab447af5cd17e6a63ee8e9b3662e36f664efe Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 4 Oct 2026 16:37:15 -0700 Subject: [PATCH] ops: retire VM 101 (DC) and VM 104 (nh3-laser) from running + backups; infra-ops login on pbs-ana/pbs-nh3; nh3-pve-2 follow-ups verified done --- docs/runbooks/pbs-pbs4-upgrade-plan.md | 3 +++ persistent-memory.md | 9 +++------ servers/nh3-pve-2/README.md | 2 +- servers/nh3-pve/README.md | 2 +- servers/pbs-ana/README.md | 2 +- servers/pbs-ana/ssh-target | 2 +- servers/pbs-nh3/README.md | 2 +- servers/pbs-nh3/ssh-target | 2 +- servers/pfi-pve/README.md | 2 +- 9 files changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/runbooks/pbs-pbs4-upgrade-plan.md b/docs/runbooks/pbs-pbs4-upgrade-plan.md index afc0184..08ebbb1 100644 --- a/docs/runbooks/pbs-pbs4-upgrade-plan.md +++ b/docs/runbooks/pbs-pbs4-upgrade-plan.md @@ -85,3 +85,6 @@ on 4 can pull from a pbs-ana still on 3 in between. first real run is the 04:00 job on 2026-10-05; tomorrow's 08:00 backup-freshness check covers `pbs/sfsrv-pve`. - Open: drop both `pre-pbs4` snapshots on/after 2026-10-05 after that 04:00 run lands; keep the two one-off vzdumps until ~2026-10-11. +- 1636 (Prime: "give yourself a login on the backup servers"): **infra-ops bootstrapped on both** (uid/gid 850, fleet key, + NOPASSWD sudo with `log_output`, same content as `playbooks/bootstrap-infra-ops-user.yaml`, applied through the guest + agent). `ssh infra-ops@10.250.50.90` / `@10.100.50.90` + `sudo -n` verified; `servers/pbs-*/ssh-target` repointed. diff --git a/persistent-memory.md b/persistent-memory.md index c85d62f..af494c7 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -119,15 +119,12 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca ### 2026-10-03/04: live now (as of 2026-10-04 ~0546 PT) +- **VM 101 PFI-ANA-DC (pfi-pve) and VM 104 nh3-laser (nh3-pve) RETIRED from running (Prime 2026-10-04 1634: "we aren't running the DC or the remote laser")**: 101 shut down cleanly, both `onboot 0`, both excluded from their nightly vzdump (last backups kept). That also ends the PVE 9 Secure Boot/VirtIO backup warnings. ⚠ If either is ever started again: enroll the Microsoft 2023 certs first (`qm enroll-efi-keys`, BitLocker check). +- **infra-ops login on pbs-ana + pbs-nh3** (Prime 2026-10-04): uid 850, fleet key, NOPASSWD + `log_output`, bootstrapped via guest agent; `servers/pbs-*/ssh-target` now infra-ops. - **PBS 3 → 4: DONE 2026-10-04 1144** (Prime via Miranda): pbs-nh3 then pbs-ana, both PBS 4.2.7 / Debian 13.7. VM backups first (pbs-ana had none anywhere), disk snapshots `pre-pbs4`, read-only datastores, detached dist-upgrades. Proven: v3→v4 sync, PVE 9→PBS 4 write (ct/113), v4→v4 sync. ⚠ My first prep killed apt mid-dpkg on both (qemu-guest-agent restarted itself under the in-band apt); repaired in ~5 min, PBS never stopped serving. **Open:** sfsrv-ana (PVE 8) 04:00 run on 10-05 is the one untested pair; drop `pre-pbs4` snapshots after it; vzdumps until ~10-11. → `docs/runbooks/pbs-pbs4-upgrade-plan.md` - **pfi-pve (Anaheim): DONE, PVE 9.2.21 / kernel 7.0.14-20 since 0656 2026-10-04** (Prime: "drive it", "proceed"). Prep fixed a real boot trap: the box boots UEFI but had `grub-pc`, so the ESP GRUB was the 2023 install's; now `grub-efi-amd64`. NIC pins, microcode, a PVE 8 reboot test (passed), then trixie. All 14 guests + services verified 0715. ⚠ Each pfi-pve reboot = ~4 min of NH3/ESH↔ANA (ana-scale) and **~20 min of LiteLLM, which also silences Miranda**. Break-glass = FortiGate WAN SSH `infra-ops@38.120.12.42` (verified). ⚠ ifupdown2 3.3 `ifreload -a -s` exits 1 on PVE's own `bridge-fd 0` — cosmetic (same on nh3-pve/esh-nas-pve). **Open: `lvremove -y pve/root-pre-pve9` on/after 2026-10-05** (asked infra-hermes); no `zpool upgrade`. → `docs/runbooks/pfi-pve-pve9-upgrade-plan.md` - **nh3-pve: DONE. ON PVE 9.2.21 / kernel 7.0.14-20 (booted 1402).** Post-check: every guest is back (the stopped 104/108 are as before), ZFS healthy, DNS, post office, albok, svos/hermes, mesh and AMT all OK. ⚠ **nh3-ml1 (CT 109) is DOWN:** no `proxmox-headers-7.0` was installed, so the nvidia 580.178.04 DKMS module was never built for 7.0, `nvidia-persistenced` failed, and CT 109 has no /dev/nvidia*. **FIXED 1409 (Prime go, `playbooks/nh3-pve-pve9-postfix.yaml`):** installed `proxmox-default-headers` + headers-7.0.14-20, nvidia DKMS built for 7.0 (580.178.04 compiles fine), nh3-ml1 back (TEI :8001/:8013 healthy, embed dim 1024); intel-microcode installed (non-free-firmware added; takes effect next boot); `pve-enterprise.sources` off; pve8to9 now 0 FAIL / 1 WARN (dkms, benign). VM 102 snapshot `pre-rootgrow-20261002` DELETED (Prime go). No `pre-pve9` snapshot had been taken. **POST-CHECK DONE: nh3-pve clean.** ⚠ Lesson: on a PVE host with DKMS, check that `proxmox-default-headers` is installed BEFORE a major upgrade, or the new kernel boots with no module. -- **nh3-pve-2** (MS-03, NH3) is live at 10.100.250.62 on nh3-mgmt: - - X710 `nic3` on USW port 25; - - AMT `nic0` with `auto` + `/etc/sysctl.d/90-amt-port.conf`; - - vmstore 913 GiB; - - subscription popup patched. - Open and NOT approved yet (Prime's call): disable the ceph enterprise apt source, apt upgrade, and one reboot to prove `.62` and nic0 survive. → `servers/nh3-pve-2/README.md` +- **nh3-pve-2** (MS-03, NH3, 10.100.250.62): **follow-ups DONE 2026-10-04** (Prime rebooted 0724; .62, nic0/AMT, CIRA connected, enterprise repos off, up to date). No guests; purpose TBD. → `servers/nh3-pve-2/README.md` - **esh-pve-2** (MS-03, ESH) is unplugged by Prime on purpose. vmstore + AMT phone-home are done. The permanent address is deferred by Prime ("address can wait"; `servers/esh-pve-2/README.md`). Run `playbooks/pve-nag-patch.yaml` on it when it is back. - **cira-tunnel-watchdog** on pfi-tacticalrmm has been live since 1232 and drops :4433 tunnels silent >180 s. Watch `journalctl -t cira-tunnel-watchdog`. - **RustDesk server (ana-docker) updated 1.1.14 → 1.1.16 at 1424** (same key; ports 21115-21119 reachable via rustdesk.phasefinal.com). Private key vaulted first as `ana-docker/rustdesk/id_ed25519`. "Change ID" is refused by the OSS server BY DESIGN (Pro feature; `rendezvous_server.rs` answers TCP RegisterPk with NOT_SUPPORT). Workaround, read at client source and untested: stop RustDesk, delete `enc_id` from RustDesk.toml, set a plain `id`, restart. → `servers/ana-docker/README.md` diff --git a/servers/nh3-pve-2/README.md b/servers/nh3-pve-2/README.md index f46c8e7..51c295e 100644 --- a/servers/nh3-pve-2/README.md +++ b/servers/nh3-pve-2/README.md @@ -6,7 +6,7 @@ possibly a dev environment for security-software work. Standalone node, not clus - **Host:** `10.100.250.62` on nh3-mgmt (`nh3-pve-2.nh3.internal`, `nh3-pve-2.phasefinal.com` in /etc/hosts). Static in `/etc/network/interfaces`, plus a UDM reservation on `38:05:25:3b:a0:a8`. `ssh infra-ops@10.100.250.62` (fleet key, NOPASSWD sudo with `log_output`). -- PVE 9.2.21, installed 2026-10-03 on site to the right disk (the 256 GB Toshiba). +- PVE 9.2.21, installed 2026-10-03 on site to the right disk (the 256 GB Toshiba). **Follow-ups DONE 2026-10-04:** Prime rebooted it at 0724 onto 7.0.14-20; verified 1635: `.62` on vmbr0, `nic0` up at 1 Gb with `arp_ignore=8` + IPv6 off, MeshCentral shows `nh3-pve-2-amt` connected (CIRA, power on), ceph + pve enterprise sources `Enabled: false`, 0 upgradable, no failed units. No guests yet. | NIC | MAC | Chip | Use | |---|---|---|---| diff --git a/servers/nh3-pve/README.md b/servers/nh3-pve/README.md index dda2866..3639712 100644 --- a/servers/nh3-pve/README.md +++ b/servers/nh3-pve/README.md @@ -27,7 +27,7 @@ after a power loss** is the column that matters in a recovery: a guest listed as | 100 | nh3-docker (`10.100.50.40`) | VM | 1 | up | | 101 | nh3-extdev (`10.100.50.42`) | VM | 1 | up | | 102 | nh3-dev (`10.100.10.50`) | VM | 1 | up | -| 104 | nh3-laser | VM | — | **stays off — on-demand** (Prime, 2026-09-24) | +| 104 | nh3-laser | VM | — | **stays off — on-demand** (Prime, 2026-09-24); `onboot 0` + excluded from vzdump 2026-10-04 (Prime: not in use). Before any start: enroll MS 2023 Secure Boot certs (`qm enroll-efi-keys 104`) | | 105 | pbs-nh3 (`10.100.50.90`) | VM | 1 | up (onboot added 2026-09-24; it had none and stayed down after the outage) | | 108 | opnsense-lab | VM | 0 | stays off (lab) | | 103 | nh3-wg | CT | 1 | up | diff --git a/servers/pbs-ana/README.md b/servers/pbs-ana/README.md index a51d278..f152f63 100644 --- a/servers/pbs-ana/README.md +++ b/servers/pbs-ana/README.md @@ -1,6 +1,6 @@ # pbs-ana -> **2026-10-04: PBS 4.2.7 on Debian 13.7 (kernel 6.12.111)**, upgraded from 3.4.8 (`docs/runbooks/pbs-pbs4-upgrade-plan.md`). PBS-on-plain-Debian install (no `proxmox-backup` meta package, Debian kernel), so `pbs3to4` FAILs one check by design. infra-ops has no login here: root ops go through the hypervisor's guest agent (`qm guest exec`). +> **2026-10-04: PBS 4.2.7 on Debian 13.7 (kernel 6.12.111)**, upgraded from 3.4.8 (`docs/runbooks/pbs-pbs4-upgrade-plan.md`). PBS-on-plain-Debian install (no `proxmox-backup` meta package, Debian kernel), so `pbs3to4` FAILs one check by design. **`ssh infra-ops@10.250.50.90`** (uid 850, fleet key, NOPASSWD sudo with `log_output`) since 2026-10-04 (Prime); the hypervisor's guest agent (`qm guest exec`) is the fallback if the network is down. Proxmox Backup Server — **fleet primary**. VM on pfi-pve with an NFS-mounted datastore on the Ana NAS. All 5 hypervisors (pfi-pve, diff --git a/servers/pbs-ana/ssh-target b/servers/pbs-ana/ssh-target index ff90f9e..a5ddb3d 100644 --- a/servers/pbs-ana/ssh-target +++ b/servers/pbs-ana/ssh-target @@ -1 +1 @@ -lkraven@10.250.50.90 +infra-ops@10.250.50.90 diff --git a/servers/pbs-nh3/README.md b/servers/pbs-nh3/README.md index 49e5d46..d195324 100644 --- a/servers/pbs-nh3/README.md +++ b/servers/pbs-nh3/README.md @@ -1,6 +1,6 @@ # pbs-nh3 -> **2026-10-04: PBS 4.2.7 on Debian 13.7 (kernel 6.12.111)**, upgraded from 3.4.8 (`docs/runbooks/pbs-pbs4-upgrade-plan.md`). PBS-on-plain-Debian install (no `proxmox-backup` meta package, Debian kernel), so `pbs3to4` FAILs one check by design. infra-ops has no login here: root ops go through the hypervisor's guest agent (`qm guest exec`). +> **2026-10-04: PBS 4.2.7 on Debian 13.7 (kernel 6.12.111)**, upgraded from 3.4.8 (`docs/runbooks/pbs-pbs4-upgrade-plan.md`). PBS-on-plain-Debian install (no `proxmox-backup` meta package, Debian kernel), so `pbs3to4` FAILs one check by design. **`ssh infra-ops@10.100.50.90`** (uid 850, fleet key, NOPASSWD sudo with `log_output`) since 2026-10-04 (Prime); the hypervisor's guest agent (`qm guest exec`) is the fallback if the network is down. Proxmox Backup Server — **DR mirror**. VM on nh3-pve with an NFS-mounted datastore on the NH3 Synology. Pulls nightly from pbs-ana diff --git a/servers/pbs-nh3/ssh-target b/servers/pbs-nh3/ssh-target index bffba21..74255c9 100644 --- a/servers/pbs-nh3/ssh-target +++ b/servers/pbs-nh3/ssh-target @@ -1 +1 @@ -lkraven@10.100.50.90 +infra-ops@10.100.50.90 diff --git a/servers/pfi-pve/README.md b/servers/pfi-pve/README.md index bef810c..9b7f92b 100644 --- a/servers/pfi-pve/README.md +++ b/servers/pfi-pve/README.md @@ -30,7 +30,7 @@ live here. Full enumeration as of 2026-05-14 (via `qm list` on the hypervisor): - **VM 100** — `pbs-ana` (Proxmox Backup Server; 10.250.50.90) — `servers/pbs-ana/` -- **VM 101** — `PFI-ANA-DC` — Active Directory domain controller; **not yet in `servers/` inventory** +- **VM 101** — `PFI-ANA-DC` — Active Directory domain controller; **not in use: shut down, `onboot 0`, excluded from vzdump since 2026-10-04 (Prime)** - **VM 102** — `ana-docker` (PFI-ANA-Docker) — `servers/ana-docker/` - **VM 103** — `PFI-SlaveBot` — **not yet in `servers/` inventory** (purpose TBD) - **VM 104** — `PFI-Mongo` — MongoDB host; **not yet in `servers/` inventory**