memory: snapshot — 2026-10-03 ~2045 (esh-nas-pve PVE 9, albok 0.1.5, QNAP LCD, open calls)
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-10-03 ~1440 PT (nh3-pve on PVE 9 and clean; RustDesk 1.1.16; esh-nas-pve PVE 9 assessed, awaiting Prime; earlier: PVE 8→9 by Prime at the console; nh3-pve-2 live on .62 with vmstore; CIRA watchdog live; PVE nag patched; tank healthy; albok 0.1.2 + Nemi timer. Prior: 2026-10-01 Parakeet/U11a.)_
|
||||
_Last updated: 2026-10-03 ~2045 PT (esh-nas-pve on PVE 9; nh3-pve on PVE 9; albok-service 0.1.5 + vh/albok pushed on Prime's go; QNAP LCD menu fixed; RustDesk 1.1.16; NH3 Mac mini asleep, fixed on site tomorrow.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -115,9 +115,9 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-10-03 ~1440 PT. The newest subsection is first; older subsections carry their own dates._
|
||||
_As of 2026-10-03 ~2045 PT. The newest subsection is first; older subsections carry their own dates._
|
||||
|
||||
### 2026-10-03: live now (as of ~1440 PT)
|
||||
### 2026-10-03: live now (as of ~2045 PT)
|
||||
|
||||
- **nh3-pve: DONE. ON PVE 9.2.21 / kernel 7.0.14-20 (booted 1402).** Post-check: every guest is back (the stopped 104/108 are as before), ZFS healthy, DNS, post office, albok, svos/hermes, mesh and AMT all OK. ⚠ **nh3-ml1 (CT 109) is DOWN:** no `proxmox-headers-7.0` was installed, so the nvidia 580.178.04 DKMS module was never built for 7.0, `nvidia-persistenced` failed, and CT 109 has no /dev/nvidia*. **FIXED 1409 (Prime go, `playbooks/nh3-pve-pve9-postfix.yaml`):** installed `proxmox-default-headers` + headers-7.0.14-20, nvidia DKMS built for 7.0 (580.178.04 compiles fine), nh3-ml1 back (TEI :8001/:8013 healthy, embed dim 1024); intel-microcode installed (non-free-firmware added; takes effect next boot); `pve-enterprise.sources` off; pve8to9 now 0 FAIL / 1 WARN (dkms, benign). VM 102 snapshot `pre-rootgrow-20261002` DELETED (Prime go). No `pre-pve9` snapshot had been taken. **POST-CHECK DONE: nh3-pve clean.** ⚠ Lesson: on a PVE host with DKMS, check that `proxmox-default-headers` is installed BEFORE a major upgrade, or the new kernel boots with no module.
|
||||
- **nh3-pve-2** (MS-03, NH3) is live at 10.100.250.62 on nh3-mgmt:
|
||||
@@ -129,9 +129,12 @@ _As of 2026-10-03 ~1440 PT. The newest subsection is first; older subsections ca
|
||||
- **esh-pve-2** (MS-03, ESH) is unplugged by Prime on purpose. vmstore + AMT phone-home are done. The permanent address is deferred by Prime ("address can wait"; `servers/esh-pve-2/README.md`). Run `playbooks/pve-nag-patch.yaml` on it when it is back.
|
||||
- **cira-tunnel-watchdog** on pfi-tacticalrmm has been live since 1232 and drops :4433 tunnels silent >180 s. Watch `journalctl -t cira-tunnel-watchdog`.
|
||||
- **RustDesk server (ana-docker) updated 1.1.14 → 1.1.16 at 1424** (same key; ports 21115-21119 reachable via rustdesk.phasefinal.com). Private key vaulted first as `ana-docker/rustdesk/id_ed25519`. "Change ID" is refused by the OSS server BY DESIGN (Pro feature; `rendezvous_server.rs` answers TCP RegisterPk with NOT_SUPPORT). Workaround, read at client source and untested: stop RustDesk, delete `enc_id` from RustDesk.toml, set a plain `id`, restart. → `servers/ana-docker/README.md`
|
||||
- **Booth vs 10.100.10.166 (Prime, ~1420): that client cannot reach the Booth.** The Booth was up and answered from 4 hosts at 1417; .166 is on nh3-dev's own subnet. Prime interrupted my check, so it is OPEN and resumes only on his word.
|
||||
- **esh-nas-pve → PVE 9: PREP + PVE 8 REBOOT TEST DONE (1533-1546, Prime on site).** Prep playbook `playbooks/esh-nas-pve-pve9-prep.yaml`; backups 103-107 on pbs-ana; came back on 6.8.12-43 with the pinned names. Two foot-guns are logged in the plan doc: VM 101's first start times out while its NFS recovers (retry), and mosquitto does not auto-restart on VM 100. **DONE 1606: esh-nas-pve is ON PVE 9.2.21 / kernel 7.0.14-20** (Prime go; ESH DNS stayed up; the /boot-LV-full trap is fixed and written up in the plan doc). Keep `nvme/ROOT@pre-pve9` about a week, then destroy it. **`pve` (esh-pve) is still on 8.4.20; its upgrade is a separate, bigger window** (DNS VM, the CT 108 mesh router, esh-ml1 with NVIDIA, so it needs `proxmox-default-headers` + a /boot check first). (Earlier assessment, kept:) Prime asked (1440) whether I can do it myself. Assessed read-only and ANSWERED. Facts: QNAP hardware, no AMT/IPMI (no remote console); PVE 8.4.20 but running kernel 6.8.12-13; up since 2026-08-18; ZFS root `nvme` booted by GRUB; systemd-boot meta present; NO NIC pins (uplink `enp5s0f0`); no DKMS. My answer: I can do all the prep remotely (vzdump, NIC pin, remove systemd-boot, root snapshot, pve8to9, a reboot test on PVE 8 first). The upgrade reboot should happen only with someone able to reach ESH, or with Prime's explicit acceptance of a no-console failure. VMs 100/101 on `pve` must be shut down first (hard NFS mounts). → `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`
|
||||
- **Booth vs 10.100.10.166 (Prime, ~1420): that client cannot reach the Booth.** The Booth was up and answered from 4 hosts at 1417; .166 is on nh3-dev's own subnet. Prime interrupted my check, so it is OPEN and resumes only on his word. Lead for later: .166 is Prime's Wi-Fi MacBook (`Mac`, 36:c7:30:13:aa:28, nh3-userland). So it is Wi-Fi → wired VM on the same /24; check the SSID's client-isolation / proxy-ARP settings first.
|
||||
- **esh-nas-pve: DONE, PVE 9.2.21 / kernel 7.0.14-20 since 1604** (Prime on site; ESH DNS never dropped; the /boot-LV-full trap was hit and fixed). Keep `nvme/ROOT@pre-pve9` about a week (until ~2026-10-10), then destroy it. `openipmi.service` fails at boot (no IPMI on the QNAP); harmless, maskable. **`pve` (esh-pve) is still on 8.4.20; separate window** (DNS VM, CT 108 mesh, esh-ml1 NVIDIA → `proxmox-default-headers` + a /boot check first). → `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`
|
||||
- **NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, Wi-Fi, 10.100.10.121) asleep since ~1407; MeshCentral + RustDesk can't reach it.** Wake-on-LAN from nh3-dev (same subnet) did NOT wake it (Wi-Fi had disassociated). Prime fixes it on site 2026-10-04: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally plus Ethernet.
|
||||
- **albok-service 0.1.5 LIVE (2039) after Prime's "push it"; 0.1.4 went out at 1611 the same way.** Each vh/albok push is a per-release go from Prime, not standing authorization. 0.1.5 = search 4.5 s → ~0.25 s. Deploy BETWEEN Nemi walks: `flock -n ~/.local/state/albok-nemi/run.lock true` (the 0.1.4 redeploy cut the 1601 walk; a catch-up walk 1612-1625 was clean). → `stacks/albok-service/README.md`
|
||||
- **QNAP front-panel LCD on esh-nas-pve fixed (1614), AWAITING Prime's visual check.** Upstream DigitalFunk/QnapLCD-Proxmox opened on a TrueNAS-only screen (`cli`) and died on Proxmox; it also hid vmbr0's IP. Our menu (host/IP, PVE version, uptime, IPs, pools) + a restarting unit: `services/qnap-lcd/`, `playbooks/qnap-lcd.yaml`; the originals are kept beside.
|
||||
- **Uptime Kuma watches the Booth (#2) but NOTHING on Miranda's chain** (svos :8770, hermes-gateway 127.0.0.1:8765); its alerts route through nh3-dev's :8096 bridge, the same host. Prime has NOT decided; recommended SVOS check + gateway heartbeat + a 2nd alert path off nh3-dev (mailrise on ana-docker).
|
||||
- **Done today, nothing open:**
|
||||
- tank verified healthy 0629, disk kept, Miranda told;
|
||||
- albok-service 0.1.2 + the Nemi hourly timer;
|
||||
|
||||
Reference in New Issue
Block a user