vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: snapshot — 2026-10-04 ~0546 (albok 0.1.6; 8 settled 09-19 entries archived)

Browse Source
This commit is contained in:
vh
2026-10-04 05:46:09 -07:00
parent 2449e8806f
commit a4b7626a13
10 changed files with 47 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+5 -13
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-10-03 ~2045 PT (esh-nas-pve on PVE 9; nh3-pve on PVE 9; albok-service 0.1.5 + vh/albok pushed on Prime's go; QNAP LCD menu fixed; RustDesk 1.1.16; NH3 Mac mini asleep, fixed on site tomorrow.)_
_Last updated: 2026-10-04 ~0546 PT (albok-service 0.1.6 live after Prime's push go; esh-nas-pve + nh3-pve on PVE 9; QNAP LCD fixed, awaiting Prime's look; NH3 Mac mini asleep, fixed on site today.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,9 +115,9 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-10-03 ~2045 PT. The newest subsection is first; older subsections carry their own dates._
_As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections carry their own dates._
### 2026-10-03: live now (as of ~2045 PT)
### 2026-10-03/04: live now (as of 2026-10-04 ~0546 PT)
- **nh3-pve: DONE. ON PVE 9.2.21 / kernel 7.0.14-20 (booted 1402).** Post-check: every guest is back (the stopped 104/108 are as before), ZFS healthy, DNS, post office, albok, svos/hermes, mesh and AMT all OK. ⚠ **nh3-ml1 (CT 109) is DOWN:** no `proxmox-headers-7.0` was installed, so the nvidia 580.178.04 DKMS module was never built for 7.0, `nvidia-persistenced` failed, and CT 109 has no /dev/nvidia*. **FIXED 1409 (Prime go, `playbooks/nh3-pve-pve9-postfix.yaml`):** installed `proxmox-default-headers` + headers-7.0.14-20, nvidia DKMS built for 7.0 (580.178.04 compiles fine), nh3-ml1 back (TEI :8001/:8013 healthy, embed dim 1024); intel-microcode installed (non-free-firmware added; takes effect next boot); `pve-enterprise.sources` off; pve8to9 now 0 FAIL / 1 WARN (dkms, benign). VM 102 snapshot `pre-rootgrow-20261002` DELETED (Prime go). No `pre-pve9` snapshot had been taken. **POST-CHECK DONE: nh3-pve clean.** ⚠ Lesson: on a PVE host with DKMS, check that `proxmox-default-headers` is installed BEFORE a major upgrade, or the new kernel boots with no module.
- **nh3-pve-2** (MS-03, NH3) is live at 10.100.250.62 on nh3-mgmt:
@@ -132,7 +132,7 @@ _As of 2026-10-03 ~2045 PT. The newest subsection is first; older subsections ca
- **Booth vs 10.100.10.166 (Prime, ~1420): that client cannot reach the Booth.** The Booth was up and answered from 4 hosts at 1417; .166 is on nh3-dev's own subnet. Prime interrupted my check, so it is OPEN and resumes only on his word. Lead for later: .166 is Prime's Wi-Fi MacBook (`Mac`, 36:c7:30:13:aa:28, nh3-userland). So it is Wi-Fi → wired VM on the same /24; check the SSID's client-isolation / proxy-ARP settings first.
- **esh-nas-pve: DONE, PVE 9.2.21 / kernel 7.0.14-20 since 1604** (Prime on site; ESH DNS never dropped; the /boot-LV-full trap was hit and fixed). Keep `nvme/ROOT@pre-pve9` about a week (until ~2026-10-10), then destroy it. `openipmi.service` fails at boot (no IPMI on the QNAP); harmless, maskable. **`pve` (esh-pve) is still on 8.4.20; separate window** (DNS VM, CT 108 mesh, esh-ml1 NVIDIA → `proxmox-default-headers` + a /boot check first). → `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`
- **NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, Wi-Fi, 10.100.10.121) asleep since ~1407; MeshCentral + RustDesk can't reach it.** Wake-on-LAN from nh3-dev (same subnet) did NOT wake it (Wi-Fi had disassociated). Prime fixes it on site 2026-10-04: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally plus Ethernet.
- **albok-service 0.1.5 LIVE (2039) after Prime's "push it"; 0.1.4 went out at 1611 the same way.** Each vh/albok push is a per-release go from Prime, not standing authorization. 0.1.5 = search 4.5 s → ~0.25 s. Deploy BETWEEN Nemi walks: `flock -n ~/.local/state/albok-nemi/run.lock true` (the 0.1.4 redeploy cut the 1601 walk; a catch-up walk 1612-1625 was clean). → `stacks/albok-service/README.md`
- **albok-service 0.1.6 LIVE (2026-10-04 0545; review fold: a locked keyword-index build); 0.1.5 at 2039 and 0.1.4 at 1611, each after Prime's "push it".** Each vh/albok push is a per-release go from Prime, not standing authorization. 0.1.5 = search 4.5 s → ~0.25 s. Deploy BETWEEN Nemi walks: `flock -n ~/.local/state/albok-nemi/run.lock true` (the 0.1.4 redeploy cut the 1601 walk; a catch-up walk 1612-1625 was clean). → `stacks/albok-service/README.md`
- **QNAP front-panel LCD on esh-nas-pve fixed (1614), AWAITING Prime's visual check.** Upstream DigitalFunk/QnapLCD-Proxmox opened on a TrueNAS-only screen (`cli`) and died on Proxmox; it also hid vmbr0's IP. Our menu (host/IP, PVE version, uptime, IPs, pools) + a restarting unit: `services/qnap-lcd/`, `playbooks/qnap-lcd.yaml`; the originals are kept beside.
- **Uptime Kuma watches the Booth (#2) but NOTHING on Miranda's chain** (svos :8770, hermes-gateway 127.0.0.1:8765); its alerts route through nh3-dev's :8096 bridge, the same host. Prime has NOT decided; recommended SVOS check + gateway heartbeat + a 2nd alert path off nh3-dev (mailrise on ana-docker).
- **Done today, nothing open:**
@@ -429,21 +429,13 @@ _As of 2026-10-03 ~2045 PT. The newest subsection is first; older subsections ca
- `[2026-09-20]` **`ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.** — infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a… → `persistent-memory.d/2026-09-20-ravenpen-com-registered-by-the-operator-the-09-18-hold-is.md`
- `[2026-09-19]` **FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed).** → `persistent-memory.d/2026-09-19-fv-is-a-dedicated-20-a-circuit-carrying-fv-ml1-and-the-r420.md`
- `[2026-09-19]` **althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning.** → `persistent-memory.d/2026-09-19-althing-3-7-0-rolled-out-to-both-infra-ops-surfaces-and-the.md`
- `[2026-09-19]` **Three agents commit as one git author, and closing that gap took three instruments to get right.** — An unattributable commit (`e43e262`) appeared in the push set between two of mine — unidentifiable from git… → `persistent-memory.d/2026-09-19-three-agents-commit-as-one-git-author-and-closing-that-gap.md`
- `[2026-09-19]` **The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking.** → `persistent-memory.d/2026-09-19-the-backup-alarm-had-no-wire-for-three-weeks-and-fixing-it.md`
- `[2026-09-19]` **elway evaluated `when:` / `creates:` / `removes:` / `changed_when:` WITHOUT the step's sudo, and it fails silently in the dangerous direction.** → `persistent-memory.d/2026-09-19-elway-evaluated-when-creates-removes-changedwhen-without.md`
- `[2026-09-19]` **ESH VM 102 (`esh-vm-workstation`) excluded from the nightly backup job — operator ruling.** — It is a Windows 11 Parsec/RDP **sandbox** (no password, no state to recover), and its vzdump had failed… → `persistent-memory.d/2026-09-19-esh-vm-102-esh-vm-workstation-excluded-from-the-nightly.md`
- `[2026-09-19]` **The ops log is BUILT — `scripts/ops-log`, automatic writers, and a detector for the path they cannot cover.** → `persistent-memory.d/2026-09-19-the-ops-log-is-built-scripts-ops-log-automatic-writers-and.md`
- `[2026-09-19]` **`infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.** — infra-ops keeps **improving infrastructure tooling** plus the hard calls; infra-hermes does **day-to-day… → `persistent-memory.d/2026-09-19-infra-hermes-is-this-session-s-assistant-and-the-division.md`
- `[2026-09-18]` ⭐⭐ **SearXNG had ONE working general web engine and every health check said fine.** 7 of 55 were enabled-by-default and six of those are dictionary/translation engines — `inactive: false` only makes an engine SELECTABLE, `disabled: false` puts it in the DEFAULT set. Now seven. ⭐ This stack tracks `:latest` ON PURPOSE (upstream ships engine-handler fixes continuously; a pin freezes breakage). ⭐ The Brave key is committed in plaintext by explicit operator decision — scoped to one low-value credential, NOT a change to the no-secrets rule. → `persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md`
- `[2026-09-18]` ⭐⭐ **althing search returned ZERO for every hyphenated query — silently, for every handle, and on this fleet that is most of our hostnames.** FTS5 read the hyphen as a column filter; `search()` caught the error, its probe passed, and it returned `[]`. Routed to forseti (they own the code, I own rollout) → **v3.6.3** deployed, 17 s bus outage. The reported symptom, a message that never arrived, was NOT a defect: the reporter's evidence file was truncated at 8 KiB. ⚠ No CI builds the post-office image. → `persistent-memory.d/2026-09-18-althing-363-hyphen-search.md`
@@ -469,7 +461,7 @@ _As of 2026-10-03 ~2045 PT. The newest subsection is first; older subsections ca
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
_172 older entries archived to archival-memory.md._
_180 older entries archived to archival-memory.md._
## Tried and abandoned