From c6b6435c5203c0ac95cf173d6fc1535893fa5a08 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 15 Sep 2026 01:07:53 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20mesh=20retirem?= =?UTF-8?q?ent=20complete,=20next=20session=20points=20at=20svos-dev=20+?= =?UTF-8?q?=20STT?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures the close-out of the fleet networking session: mesh membership retired for both fv-ml1 and nh3-dev, leaving six nodes that each have a job, with fv-ml1 carrying a break-glass rejoin instead of standing membership and exactly one live reusable pre-auth key left fleet-wide. In-flight rewritten to lead with the two jobs the operator named for the next session -- drain the svos-dev message that has been unread since 00:52, then stand up an STT service from nothing -- so a fresh context opens on the work rather than on the history. --- persistent-memory.md | 74 +++++++++++++++++++++++--------------------- 1 file changed, 39 insertions(+), 35 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index 2f72662..d7d8b56 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-15 ~00:50 PT (FV CROSS-SITE ROUTING FIXED — one OPNsense NAT rule was scoped to Anaheim only; four rules now cover NH3/ESH/IRV/ANA from the whole FV /24. Dead man's switch live on fv-ml1. Fleet docker trees normalized to root:docker setgid + conventions pinned. ESPHome modernised. ⚠ I rebooted the FV firewall by probing API endpoints — 3.5 min site outage.)_ +_Last updated: 2026-09-15 ~01:15 PT (FV cross-site routing FIXED; mesh membership RETIRED for both fv-ml1 and nh3-dev — six nodes remain, each with a job; fv-ml1 carries a break-glass rejoin instead. Next session: drain svos-dev, then stand up an STT.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -115,50 +115,54 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-09-15 ~00:50 PT._ +_As of 2026-09-15 ~01:15 PT._ -**fv-ml1 seat topology is UNCHANGED from the 2026-09-14 rebalance** — see the table in -that entry. This session was fleet networking + hygiene, not seats. +### ⭐ NEXT SESSION'S NAMED WORK (operator, at snapshot time) +1. **Drain svos-dev** — one althing message unread since 00:52 PT, pinged five times, + never read. `/althing:inbox` first. +2. **Stand up an STT** (speech-to-text) service. Nothing started: no placement decided, + no model chosen, no stack authored. Greenfield. -### FV cross-site routing — FIXED (the session's main thread) -fv-ml1 reaches **NH3, ESH, Anaheim, Irvine, mesh and internet**. Four outbound-NAT rules -on the FV gateway, all `src=10.251.50.0/24`, dst `10.100.0.0/16` / `10.0.0.0/16` / -`10.6.110.0/24` / `10.250.0.0/16`. Root cause was the 2026-09-13 rule being scoped to -Anaheim only. Commits `fa04f45`, `0ab9da5`. +### Fleet networking — closed out this session +- **FV cross-site routing FIXED.** fv-ml1 reaches NH3/ESH/ANA/IRV/internet via outbound + SNAT on the FV gateway. Four rules, all `src=10.251.50.0/24`. `fa04f45`, `0ab9da5`. +- **Mesh membership retired for fv-ml1 AND nh3-dev** (`959a743`, `a65cdf6`). Six nodes + remain, each with a job: nh3-scale, esh-scale, ana-scale, vb-gateway, irv-ml1 (Irvine's + own router), the operator's MacBook Air. The nh3-scale masquerade exception nh3-dev + required was reverted with it. +- **fv-ml1 has break-glass instead of membership** — `fv-mesh-watchdog` joins the mesh + when nh3-dev *and* nh3-docker go unreachable while the WAN is up. Proven off-mesh end + to end. Key: dedicated 1-year reusable, headscale ID 8, expires 2027-09-15, vaulted + `fv-ml1/headscale-breakglass-key`. +- **Exactly one live reusable pre-auth key fleet-wide** (that one). The two stale FV + cutover keys were expired. -### Safety net at FV -- **Break-glass mesh path on fv-ml1** — the box is now OFF the mesh; `fv-mesh-watchdog` - (1/min) JOINS it when nh3-dev/nh3-docker go unreachable while the WAN is up. Verified - end to end off-mesh: joined as `100.64.0.10`, answered ping+ssh, closed cleanly. -- **FV firewall config in the nightly restic run** via ana-docker's pre-backup hook - (`80d982d`); `infra-ops` now has an SSH key + `/bin/sh` on the gateway, so the - allowlisted WAN path gives a shell rather than API-only. - -### Fleet hygiene landed -`root:docker 2775` setgid deploy trees on all 5 hosts + `0777` cleared (`826a63b`); -conventions pinned in `docs/pfi/fleet-conventions.md` with a read-only audit playbook -(`abef67a`); `linus` deleted and `llmuser` de-privileged (`ce7b07f`); nh3-dev retired from the mesh and its masquerade -exception reverted with it (`9dbd829` superseded); ESPHome pinned/relocated/rotated -(`d1769ed` ff); `kb` KB-search tool shipped (`68fa80f`). +### Also landed this session +`root:docker 2775` setgid deploy trees on 5 hosts + `0777` cleared (`826a63b`); +conventions pinned in `docs/pfi/fleet-conventions.md` + audit playbook (`abef67a`); +`linus` deleted, `llmuser` de-privileged (`ce7b07f`); FV firewall config in the nightly +restic run (`80d982d`); `infra-ops` SSH key on the OPNsense gateway; ESPHome pinned and +hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`). ### Open loose ends (none blocking) -- ✅ **Mesh membership retired for both nh3-dev and fv-ml1** (2026-09-15). Six nodes remain: - the three site routers, vb-gateway, irv-ml1 (Irvine's own router) and the MacBook Air. - Every one of those has a job; nothing is enrolled "just in case" any more. -- ✅ **Stale reusable pre-auth keys retired** (IDs 5, 6 expired). The mesh now has exactly - one live reusable key: the dedicated fv-ml1 break-glass key, ID 8, expires 2027-09-15, - vaulted `fv-ml1/headscale-breakglass-key`. -- **FV WAN admin is plain HTTP**, no TLS. Allowlisted to ESH/ANA/NH3 egress — ⚠ pinned to - **egress IPs**, so a WAN change at any site silently removes the fallback. -- **Legacy `/32` ANA NAT rule** redundant and invisible to `source_nat/search_rule`; - delete from the UI. -- **`10.251.250.0/24` (BMC/mgmt) is not covered** by the FV SNAT rules — inbound works. -- **No headscale ACL policy loaded** → default-allow across the mesh. +- **nh3-dev has no break-glass watchdog, by choice** — NH3 is a populated site with other + boxes to diagnose from, unlike FV where a single gateway is the only path. Reversible: + the script is in `servers/fv-ml1/`. +- **Legacy `/32` ANA NAT rule** redundant and invisible to `source_nat/search_rule` — + delete from the OPNsense UI. +- **FV WAN admin is plain HTTP**, allowlisted to ESH/ANA/NH3 — ⚠ pinned to **egress IPs**, + so a WAN change at any site silently removes the fallback. +- **`10.251.250.0/24` (BMC/mgmt)** outside the FV SNAT scope — inbound works. +- **No headscale ACL policy loaded** → mesh is default-allow. - **Prior-session items still open:** FV site-visit power measurements in `docs/runbooks/fv-site-dark-20260913.md`; mog-sec blue-team seat fate; deferred park items. +- ⚠ **fv-ml1 seat topology unchanged** from the 2026-09-14 rebalance — this session touched + no seats. ## Recent decisions +- `[2026-09-15]` **Mesh membership retired for fv-ml1 and nh3-dev — six nodes left, each with a job.** fv-ml1 gets break-glass rejoin instead of standing membership; nh3-dev's retirement also removed the nh3-scale masquerade exception it had required. Exactly one live reusable pre-auth key remains fleet-wide. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md` + - `[2026-09-15]` **FV cross-site routing fixed — one OPNsense outbound-NAT rule had been scoped to Anaheim only.** fv-ml1 now reaches NH3/ESH/IRV/ANA/mesh/internet; four rules, all `src=10.251.50.0/24`. The diagnostic signature is the valuable part: every layer looks correct and the discriminator is that *every other site pair works*. → `persistent-memory.d/2026-09-15-fv-cross-site-snat.md` - `[2026-09-15]` **Break-glass mesh path on fv-ml1** — inverted from a restore-watchdog on the operator's suggestion: the box is OFF the mesh and the watchdog JOINS it on fleet loss. Exposed a rejoin key expiring in 4 days; replaced with a dedicated 1-year key and the two stale reusable keys retired. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md`