vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: snapshot — mesh retirement complete, next session points at svos-dev + STT

Browse Source 
Captures the close-out of the fleet networking session: mesh membership retired
for both fv-ml1 and nh3-dev, leaving six nodes that each have a job, with
fv-ml1 carrying a break-glass rejoin instead of standing membership and exactly
one live reusable pre-auth key left fleet-wide.

In-flight rewritten to lead with the two jobs the operator named for the next
session -- drain the svos-dev message that has been unread since 00:52, then
stand up an STT service from nothing -- so a fresh context opens on the work
rather than on the history.
This commit is contained in:
vh
2026-09-15 01:07:53 -07:00
parent a65cdf65d9
commit c6b6435c52
1 changed files with 39 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+39 -35
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-15 ~00:50 PT (FV CROSS-SITE ROUTING FIXED — one OPNsense NAT rule was scoped to Anaheim only; four rules now cover NH3/ESH/IRV/ANA from the whole FV /24. Dead man's switch live on fv-ml1. Fleet docker trees normalized to root:docker setgid + conventions pinned. ESPHome modernised. ⚠ I rebooted the FV firewall by probing API endpoints — 3.5 min site outage.)_
_Last updated: 2026-09-15 ~01:15 PT (FV cross-site routing FIXED; mesh membership RETIRED for both fv-ml1 and nh3-dev — six nodes remain, each with a job; fv-ml1 carries a break-glass rejoin instead. Next session: drain svos-dev, then stand up an STT.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,50 +115,54 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-09-15 ~00:50 PT._
_As of 2026-09-15 ~01:15 PT._
**fv-ml1 seat topology is UNCHANGED from the 2026-09-14 rebalance** — see the table in
that entry. This session was fleet networking + hygiene, not seats.
### ⭐ NEXT SESSION'S NAMED WORK (operator, at snapshot time)
1. **Drain svos-dev** — one althing message unread since 00:52 PT, pinged five times,
never read. `/althing:inbox` first.
2. **Stand up an STT** (speech-to-text) service. Nothing started: no placement decided,
no model chosen, no stack authored. Greenfield.
### FV cross-site routing — FIXED (the session's main thread)
fv-ml1 reaches **NH3, ESH, Anaheim, Irvine, mesh and internet**. Four outbound-NAT rules
on the FV gateway, all `src=10.251.50.0/24`, dst `10.100.0.0/16` / `10.0.0.0/16` /
`10.6.110.0/24` / `10.250.0.0/16`. Root cause was the 2026-09-13 rule being scoped to
Anaheim only. Commits `fa04f45`, `0ab9da5`.
### Fleet networking — closed out this session
- **FV cross-site routing FIXED.** fv-ml1 reaches NH3/ESH/ANA/IRV/internet via outbound
SNAT on the FV gateway. Four rules, all `src=10.251.50.0/24`. `fa04f45`, `0ab9da5`.
- **Mesh membership retired for fv-ml1 AND nh3-dev** (`959a743`, `a65cdf6`). Six nodes
remain, each with a job: nh3-scale, esh-scale, ana-scale, vb-gateway, irv-ml1 (Irvine's
own router), the operator's MacBook Air. The nh3-scale masquerade exception nh3-dev
required was reverted with it.
- **fv-ml1 has break-glass instead of membership** — `fv-mesh-watchdog` joins the mesh
when nh3-dev *and* nh3-docker go unreachable while the WAN is up. Proven off-mesh end
to end. Key: dedicated 1-year reusable, headscale ID 8, expires 2027-09-15, vaulted
`fv-ml1/headscale-breakglass-key`.
- **Exactly one live reusable pre-auth key fleet-wide** (that one). The two stale FV
cutover keys were expired.
### Safety net at FV
- **Break-glass mesh path on fv-ml1** — the box is now OFF the mesh; `fv-mesh-watchdog`
(1/min) JOINS it when nh3-dev/nh3-docker go unreachable while the WAN is up. Verified
end to end off-mesh: joined as `100.64.0.10`, answered ping+ssh, closed cleanly.
- **FV firewall config in the nightly restic run** via ana-docker's pre-backup hook
(`80d982d`); `infra-ops` now has an SSH key + `/bin/sh` on the gateway, so the
allowlisted WAN path gives a shell rather than API-only.
### Fleet hygiene landed
`root:docker 2775` setgid deploy trees on all 5 hosts + `0777` cleared (`826a63b`);
conventions pinned in `docs/pfi/fleet-conventions.md` with a read-only audit playbook
(`abef67a`); `linus` deleted and `llmuser` de-privileged (`ce7b07f`); nh3-dev retired from the mesh and its masquerade
exception reverted with it (`9dbd829` superseded); ESPHome pinned/relocated/rotated
(`d1769ed` ff); `kb` KB-search tool shipped (`68fa80f`).
### Also landed this session
`root:docker 2775` setgid deploy trees on 5 hosts + `0777` cleared (`826a63b`);
conventions pinned in `docs/pfi/fleet-conventions.md` + audit playbook (`abef67a`);
`linus` deleted, `llmuser` de-privileged (`ce7b07f`); FV firewall config in the nightly
restic run (`80d982d`); `infra-ops` SSH key on the OPNsense gateway; ESPHome pinned and
hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`).
### Open loose ends (none blocking)
- ✅ **Mesh membership retired for both nh3-dev and fv-ml1** (2026-09-15). Six nodes remain:
the three site routers, vb-gateway, irv-ml1 (Irvine's own router) and the MacBook Air.
Every one of those has a job; nothing is enrolled "just in case" any more.
- ✅ **Stale reusable pre-auth keys retired** (IDs 5, 6 expired). The mesh now has exactly
one live reusable key: the dedicated fv-ml1 break-glass key, ID 8, expires 2027-09-15,
vaulted `fv-ml1/headscale-breakglass-key`.
- **FV WAN admin is plain HTTP**, no TLS. Allowlisted to ESH/ANA/NH3 egress — ⚠ pinned to
**egress IPs**, so a WAN change at any site silently removes the fallback.
- **Legacy `/32` ANA NAT rule** redundant and invisible to `source_nat/search_rule`;
delete from the UI.
- **`10.251.250.0/24` (BMC/mgmt) is not covered** by the FV SNAT rules — inbound works.
- **No headscale ACL policy loaded** → default-allow across the mesh.
- **nh3-dev has no break-glass watchdog, by choice** — NH3 is a populated site with other
boxes to diagnose from, unlike FV where a single gateway is the only path. Reversible:
the script is in `servers/fv-ml1/`.
- **Legacy `/32` ANA NAT rule** redundant and invisible to `source_nat/search_rule` —
delete from the OPNsense UI.
- **FV WAN admin is plain HTTP**, allowlisted to ESH/ANA/NH3 — ⚠ pinned to **egress IPs**,
so a WAN change at any site silently removes the fallback.
- **`10.251.250.0/24` (BMC/mgmt)** outside the FV SNAT scope — inbound works.
- **No headscale ACL policy loaded** → mesh is default-allow.
- **Prior-session items still open:** FV site-visit power measurements in
`docs/runbooks/fv-site-dark-20260913.md`; mog-sec blue-team seat fate; deferred park items.
- ⚠ **fv-ml1 seat topology unchanged** from the 2026-09-14 rebalance — this session touched
no seats.
## Recent decisions
- `[2026-09-15]` **Mesh membership retired for fv-ml1 and nh3-dev — six nodes left, each with a job.** fv-ml1 gets break-glass rejoin instead of standing membership; nh3-dev's retirement also removed the nh3-scale masquerade exception it had required. Exactly one live reusable pre-auth key remains fleet-wide. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md`
- `[2026-09-15]` **FV cross-site routing fixed — one OPNsense outbound-NAT rule had been scoped to Anaheim only.** fv-ml1 now reaches NH3/ESH/IRV/ANA/mesh/internet; four rules, all `src=10.251.50.0/24`. The diagnostic signature is the valuable part: every layer looks correct and the discriminator is that *every other site pair works*. → `persistent-memory.d/2026-09-15-fv-cross-site-snat.md`
- `[2026-09-15]` **Break-glass mesh path on fv-ml1** — inverted from a restore-watchdog on the operator's suggestion: the box is OFF the mesh and the watchdog JOINS it on fleet loss. Exposed a rejoin key expiring in 4 days; replaced with a dedicated 1-year key and the two stale reusable keys retired. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md`