memory: snapshot — 2026-10-03 in-flight (nh3-pve 8→9 under way), two-tier split of 27 entries, 7 archived

2026-10-03 13:36:42 -07:00
parent 4b2a81bf39
commit 5abd43e1c8
36 changed files with 465 additions and 336 deletions
@@ -13243,3 +13243,319 @@ ruled. heid also found and killed two live instructions in their own persistent-
 fresh session to dispatch `--groa-transport http`; either would have resumed a stopped experiment
 after a context reset.
  _Archived 2026-10-01._
 ## Recent decisions (archived 2026-10-03 batch)
 # [2026-09-18] NH3↔Anaheim was DERP-relayed, not direct — a UDP 41641 port-forward on ana-gw fixed it
 Every NH3→Anaheim flow had been crossing Tailscale's LA DERP relay rather than a
 direct path, long enough to have carried **78 GB tx on the NH3 side alone**. DERP
 is a deliberately throttled fallback — a reachability mechanism, not a data plane —
 so this imposed both a fixed round-trip penalty and a bandwidth ceiling on LiteLLM,
 Beszel, task-board, vor and the Henge alike.
 ## How it surfaced, and how it did NOT
 tts-dev reported ext-stt transcriptions 4–18× slower through the LiteLLM gateway
 than straight at the Parakeet host, with a ~1.9 s fixed floor plus a per-byte cost.
 They had already cleared GPU contention, the ASR itself and output quality with
 real controls.
 **No monitor caught it.** Beszel had every host green. Nothing on this fleet checks
 whether a mesh path is direct.
 ## The diagnosis, and two wrong causes published on the way
 Their hypothesis was a dead first target in a LiteLLM fallback list. Checked: `ext-stt`
 has exactly ONE deployment, no fallback, nothing to retry against. Langfuse looked like
 a great candidate for a fixed floor — env vars still set, container gone — but it was
 retired from the callback list 2026-06-20 and nothing calls it per-request.
 The decisive arm was one neither of us had run: **measure from ana-docker itself**, so
 both arms share the identical onward path to FV.
    ana-docker -> fv-ml1 DIRECT        0.289 / 0.218 / 0.273 / 0.220 s
    ana-docker -> gateway -> fv-ml1    0.463 / 0.211 / 0.222 / 0.222 s
 The gateway adds ~0. tts-dev measured 2618 ms for that clip; from the gateway's own
 host it is 220 ms. Their two arms differed in TWO things — whether LiteLLM was in the
 path AND which destination the packets crossed the network to reach — and the entire
 delta was attributed to the variable of interest.
    nh3-dev -> fv-ml1  direct        0.269 / 0.219 / 0.216 s
    nh3-dev -> ANA gateway           1.449 / 1.399 / 1.404 s
    nh3-scale$ tailscale ping 100.64.0.3
      pong via DERP(lax) in 507ms, 518, 521, 522, 373, 423, 464, 493
      direct connection not established
 ## Why hole-punching failed
 ana-scale advertised `38.120.12.42:41641`, but its netcheck mapped to `:60798` — the
 Anaheim NAT was not preserving the port — and `PortMapping` was empty, so no UPnP or
 NAT-PMP was establishing one. `MappingVariesByDestIP: false`, so not symmetric NAT;
 simply no reachable inbound endpoint. ESH↔Anaheim and NH3↔ESH were already direct;
 only this pair failed.
 ## The fix — four ADDITIVE objects on ana-gw (10.250.0.1)
 ⚠ **This box runs `central-nat enable`, so a policy's `dstaddr` is the REAL internal
 address and not the VIP.** The existing `wg-to-ana-wg` VIP+policy pair is the house
 template; writing what a FortiGate normally wants would have failed.
 | object | value |
 |---|---|
 | `firewall address` | `ana-scale-ip` → 10.250.50.45/32, iface `servers` |
 | `firewall service custom` | `Tailscale-41641` → `udp-portrange 41641` |
 | `firewall vip` | `tailscale-to-ana-scale` → 38.120.12.42:41641 udp → 10.250.50.45:41641, extintf wan1 |
 | `firewall policy` id 75 | wan1→servers, all→`ana-scale-ip`, `Tailscale-41641`, accept |
 ## Measured before → after
 | path | before | after |
 |---|---|---|
 | `tailscale ping` nh3-scale→ana-scale | 373–522 ms via DERP(lax) | **6 ms direct** |
 | STT via the ANA gateway (96 kB clip) | 1.399–1.449 s | **0.237–0.270 s** |
 | Beszel HTTP nh3-dev→ana-docker | 0.94–1.29 s | **0.014–0.016 s** |
 tts-dev confirmed independently from talk's own vantage: 2618 → 249 ms and 4119 →
 548 ms, converged with their direct arm.
 ## Operational notes for next time
 ⚠ **This edge has NO out-of-band access.** Back up first — `ssh infra-ops@10.250.0.1
 'show' > file` captures the full non-default config (11,320 lines) without needing a
 tftp server; kept at `~/backups/ana-gw-config-BEFORE-<stamp>.txt`. Credentials vaulted
 at `fortigate/ana-gw-infra-ops-password`, reached with
 `sshpass -e ssh -o PubkeyAuthentication=no`. **Additive objects only** — never edit an
 existing policy or interface over a 327 ms link you cannot recover.
 ⚠ **irv-ml1 (100.64.0.6) remains `relay "lax"`.** Same class, different site, not fixed
 by this. Measured 24-25 ms on HTTP from nh3-dev, so it is not costing what Anaheim was.
 I warned tts-dev their TTS stack was paying for it; they measured and I was wrong.
 Documented in `docs/pfi/headscale-mesh-plan.md`; commit `5a9fad8`. See
 [[2026-09-18-fleet-dns-ring-and-ratelimit]] — the ANA AdGuard only became a viable
 cross-site DNS backup because of this fix.
  _Archived 2026-10-03._
 # [2026-09-18] `.internal` DNS was failing ~10% of lookups — two causes, both fleet-wide
 Two independent faults, fixed in order. Together they were costing roughly one
 `.internal` lookup in ten either a hard failure or a five-second stall, on every
 DHCP client at every site.
 ## Fault 1 — a PUBLIC resolver was the fallback for a PRIVATE zone
 DHCP handed out `10.100.50.40, 1.1.1.1` on both NH3 VLANs. When the internal
 resolver missed a packet, the resolver waited out its timeout, fell through to
 Cloudflare, and got an **authoritative NXDOMAIN** — so a transient miss became a
 definitive "no such host" rather than a retry. Binary failure: instant, or five
 seconds then `gaierror`.
 **Anaheim was worse.** Its FortiGate handed clients *itself* as resolver and
 forwarded to `1.1.1.1`/`1.0.0.1`, so **no ANA host could resolve `.internal` at
 all** — `ana-docker`, which HOSTS the Anaheim AdGuard, had `1.1.1.1` in its own
 `resolv.conf`.
 ⚠ **`/etc/resolv.conf` on these hosts is DHCP-managed** (dhclient on ens18). Hand
 editing it survives exactly until the next lease renewal — tts-dev's original
 framing was "fix nh3-dev's resolv.conf", which would have worked, been verified,
 and then silently reverted. Change it at the source:
 - **UniFi**: `dhcpd_dns_1` / `dhcpd_dns_2` per network object.
 - **FortiGate**: `config system dhcp server`, `set dns-service specify` +
  `dns-server1` / `dns-server2`.
 - Then `sudo dhclient -1 -v ens18` to pick it up **without releasing** the lease.
 **Operator's ruling was broader than the proposal**: each site's backup resolver
 should be *another site's* resolver. ESH already worked this way; the other two
 were set, and the ring closes.
    ESH  primary 10.0.50.45    backup 10.100.50.40  (NH3)   [pre-existing]
    NH3  primary 10.100.50.40  backup 10.250.50.70  (ANA)   [changed]
    ANA  primary 10.250.50.70  backup 10.0.50.45    (ESH)   [changed]
 All six members verified to answer the private zone. ⚠ The SureFire tenant DHCP
 server (`sfsrv`) and the fortilink switch-management server were deliberately left
 alone — tenant property.
 **Result: hard failures 3/40 → 0/40. The 5 s stalls remained.**
 ## Fault 2 — THE ROOT CAUSE: AdGuard rate-limiting, identical on all three
    ratelimit: 20
    ratelimit_subnet_len_ipv4: 24
    ratelimit_whitelist: []
 **20 queries per second shared across an entire /24 — per subnet, not per client.**
 Every host on a VLAN draws from one bucket, so one busy container starves the rest,
 and anything over the line is **silently dropped**, costing the client its full 5 s
 resolver timeout.
 Demonstrated rather than inferred — 60 concurrent queries at one resolver:
 | | before | after |
 |---|---|---|
 | median | 5044 ms | 72 ms |
 | timed out (>4 s) | **40 of 60** | **0 of 60** |
 Set to `ratelimit: 0` on all three. These are LAN-only resolvers behind three
 firewalls; that setting exists to blunt internet-side DNS amplification, which is
 not this.
 ⚠ **Restarting AdGuard is required for a config change and briefly drops that site's
 primary resolver — do the three ONE AT A TIME** so the cross-site ring always has a
 live member. That the ring made this rollout safe is the point of having built it
 first, an hour earlier.
 ## End-to-end, the original symptom
 40 TCP connects to a service by name vs by IP; the by-IP arm is the control.
 | | original | after ring only | after ratelimit 0 |
 |---|---|---|---|
 | by NAME p90 | 120 ms | 5017 ms | **13.5 ms** |
 | by NAME >1 s | 3/40 | 5/40 | **0/40** |
 | by NAME failed | 2/40 | 0/40 | **0/40** |
 | by IP p90 (control) | 25 ms | — | 11.7 ms |
 tts-dev confirmed from talk's own request path: name and IP **1.5 ms apart at the
 median**. Resolution has stopped being a cost rather than become a smaller one.
 ## What this says about monitoring
 Nothing on this fleet watches DNS success rate. The fleet's actual DNS monitor
 today was a voice loop with a 1545 ms budget whose owner chose to measure instead
 of adapting around the problem. Both faults were invisible to Beszel and Uptime
 Kuma — every host was up, every service answered.
 Related: [[2026-09-18-nh3-ana-derp-relay]] — the ANA AdGuard is only a sensible
 cross-site backup because that fix took NH3→ANA from ~1.2 s to ~15 ms the same
 afternoon. The two changes compose.
  _Archived 2026-10-03._
 # [2026-09-18] FleetTools — one capability index, autoloaded by three agent families
 Every agent on nh3-dev needs the same answers — what runs here, how do I call it,
 what will bite me — but that knowledge lived in `~/.claude/CLAUDE.md`, which only
 Claude sessions load, interleaved with operator preferences other families have no
 use for.
 ## Shape
 **Two-tier, matching the persistent-memory split.** `FLEETTOOLS.md` is a ~135-line
 index an agent reads whole; each entry links to a detail file it opens only when it
 needs that tool. Reading the index costs about a fifth of reading the tree.
 - Canonical: `docs/fleettools/` in this repo (git-tracked)
 - Discovery: `~/FLEETTOOLS.md` → symlink to the index
 - **Detail links are ABSOLUTE paths**, not relative markdown — a non-Claude agent
  `cat`s the path rather than following a link, and may be anywhere on the filesystem.
 Covers: althing, the Booth, `secret`, LiteLLM, inference seats + Asset Engine,
 speech, Arbo, elway, fleet SSH, observability, graphify, Playwright, the Henge, and
 a segregated Claude-only file (ratecheck, remote-ssh MCP, task-board, skills).
 ## Autoload — verified per family, not assumed
 - **Codex** reads `$CODEX_HOME/AGENTS.md` — the binary carries the string
  `"Failed to read global AGENTS.md instructions from"`.
 - **Grok** always scans `$GROK_HOME/rules/` and loads **every `*.md` in it regardless
  of name** — from its own embedded docs.
 Both locations were empty. `AGENT-BOOTSTRAP.md` is **symlinked** into each rather than
 copied — one file, three families, no drift surface.
 ⚠ Grok also scans `~/.claude/rules/` and recognises `CLAUDE.md` via harness-compat
 settings. That directory is empty today, but Grok can see Claude's config.
 ## Rule zero, which is the load-bearing part
 **Query live inventories, never a written list.** The index points at
 Homepage `/api/services`, asset-engine `/api/v1/services`, LiteLLM `/v1/models`, and
 every FastAPI seat's `/openapi.json`. A copied service table would be stale within a
 month — the LiteLLM roster was already at 40 models where the old CLAUDE.md note said
 ~30.
 ## The CLAUDE.md swap
 `~/.claude/CLAUDE.md`'s "Global tools available" section went from **231 lines to 39**
 (file 1218 → 1026, a 16% cut to what every Claude session loads). Three things stayed
 inline deliberately because they govern BEHAVIOUR rather than lookup: run `ratecheck`
 before expensive work, post operator-facing links to the Booth board, and the vault is
 the credential source of truth. **A rule behind a file read is a rule that stops
 firing.** Backup at `~/.claude/CLAUDE.md.bak-20260918-074048`.
 The shared all-agents LiteLLM key stays inline there too — every session needs it and a
 vault round-trip measured over two minutes — but it is now ALSO vaulted at
 `litellm/all-agents-shared-key`, since it had been single-copy.
 Commits `53c3e80` `21d24c5`.
  _Archived 2026-10-03._
 # `[2026-09-18]` Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy
 **Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy** (operator-approved). Verified empirically — `is_dir()`/`exists()` are stat-based and follow symlinks — and the copy had ALREADY drifted (README, since 09-15). Pre-restart gate is `uv run python scripts/sync_plugin.py --check`; `hermes plugins list` is confirmation-after, never permission-before. Copy kept at `~/backups/svos_miranda-copy-20260918-1444`. ⚠ The install now tracks a working tree — an uncommitted edit in svos is what loads at the next restart.
  _Archived 2026-10-03._
 # `[2026-09-18]` Worldtree's `env.sh` secrets are vaulted
 **Worldtree's `env.sh` secrets are vaulted** — 10 entries under `worldtree/` (gitea, matrix as/hs, openai, uv-index, vastblueai, wt-admin demo+personal, z-ai, zai). ⚠ `Z_AI_API_KEY` and `ZAI_API_KEY` are DIFFERENT keys despite the near-identical names (fp `26bde3fa` vs `a4884152`). `ANTHROPIC_API_KEY` was empty and skipped.
  _Archived 2026-10-03._
 ## Tried and abandoned (archived 2026-10-03 batch)
 # [2026-09-18] SearXNG's ESH SOCKS5 egress — one day live, reverted, and it fixed nothing
 On 2026-09-17, at operator request, SearXNG's search egress was moved from
 nh3-docker's direct residential path to `socks5h://10.0.50.65:1080` on **esh-scale**
 (CT 108 on esh-pve) — an application-level proxy, no host route or exit-node change.
 Reverted 2026-09-18.
 ## Why it was reverted, and why the STATED reason was wrong
 It was reverted on a diagnosis that turned out to be false: that the move had cost
 three of four search engines to CAPTCHAs. Reverting to direct NH3 egress produced a
 **byte-identical** result — same three engines down — which falsified it. A live
 `!ddg` bang probe on a freshly restarted container also CAPTCHA'd, ruling out stale
 suspension timers too.
 The real cause was elsewhere (see [[2026-09-18-searxng-one-engine-to-seven]]).
 **The revert still stands on its own merits**: ESH egress bought no measurable
 improvement while making all fleet search depend on ESH WAN and mesh availability.
 The simpler configuration is the better one. It is simply not the fix for the engines.
 ## What was built, because it was built correctly
 The proxy side was not at fault and is worth keeping as a pattern. `microsocks` ran
 as `nobody` under `searxng-egress.service`, bound `10.0.50.65:1080` only, and allowed
 source `10.100.50.40` alone — everything else had to supply a password regenerated at
 each start and never distributed. Allowed-host egress and denied-host rejection were
 both tested. The unit is kept at `configs/esh-scale/searxng-egress.service`; the
 service is **stopped and disabled** on esh-scale, and `tailscaled` there was not
 touched (CT 108 is ESH's whole-site mesh SPOF).
 Measured egress was `128.177.138.182` in use and `154.50.58.126` at cutover — ESH's
 WAN address moves and nothing pins it, so the number in the stack README is historical.
 ## The transferable lesson
 ⚠ **I asserted causation from correlation with no baseline.** The only evidence that
 residential egress avoided CAPTCHAs was a config comment dated 2026-09-03, which was
 no longer true of that address. Five samples of the post-change state and zero of the
 working state is not a comparison. The rollback WAS the counterfactual, and it
 falsified the claim I had already published in a commit message.
 This was the first of three wrong causal attributions in a single afternoon. The
 common shape: measure after a change, attribute the delta to *my* change, never check
 what else moved.
 Commits `156e126` (applied), `1a35181` (reverted).
  _Archived 2026-10-03._
 # `[2026-09-18]` `api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings
 **`api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings** — this build has NO `!ENV` YAML constructor, so the file was unparseable and the container crash-looped ten times with fleet search down. There is no env-var path into that file at all: the loader reads only `SEARXNG_SETTINGS_PATH` and the entrypoint substitutes only `ultrasecretkey`. Literal or nothing.
  _Archived 2026-10-03._
@@ -0,0 +1,3 @@
 # `nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction
 `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction.** brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`.
@@ -1,3 +0,0 @@
 # `[2026-09-18]` `api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings
 **`api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings** — this build has NO `!ENV` YAML constructor, so the file was unparseable and the container crash-looped ten times with fleet search down. There is no env-var path into that file at all: the loader reads only `SEARXNG_SETTINGS_PATH` and the entrypoint substitutes only `ultrasecretkey`. Literal or nothing.
@@ -1,94 +0,0 @@
 # [2026-09-18] `.internal` DNS was failing ~10% of lookups — two causes, both fleet-wide
 Two independent faults, fixed in order. Together they were costing roughly one
 `.internal` lookup in ten either a hard failure or a five-second stall, on every
 DHCP client at every site.
 ## Fault 1 — a PUBLIC resolver was the fallback for a PRIVATE zone
 DHCP handed out `10.100.50.40, 1.1.1.1` on both NH3 VLANs. When the internal
 resolver missed a packet, the resolver waited out its timeout, fell through to
 Cloudflare, and got an **authoritative NXDOMAIN** — so a transient miss became a
 definitive "no such host" rather than a retry. Binary failure: instant, or five
 seconds then `gaierror`.
 **Anaheim was worse.** Its FortiGate handed clients *itself* as resolver and
 forwarded to `1.1.1.1`/`1.0.0.1`, so **no ANA host could resolve `.internal` at
 all** — `ana-docker`, which HOSTS the Anaheim AdGuard, had `1.1.1.1` in its own
 `resolv.conf`.
 ⚠ **`/etc/resolv.conf` on these hosts is DHCP-managed** (dhclient on ens18). Hand
 editing it survives exactly until the next lease renewal — tts-dev's original
 framing was "fix nh3-dev's resolv.conf", which would have worked, been verified,
 and then silently reverted. Change it at the source:
 - **UniFi**: `dhcpd_dns_1` / `dhcpd_dns_2` per network object.
 - **FortiGate**: `config system dhcp server`, `set dns-service specify` +
  `dns-server1` / `dns-server2`.
 - Then `sudo dhclient -1 -v ens18` to pick it up **without releasing** the lease.
 **Operator's ruling was broader than the proposal**: each site's backup resolver
 should be *another site's* resolver. ESH already worked this way; the other two
 were set, and the ring closes.
    ESH  primary 10.0.50.45    backup 10.100.50.40  (NH3)   [pre-existing]
    NH3  primary 10.100.50.40  backup 10.250.50.70  (ANA)   [changed]
    ANA  primary 10.250.50.70  backup 10.0.50.45    (ESH)   [changed]
 All six members verified to answer the private zone. ⚠ The SureFire tenant DHCP
 server (`sfsrv`) and the fortilink switch-management server were deliberately left
 alone — tenant property.
 **Result: hard failures 3/40 → 0/40. The 5 s stalls remained.**
 ## Fault 2 — THE ROOT CAUSE: AdGuard rate-limiting, identical on all three
    ratelimit: 20
    ratelimit_subnet_len_ipv4: 24
    ratelimit_whitelist: []
 **20 queries per second shared across an entire /24 — per subnet, not per client.**
 Every host on a VLAN draws from one bucket, so one busy container starves the rest,
 and anything over the line is **silently dropped**, costing the client its full 5 s
 resolver timeout.
 Demonstrated rather than inferred — 60 concurrent queries at one resolver:
 | | before | after |
 |---|---|---|
 | median | 5044 ms | 72 ms |
 | timed out (>4 s) | **40 of 60** | **0 of 60** |
 Set to `ratelimit: 0` on all three. These are LAN-only resolvers behind three
 firewalls; that setting exists to blunt internet-side DNS amplification, which is
 not this.
 ⚠ **Restarting AdGuard is required for a config change and briefly drops that site's
 primary resolver — do the three ONE AT A TIME** so the cross-site ring always has a
 live member. That the ring made this rollout safe is the point of having built it
 first, an hour earlier.
 ## End-to-end, the original symptom
 40 TCP connects to a service by name vs by IP; the by-IP arm is the control.
 | | original | after ring only | after ratelimit 0 |
 |---|---|---|---|
 | by NAME p90 | 120 ms | 5017 ms | **13.5 ms** |
 | by NAME >1 s | 3/40 | 5/40 | **0/40** |
 | by NAME failed | 2/40 | 0/40 | **0/40** |
 | by IP p90 (control) | 25 ms | — | 11.7 ms |
 tts-dev confirmed from talk's own request path: name and IP **1.5 ms apart at the
 median**. Resolution has stopped being a cost rather than become a smaller one.
 ## What this says about monitoring
 Nothing on this fleet watches DNS success rate. The fleet's actual DNS monitor
 today was a voice loop with a 1545 ms budget whose owner chose to measure instead
 of adapting around the problem. Both faults were invisible to Beszel and Uptime
 Kuma — every host was up, every service answered.
 Related: [[2026-09-18-nh3-ana-derp-relay]] — the ANA AdGuard is only a sensible
 cross-site backup because that fix took NH3→ANA from ~1.2 s to ~15 ms the same
 afternoon. The two changes compose.
@@ -1,57 +0,0 @@
 # [2026-09-18] FleetTools — one capability index, autoloaded by three agent families
 Every agent on nh3-dev needs the same answers — what runs here, how do I call it,
 what will bite me — but that knowledge lived in `~/.claude/CLAUDE.md`, which only
 Claude sessions load, interleaved with operator preferences other families have no
 use for.
 ## Shape
 **Two-tier, matching the persistent-memory split.** `FLEETTOOLS.md` is a ~135-line
 index an agent reads whole; each entry links to a detail file it opens only when it
 needs that tool. Reading the index costs about a fifth of reading the tree.
 - Canonical: `docs/fleettools/` in this repo (git-tracked)
 - Discovery: `~/FLEETTOOLS.md` → symlink to the index
 - **Detail links are ABSOLUTE paths**, not relative markdown — a non-Claude agent
  `cat`s the path rather than following a link, and may be anywhere on the filesystem.
 Covers: althing, the Booth, `secret`, LiteLLM, inference seats + Asset Engine,
 speech, Arbo, elway, fleet SSH, observability, graphify, Playwright, the Henge, and
 a segregated Claude-only file (ratecheck, remote-ssh MCP, task-board, skills).
 ## Autoload — verified per family, not assumed
 - **Codex** reads `$CODEX_HOME/AGENTS.md` — the binary carries the string
  `"Failed to read global AGENTS.md instructions from"`.
 - **Grok** always scans `$GROK_HOME/rules/` and loads **every `*.md` in it regardless
  of name** — from its own embedded docs.
 Both locations were empty. `AGENT-BOOTSTRAP.md` is **symlinked** into each rather than
 copied — one file, three families, no drift surface.
 ⚠ Grok also scans `~/.claude/rules/` and recognises `CLAUDE.md` via harness-compat
 settings. That directory is empty today, but Grok can see Claude's config.
 ## Rule zero, which is the load-bearing part
 **Query live inventories, never a written list.** The index points at
 Homepage `/api/services`, asset-engine `/api/v1/services`, LiteLLM `/v1/models`, and
 every FastAPI seat's `/openapi.json`. A copied service table would be stale within a
 month — the LiteLLM roster was already at 40 models where the old CLAUDE.md note said
 ~30.
 ## The CLAUDE.md swap
 `~/.claude/CLAUDE.md`'s "Global tools available" section went from **231 lines to 39**
 (file 1218 → 1026, a 16% cut to what every Claude session loads). Three things stayed
 inline deliberately because they govern BEHAVIOUR rather than lookup: run `ratecheck`
 before expensive work, post operator-facing links to the Booth board, and the vault is
 the credential source of truth. **A rule behind a file read is a rule that stops
 firing.** Backup at `~/.claude/CLAUDE.md.bak-20260918-074048`.
 The shared all-agents LiteLLM key stays inline there too — every session needs it and a
 vault round-trip measured over two minutes — but it is now ALSO vaulted at
 `litellm/all-agents-shared-key`, since it had been single-copy.
 Commits `53c3e80` `21d24c5`.
@@ -1,3 +0,0 @@
 # `[2026-09-18]` Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy
 **Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy** (operator-approved). Verified empirically — `is_dir()`/`exists()` are stat-based and follow symlinks — and the copy had ALREADY drifted (README, since 09-15). Pre-restart gate is `uv run python scripts/sync_plugin.py --check`; `hermes plugins list` is confirmation-after, never permission-before. Copy kept at `~/backups/svos_miranda-copy-20260918-1444`. ⚠ The install now tracks a working tree — an uncommitted edit in svos is what loads at the next restart.
@@ -1,91 +0,0 @@
 # [2026-09-18] NH3↔Anaheim was DERP-relayed, not direct — a UDP 41641 port-forward on ana-gw fixed it
 Every NH3→Anaheim flow had been crossing Tailscale's LA DERP relay rather than a
 direct path, long enough to have carried **78 GB tx on the NH3 side alone**. DERP
 is a deliberately throttled fallback — a reachability mechanism, not a data plane —
 so this imposed both a fixed round-trip penalty and a bandwidth ceiling on LiteLLM,
 Beszel, task-board, vor and the Henge alike.
 ## How it surfaced, and how it did NOT
 tts-dev reported ext-stt transcriptions 4–18× slower through the LiteLLM gateway
 than straight at the Parakeet host, with a ~1.9 s fixed floor plus a per-byte cost.
 They had already cleared GPU contention, the ASR itself and output quality with
 real controls.
 **No monitor caught it.** Beszel had every host green. Nothing on this fleet checks
 whether a mesh path is direct.
 ## The diagnosis, and two wrong causes published on the way
 Their hypothesis was a dead first target in a LiteLLM fallback list. Checked: `ext-stt`
 has exactly ONE deployment, no fallback, nothing to retry against. Langfuse looked like
 a great candidate for a fixed floor — env vars still set, container gone — but it was
 retired from the callback list 2026-06-20 and nothing calls it per-request.
 The decisive arm was one neither of us had run: **measure from ana-docker itself**, so
 both arms share the identical onward path to FV.
    ana-docker -> fv-ml1 DIRECT        0.289 / 0.218 / 0.273 / 0.220 s
    ana-docker -> gateway -> fv-ml1    0.463 / 0.211 / 0.222 / 0.222 s
 The gateway adds ~0. tts-dev measured 2618 ms for that clip; from the gateway's own
 host it is 220 ms. Their two arms differed in TWO things — whether LiteLLM was in the
 path AND which destination the packets crossed the network to reach — and the entire
 delta was attributed to the variable of interest.
    nh3-dev -> fv-ml1  direct        0.269 / 0.219 / 0.216 s
    nh3-dev -> ANA gateway           1.449 / 1.399 / 1.404 s
    nh3-scale$ tailscale ping 100.64.0.3
      pong via DERP(lax) in 507ms, 518, 521, 522, 373, 423, 464, 493
      direct connection not established
 ## Why hole-punching failed
 ana-scale advertised `38.120.12.42:41641`, but its netcheck mapped to `:60798` — the
 Anaheim NAT was not preserving the port — and `PortMapping` was empty, so no UPnP or
 NAT-PMP was establishing one. `MappingVariesByDestIP: false`, so not symmetric NAT;
 simply no reachable inbound endpoint. ESH↔Anaheim and NH3↔ESH were already direct;
 only this pair failed.
 ## The fix — four ADDITIVE objects on ana-gw (10.250.0.1)
 ⚠ **This box runs `central-nat enable`, so a policy's `dstaddr` is the REAL internal
 address and not the VIP.** The existing `wg-to-ana-wg` VIP+policy pair is the house
 template; writing what a FortiGate normally wants would have failed.
 | object | value |
 |---|---|
 | `firewall address` | `ana-scale-ip` → 10.250.50.45/32, iface `servers` |
 | `firewall service custom` | `Tailscale-41641` → `udp-portrange 41641` |
 | `firewall vip` | `tailscale-to-ana-scale` → 38.120.12.42:41641 udp → 10.250.50.45:41641, extintf wan1 |
 | `firewall policy` id 75 | wan1→servers, all→`ana-scale-ip`, `Tailscale-41641`, accept |
 ## Measured before → after
 | path | before | after |
 |---|---|---|
 | `tailscale ping` nh3-scale→ana-scale | 373–522 ms via DERP(lax) | **6 ms direct** |
 | STT via the ANA gateway (96 kB clip) | 1.399–1.449 s | **0.237–0.270 s** |
 | Beszel HTTP nh3-dev→ana-docker | 0.94–1.29 s | **0.014–0.016 s** |
 tts-dev confirmed independently from talk's own vantage: 2618 → 249 ms and 4119 →
 548 ms, converged with their direct arm.
 ## Operational notes for next time
 ⚠ **This edge has NO out-of-band access.** Back up first — `ssh infra-ops@10.250.0.1
 'show' > file` captures the full non-default config (11,320 lines) without needing a
 tftp server; kept at `~/backups/ana-gw-config-BEFORE-<stamp>.txt`. Credentials vaulted
 at `fortigate/ana-gw-infra-ops-password`, reached with
 `sshpass -e ssh -o PubkeyAuthentication=no`. **Additive objects only** — never edit an
 existing policy or interface over a 327 ms link you cannot recover.
 ⚠ **irv-ml1 (100.64.0.6) remains `relay "lax"`.** Same class, different site, not fixed
 by this. Measured 24-25 ms on HTTP from nh3-dev, so it is not costing what Anaheim was.
 I warned tts-dev their TTS stack was paying for it; they measured and I was wrong.
 Documented in `docs/pfi/headscale-mesh-plan.md`; commit `5a9fad8`. See
 [[2026-09-18-fleet-dns-ring-and-ratelimit]] — the ANA AdGuard only became a viable
 cross-site DNS backup because of this fix.
@@ -1,47 +0,0 @@
 # [2026-09-18] SearXNG's ESH SOCKS5 egress — one day live, reverted, and it fixed nothing
 On 2026-09-17, at operator request, SearXNG's search egress was moved from
 nh3-docker's direct residential path to `socks5h://10.0.50.65:1080` on **esh-scale**
 (CT 108 on esh-pve) — an application-level proxy, no host route or exit-node change.
 Reverted 2026-09-18.
 ## Why it was reverted, and why the STATED reason was wrong
 It was reverted on a diagnosis that turned out to be false: that the move had cost
 three of four search engines to CAPTCHAs. Reverting to direct NH3 egress produced a
 **byte-identical** result — same three engines down — which falsified it. A live
 `!ddg` bang probe on a freshly restarted container also CAPTCHA'd, ruling out stale
 suspension timers too.
 The real cause was elsewhere (see [[2026-09-18-searxng-one-engine-to-seven]]).
 **The revert still stands on its own merits**: ESH egress bought no measurable
 improvement while making all fleet search depend on ESH WAN and mesh availability.
 The simpler configuration is the better one. It is simply not the fix for the engines.
 ## What was built, because it was built correctly
 The proxy side was not at fault and is worth keeping as a pattern. `microsocks` ran
 as `nobody` under `searxng-egress.service`, bound `10.0.50.65:1080` only, and allowed
 source `10.100.50.40` alone — everything else had to supply a password regenerated at
 each start and never distributed. Allowed-host egress and denied-host rejection were
 both tested. The unit is kept at `configs/esh-scale/searxng-egress.service`; the
 service is **stopped and disabled** on esh-scale, and `tailscaled` there was not
 touched (CT 108 is ESH's whole-site mesh SPOF).
 Measured egress was `128.177.138.182` in use and `154.50.58.126` at cutover — ESH's
 WAN address moves and nothing pins it, so the number in the stack README is historical.
 ## The transferable lesson
 ⚠ **I asserted causation from correlation with no baseline.** The only evidence that
 residential egress avoided CAPTCHAs was a config comment dated 2026-09-03, which was
 no longer true of that address. Five samples of the post-change state and zero of the
 working state is not a comparison. The rollback WAS the counterfactual, and it
 falsified the claim I had already published in a commit message.
 This was the first of three wrong causal attributions in a single afternoon. The
 common shape: measure after a change, attribute the delta to *my* change, never check
 what else moved.
 Commits `156e126` (applied), `1a35181` (reverted).
@@ -1,3 +0,0 @@
 # `[2026-09-18]` Worldtree's `env.sh` secrets are vaulted
 **Worldtree's `env.sh` secrets are vaulted** — 10 entries under `worldtree/` (gitea, matrix as/hs, openai, uv-index, vastblueai, wt-admin demo+personal, z-ai, zai). ⚠ `Z_AI_API_KEY` and `ZAI_API_KEY` are DIFFERENT keys despite the near-identical names (fp `26bde3fa` vs `a4884152`). `ANTHROPIC_API_KEY` was empty and skipped.
@@ -0,0 +1,3 @@
 # `headscale-ddns` exited 1 silently and the alarm carried no cause
 `[2026-09-22]` **`headscale-ddns` exited 1 silently and the alarm carried no cause** — both failure paths were `|| exit 1` with no message. Now every exit names its reason and the WAN lookup retries 3×. The script was also untracked; a fix to what every mesh client resolves through lived on one disk. `30517fd`.
@@ -0,0 +1,3 @@
 # A TCP connect as the "is Blender ready" probe
 `[2026-09-27]` **A TCP connect as the "is Blender ready" probe.** docker-proxy accepts on a published port before the app behind it listens, so it said "answering" while Blender was still loading. The probe now asks the add-on to `ping`. The same trap applies to any service behind a published port.
@@ -0,0 +1,3 @@
 # hermes-gateway restarted 0401 for highseat-dev
 `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** (SVOS v2.1.12: `propose_decision` gained `seat_up`, and Hermes reads the plugin only at start). The plugin load was verified at file level; the end-to-end proof is Miranda's first seat_up card. Enabling `zellij-fleet@Claude` at boot remains Prime's call.
@@ -0,0 +1,3 @@
 # Relying on the build cache surviving on esh-ml1
 `[2026-09-27]` **Relying on the build cache surviving on esh-ml1** — the augaman v0.1.3 build missed its dependency layer there (fv-ml1 hit it; reason unexplained, suspect the `docker builder prune` runs) and the rootfs peaked at 90%. Check for ≥15 GB free, or remove the old image, before building there.
@@ -0,0 +1,3 @@
 # The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel
 `[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel.** Every tool worked except `get_viewport_screenshot`, which had Blender write a file for the SERVER to read ("Screenshot file was not created"). The server now runs inside the Blender container over ssh+docker-exec stdio. That also means no port is published. → `stacks/blender/README.md`
@@ -0,0 +1,3 @@
 # `uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project
 `[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** — the export drops uv's per-package index routing; with `--extra-index-url … --index-strategy unsafe-best-match`, triton came from the pytorch index and failed the lock's hash. Keep `uv sync` and blank the project version in a deps-only stage instead (`services/semif-serve/Dockerfile`).
@@ -0,0 +1,3 @@
 # Shorter Parakeet slices as Scriberr's memory fix
 `[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix.** I predicted ~6× less memory from the attention-matrix arithmetic. Measured, 300 → 120 s only went 9,384 → 6,510 MiB: a ~5.6 GB fixed floor dominates. `expandable_segments:True` was the real lever (5,496). Measure the process peak; never extrapolate it from one tensor.
@@ -0,0 +1,3 @@
 # ⚠ Gitea's `[webhook] ALLOWED_HOST_LIST = external, 10.0.0.0/8` silently REJECTS headscale mesh IPs (100.64.0.0/10)
 `[2026-10-01]` **⚠ Gitea's `[webhook] ALLOWED_HOST_LIST = external, 10.0.0.0/8` silently REJECTS headscale mesh IPs (100.64.0.0/10).** The test API still returns 204 and nothing arrives. The vh/arbo hook therefore targets irv-ml1's LAN `10.6.110.50:9009`; the secret was re-set and a delivery is verified (deploy ran 04:41). A Gitea webhook PATCH without `secret` and `branch_filter` drops both, so always resend them.
@@ -0,0 +1,3 @@
 # MIA (Make-It-Animatable v2) auto-rigger LIVE on fv-ml1 GPU 3, one-shot (Prime via dread-dev)
 `[2026-10-01]` **MIA (Make-It-Animatable v2) auto-rigger LIVE on fv-ml1 GPU 3, one-shot (Prime via dread-dev).** `scripts/mia-run --job DIR -- name=in.glb ...`, image `local/mia:0.1.0` (10.5 GB), weights in the shared HF cache at pinned revisions (sha256-verified). Acceptance: 3.9–4.7 s a mesh (median of 3), 12.7 s model load, peak 3,394 MiB; seeded runs bit-identical; GPU-vs-CPU distances the same size as sampling noise (positive control: unseeded GPU). Skeleton template is a SUBSTITUTE (gated HF dataset `jasongzy/Mixamo`, terms not accepted; Prime's call). ⚠ fv-ml1 zroot at 85% after the build. → `stacks/mia/README.md`
@@ -0,0 +1,3 @@
 # Prime: "clean up the 1700+ backups" — dev-backup retention FIXED, 1,740 husks removed
 `[2026-10-01]` **Prime: "clean up the 1700+ backups" — dev-backup retention FIXED, 1,740 husks removed.** They were never 1,788 real snapshots: the prune had deleted everything except one 0555 dir (`vastblue/praxis/references/PraxisPM_Rev0_07.11.26`, 17 files hardlinked into every snapshot), so each old dir was a 22-entry husk and only the newest 48 were complete. Removing them lost no history and freed ~0 bytes (same inodes, link count 1,788). Deletion ran from a generated list of 1,740 literal paths, each verified as a husk first, with 0 errors. The script now runs `chmod -R u+w` before `rm`, logs the error count and the kept count, and fails the unit (exit 3) on a bad prune or exit 1 on a failed rsync. Positive control: a manual run at 0556 pruned the full snapshot `2026-09-29_0602` with 0 errors, leaving 48. Retention stays the designed 48 hourly; the 4.6 GB log rotated at 0530 is kept, compressed to 99 MB, at `~/.config/dev-backup/dev-backup.log.20261001-0530.gz` until someone deletes it.
@@ -0,0 +1,3 @@
 # Prime: delete the bench leftovers, no upstream for Scriberr, push
 `[2026-10-01]` **Prime: delete the bench leftovers, no upstream for Scriberr, push.** DONE: 7 HF revisions deleted through huggingface_hub's cache API (25.1 GB; the parakeet 1.1B/ctc/v2 models and whisper-large-v3; **unified-en KEPT, the live seat mounts it**), plus `/tank/spikes/scriberr-slicer` (including the private copies of Prime's recordings) and `/tank/spikes/parakeet-ab`. The Scriberr upstream PR text was dropped (6b66207). Both repos pushed.
@@ -0,0 +1,3 @@
 # Size + mtime scans to pick reclaimable "staging" on a model store
 `[2026-10-01]` **Size + mtime scans to pick reclaimable "staging" on a model store** (infra-hermes, irv-ml1). `_inbound/retro-diffusion` looked like staging leftovers but is the TARGET of 47 model-store symlinks, holding a purchased bundle used by a live commission. Resolve symlinks INTO any staging dir before proposing to delete it.
@@ -0,0 +1,3 @@
 # vh/arbo push webhook repointed from the retired wg0 lifeline `10.100.79.3:9009` to irv-ml1's mesh address `100.64.0.6:9009`
 `[2026-10-01]` **vh/arbo push webhook repointed from the retired wg0 lifeline `10.100.79.3:9009` to irv-ml1's mesh address `100.64.0.6:9009`.** It had been dead since 09-06 (comfy-dev report). No other of the 109 repos' hooks pointed at 10.100.79.x. The Actions runner `irv-ml1-arbo` was fine; run 677 failed, likely colliding with a hand deploy. The secret was not resent: if the next push shows an HMAC rejection, re-set it.
@@ -0,0 +1,3 @@
 # albok-service 0.1.0 LIVE on nh3-docker (albok-dev ask, operator-approved): `http://albok.nh3.internal:8392`
 `[2026-10-02]` **albok-service 0.1.0 LIVE on nh3-docker (albok-dev ask, operator-approved): `http://albok.nh3.internal:8392`** (8390 there is the post office). Store/private on local ext4 under /srv/albok, uid 1500, read groups 1510/1511 (container gets a mounted /etc/group + group_add so its getgrnam/chgrp work). Scoped LiteLLM key `albok-service` (qwen3-embedding only) + bootstrap admin token in the vault under `albok/`. Upgraded to **0.1.1** (e349d50) at 1805: /health now `ok` (0.1.0's always-degraded canary bug fixed); albok-dev smoke-tested ingest/search/readback (token vaulted `albok/token-albok-dev`). → `stacks/albok-service/README.md`
@@ -0,0 +1,3 @@
 # Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684)
 `[2026-10-02]` **Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684).** Their b193 release commit c2d87263 swept 60 staged deletions into the tree, so the demo api crash-looped. I recreated `worldtree-api` only (`compose up -d --no-deps`) on the `.env`-pinned fa8bc51cc064 (compose.yaml identical at both shas): healthy in 35 s at 15:16Z. ⚠ Their deploy health gate does NOT restore the old container on failure (it only withholds `:latest`); flagged to them. Fix-forward 7ab6ae40 (tag v1.0.0b193 moved onto it) deployed via CI 15:22Z and verified healthy, same seven retired-key WARNINGs; the deploy-gate gap is Worldtree #423.
@@ -0,0 +1,3 @@
 # esh-pve-2 (MS-03 at ESH, future `esh-dev` host) onboarded: infra-ops (Prime), 1 TB → LVM-thin `vmstore`, AMT phoning home as `esh-pve-2-amt`
 `[2026-10-02]` **esh-pve-2 (MS-03 at ESH, future `esh-dev` host) onboarded: infra-ops (Prime), 1 TB → LVM-thin `vmstore`, AMT phoning home as `esh-pve-2-amt`.** The stale firmware boot entry for an old install on the 1 TB was deleted; that drive was wiped and pvesh-created as `vmstore` (913 GiB; `playbooks/esh-pve-2-disk-prep.yaml`); the reboot onto kernel 7.0.14-20 is verified. AMT 21 (ACM, same MEBx pw, vaulted `esh-pve-2/amt-admin`) shares nic1 (I226-LM), its MAC and its DHCP IP with the host: KVM on, OptIn 0, listener on, then `amt-cira-setup.py --apply`. ⚠ Its FIRST CIRA connect crashed MeshCentral once (~7 s, agents back; MeshCentral 1.2.0 race in mpsserver.js, read at source); credentials were then picked up by dropping only its tunnel (`ss -K`), with no restart. Address 10.0.10.70 is TEMPORARY (Prime: can wait) and must become a reservation on MAC `38:05:25:3b:9c:12`. → `servers/esh-pve-2/README.md`, `servers/pfi-tacticalrmm/README.md`
@@ -0,0 +1,3 @@
 # ESH `tank` (esh-nas-pve, 175 T raidz2×2): was DEGRADED since ~2026-08-20 (found while planning PVE 9); HEALTHY since 2026-10-03 0629
 `[2026-10-02]` ✅ **ESH `tank` (esh-nas-pve, 175 T raidz2×2): was DEGRADED since ~2026-08-20 (found while planning PVE 9); HEALTHY since 2026-10-03 0629.** raidz2-0 disk `wwn-0x5000c500c91df554`: 36 CKSUM errors; Aug 20 resilver logged 6.68 M errors; no known data errors; SMART clean. **Prime (via Miranda) 1925: verify, then clear + full scrub.** Verified residual: the disk was REPLACED 2026-08-20 09:50 (zpool history), resilver ended with 6.68 M errors, and since the last clear only 21 checksum events (Aug 21 02:05 ×20, Aug 29 02:50 ×1), none in 34 days. `zpool clear` 1926 → ONLINE; full scrub running (background watcher); replace the disk if errors return. **UPDATE 2331: the scrub is REPAIRING that disk: 6.19 M CKSUM, 198 G repaired at 72%, 0 read/write errors, still no known data errors, ETA ~0107.** History explains it: after the Aug 20 replace, the resilver ended `errors=6676436`, someone ran only an error-scrub (`zpool scrub -e`, 4 s) and then `zpool clear` 19:32, so NO full scrub ran until today, and those blocks sat bad on the new disk for 6 weeks. SMART still clean (0 realloc/pending/uncorrect/CRC, no errors logged), no kernel I/O errors. Read: residual repair, not a failing disk (inferred). Discriminator: clear + a SECOND full scrub; any CKSUM on it → replace. ⚠ The watcher's error check was blind (could not parse `6.19M` as an integer); it still fires on completion. **0106 scrub DONE: repaired 198G, 0 errors, no known data errors; that disk 6,494,530 CKSUM exact (97% of the 6,676,436 the Aug 20 resilver failed on), last checksum ereport 21:26:27 = none in the scrub's final 3h40m.** **0118: SECOND full scrub started WITHOUT `zpool clear`**, so any new error shows as count > 6494530 (`zpool status -p`); watcher tested with +1/null/finished controls; ETA ~0700. **RESOLVED 0627: second scrub repaired 0B with 0 errors; the disk's CKSUM stayed at 6494530 (zero new); SMART clean; no kernel I/O errors. Verdict: residue of the Aug 20 resilver, now repaired; DISK KEPT. `zpool clear` 0629 → `pool 'tank' is healthy`.** Reported to Miranda once. **PVE 8→9 plan for esh-pve-cluster written, NOT executed** (Prime via Miranda): `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`. Also found: remote access to ESH rides CT 108 on `pve` (no remote recovery if it fails to boot); both nodes fail pve8to9 on the systemd-boot meta-package.
@@ -0,0 +1,3 @@
 # nh3-dev root grown 250 → 378 GB (Prime resized scsi0; I grew the guest online, no reboot)
 `[2026-10-02]` **nh3-dev root grown 250 → 378 GB (Prime resized scsi0; I grew the guest online, no reboot).** Root 372 GB, 58%, 150 GB free, after the 85% alert fired twice in 14 h (uv cache + agent venvs). Swap moved to a 4 GB `/swapfile` (the old `sda5` blocked growth), `RESUME=none`, all initrds rebuilt (`playbooks/nh3-dev-grow-root.yaml`). ⚠ **TODO: delete VM snapshot `pre-rootgrow-20261002` on nh3-pve after the next NATURAL reboot (Prime 2026-10-02: no test reboot).** Trigger: `uptime -s` on nh3-dev later than 2026-10-02 0733. Then check the boot was clean (`swapon --show` = /swapfile; `systemd-analyze` shows no ~30 s stall; `journalctl -b | grep -i resume` has no 'waiting for resume device'), and only then `qm delsnapshot 102 pre-rootgrow-20261002`.
@@ -0,0 +1,3 @@
 # nh3-pve-2 status at end of day: AMT done, Proxmox on the WRONG NVMe; Prime reinstalls ON SITE 2026-10-03
 `[2026-10-02]` **nh3-pve-2 status at end of day: AMT done, Proxmox on the WRONG NVMe; Prime reinstalls ON SITE 2026-10-03.** IDE-R from MeshCentral (ISO `proxmox-ve_9.2-1.iso` in lkraven's My Files, kept) installed it but then stalled across the internet; the 4K dummy plug blacks the AMT console once Linux takes the display (8-bit/grayscale encoding or a 1080p plug fixes it); the Realtek 10G (`…:a0:a9`, USW port 1) only links during firmware, so it likely has no driver. **On-site checklist:** USB-stick install → pick the right disk under Target Harddisk → Options; host NIC = i226 (shared with AMT) or an X710 SFP+, not the Realtek; then I wipe the old disk (`wipefs` + `zpool labelclear` if ZFS: two `rpool`s collide), reserve .62, onboard `infra-ops`, keep the AMT port admin-UP in Linux (MS-01 lesson), swap in a 1080p plug.
@@ -0,0 +1,3 @@
 # nh3-pve's AMT is in TacticalRMM's MeshCentral (group `PFI-AMT`, device `nh3-pve-amt`)
 `[2026-10-02]` **nh3-pve's AMT is in TacticalRMM's MeshCentral (group `PFI-AMT`, device `nh3-pve-amt`).** The adds had failed silently because TRMM installs MeshCentral `WANonly` (meshuser.js:2682 drops AMT adds). Prime ruled hybrid; switched 0812, 14/14 agents back, AMT connected at once (16.1.25, TLS fine). **UPDATE 0859: it now PHONES HOME (CIRA)** — MeshCentral mpsPass, ana-gw VIP/policy 76 on 4433, AMT settings via `scripts/amt-cira-setup.py`, AMT moved static → DHCP (Intel: CIRA needs DHCP; reservation keeps .61). Tunnel is independent of nh3-pve. ⚠ While phoning home the AMT REFUSES LAN management (:16993 dark), so manage it via MeshCentral; revert body `servers/nh3-pve/amt-ethernet-static-revert.xml`. 2FA still not forced. Prime's MeshCentral login token is vaulted `pfi-tacticalrmm/meshcentral-login-token`. → `servers/pfi-tacticalrmm/README.md`
@@ -0,0 +1,3 @@
 # Prime: dev-backup gets dailies + weeklies — DONE
 `[2026-10-02]` **Prime: dev-backup gets dailies + weeklies — DONE.** Retention is now 48 hourly + newest of 30 days + newest of 12 ISO weeks (`retention.py` beside the script; second path guard on the NAS side; unit fails if kept ≠ expected). Live run 0739: deleted 1, 0 errors, 48 kept = expected. History before 09-30 was already gone; dailies accumulate from today.
@@ -0,0 +1,3 @@
 # Prime: MS-03s get Proxmox; dummy plugs in hand. One MS-03 is being deployed as `nh3-pve-2` at NH3 (Prime, ~0910)
 `[2026-10-02]` **Prime: MS-03s get Proxmox; dummy plugs in hand. One MS-03 is being deployed as `nh3-pve-2` at NH3 (Prime, ~0910).** Plan: its AMT on DHCP (needed for phone-home) with a UDM reservation, proposed 10.100.250.63 / host 10.100.250.62 (static in PVE + reservation); configure KVM/opt-in over LAN FIRST, then `scripts/amt-cira-setup.py` (LAN goes dark after). MS-03 = i226-LM vPro 2.5G + RTL8127 10G RJ45 + 2× X710 SFP+. **Cabled 2026-10-02 1435 on UDM port 5: AMT 21.0.6 answers (TLS-only, :16993/:664), MAC 38:05:25:3b:a0:a6, sharing the factory Windows' DHCP address (WIN-94HJ50P1LUE).** Port 5 is now native nh3-mgmt (copy of port 6); reservation nh3-pve-2-amt = 10.100.250.63; DNS added. It moved to .63 at 14:47 (replug/reboot). Then (MEBx password = nh3-pve's, vaulted `nh3-pve-2/amt-admin`): KVM on, listener on, OptIn 0 (ACM), `amt-cira-setup.py --apply` → **phoning home at once**; MeshCentral device `nh3-pve-2-amt` (creds + tls=1, needed a MeshCentral restart to log in) shows AMT 21.0.6, power on. LAN :16993 dark by design. Remote install: MeshCentral's embedded MeshCommander (device → Intel AMT tab) has IDE-R; AMT_RedirectionService 32771 = IDER+SOL enabled. **Roles (Prime ~0915): both MS-03 run Proxmox. The other one hosts `esh-dev` at ESH, which will INHERIT MOST OF nh3-dev's SESSIONS (a migration, not yet planned). nh3-pve-2's purpose is TBD ON PURPOSE (high-powered PVE host; possibly a dev environment for security software).** Do not assign it a role. When the esh-dev move is planned, inventory what is anchored to nh3-dev first: the althing herald, svos/hermes-gateway (Miranda's channel), the Booth, the fleet TLS caddy and the `*.nh3.phasefinal.com` rewrite to 10.100.10.50, dev-backup, ttyd/zellij seats, and the `nh3-dev/` vault namespace. On arrival: check the NIC chipset (I226-LM = keep the AMT port admin-UP), fit a plug on each, then the parked AMT follow-ups.
@@ -0,0 +1,3 @@
 # albok-service 0.1.2 LIVE on nh3-docker (albok-dev ask 6955, 0251), fixing a leak that had broken 0.1.1
 `[2026-10-03]` **albok-service 0.1.2 LIVE on nh3-docker (albok-dev ask 6955, 0251), fixing a leak that had broken 0.1.1.** 0.1.1 leaked a Chroma client per health probe: after 9 h it held 1018 of 1024 fds and `/health`/`/search` answered 500. 0.1.2 (`sha256:6f91a33f…`, built from albok-dev's LOCAL `544ecfd` before the push; origin's tags verified = `544ecfd` at 1000) holds 59 fds, 0 deleted, over 30 health calls. Added restricted wing `personal/agent-feedback` (group `albok-feedback` 1512, default-deny). Nemi credentials: service token re-minted with agent-feedback grants (`albok/token-nemi`; old token still live until albok-dev revokes it) and LiteLLM key `albok-nemi` (gen-small/summarizer/qwen3-embedding; `albok/litellm-key-nemi`). **Nemi timer INSTALLED 0559** (user units `albok-nemi.{service,timer}` on nh3-dev, hourly; OnFailure → infra-ops inbox; `services/albok-nemi/`). Gate: albok-dev's steady-state walk, median 594 s over n=3; my test walk through the unit took 589 s with exit 0. ⚠ Reports grow ~55 MB/day (retention asked of albok-dev); manual walks go through `systemctl --user start albok-nemi.service` (no overlap). → `stacks/albok-service/README.md`
@@ -0,0 +1,3 @@
 # CIRA tunnel watchdog LIVE on pfi-tacticalrmm (Prime go, 1232)
 `[2026-10-03]` **CIRA tunnel watchdog LIVE on pfi-tacticalrmm (Prime go, 1232)** fixes MeshCentral "HW Connect stuck at Setup": after a power event AMT leaves its old tunnel half-open, and MeshCentral's relay picks it (3 of 3 episodes: esh-pve-2 twice, nh3-pve-2 once). A timer runs every minute and `ss -K`s any :4433 socket silent >180 s (healthy tunnels 6–30 s). Controls passed. → `services/cira-tunnel-watchdog/`. **Same day: nh3-pve-2's host moved to its X710 10G port** (`38:05:25:3b:a0:a8`, USW Pro 24 port 25 now native nh3-mgmt + tagged auto, reservation and DNS `nh3-pve-2.nh3.internal` = 10.100.250.62). ⚠ At 1233 its AMT (`:a6`, UDM port 5) had NO link, probably because the I226 port has no `auto` line now (MS-01 lesson, unconfirmed); flagged to Prime at 1235.
@@ -0,0 +1,3 @@
 # nh3-pve-2 finished (Prime on site): host 10.100.250.62 on nh3-mgmt via X710 `nic3`, AMT on `nic0` with `auto` kept up, 1 TB wiped into `vmstore` (913 GiB), stale boot entry gone
 `[2026-10-03]` **nh3-pve-2 finished (Prime on site): host 10.100.250.62 on nh3-mgmt via X710 `nic3`, AMT on `nic0` with `auto` kept up, 1 TB wiped into `vmstore` (913 GiB), stale boot entry gone.** My remote re-IP at 1259 failed: a live ifreload read the new vmbr0 as "not a bridge" and downed it. My rollback timer then restored the old /etc/hosts. Prime fixed the network at the console at 1312; I fixed `auto nic0` and /etc/hosts at 1316 (no reload). Lesson: auto-memory `feedback_no_remote_reip_thrash_when_operator_has_console`. → `servers/nh3-pve-2/README.md`
@@ -0,0 +1,3 @@
 # Worldtree memory-gate duty REFRAMED: regression check, not a gate
 `[2026-10-03]` **Worldtree memory-gate duty REFRAMED: regression check, not a gate** (infra-ops msg 01M413W4). The 3-PASS streak completed 10-02 and U11b step 5 ran 2026-10-02 14:54:03Z on Prime's go; 20261003T143101Z was the first post-deletion PASS — recall held without legacy data. Keep the daily batch until worldtree-dev says retire. A FAIL now = recall regression on the LIVE plane (no streak to reset): send infra-ops AND worldtree-dev directly. Cron prompt and skill updated same-day.
@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
-_Last updated: 2026-10-01 ~0446 PT (gen-small OOM fixed by parakeet-nemo 0.1.1, with a hard cap and cache return; Prime: leave the gen-small KV and graphs-off as is; arbo webhook fixed (Gitea allow-list); leftovers deleted, repos pushed. Prior: Parakeet seat → unified-en LIVE, U11a off, SemIf → intern-decision, Scriberr GPU 3 + patches.)_
+_Last updated: 2026-10-03 ~1335 PT (nh3-pve PVE 8→9 upgrade by Prime at the console; nh3-pve-2 live on .62 with vmstore; CIRA watchdog live; PVE nag patched; tank healthy; albok 0.1.2 + Nemi timer. Prior: 2026-10-01 Parakeet/U11a.)_
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,7 +115,43 @@ no longer deployed sidecars here. See Recent decisions.)
 ## Current state / in-flight
-_As of 2026-10-01 ~0446 PT._
+_As of 2026-10-03 ~1335 PT. The newest subsection is first; older subsections carry their own dates._
 ### 2026-10-03: live now (as of ~1335 PT)
 - **nh3-pve PVE 8.4.1 → 9 upgrade: Prime is running it at the console now.** Steps given to him:
  - latest 8.4, then `pve8to9 --full`;
  - `apt remove systemd-boot` (safe: it boots GRUB via proxmox-boot-tool);
  - `zfs snapshot -r rpool/ROOT@pre-pve9`;
  - bookworm → trixie in `/etc/apt/sources.list`, then `apt dist-upgrade`;
  - **before the reboot, `dkms status` must show nvidia 580.178.04 built for the NEW kernel** (nh3-ml1's GPU);
  - reboot.
  NIC names are already pinned (`/etc/systemd/network/10-pin-*.link`). **nh3-dev (VM 102, this session's host) goes down with it.**
 - **Rollback for nh3-pve:** the ZFS snapshot, from a rescue boot. Never run `zpool upgrade rpool`.
 - **POST-CHECK owed once nh3-pve and nh3-dev are back:**
  - every guest is running: VMs 100 nh3-docker1, 101 nh3-extdev, 102 nh3-dev, 105 pbs-nh3 (104 nh3-laser and 108 opnsense-lab were already stopped); CTs 103 nh3-wg, 106 nh3-headscale, 107 nh3-scale, 109 nh3-ml1;
  - NH3 DNS (AdGuard on nh3-docker 10.100.50.40);
  - the althing post office (:8390) and albok-service (:8392);
  - the mesh (headscale, nh3-scale);
  - Miranda's channel (svos :8770 + hermes-gateway on nh3-dev);
  - nh3-ml1's GPU (`nvidia-smi` in CT 109);
  - `nh3-pve-amt` connected in MeshCentral;
  - Beszel;
  - `pveversion` shows 9.x.
 - **nh3-pve-2** (MS-03, NH3) is live at 10.100.250.62 on nh3-mgmt:
  - X710 `nic3` on USW port 25;
  - AMT `nic0` with `auto` + `/etc/sysctl.d/90-amt-port.conf`;
  - vmstore 913 GiB;
  - subscription popup patched.
  Open and NOT approved yet (Prime's call): disable the ceph enterprise apt source, apt upgrade, and one reboot to prove `.62` and nic0 survive. → `servers/nh3-pve-2/README.md`
 - **esh-pve-2** (MS-03, ESH) is unplugged by Prime on purpose. vmstore + AMT phone-home are done. The permanent address is deferred by Prime ("address can wait"; `servers/esh-pve-2/README.md`). Run `playbooks/pve-nag-patch.yaml` on it when it is back.
 - **cira-tunnel-watchdog** on pfi-tacticalrmm has been live since 1232 and drops :4433 tunnels silent >180 s. Watch `journalctl -t cira-tunnel-watchdog`.
 - **Done today, nothing open:**
  - tank verified healthy 0629, disk kept, Miranda told;
  - albok-service 0.1.2 + the Nemi hourly timer;
  - the post-deletion Worldtree gate (infra-hermes runs it as a regression check);
  - the PVE subscription popup patched on nh3-pve, nh3-pve-2, pfi-pve and esh-pve.
 - **ESH PVE 8→9 plan:** written, NOT executed (needs Prime's green light). `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`. The tank blocker is cleared.
 ### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)
@@ -287,25 +323,26 @@ _As of 2026-10-01 ~0446 PT._
 ## Recent decisions
- `[2026-10-03]` **nh3-pve-2 finished (Prime on site): host 10.100.250.62 on nh3-mgmt via X710 `nic3`, AMT on `nic0` with `auto` kept up, 1 TB wiped into `vmstore` (913 GiB), stale boot entry gone.** My remote re-IP at 1259 failed: a live ifreload read the new vmbr0 as "not a bridge" and downed it. My rollback timer then restored the old /etc/hosts. Prime fixed the network at the console at 1312; I fixed `auto nic0` and /etc/hosts at 1316 (no reload). Lesson: auto-memory `feedback_no_remote_reip_thrash_when_operator_has_console`. → `servers/nh3-pve-2/README.md`
+- `[2026-10-03]` **PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host")**: an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → `services/pve-nag-patch/README.md`
- `[2026-10-03]` **CIRA tunnel watchdog LIVE on pfi-tacticalrmm (Prime go, 1232)** fixes MeshCentral "HW Connect stuck at Setup": after a power event AMT leaves its old tunnel half-open, and MeshCentral's relay picks it (3 of 3 episodes: esh-pve-2 twice, nh3-pve-2 once). A timer runs every minute and `ss -K`s any :4433 socket silent >180 s (healthy tunnels 6–30 s). Controls passed. → `services/cira-tunnel-watchdog/`. **Same day: nh3-pve-2's host moved to its X710 10G port** (`38:05:25:3b:a0:a8`, USW Pro 24 port 25 now native nh3-mgmt + tagged auto, reservation and DNS `nh3-pve-2.nh3.internal` = 10.100.250.62). ⚠ At 1233 its AMT (`:a6`, UDM port 5) had NO link, probably because the I226 port has no `auto` line now (MS-01 lesson, unconfirmed); flagged to Prime at 1235.
+- `[2026-10-03]` **nh3-pve-2 finished (Prime on site): host 10.100.250.62 on nh3-mgmt via X710 `nic3`, AMT on `nic0` with `auto` kept up, 1 TB wiped into `vmstore` (913 GiB), stale boot entry gone** → `persistent-memory.d/2026-10-03-nh3-pve-2-finished-prime-on-site-host-10-100-250.md`
- `[2026-10-03]` **Worldtree memory-gate duty REFRAMED: regression check, not a gate** (infra-ops msg 01M413W4). The 3-PASS streak completed 10-02 and U11b step 5 ran 2026-10-02 14:54:03Z on Prime's go; 20261003T143101Z was the first post-deletion PASS — recall held without legacy data. Keep the daily batch until worldtree-dev says retire. A FAIL now = recall regression on the LIVE plane (no streak to reset): send infra-ops AND worldtree-dev directly. Cron prompt and skill updated same-day.
+- `[2026-10-03]` **CIRA tunnel watchdog LIVE on pfi-tacticalrmm (Prime go, 1232)** → `persistent-memory.d/2026-10-03-cira-tunnel-watchdog-live-on-pfi-tacticalrmm-prime-go-1232.md`
- `[2026-10-03]` **albok-service 0.1.2 LIVE on nh3-docker (albok-dev ask 6955, 0251), fixing a leak that had broken 0.1.1.** 0.1.1 leaked a Chroma client per health probe: after 9 h it held 1018 of 1024 fds and `/health`/`/search` answered 500. 0.1.2 (`sha256:6f91a33f…`, built from albok-dev's LOCAL `544ecfd` before the push; origin's tags verified = `544ecfd` at 1000) holds 59 fds, 0 deleted, over 30 health calls. Added restricted wing `personal/agent-feedback` (group `albok-feedback` 1512, default-deny). Nemi credentials: service token re-minted with agent-feedback grants (`albok/token-nemi`; old token still live until albok-dev revokes it) and LiteLLM key `albok-nemi` (gen-small/summarizer/qwen3-embedding; `albok/litellm-key-nemi`). **Nemi timer INSTALLED 0559** (user units `albok-nemi.{service,timer}` on nh3-dev, hourly; OnFailure → infra-ops inbox; `services/albok-nemi/`). Gate: albok-dev's steady-state walk, median 594 s over n=3; my test walk through the unit took 589 s with exit 0. ⚠ Reports grow ~55 MB/day (retention asked of albok-dev); manual walks go through `systemctl --user start albok-nemi.service` (no overlap). → `stacks/albok-service/README.md`
+- `[2026-10-03]` **Worldtree memory-gate duty REFRAMED: regression check, not a gate** → `persistent-memory.d/2026-10-03-worldtree-memory-gate-duty-reframed-regression-check-not-a-gate.md`
- `[2026-10-02]` **esh-pve-2 (MS-03 at ESH, future `esh-dev` host) onboarded: infra-ops (Prime), 1 TB → LVM-thin `vmstore`, AMT phoning home as `esh-pve-2-amt`.** The stale firmware boot entry for an old install on the 1 TB was deleted; that drive was wiped and pvesh-created as `vmstore` (913 GiB; `playbooks/esh-pve-2-disk-prep.yaml`); the reboot onto kernel 7.0.14-20 is verified. AMT 21 (ACM, same MEBx pw, vaulted `esh-pve-2/amt-admin`) shares nic1 (I226-LM), its MAC and its DHCP IP with the host: KVM on, OptIn 0, listener on, then `amt-cira-setup.py --apply`. ⚠ Its FIRST CIRA connect crashed MeshCentral once (~7 s, agents back; MeshCentral 1.2.0 race in mpsserver.js, read at source); credentials were then picked up by dropping only its tunnel (`ss -K`), with no restart. Address 10.0.10.70 is TEMPORARY (Prime: can wait) and must become a reservation on MAC `38:05:25:3b:9c:12`. → `servers/esh-pve-2/README.md`, `servers/pfi-tacticalrmm/README.md`
+- `[2026-10-03]` **albok-service 0.1.2 LIVE on nh3-docker (albok-dev ask 6955, 0251), fixing a leak that had broken 0.1.1** → `persistent-memory.d/2026-10-03-albok-service-0-1-2-live-on-nh3-docker-albok-dev-ask.md`
- `[2026-10-02]` **Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684).** Their b193 release commit c2d87263 swept 60 staged deletions into the tree, so the demo api crash-looped. I recreated `worldtree-api` only (`compose up -d --no-deps`) on the `.env`-pinned fa8bc51cc064 (compose.yaml identical at both shas): healthy in 35 s at 15:16Z. ⚠ Their deploy health gate does NOT restore the old container on failure (it only withholds `:latest`); flagged to them. Fix-forward 7ab6ae40 (tag v1.0.0b193 moved onto it) deployed via CI 15:22Z and verified healthy, same seven retired-key WARNINGs; the deploy-gate gap is Worldtree #423.
+- `[2026-10-02]` **esh-pve-2 (MS-03 at ESH, future `esh-dev` host) onboarded: infra-ops (Prime), 1 TB → LVM-thin `vmstore`, AMT phoning home as `esh-pve-2-amt`** → `persistent-memory.d/2026-10-02-esh-pve-2-ms-03-at-esh-future-esh-dev-host-onboarded-infra-ops.md`
- `[2026-10-02]` **nh3-pve's AMT is in TacticalRMM's MeshCentral (group `PFI-AMT`, device `nh3-pve-amt`).** The adds had failed silently because TRMM installs MeshCentral `WANonly` (meshuser.js:2682 drops AMT adds). Prime ruled hybrid; switched 0812, 14/14 agents back, AMT connected at once (16.1.25, TLS fine). **UPDATE 0859: it now PHONES HOME (CIRA)** — MeshCentral mpsPass, ana-gw VIP/policy 76 on 4433, AMT settings via `scripts/amt-cira-setup.py`, AMT moved static → DHCP (Intel: CIRA needs DHCP; reservation keeps .61). Tunnel is independent of nh3-pve. ⚠ While phoning home the AMT REFUSES LAN management (:16993 dark), so manage it via MeshCentral; revert body `servers/nh3-pve/amt-ethernet-static-revert.xml`. 2FA still not forced. Prime's MeshCentral login token is vaulted `pfi-tacticalrmm/meshcentral-login-token`. → `servers/pfi-tacticalrmm/README.md`
+- `[2026-10-02]` **Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684)** → `persistent-memory.d/2026-10-02-demo-outage-13-min-rolled-back-worldtree-dev-urgent-6684.md`
- `[2026-10-02]` **Prime: dev-backup gets dailies + weeklies — DONE.** Retention is now 48 hourly + newest of 30 days + newest of 12 ISO weeks (`retention.py` beside the script; second path guard on the NAS side; unit fails if kept ≠ expected). Live run 0739: deleted 1, 0 errors, 48 kept = expected. History before 09-30 was already gone; dailies accumulate from today.
+- `[2026-10-02]` **nh3-pve's AMT is in TacticalRMM's MeshCentral (group `PFI-AMT`, device `nh3-pve-amt`)** → `persistent-memory.d/2026-10-02-nh3-pve-s-amt-is-in-tacticalrmm-s-meshcentral-group.md`
- `[2026-10-02]` ✅ **ESH `tank` (esh-nas-pve, 175 T raidz2×2): was DEGRADED since ~2026-08-20 (found while planning PVE 9); HEALTHY since 2026-10-03 0629.** raidz2-0 disk `wwn-0x5000c500c91df554`: 36 CKSUM errors; Aug 20 resilver logged 6.68 M errors; no known data errors; SMART clean. **Prime (via Miranda) 1925: verify, then clear + full scrub.** Verified residual: the disk was REPLACED 2026-08-20 09:50 (zpool history), resilver ended with 6.68 M errors, and since the last clear only 21 checksum events (Aug 21 02:05 ×20, Aug 29 02:50 ×1), none in 34 days. `zpool clear` 1926 → ONLINE; full scrub running (background watcher); replace the disk if errors return. **UPDATE 2331: the scrub is REPAIRING that disk: 6.19 M CKSUM, 198 G repaired at 72%, 0 read/write errors, still no known data errors, ETA ~0107.** History explains it: after the Aug 20 replace, the resilver ended `errors=6676436`, someone ran only an error-scrub (`zpool scrub -e`, 4 s) and then `zpool clear` 19:32, so NO full scrub ran until today, and those blocks sat bad on the new disk for 6 weeks. SMART still clean (0 realloc/pending/uncorrect/CRC, no errors logged), no kernel I/O errors. Read: residual repair, not a failing disk (inferred). Discriminator: clear + a SECOND full scrub; any CKSUM on it → replace. ⚠ The watcher's error check was blind (could not parse `6.19M` as an integer); it still fires on completion. **0106 scrub DONE: repaired 198G, 0 errors, no known data errors; that disk 6,494,530 CKSUM exact (97% of the 6,676,436 the Aug 20 resilver failed on), last checksum ereport 21:26:27 = none in the scrub's final 3h40m.** **0118: SECOND full scrub started WITHOUT `zpool clear`**, so any new error shows as count > 6494530 (`zpool status -p`); watcher tested with +1/null/finished controls; ETA ~0700. **RESOLVED 0627: second scrub repaired 0B with 0 errors; the disk's CKSUM stayed at 6494530 (zero new); SMART clean; no kernel I/O errors. Verdict: residue of the Aug 20 resilver, now repaired; DISK KEPT. `zpool clear` 0629 → `pool 'tank' is healthy`.** Reported to Miranda once. **PVE 8→9 plan for esh-pve-cluster written, NOT executed** (Prime via Miranda): `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`. Also found: remote access to ESH rides CT 108 on `pve` (no remote recovery if it fails to boot); both nodes fail pve8to9 on the systemd-boot meta-package.
+- `[2026-10-02]` **Prime: dev-backup gets dailies + weeklies — DONE** → `persistent-memory.d/2026-10-02-prime-dev-backup-gets-dailies-weeklies-done.md`
- `[2026-10-02]` **albok-service 0.1.0 LIVE on nh3-docker (albok-dev ask, operator-approved): `http://albok.nh3.internal:8392`** (8390 there is the post office). Store/private on local ext4 under /srv/albok, uid 1500, read groups 1510/1511 (container gets a mounted /etc/group + group_add so its getgrnam/chgrp work). Scoped LiteLLM key `albok-service` (qwen3-embedding only) + bootstrap admin token in the vault under `albok/`. Upgraded to **0.1.1** (e349d50) at 1805: /health now `ok` (0.1.0's always-degraded canary bug fixed); albok-dev smoke-tested ingest/search/readback (token vaulted `albok/token-albok-dev`). → `stacks/albok-service/README.md`
+- `[2026-10-02]` **ESH `tank` (esh-nas-pve, 175 T raidz2×2): was DEGRADED since ~2026-08-20 (found while planning PVE 9); HEALTHY since 2026-10-03 0629** → `persistent-memory.d/2026-10-02-esh-tank-esh-nas-pve-175-t-raidz2-2-was-degraded.md`
- `[2026-10-02]` **nh3-pve-2 status at end of day: AMT done, Proxmox on the WRONG NVMe; Prime reinstalls ON SITE 2026-10-03.** IDE-R from MeshCentral (ISO `proxmox-ve_9.2-1.iso` in lkraven's My Files, kept) installed it but then stalled across the internet; the 4K dummy plug blacks the AMT console once Linux takes the display (8-bit/grayscale encoding or a 1080p plug fixes it); the Realtek 10G (`…:a0:a9`, USW port 1) only links during firmware, so it likely has no driver. **On-site checklist:** USB-stick install → pick the right disk under Target Harddisk → Options; host NIC = i226 (shared with AMT) or an X710 SFP+, not the Realtek; then I wipe the old disk (`wipefs` + `zpool labelclear` if ZFS: two `rpool`s collide), reserve .62, onboard `infra-ops`, keep the AMT port admin-UP in Linux (MS-01 lesson), swap in a 1080p plug.
+- `[2026-10-02]` **albok-service 0.1.0 LIVE on nh3-docker (albok-dev ask, operator-approved): `http://albok.nh3.internal:8392`** → `persistent-memory.d/2026-10-02-albok-service-0-1-0-live-on-nh3-docker-albok-dev-ask.md`
- `[2026-10-02]` **Prime: MS-03s get Proxmox; dummy plugs in hand. One MS-03 is being deployed as `nh3-pve-2` at NH3 (Prime, ~0910).** Plan: its AMT on DHCP (needed for phone-home) with a UDM reservation, proposed 10.100.250.63 / host 10.100.250.62 (static in PVE + reservation); configure KVM/opt-in over LAN FIRST, then `scripts/amt-cira-setup.py` (LAN goes dark after). MS-03 = i226-LM vPro 2.5G + RTL8127 10G RJ45 + 2× X710 SFP+. **Cabled 2026-10-02 1435 on UDM port 5: AMT 21.0.6 answers (TLS-only, :16993/:664), MAC 38:05:25:3b:a0:a6, sharing the factory Windows' DHCP address (WIN-94HJ50P1LUE).** Port 5 is now native nh3-mgmt (copy of port 6); reservation nh3-pve-2-amt = 10.100.250.63; DNS added. It moved to .63 at 14:47 (replug/reboot). Then (MEBx password = nh3-pve's, vaulted `nh3-pve-2/amt-admin`): KVM on, listener on, OptIn 0 (ACM), `amt-cira-setup.py --apply` → **phoning home at once**; MeshCentral device `nh3-pve-2-amt` (creds + tls=1, needed a MeshCentral restart to log in) shows AMT 21.0.6, power on. LAN :16993 dark by design. Remote install: MeshCentral's embedded MeshCommander (device → Intel AMT tab) has IDE-R; AMT_RedirectionService 32771 = IDER+SOL enabled. **Roles (Prime ~0915): both MS-03 run Proxmox. The other one hosts `esh-dev` at ESH, which will INHERIT MOST OF nh3-dev's SESSIONS (a migration, not yet planned). nh3-pve-2's purpose is TBD ON PURPOSE (high-powered PVE host; possibly a dev environment for security software).** Do not assign it a role. When the esh-dev move is planned, inventory what is anchored to nh3-dev first: the althing herald, svos/hermes-gateway (Miranda's channel), the Booth, the fleet TLS caddy and the `*.nh3.phasefinal.com` rewrite to 10.100.10.50, dev-backup, ttyd/zellij seats, and the `nh3-dev/` vault namespace. On arrival: check the NIC chipset (I226-LM = keep the AMT port admin-UP), fit a plug on each, then the parked AMT follow-ups.
+- `[2026-10-02]` **nh3-pve-2 status at end of day: AMT done, Proxmox on the WRONG NVMe; Prime reinstalls ON SITE 2026-10-03** → `persistent-memory.d/2026-10-02-nh3-pve-2-status-at-end-of-day-amt-done-proxmox.md`
- `[2026-10-02]` **nh3-dev root grown 250 → 378 GB (Prime resized scsi0; I grew the guest online, no reboot).** Root 372 GB, 58%, 150 GB free, after the 85% alert fired twice in 14 h (uv cache + agent venvs). Swap moved to a 4 GB `/swapfile` (the old `sda5` blocked growth), `RESUME=none`, all initrds rebuilt (`playbooks/nh3-dev-grow-root.yaml`). ⚠ **TODO: delete VM snapshot `pre-rootgrow-20261002` on nh3-pve after the next NATURAL reboot (Prime 2026-10-02: no test reboot).** Trigger: `uptime -s` on nh3-dev later than 2026-10-02 0733. Then check the boot was clean (`swapon --show` = /swapfile; `systemd-analyze` shows no ~30 s stall; `journalctl -b | grep -i resume` has no 'waiting for resume device'), and only then `qm delsnapshot 102 pre-rootgrow-20261002`.
+- `[2026-10-02]` **Prime: MS-03s get Proxmox; dummy plugs in hand. One MS-03 is being deployed as `nh3-pve-2` at NH3 (Prime, ~0910)** → `persistent-memory.d/2026-10-02-prime-ms-03s-get-proxmox-dummy-plugs-in-hand-one.md`
- `[2026-10-01]` **MIA (Make-It-Animatable v2) auto-rigger LIVE on fv-ml1 GPU 3, one-shot (Prime via dread-dev).** `scripts/mia-run --job DIR -- name=in.glb ...`, image `local/mia:0.1.0` (10.5 GB), weights in the shared HF cache at pinned revisions (sha256-verified). Acceptance: 3.9–4.7 s a mesh (median of 3), 12.7 s model load, peak 3,394 MiB; seeded runs bit-identical; GPU-vs-CPU distances the same size as sampling noise (positive control: unseeded GPU). Skeleton template is a SUBSTITUTE (gated HF dataset `jasongzy/Mixamo`, terms not accepted; Prime's call). ⚠ fv-ml1 zroot at 85% after the build. → `stacks/mia/README.md`
+- `[2026-10-02]` **nh3-dev root grown 250 → 378 GB (Prime resized scsi0; I grew the guest online, no reboot)** → `persistent-memory.d/2026-10-02-nh3-dev-root-grown-250-378-gb-prime-resized-scsi0.md`
- `[2026-10-01]` **Prime: "clean up the 1700+ backups" — dev-backup retention FIXED, 1,740 husks removed.** They were never 1,788 real snapshots: the prune had deleted everything except one 0555 dir (`vastblue/praxis/references/PraxisPM_Rev0_07.11.26`, 17 files hardlinked into every snapshot), so each old dir was a 22-entry husk and only the newest 48 were complete. Removing them lost no history and freed ~0 bytes (same inodes, link count 1,788). Deletion ran from a generated list of 1,740 literal paths, each verified as a husk first, with 0 errors. The script now runs `chmod -R u+w` before `rm`, logs the error count and the kept count, and fails the unit (exit 3) on a bad prune or exit 1 on a failed rsync. Positive control: a manual run at 0556 pruned the full snapshot `2026-09-29_0602` with 0 errors, leaving 48. Retention stays the designed 48 hourly; the 4.6 GB log rotated at 0530 is kept, compressed to 99 MB, at `~/.config/dev-backup/dev-backup.log.20261001-0530.gz` until someone deletes it.
+- `[2026-10-01]` **MIA (Make-It-Animatable v2) auto-rigger LIVE on fv-ml1 GPU 3, one-shot (Prime via dread-dev)** → `persistent-memory.d/2026-10-01-mia-make-it-animatable-v2-auto-rigger-live-on-fv-ml1-gpu-3.md`
 - `[2026-10-01]` **Prime: "clean up the 1700+ backups" — dev-backup retention FIXED, 1,740 husks removed** → `persistent-memory.d/2026-10-01-prime-clean-up-the-1700-backups-dev-backup-retention-fixed.md`
 - `[2026-10-01]` **Prime: gen-small keeps its 8 GiB KV pin, and parakeet-nemo keeps CUDA graphs OFF** (~2–8 ms at short clips, accepted). GPU 0's ~1 GB of spare memory is enough with the seat's hard 3,840 MiB cap.
- `[2026-10-01]` **⚠ Gitea's `[webhook] ALLOWED_HOST_LIST = external, 10.0.0.0/8` silently REJECTS headscale mesh IPs (100.64.0.0/10).** The test API still returns 204 and nothing arrives. The vh/arbo hook therefore targets irv-ml1's LAN `10.6.110.50:9009`; the secret was re-set and a delivery is verified (deploy ran 04:41). A Gitea webhook PATCH without `secret` and `branch_filter` drops both, so always resend them.
+- `[2026-10-01]` **⚠ Gitea's `[webhook] ALLOWED_HOST_LIST = external, 10.0.0.0/8` silently REJECTS headscale mesh IPs (100.64.0.0/10)** → `persistent-memory.d/2026-10-01-gitea-s-webhook-allowed-host-list-external-10-0.md`
- `[2026-10-01]` **vh/arbo push webhook repointed from the retired wg0 lifeline `10.100.79.3:9009` to irv-ml1's mesh address `100.64.0.6:9009`.** It had been dead since 09-06 (comfy-dev report). No other of the 109 repos' hooks pointed at 10.100.79.x. The Actions runner `irv-ml1-arbo` was fine; run 677 failed, likely colliding with a hand deploy. The secret was not resent: if the next push shows an HMAC rejection, re-set it.
+- `[2026-10-01]` **vh/arbo push webhook repointed from the retired wg0 lifeline `10.100.79.3:9009` to irv-ml1's mesh address `100.64.0.6:9009`** → `persistent-memory.d/2026-10-01-vh-arbo-push-webhook-repointed-from-the-retired-wg0.md`
- `[2026-10-01]` **Prime: delete the bench leftovers, no upstream for Scriberr, push.** DONE: 7 HF revisions deleted through huggingface_hub's cache API (25.1 GB; the parakeet 1.1B/ctc/v2 models and whisper-large-v3; **unified-en KEPT, the live seat mounts it**), plus `/tank/spikes/scriberr-slicer` (including the private copies of Prime's recordings) and `/tank/spikes/parakeet-ab`. The Scriberr upstream PR text was dropped (6b66207). Both repos pushed.
+- `[2026-10-01]` **Prime: delete the bench leftovers, no upstream for Scriberr, push** → `persistent-memory.d/2026-10-01-prime-delete-the-bench-leftovers-no-upstream-for-scriberr.md`
 - `[2026-10-01]` **irv-ml1 /storetank reclaim done:** Prime ruled through comfy-dev, which deleted 84 files of its own (260 → 299 GB free). Tiers B/C/D got no ruling (infra-hermes thread `01M3TCSYRSFNAPA9BPQTMFQ6KJ`).
 - `[2026-09-30]` **Parakeet speech seat → `parakeet-unified-en-0.6b` under NeMo (bf16) APPROVED by Prime, NVIDIA Open Model License accepted. DONE 2026-10-01 0126 by infra-hermes; infra-ops audit passed 0137.** → `persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md`
 - `[2026-09-30]` **Worldtree U11a: legacy memory plane OFF on demo and personal. The U11b data deletion is gated on 3 consecutive PASS and step 5 is mine; the legacy archive must be destroyed by 2026-10-30.** → `persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md`
@@ -316,7 +353,7 @@ _As of 2026-10-01 ~0446 PT._
 - `[2026-09-28]` **Bonsai ternary vs Q4_K_XL at concurrency on the 275 W card (1.93x at N=1 falls to 1.06x at N=8, 1.21x with the MMVQ fix); the weights are acquired.** → `persistent-memory.d/2026-09-28-bonsai-ternary-spike.md`
 - `[2026-09-28]` **blender-run gained `--cpu` (no GPU attached) and a fixed hostname `fv-ml1-blender` (draupnir).** The design stage never renders, so it stays off GPU 3.
 - `[2026-09-28]` **Blender extensions live in a read-only System repo built from a sha256 lock, enabled by a hook, opt-in for blender-run (`--extensions`).** SurfacePsycho's eval() is patched to literal_eval (a proven safe-mode escape). → `stacks/blender/README.md` § Extensions
- `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** (SVOS v2.1.12: `propose_decision` gained `seat_up`, and Hermes reads the plugin only at start). The plugin load was verified at file level; the end-to-end proof is Miranda's first seat_up card. Enabling `zellij-fleet@Claude` at boot remains Prime's call.
+- `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** → `persistent-memory.d/2026-09-27-hermes-gateway-restarted-0401-for-highseat-dev.md`
 - `[2026-09-27]` **SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime):** wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → `persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md`
 - `[2026-09-27]` **Blender 5.2 on fv-ml1 GPU 3 (on demand), agent-driven via mcp-for-blender running in-container over ssh stdio; safe mode on, no published port.** → `stacks/blender/README.md`
 - `[2026-09-27]` **esh-docker-vm host→HA traffic needs a /32 over macvlan-shim (if-up.d hook); without it HA silently loses MQTT (3× since August).** → `servers/esh-docker-vm/README.md`
@@ -370,7 +407,7 @@ _As of 2026-10-01 ~0446 PT._
 - `[2026-09-22]` **The acceptance probe for a guard must not be able to destroy what it tests** — (infra-hermes). → `persistent-memory.d/2026-09-22-the-acceptance-probe-for-a-guard-must-not-be-able-to.md`
 - `[2026-09-22]` ⚠ **`{"sent": true}` is a claim about transmission, never about effect.** `pane_send` structurally cannot deliver a harness command (every relay is prefixed with the card id), so D-0010 promised an unachievable `/clear` and its receipt reported success. Consumed an operator approval. → `persistent-memory.d/2026-09-22-instrument-errors.md`
 - `[2026-09-22]` **D-0010/D-0011 were misrouted to this seat by `pane_find` matching a ROLLING PANE TITLE.** — Genuine and operator-approved, wrong seat; `fleet_telemetry` held the right mapping and carries the warning… → `persistent-memory.d/2026-09-22-d-0010-d-0011-were-misrouted-to-this-seat-by-panefind.md`
- `[2026-09-22]` **`headscale-ddns` exited 1 silently and the alarm carried no cause** — both failure paths were `|| exit 1` with no message. Now every exit names its reason and the WAN lookup retries 3×. The script was also untracked; a fix to what every mesh client resolves through lived on one disk. `30517fd`.
+- `[2026-09-22]` **`headscale-ddns` exited 1 silently and the alarm carried no cause** → `persistent-memory.d/2026-09-22-headscale-ddns-exited-1-silently-and-the-alarm-carried-no.md`
 - `[2026-09-22]` ⭐⭐ **Five instrument errors in one day, and the shape is one thing: a tool that enumerates "things that are fine" has selected against its own subject.** `--state=running` skipped the units most needing hooks; `awk '{print $1}'` dropped systemd's `●`-decorated FAILED rows; `grep -ic restic` on the wrapper missed the check script; `restic ls`'s header line made an absent path read as "blobs gone"; and a cooldown test that invoked before failing the unit. **Every one reported cleanly while looking at the wrong thing.** The rule is not "verify" — it is *verify, then ask what the verification could not have seen*. → `persistent-memory.d/2026-09-22-instrument-errors.md`
 - `[2026-09-22]` **Fleet alert bridge generalized** — `beszel-althing` → `althing-alert-bridge`, route registry (`/beszel` + `/kuma`), each with its own… → `persistent-memory.d/2026-09-22-fleet-alert-bridge-generalized.md`
@@ -418,13 +455,8 @@ _As of 2026-10-01 ~0446 PT._
 - `[2026-09-19]` **`infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.** — infra-ops keeps **improving infrastructure tooling** plus the hard calls; infra-hermes does **day-to-day… → `persistent-memory.d/2026-09-19-infra-hermes-is-this-session-s-assistant-and-the-division.md`
 - `[2026-09-18]` ⭐⭐⭐ **NH3↔Anaheim had been running over a throttled DERP relay, not a direct path — 78 GB of fleet traffic on someone else's free infrastructure.** Four additive objects on ana-gw gave ana-scale a stable inbound UDP 41641 endpoint; `tailscale ping` 373–522 ms → **6 ms direct**, cross-site HTTP 1.2 s → 0.015 s, STT via the ANA gateway 1.4 s → 0.25 s. ⚠ That box runs `central-nat`, so a policy `dstaddr` is the REAL internal address, not the VIP. No OOB access — back up with `show` to a local file and make additive changes ONLY. irv-ml1 still relayed. → `persistent-memory.d/2026-09-18-nh3-ana-derp-relay.md`
 - `[2026-09-18]` ⭐⭐⭐ **`.internal` DNS was failing ~10% of lookups fleet-wide, from two independent causes.** A PUBLIC resolver was the fallback for a PRIVATE zone (Cloudflare answers NXDOMAIN authoritatively, so a transient miss became a hard failure) — now a cross-site ring, each site local-first with a different site as backup. Then the root cause: all three AdGuards shipped `ratelimit: 20` shared across an entire **/24**, silently dropping queries at 5 s each. Set to 0. Hard failures 3/40 → 0/40; burst timeouts 40/60 → 0/60. ⚠ `resolv.conf` is DHCP-managed — change it at the UDM/FortiGate, not the file. → `persistent-memory.d/2026-09-18-fleet-dns-ring-and-ratelimit.md`
 - `[2026-09-18]` ⭐⭐ **SearXNG had ONE working general web engine and every health check said fine.** 7 of 55 were enabled-by-default and six of those are dictionary/translation engines — `inactive: false` only makes an engine SELECTABLE, `disabled: false` puts it in the DEFAULT set. Now seven. ⭐ This stack tracks `:latest` ON PURPOSE (upstream ships engine-handler fixes continuously; a pin freezes breakage). ⭐ The Brave key is committed in plaintext by explicit operator decision — scoped to one low-value credential, NOT a change to the no-secrets rule. → `persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md`
 - `[2026-09-18]` ⭐⭐ **althing search returned ZERO for every hyphenated query — silently, for every handle, and on this fleet that is most of our hostnames.** FTS5 read the hyphen as a column filter; `search()` caught the error, its probe passed, and it returned `[]`. Routed to forseti (they own the code, I own rollout) → **v3.6.3** deployed, 17 s bus outage. The reported symptom, a message that never arrived, was NOT a defect: the reporter's evidence file was truncated at 8 KiB. ⚠ No CI builds the post-office image. → `persistent-memory.d/2026-09-18-althing-363-hyphen-search.md`
 - `[2026-09-18]` ⭐ **FleetTools: one capability index, autoloaded by Claude, Codex and Grok from a single symlinked file.** `docs/fleettools/` + `~/FLEETTOOLS.md`; absolute detail paths because a non-Claude agent cats them. Rule zero is query-live-inventories-never-a-written-list. The global CLAUDE.md tools section went 231 lines → 39, keeping only the three rules that govern behaviour rather than lookup. → `persistent-memory.d/2026-09-18-fleettools-agent-index.md`
 - `[2026-09-18]` **Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy** — (operator-approved). → `persistent-memory.d/2026-09-18-miranda-s-hermes-plugin-install-is-now-a-symlink-to-the.md`
 - `[2026-09-18]` **Worldtree's `env.sh` secrets are vaulted** — 10 entries under `worldtree/` (gitea, matrix as/hs, openai, uv-index, vastblueai, wt-admin demo+personal… → `persistent-memory.d/2026-09-18-worldtree-s-env-sh-secrets-are-vaulted.md`
 - `[2026-09-17]` **`dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.** → `persistent-memory.d/2026-09-17-dragonfireacoustics-com-expires-2026-10-30-six-weeks-at.md`
  ⚠ **If it is ever transferred, DNS does NOT come with the registration** — the nameservers are eNom's `name-services.com` and the zone must be recreated first or mail dies. The whole zone is two facts plus a landmine: `*` (WILDCARD) → 199.250.192.76 which is **dead** (no HTTP at all, and it is what the apex/mail/webmail/admin/ftp all answer with), `www` → 38.120.12.45 (us), and **7 Google Workspace MX records that must not be lost**. No DNSSEC (`delegationSigned: false`), so no transfer complication. ⚠ Also found: **no SPF and no DMARC** at the apex on a Google Workspace domain — a live deliverability problem independent of everything else. Transfer gate is the **TAC/EPP code from the eNom account**, not the lock; the missing lock is not authorization. 60-day rule is satisfied (last changed 2025-10-24).
@@ -445,26 +477,26 @@ _As of 2026-10-01 ~0446 PT._
 - `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** → `persistent-memory.d/2026-08-25-fused-moe-kernel-path-deferred-tracked-at-park-fused-moe.md`
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction.** brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`.
+- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
-_167 older entries archived to archival-memory.md._
+_172 older entries archived to archival-memory.md._
 ## Tried and abandoned
- `[2026-10-01]` **Size + mtime scans to pick reclaimable "staging" on a model store** (infra-hermes, irv-ml1). `_inbound/retro-diffusion` looked like staging leftovers but is the TARGET of 47 model-store symlinks, holding a purchased bundle used by a live commission. Resolve symlinks INTO any staging dir before proposing to delete it.
+- `[2026-10-01]` **Size + mtime scans to pick reclaimable "staging" on a model store** → `persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md`
- `[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix.** I predicted ~6× less memory from the attention-matrix arithmetic. Measured, 300 → 120 s only went 9,384 → 6,510 MiB: a ~5.6 GB fixed floor dominates. `expandable_segments:True` was the real lever (5,496). Measure the process peak; never extrapolate it from one tensor.
+- `[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix** → `persistent-memory.d/2026-09-30-shorter-parakeet-slices-as-scriberr-s-memory-fix.md`
 - `[2026-09-30]` **Whole-file local-attention Parakeet in Scriberr (context 255/255).** OOM past 16 GB on a 35-min file. Local attention inside chunks is also non-deterministic run to run.
 - `[2026-09-30]` **Start-time midpoint stitching of overlapped Parakeet chunks.** It duplicated a word at 26 of 108 stitches, because Parakeet timestamps a post-pause word anywhere inside the pause. Hand over at a word both chunks agree on instead.
 - `[2026-09-30]` **int8 ONNX (sherpa-onnx) as the low-latency Parakeet runtime.** The int8 graph runs on ONE CPU thread with the GPU at 2–9%. Unified-en int8 was slower than the seat; fp32 ONNX was 4–12× faster, and NeMo was fastest.
 - `[2026-09-30]` **GPU budgets computed as total − used.** nvidia-smi `Free` is ~640 MiB lower per card (driver reserve). Budget from `Free`.
- `[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel.** Every tool worked except `get_viewport_screenshot`, which had Blender write a file for the SERVER to read ("Screenshot file was not created"). The server now runs inside the Blender container over ssh+docker-exec stdio. That also means no port is published. → `stacks/blender/README.md`
+- `[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel** → `persistent-memory.d/2026-09-27-the-blender-mcp-server-on-nh3-dev-reaching-the-add-on.md`
- `[2026-09-27]` **A TCP connect as the "is Blender ready" probe.** docker-proxy accepts on a published port before the app behind it listens, so it said "answering" while Blender was still loading. The probe now asks the add-on to `ping`. The same trap applies to any service behind a published port.
+- `[2026-09-27]` **A TCP connect as the "is Blender ready" probe** → `persistent-memory.d/2026-09-27-a-tcp-connect-as-the-is-blender-ready-probe.md`
 - `[2026-09-27]` **`log.exception()` in a GPU failure path** — the record keeps `exc_info`, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log `traceback.format_exc()` text instead (semif-serve `engine._guard`).
 - `[2026-09-27]` **fla/triton in a slim image without gcc** — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra.
- `[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** — the export drops uv's per-package index routing; with `--extra-index-url … --index-strategy unsafe-best-match`, triton came from the pytorch index and failed the lock's hash. Keep `uv sync` and blank the project version in a deps-only stage instead (`services/semif-serve/Dockerfile`).
+- `[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** → `persistent-memory.d/2026-09-27-uv-export-requirements-txt-uv-pip-install-for-a.md`
 - `[2026-09-27]` **Translating a CUDA OOM by raising inside `except` (or `from exc`)** — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, `gc.collect()` before `empty_cache()`.
 - `[2026-09-27]` **`deploy-stack.sh --yes` / `dns-sync.py` fed a blind `y`** — the harness refuses a blind apply. Review with `echo n |` first, then apply with `echo y |`.
- `[2026-09-27]` **Relying on the build cache surviving on esh-ml1** — the augaman v0.1.3 build missed its dependency layer there (fv-ml1 hit it; reason unexplained, suspect the `docker builder prune` runs) and the rootfs peaked at 90%. Check for ≥15 GB free, or remove the old image, before building there.
+- `[2026-09-27]` **Relying on the build cache surviving on esh-ml1** → `persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md`
 - `[2026-09-26]` **Greedy exact-match as parity for a small generative seat (coder)** — same-host repeats matched only 30–38% (run B hit the prefix cache, so a different numeric path; near-tie… → `persistent-memory.d/2026-09-26-greedy-exact-match-as-parity-for-a-small-generative-seat.md`
 - `[2026-09-26]` **zsh traps in ad-hoc test loops:** `set -- $a` does NOT word-split, so alert POSTs went out with empty fields; and `local path=…` inside a function CLOBBERS `$PATH` (zsh's tied array), so the failover test ran nothing while TEI sat stopped. Use other names and `${=var}`.
 - `[2026-09-25]` **Concluding "not on the UDM" from a port table read 15 s after link-up** — UniFi polls (~60 s); AMT was on port 6 as Prime said. Wait a poll, and prefer a which-port-lists-the-MAC check. auto-memory `feedback_polled_stats_lag_the_event`.
@@ -475,8 +507,6 @@ _167 older entries archived to archival-memory.md._
 - `[2026-09-21]` **Using directory mtime as a liveness test when pruning session scratchpads** — `find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7` deleted an ACTIVE session's working… → `persistent-memory.d/2026-09-21-using-directory-mtime-as-a-liveness-test-when-pruning.md`
 - `[2026-09-18]` **Routing SearXNG's egress through a SOCKS5 proxy on esh-scale** — one day live, reverted. It fixed nothing, and the reason I gave for reverting it was itself wrong: the rollback was the counterfactual and it falsified my own published claim. Kept reverted on its own merits (no measurable gain, added a hard ESH dependency for all fleet search). → `persistent-memory.d/2026-09-18-searxng-esh-egress-reverted.md`
 - `[2026-09-18]` **`api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings** — this build has NO `!ENV` YAML constructor, so the file was unparseable and the container crash-looped ten… → `persistent-memory.d/2026-09-18-apikey-env-searxngbraveapikey-in-searxng-settings.md`
-_118 older entries archived to archival-memory.md._
+_120 older entries archived to archival-memory.md._
		`@@ -0,0 +1,3 @@`
							# `nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction

							`[2026-08-24]` `nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction. brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`.
		`@@ -1,3 +0,0 @@`
			# `[2026-09-18]` `api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings

			`api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings — this build has NO `!ENV` YAML constructor, so the file was unparseable and the container crash-looped ten times with fleet search down. There is no env-var path into that file at all: the loader reads only `SEARXNG_SETTINGS_PATH` and the entrypoint substitutes only `ultrasecretkey`. Literal or nothing.
		`@@ -1,3 +0,0 @@`
			# `[2026-09-18]` Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy

			Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy (operator-approved). Verified empirically — `is_dir()`/`exists()` are stat-based and follow symlinks — and the copy had ALREADY drifted (README, since 09-15). Pre-restart gate is `uv run python scripts/sync_plugin.py --check`; `hermes plugins list` is confirmation-after, never permission-before. Copy kept at `~/backups/svos_miranda-copy-20260918-1444`. ⚠ The install now tracks a working tree — an uncommitted edit in svos is what loads at the next restart.
		`@@ -1,3 +0,0 @@`
			# `[2026-09-18]` Worldtree's `env.sh` secrets are vaulted

			Worldtree's `env.sh` secrets are vaulted — 10 entries under `worldtree/` (gitea, matrix as/hs, openai, uv-index, vastblueai, wt-admin demo+personal, z-ai, zai). ⚠ `Z_AI_API_KEY` and `ZAI_API_KEY` are DIFFERENT keys despite the near-identical names (fp `26bde3fa` vs `a4884152`). `ANTHROPIC_API_KEY` was empty and skipped.
		`@@ -0,0 +1,3 @@`
							# `headscale-ddns` exited 1 silently and the alarm carried no cause

							`[2026-09-22]` `headscale-ddns` exited 1 silently and the alarm carried no cause — both failure paths were `\|\| exit 1` with no message. Now every exit names its reason and the WAN lookup retries 3×. The script was also untracked; a fix to what every mesh client resolves through lived on one disk. `30517fd`.
		`@@ -0,0 +1,3 @@`
							`# A TCP connect as the "is Blender ready" probe`

							`[2026-09-27]` A TCP connect as the "is Blender ready" probe. docker-proxy accepts on a published port before the app behind it listens, so it said "answering" while Blender was still loading. The probe now asks the add-on to `ping`. The same trap applies to any service behind a published port.
		`@@ -0,0 +1,3 @@`
							`# hermes-gateway restarted 0401 for highseat-dev`

							`[2026-09-27]` hermes-gateway restarted 0401 for highseat-dev (SVOS v2.1.12: `propose_decision` gained `seat_up`, and Hermes reads the plugin only at start). The plugin load was verified at file level; the end-to-end proof is Miranda's first seat_up card. Enabling `zellij-fleet@Claude` at boot remains Prime's call.
		`@@ -0,0 +1,3 @@`
							`# Relying on the build cache surviving on esh-ml1`

							`[2026-09-27]` Relying on the build cache surviving on esh-ml1 — the augaman v0.1.3 build missed its dependency layer there (fv-ml1 hit it; reason unexplained, suspect the `docker builder prune` runs) and the rootfs peaked at 90%. Check for ≥15 GB free, or remove the old image, before building there.
		`@@ -0,0 +1,3 @@`
							`# The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel`

							`[2026-09-27]` The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel. Every tool worked except `get_viewport_screenshot`, which had Blender write a file for the SERVER to read ("Screenshot file was not created"). The server now runs inside the Blender container over ssh+docker-exec stdio. That also means no port is published. → `stacks/blender/README.md`
		`@@ -0,0 +1,3 @@`
							# `uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project

							`[2026-09-27]` `uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project — the export drops uv's per-package index routing; with `--extra-index-url … --index-strategy unsafe-best-match`, triton came from the pytorch index and failed the lock's hash. Keep `uv sync` and blank the project version in a deps-only stage instead (`services/semif-serve/Dockerfile`).