memory: snapshot — Parakeet STT, svos_miranda live, talk v10, address sweep, secrets fix

Closes both of the previous session's named jobs and six unplanned pieces of work.
Nothing is in flight and nothing is blocked.

Parakeet STT live on fv-ml1 GPU 0 behind LiteLLM ext-stt and whisper-1; GPU 3 is now
a documented reserve after the operator caught an 800 MiB seat parked on the one
pristine 96 GB card. svos_miranda enabled and Miranda serving, with
agent.disabled_toolsets deleted and staying out by operator ruling. talk v10
deployed as the STT seat's first consumer. The irv-ml1 dead-address sweep is
complete at 0 of 112 Homepage cards, having turned up four live breakages on other
hosts. The secrets-broker concurrency bug is fixed, and ~/.local/bin/secret is a
symlink rather than a stale copy.

Auto-archival fired at 433 lines but moved only one entry: three of the four
candidates old enough to qualify carry open deferred-work pointers - a park id, an
althing thread, and an explicit 'untracked by operator choice' - and the guard held
them. The index stays over cap at 389 lines, which is the correct trade: nearly
every entry is genuinely under fourteen days old.

The generated handoff needed correcting in-session before it shipped. The model had
turned three operator-deferred items into a to-do list and invited the next session
to commit files that predate this one. Both would have read as instructions to a
fresh context, which is the durable-false-warning failure this session spent the day
documenting.
This commit is contained in:
2026-09-15 09:26:26 -07:00
parent f4320ff57e
commit 653f7fb939
2 changed files with 17 additions and 57 deletions
+4
View File
@@ -4,6 +4,10 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
## Recent decisions (archived)
- `[2026-09-01]` **Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`.
_Archived 2026-09-15._
- `[2026-08-15]` **RP-seat direction: KEEP MeroMero on `char-rp`; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator).** Evaluated `TheDrummer/Artemis-31B-v1.1` — mechanically a drop-in (same `google/gemma-4-31B-it` base, identical 1188-tensor/356-vision census, same missing-`preprocessor_config.json` trick), so it's purely a quality call, and our own survey already ranked MeroMero **#1** vs Artemis **#6**; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. **MTP verified impossible on both** (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). **But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter** — vLLM 0.24 supports `eagle3` + `gemma4_mtp`, and real drafters exist: `google/gemma-4-31B-it-assistant` (0.94 GB, 4-layer, 761K dl), `RedHatAI/gemma-4-31B-it-speculator.eagle3` (4.47 GB), `AEON-7/…eagle3-NVFP4` (3.53 GB). ⚠ all list their verifier as **stock** gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). **Why the Dark-Scarlett 3.8 plan is the strong one:** DS is Qwen3.6-based today, so a 3.8 respin lands on the *gen seat's* architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: `from_pretrained` **silently drops MTP heads during finetuning** (verify 15 `mtp.*` tensors in the index; graft from stock if absent), and DS v1.0 required the `Qwen3_5ForConditionalGeneration` **wrapper class** to save a config vLLM/SGLang accept. Both in `docs/pfi/model-quantization-playbook.md`.
+13 -57
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-15 ~01:45 PT (Parakeet STT LIVE on fv-ml1 GPU 3 + LiteLLM `ext-stt`/`whisper-1`; svos_miranda plugin validated and its blocker fixed by svos-dev — enabling is the operator's call.)_
_Last updated: 2026-09-15 ~09:30 PT (Parakeet STT live on fv-ml1 GPU 0 + LiteLLM `ext-stt`; `svos_miranda` LIVE in Hermes; talk v10 deployed; irv-ml1 dead-address sweep COMPLETE; secrets-broker concurrency bug fixed. Nothing blocked, nothing mid-flight.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,63 +115,21 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-09-15 ~01:15 PT._
_As of 2026-09-15 ~09:30 PT._
### ⭐ BOTH NAMED JOBS CLOSED (2026-09-15 ~01:45 PT)
1. **svos-dev drained** — four messages, thread closed from their end. Ran
`hermes plugins validate/doctor/compat` on `svos_miranda`; found and A/B-proved a
load blocker (absolute intra-package imports), which svos-dev fixed at `c964e64`.
**Enabling is still pending and is the OPERATOR'S call** — see the decision row below.
2. **STT stood up** — Parakeet on fv-ml1 GPU 3, `ext-stt` / `whisper-1` in LiteLLM.
Not greenfield after all: `stacks/parakeet/` already existed (sherpa-onnx + our own
FastAPI wrapper, previously on irv-ml1) and was retargeted rather than rewritten.
### Nothing is blocked and nothing is mid-flight. Both of the previous session's named jobs closed, plus six unplanned pieces of work.
### ⏳ OPEN — svos_miranda enable (needs the operator)
Everything technical is done. Enabling requires editing the operator's live
`~/.hermes/config.yaml` (this session was guardrailed out of that write, then
unblocked — but the go/no-go is his) and svos-dev restarts `:8770` after, which takes
the SVOS board down until the roster verifies. Sequence when he says go: install to
`~/.hermes/plugins/svos_miranda`; add `plugins.enabled` + `entries.svos_miranda.settings`
(dispatch key from `secret get nh3-dev/svos/bifrost-dispatch-key`, `repo_read_enabled: true`);
set `platform_toolsets["api_server"] = ["svos_miranda"]` and `agent.disabled_toolsets`;
**re-derive that list after registration** from `GET http://127.0.0.1:8765/v1/toolsets`
(28 rows now, `svos_miranda` joins once it registers); hand off to svos-dev for the restart.
**Closed this session:**
1. **Parakeet STT live** — fv-ml1 **GPU 0** (not GPU 3), port 8300, v3 int8 25-language model, behind LiteLLM `ext-stt` / `whisper-1`. ⚠ Placed on GPU 3 first; operator corrected it — a ~800 MiB seat belongs on the card with the most uncommitted headroom, not on the one pristine 96 GB card, because vLLM sizes KV against TOTAL VRAM. **GPU 3 is now a deliberate reserve at 2 MiB.**
2. **`svos_miranda` LIVE in Hermes** — gateway restarted, 29 toolsets, Miranda scoped to exactly 8 tools, operator's own surface intact at 46. ⚠ `agent.disabled_toolsets` is DELETED and stays out (operator: "i dont want the tools disabled everywhere"); svos-dev fixed their roster check at `c9d2a96`. SVOS restarted itself; both roster lines verified.
3. **talk v10 deployed** on nh3-dev :8092 — push-to-talk STT through `ext-stt`, barge-in. First consumer of the Parakeet seat.
4. **irv-ml1 dead-address sweep DONE** — 0 of 112 Homepage cards on `10.100.79.3`, was 9. Found and fixed four live breakages on OTHER hosts (Open WebUI TTS, asset-engine, skaldsong x2).
5. **`secret` concurrency bug fixed** — parallel `secret get` returned empty with exit 0. Command-level lock + empty-value guard + `find()` no longer coercing empty stdout to `[]`. `~/.local/bin/secret` is now a symlink, was a stale copy.
6. **Retired:** irv-ml1 parakeet (lost tts-dev's bench) and voice-studio (dots obsoleted by Breeze).
### Fleet networking — closed out this session
- **FV cross-site routing FIXED.** fv-ml1 reaches NH3/ESH/ANA/IRV/internet via outbound
SNAT on the FV gateway. Four rules, all `src=10.251.50.0/24`. `fa04f45`, `0ab9da5`.
- **Mesh membership retired for fv-ml1 AND nh3-dev** (`959a743`, `a65cdf6`). Six nodes
remain, each with a job: nh3-scale, esh-scale, ana-scale, vb-gateway, irv-ml1 (Irvine's
own router), the operator's MacBook Air. The nh3-scale masquerade exception nh3-dev
required was reverted with it.
- **fv-ml1 has break-glass instead of membership** — `fv-mesh-watchdog` joins the mesh
when nh3-dev *and* nh3-docker go unreachable while the WAN is up. Proven off-mesh end
to end. Key: dedicated 1-year reusable, headscale ID 8, expires 2027-09-15, vaulted
`fv-ml1/headscale-breakglass-key`.
- **Exactly one live reusable pre-auth key fleet-wide** (that one). The two stale FV
cutover keys were expired.
**Open, all operator-deferred, none blocking:** the AI-tab Dormant regrouping (belayed), `nconnect=8` on /mnt/smithy (deferred), fused MoE kernel path (park id 47). `speaches`'s label claims `:8204`, which is breeze-tts's live port — a latent conflict if anyone starts it.
### Also landed this session
`root:docker 2775` setgid deploy trees on 5 hosts + `0777` cleared (`826a63b`);
conventions pinned in `docs/pfi/fleet-conventions.md` + audit playbook (`abef67a`);
`linus` deleted, `llmuser` de-privileged (`ce7b07f`); FV firewall config in the nightly
restic run (`80d982d`); `infra-ops` SSH key on the OPNsense gateway; ESPHome pinned and
hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`).
### Open loose ends (none blocking)
- **nh3-dev has no break-glass watchdog, by choice** — NH3 is a populated site with other
boxes to diagnose from, unlike FV where a single gateway is the only path. Reversible:
the script is in `servers/fv-ml1/`.
- **Legacy `/32` ANA NAT rule** redundant and invisible to `source_nat/search_rule`
delete from the OPNsense UI.
- **FV WAN admin is plain HTTP**, allowlisted to ESH/ANA/NH3 — ⚠ pinned to **egress IPs**,
so a WAN change at any site silently removes the fallback.
- **`10.251.250.0/24` (BMC/mgmt)** outside the FV SNAT scope — inbound works.
- **No headscale ACL policy loaded** → mesh is default-allow.
- **Prior-session items still open:** FV site-visit power measurements in
`docs/runbooks/fv-site-dark-20260913.md`; mog-sec blue-team seat fate; deferred park items.
-**fv-ml1 seat topology unchanged** from the 2026-09-14 rebalance — this session touched
no seats.
**Uncommitted:** `graphify-out/GRAPH_REPORT.md` and `scripts/seat-inventory.py` were modified before this session began; untouched and deliberately not committed.
## Recent decisions
@@ -408,8 +366,6 @@ hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`).
- `[2026-09-02]` **althing 3.2.0→3.2.4 deployed, and ALTHING DEPLOY IS FOUR SURFACES not three.** The fourth (plugin) had no runbook step and was frozen at Aug 28 — **missing the SessionStart/SessionEnd hooks and `pane-route.sh` entirely**, so "CC seats re-declare automatically" was never true here. Now one command (`scripts/deploy-althing.sh`). ⚠ `uv tool install .` **without `--force` is a silent no-op**. ⚠ **A missing deploy surface presents as "the migration needs manual work", not as an error.**`persistent-memory.d/2026-09-01-althing-320-deploy.md`
- `[2026-09-01]` **Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`.
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
@@ -418,7 +374,7 @@ hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`).
- `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
_8 older entries archived to archival-memory.md._
_9 older entries archived to archival-memory.md._
## Tried and abandoned