diff --git a/archival-memory.md b/archival-memory.md index 6b5cb96..d8c2e1b 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -4,6 +4,10 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re ## Recent decisions (archived) +- `[2026-09-01]` **Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`. + _Archived 2026-09-15._ + + - `[2026-08-15]` **RP-seat direction: KEEP MeroMero on `char-rp`; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator).** Evaluated `TheDrummer/Artemis-31B-v1.1` — mechanically a drop-in (same `google/gemma-4-31B-it` base, identical 1188-tensor/356-vision census, same missing-`preprocessor_config.json` trick), so it's purely a quality call, and our own survey already ranked MeroMero **#1** vs Artemis **#6**; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. **MTP verified impossible on both** (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). **But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter** — vLLM 0.24 supports `eagle3` + `gemma4_mtp`, and real drafters exist: `google/gemma-4-31B-it-assistant` (0.94 GB, 4-layer, 761K dl), `RedHatAI/gemma-4-31B-it-speculator.eagle3` (4.47 GB), `AEON-7/…eagle3-NVFP4` (3.53 GB). ⚠ all list their verifier as **stock** gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). **Why the Dark-Scarlett 3.8 plan is the strong one:** DS is Qwen3.6-based today, so a 3.8 respin lands on the *gen seat's* architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: `from_pretrained` **silently drops MTP heads during finetuning** (verify 15 `mtp.*` tensors in the index; graft from stock if absent), and DS v1.0 required the `Qwen3_5ForConditionalGeneration` **wrapper class** to save a config vLLM/SGLang accept. Both in `docs/pfi/model-quantization-playbook.md`. diff --git a/persistent-memory.md b/persistent-memory.md index f889cd7..d4c2e51 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-15 ~01:45 PT (Parakeet STT LIVE on fv-ml1 GPU 3 + LiteLLM `ext-stt`/`whisper-1`; svos_miranda plugin validated and its blocker fixed by svos-dev — enabling is the operator's call.)_ +_Last updated: 2026-09-15 ~09:30 PT (Parakeet STT live on fv-ml1 GPU 0 + LiteLLM `ext-stt`; `svos_miranda` LIVE in Hermes; talk v10 deployed; irv-ml1 dead-address sweep COMPLETE; secrets-broker concurrency bug fixed. Nothing blocked, nothing mid-flight.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -115,63 +115,21 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-09-15 ~01:15 PT._ +_As of 2026-09-15 ~09:30 PT._ -### ⭐ BOTH NAMED JOBS CLOSED (2026-09-15 ~01:45 PT) -1. **svos-dev drained** — four messages, thread closed from their end. Ran - `hermes plugins validate/doctor/compat` on `svos_miranda`; found and A/B-proved a - load blocker (absolute intra-package imports), which svos-dev fixed at `c964e64`. - ⚠ **Enabling is still pending and is the OPERATOR'S call** — see the decision row below. -2. **STT stood up** — Parakeet on fv-ml1 GPU 3, `ext-stt` / `whisper-1` in LiteLLM. - Not greenfield after all: `stacks/parakeet/` already existed (sherpa-onnx + our own - FastAPI wrapper, previously on irv-ml1) and was retargeted rather than rewritten. +### Nothing is blocked and nothing is mid-flight. Both of the previous session's named jobs closed, plus six unplanned pieces of work. -### ⏳ OPEN — svos_miranda enable (needs the operator) -Everything technical is done. Enabling requires editing the operator's live -`~/.hermes/config.yaml` (this session was guardrailed out of that write, then -unblocked — but the go/no-go is his) and svos-dev restarts `:8770` after, which takes -the SVOS board down until the roster verifies. Sequence when he says go: install to -`~/.hermes/plugins/svos_miranda`; add `plugins.enabled` + `entries.svos_miranda.settings` -(dispatch key from `secret get nh3-dev/svos/bifrost-dispatch-key`, `repo_read_enabled: true`); -set `platform_toolsets["api_server"] = ["svos_miranda"]` and `agent.disabled_toolsets`; -**re-derive that list after registration** from `GET http://127.0.0.1:8765/v1/toolsets` -(28 rows now, `svos_miranda` joins once it registers); hand off to svos-dev for the restart. +**Closed this session:** +1. **Parakeet STT live** — fv-ml1 **GPU 0** (not GPU 3), port 8300, v3 int8 25-language model, behind LiteLLM `ext-stt` / `whisper-1`. ⚠ Placed on GPU 3 first; operator corrected it — a ~800 MiB seat belongs on the card with the most uncommitted headroom, not on the one pristine 96 GB card, because vLLM sizes KV against TOTAL VRAM. **GPU 3 is now a deliberate reserve at 2 MiB.** +2. **`svos_miranda` LIVE in Hermes** — gateway restarted, 29 toolsets, Miranda scoped to exactly 8 tools, operator's own surface intact at 46. ⚠ `agent.disabled_toolsets` is DELETED and stays out (operator: "i dont want the tools disabled everywhere"); svos-dev fixed their roster check at `c9d2a96`. SVOS restarted itself; both roster lines verified. +3. **talk v10 deployed** on nh3-dev :8092 — push-to-talk STT through `ext-stt`, barge-in. First consumer of the Parakeet seat. +4. **irv-ml1 dead-address sweep DONE** — 0 of 112 Homepage cards on `10.100.79.3`, was 9. Found and fixed four live breakages on OTHER hosts (Open WebUI TTS, asset-engine, skaldsong x2). +5. **`secret` concurrency bug fixed** — parallel `secret get` returned empty with exit 0. Command-level lock + empty-value guard + `find()` no longer coercing empty stdout to `[]`. `~/.local/bin/secret` is now a symlink, was a stale copy. +6. **Retired:** irv-ml1 parakeet (lost tts-dev's bench) and voice-studio (dots obsoleted by Breeze). -### Fleet networking — closed out this session -- **FV cross-site routing FIXED.** fv-ml1 reaches NH3/ESH/ANA/IRV/internet via outbound - SNAT on the FV gateway. Four rules, all `src=10.251.50.0/24`. `fa04f45`, `0ab9da5`. -- **Mesh membership retired for fv-ml1 AND nh3-dev** (`959a743`, `a65cdf6`). Six nodes - remain, each with a job: nh3-scale, esh-scale, ana-scale, vb-gateway, irv-ml1 (Irvine's - own router), the operator's MacBook Air. The nh3-scale masquerade exception nh3-dev - required was reverted with it. -- **fv-ml1 has break-glass instead of membership** — `fv-mesh-watchdog` joins the mesh - when nh3-dev *and* nh3-docker go unreachable while the WAN is up. Proven off-mesh end - to end. Key: dedicated 1-year reusable, headscale ID 8, expires 2027-09-15, vaulted - `fv-ml1/headscale-breakglass-key`. -- **Exactly one live reusable pre-auth key fleet-wide** (that one). The two stale FV - cutover keys were expired. +**Open, all operator-deferred, none blocking:** the AI-tab Dormant regrouping (belayed), `nconnect=8` on /mnt/smithy (deferred), fused MoE kernel path (park id 47). `speaches`'s label claims `:8204`, which is breeze-tts's live port — a latent conflict if anyone starts it. -### Also landed this session -`root:docker 2775` setgid deploy trees on 5 hosts + `0777` cleared (`826a63b`); -conventions pinned in `docs/pfi/fleet-conventions.md` + audit playbook (`abef67a`); -`linus` deleted, `llmuser` de-privileged (`ce7b07f`); FV firewall config in the nightly -restic run (`80d982d`); `infra-ops` SSH key on the OPNsense gateway; ESPHome pinned and -hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`). - -### Open loose ends (none blocking) -- **nh3-dev has no break-glass watchdog, by choice** — NH3 is a populated site with other - boxes to diagnose from, unlike FV where a single gateway is the only path. Reversible: - the script is in `servers/fv-ml1/`. -- **Legacy `/32` ANA NAT rule** redundant and invisible to `source_nat/search_rule` — - delete from the OPNsense UI. -- **FV WAN admin is plain HTTP**, allowlisted to ESH/ANA/NH3 — ⚠ pinned to **egress IPs**, - so a WAN change at any site silently removes the fallback. -- **`10.251.250.0/24` (BMC/mgmt)** outside the FV SNAT scope — inbound works. -- **No headscale ACL policy loaded** → mesh is default-allow. -- **Prior-session items still open:** FV site-visit power measurements in - `docs/runbooks/fv-site-dark-20260913.md`; mog-sec blue-team seat fate; deferred park items. -- ⚠ **fv-ml1 seat topology unchanged** from the 2026-09-14 rebalance — this session touched - no seats. +**Uncommitted:** `graphify-out/GRAPH_REPORT.md` and `scripts/seat-inventory.py` were modified before this session began; untouched and deliberately not committed. ## Recent decisions @@ -408,8 +366,6 @@ hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`). - `[2026-09-02]` **althing 3.2.0→3.2.4 deployed, and ALTHING DEPLOY IS FOUR SURFACES not three.** The fourth (plugin) had no runbook step and was frozen at Aug 28 — **missing the SessionStart/SessionEnd hooks and `pane-route.sh` entirely**, so "CC seats re-declare automatically" was never true here. Now one command (`scripts/deploy-althing.sh`). ⚠ `uv tool install .` **without `--force` is a silent no-op**. ⚠ **A missing deploy surface presents as "the migration needs manual work", not as an error.** → `persistent-memory.d/2026-09-01-althing-320-deploy.md` -- `[2026-09-01]` **Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`. - - `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312. @@ -418,7 +374,7 @@ hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`). - `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now"). -_8 older entries archived to archival-memory.md._ +_9 older entries archived to archival-memory.md._ ## Tried and abandoned