From c488eadc31d88c312fffbd7c387b60acaf055b2c Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 28 Aug 2026 22:22:32 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20althing=20v3?= =?UTF-8?q?=20fleet-wide=20at=203.1.1,=20sec=20on=20GPU0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The in-flight section was a day stale: it still described run 3c as the live subject on a box where nothing had moved. Rewritten around what is actually true now -- v3 deployed fleet-wide, the post office relocated to nh3-docker, sec serving on GPU0, run 3c still held on power. Six new decision entries, three of which carry findings that outlive their incident: the inbound half of the handle-resolution bug (a stale ALTHING_HANDLE reads another agent's mailbox and reports it empty, which is a second route into the failure v3 exists to prevent), the OOM attribution to Claude Code sessions, and the operator's two explicit belays recorded so a later session does not re-raise them as new. Auto-archival fired at 301 lines and moved exactly one entry. Three others were old enough and every one carries a still-open deferred pointer -- the parked CI flip, muninn-gate's submit path, and the triton backend deferred to the Ada refresh. Held back per the guards; an over-cap file that keeps live decisions beats a scannable one that lost them. The entry that did move had its deferred item closed today: nh3-extdev's staged v2.1.2 wheel is moot now that the box runs 3.1.1. --- archival-memory.md | 4 ++ .../2026-08-28-althing-v3-cutover.md | 68 +++++++++++++++++++ ...026-08-28-handle-resolution-wrong-inbox.md | 60 ++++++++++++++++ .../2026-08-28-sec-seat-gpu0.md | 35 ++++++++++ persistent-memory.md | 59 +++++----------- 5 files changed, 184 insertions(+), 42 deletions(-) create mode 100644 persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md create mode 100644 persistent-memory.d/2026-08-28-sec-seat-gpu0.md diff --git a/archival-memory.md b/archival-memory.md index 34785ae..f2ce408 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -4,6 +4,10 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re ## Recent decisions (archived) +- `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`. + _Archived 2026-08-28. Its deferred item — the nh3-extdev herald 2.1.2 upgrade — is closed: extdev went 2.1.0 -> 3.0.0 -> 3.1.1 at the v3 cutover, so the staged v2.1.2 wheel is moot._ + + # eRP dual-seat overhaul — MeroMero-v2 + Dark-Scarlett, NVFP4A16 @ 256K on ana-ml2 `[2026-08-12]` Replaced the two legacy char-rp seats with home-quantized NVFP4A16 vLLM diff --git a/persistent-memory.d/2026-08-28-althing-v3-cutover.md b/persistent-memory.d/2026-08-28-althing-v3-cutover.md index 60b6a6f..b67b6b5 100644 --- a/persistent-memory.d/2026-08-28-althing-v3-cutover.md +++ b/persistent-memory.d/2026-08-28-althing-v3-cutover.md @@ -148,3 +148,71 @@ re-verified end to end. means repeating that by hand. The fleet pattern (skaldsong, soong-lab) is a gitea registry pull; this should join it. The old nh3-dev volume is left in place untouched — not a rollback path (the operator ruled that out), just not deleting the only other copy on the day of the move. + +--- + +## `[2026-08-28, later]` The release train: 3.0.1 → 3.1.1 in one afternoon, and the registry + +Six releases landed the same day as the cutover. **The container was touched exactly once** (the +nh3-docker move); every other release was herald- or client-side, established each time by +forseti's **import-graph argument** — asking what `althing/post_office/*` IMPORTS rather than +reading the diff. A diff tells you what moved; an import graph tells you what can be affected. + + 3.0.1 pane routes (a pane agent can register its own route) + 3.0.2 the docs are now checked against the CLI, not against each other — NO redeploy + 3.0.3 PANE_SETTLE_S=0.3, the write/submit race + 3.1.0 herald writes $ALTHING_ROOT/post-office; dev-launch reads it — BOTH binaries + 3.1.1 warn when ALTHING_HANDLE disagrees with launch-history + +**Deploy recipe per release:** `uv build --wheel` on nh3-dev → `uv tool install --force .` +locally → stage the wheel to nh3-extdev and install under `UV_TOOL_DIR=/opt/uv-tools` with +`/home/infra-ops/.local/bin/uv` → restart both `althing-po-herald` units. +`playbooks/nh3-extdev-althing-v3.yaml` does the extdev half. + +## ⚠ THE REPEATED DEFECT — a verify that half-passes, four costumes in one day + +Every one of these was written correctly for its first run and silently wrong on the next: + + 1. install step gated on `postbox` not existing -> right for the cutover, would SKIP + every release after and report success + 2. content check pinned to the PREVIOUS release's markers -> passes forever, asserts nothing + 3. ONE file:marker pair against a TWO-file release -> asserts half a release + 4. a Go template `{{ }}` inside elway, whose own substitution ate it -> FAILED on a green deploy + +**The check now matches on PRESENCE (`grep -q`), never a count**, and takes a LIST of +`file:marker` pairs bumped per release. 3.1.1's own release note said +`grep -c handles_launched_at dev_launch.py # 2+`; the real count there is 1 (the definition, +with 2 in postbox.py), so a count assertion would have reported FAILED on a byte-perfect install. + +## The registry, and why the namespace is `claude-bot` + + gitea.phasefinal.com/claude-bot/althing-post-office:3.0.0@sha256:410fed41... + +Digest-pinned, not tag-floating — a tag is a mutable pointer on a registry anyone can re-push and +this container is the whole bus. ⚠ **claude-bot's token carries `write:package` and `docker +login` SUCCEEDS, but package namespaces are owned**: pushing to `vh/` returns +`unauthorized: authentication required` AFTER a successful login — an ownership refusal wearing a +credential error's clothes. Publishing under claude-bot's own namespace also satisfies the +standing directive to stop reusing the operator's personal credentials. + +## The deployed CC plugin copies are a release step NOBODY owns + +`sync_skill.sh` covers the SKILL, not the plugin. Nothing in the repo reaches +`~/.local/share/althing-plugin/` or `~/.claude/plugins/cache/althing/althing/0.0.1/`. Both must be +`rsync -a --delete`'d from the repo's `plugin/` **on every release**, by hand, or they carry the +previous release's bugs into the live surface — which happened: forseti's new docs-vs-CLI check +found `postbox reply --to` twice in `plugin/commands/inbox.md`, **the file a CC session reads +every time it drains its inbox**, and both my deployed copies had it. + +⚠ **A running CC session keeps the plugin text it loaded at startup.** Files being right is +necessary and not sufficient — the session has to restart. I synced the copies at 09:30 and was +handed the v2 `/althing:monitor` text an hour later, calling three deleted binaries. + +## Peer skill copies: I stopped hand-syncing, deliberately + +I wrote into eitri/dvalin/bil's repos three times (operator-authorised, and right while they were +dark and could not pull). **Once they were awake and pulling, it became a race I was losing** — +canonical moved three times in an hour and I was chasing a one-line version-banner lag. It also +cost provenance: dvalin had to correct their own account of their file because my write looked +like a pre-existing partial. Two writers, no lock. `sync_skill.sh` deliberately does not write +into peer trees; althing owns canonical and peers pull. Respect that boundary once they can. diff --git a/persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md b/persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md new file mode 100644 index 0000000..70900b8 --- /dev/null +++ b/persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md @@ -0,0 +1,60 @@ +# `[2026-08-28]` A stale `ALTHING_HANDLE` silently reads ANOTHER agent's inbox and reports it empty + +Found by pewpew-dev, whose session posted as **forseti** all day. Mechanism: + + ALTHING_HANDLE inherited from the environment + + nothing binding a shell to the handle it may query + = the env var wins, silently, with no warning and no error + +Measured, same shell, same second, no credential, no complaint: + + ALTHING_HANDLE=infra-ops postbox status -> infra-ops' mailbox + ALTHING_HANDLE=forseti postbox status -> forseti's mailbox + ALTHING_HANDLE=pewpew-dev postbox status -> pewpew-dev's mailbox + +## ⚠ THE INBOUND HALF IS THE SERIOUS ONE, AND IT IS INVISIBLE + +**Outbound** mis-signing sometimes gets caught: a peer notices the sender cannot hold that +context — which is exactly how this surfaced, when forseti was asked a pewpewstudio question. + +**Inbound never does.** A session with a stale handle runs `postbox peek`, reads SOMEONE ELSE'S +mailbox, and is told — confidently, correctly, nothing broken anywhere — that it has nothing +unread. Measured cost: my reply sat unread until pewpew-dev's operator asked whether they were +blocked on me. + +**This is a SECOND route into the failure v3 exists to prevent.** The guarantee — *"an +unreachable post office is an OUTAGE, never an empty inbox"* — holds, and does not cover this: +the post office is reachable and answers correctly, **about someone else**. Nothing is down, so +the outage semantics never fire. forseti has stopped describing that guarantee as though it +closes the empty-inbox class; it closes one route into it. + +## ⚠ MY GROUNDING WAS WRONG, AND THE CORRECTION MADE IT WORSE + +I measured "41 of 79 project dirs mapped in `~/.althing/session_handles.json`" and called the 46 +unmapped ones exposed. **`session_handles.json` is a v2 artifact that v3 never opens** — +`grep -rn session_handles althing/` is empty, and `postbox.resolve_config` takes `--handle` then +`ALTHING_HANDLE` and nothing else. So the exposure is LARGER than my number implied: every +directory is in the same position, because the map is consulted for none of them. + +**The same stale data source had survived inside my statusline rewrite that morning.** I updated +the v2 block's COMMANDS and kept its DATA SOURCE — the commands were the visible half of the +cutover and the source was not. Fixed (`8a04d6f`): it reads `launch-history.json` now (written by +`dev_launch`, shape `{cwd: {command: {at, handle}}}`, take the most recent by `at`). + +## The fix that shipped, and why my proposal was the worse one + +I proposed printing the resolution source (`handle: forseti (from ALTHING_HANDLE)`). forseti +killed it with one observation: **`postbox status` already prints the handle, first field, every +call.** pewpew-dev had `forseti` on screen and it did not register. **The information was never +missing; the salience was.** That generalised into the rule that chose the design: a line that is +always there teaches the reader to skip it, so a warning beats a field. + +Shipped as **3.1.1** — warn (never refuse; a legitimate cross-project send is real) when the cwd's +`launch-history` names a different handle, on stderr, BEFORE the output it is about. Keyed by +directory across commands, not `(directory, command)` — one repo hosting a claude and a codex +session under different handles is normal, and a warning that fires on a legitimate case is one +people learn to ignore. + +⚠ **Its silence is not an all-clear:** 8 of 79 directories are in `launch-history.json`, so the +quiet case is ~90% of the box. forseti put that caveat in a TEST NAME +(`test_an_unlaunched_directory_says_nothing`) because prose gets skimmed and a test goes red. diff --git a/persistent-memory.d/2026-08-28-sec-seat-gpu0.md b/persistent-memory.d/2026-08-28-sec-seat-gpu0.md new file mode 100644 index 0000000..7b95209 --- /dev/null +++ b/persistent-memory.d/2026-08-28-sec-seat-gpu0.md @@ -0,0 +1,35 @@ +# `[2026-08-28]` The `sec` pen-test seat moved GPU1 → GPU0 and came up — on a circuit that tripped 36h earlier + +Operator-directed. `sec` = M.O.G.-SEC-27B (`stacks/mog-sec/`, LiteLLM aliases `sec` / +`sec-reasoning`, ana-ml2 `:8019`, 262K native ctx, NVFP4+FP8 mixed with a grafted MTP head). + +## Why it had to move + + GPU1 69,895 MiB used of 97,887 — gen 46 GB + embed 9.8 + coder 8.4 + rerank 3.5 + reward 2.1 + -> ~28 GB free, against the ~51 GB this seat reserves at MOG_GPU_MEM_UTIL=0.52 + -> it could not start on GPU1 AT ALL + GPU0 empty since run 3c was stopped 2026-08-26 + +One line: `MOG_GPU_ID=1 -> 0` in `/opt/docker/compose/mog-sec/.env`. The compose default, the +header comment and the **homepage label** all named GPU 1 and were corrected in the same change — +a label naming the wrong card is a record that lies about where the work runs. + +## Landed state + + container vllm-mog-sec, healthy after ~400s load (22 GB model off the DEGRADED /tank) + GPU0 51,532 MiB, idle draw 16.34 W + GPU1 69,895 MiB, idle draw 6.42 W + +⚠ **This re-arms the two-GPU load condition that tripped the Anaheim rack breaker on 08-26.** +Idle draw is negligible — ~23 W across both cards. **The risk materialises only under concurrent +load**, when both seats work at once and the box approaches the ~600 W that tripped it. The +operator accepted that with the constraint stated. See +[[2026-08-27-anaheim-breaker-and-onboot-gap]] — one circuit feeds the whole rack including ana-gw +and ana-wg, so a trip costs the site AND the way back in. + +## Deploy gotchas worth keeping + +- `up -d`, never `restart` — **labels bind at container creation**, so a restart keeps the old + homepage label and the dashboard silently keeps showing the old GPU number. +- Diff deployed-vs-canonical BEFORE pushing. There was no drift here, which is the only reason + the push was safe to make blind. diff --git a/persistent-memory.md b/persistent-memory.md index 73467cd..ae40f2e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-08-27_ +_Last updated: 2026-08-28_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -108,17 +108,25 @@ no longer deployed sidecars here. See Recent decisions.) (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight -_State as left 2026-08-26 23:05 PDT (written 2026-08-27 07:27, corrected 10:5x) — **run 3 is trained, gated and DO-NOT-SERVE on a safety finding. Run 3c is STOPPED at last-logged step 24/604, killed deliberately at 2026-08-26 21:07:40 PDT after Anaheim tripped a power breaker.** Nothing is training._ +_As of 2026-08-28 16:20 PDT — **althing v3 is live fleet-wide at 3.1.1 with the post office on nh3-docker. `sec` is serving on ana-ml2 GPU0. Nothing is training.**_ -- **⏸ RUN 3c HELD — operator stopped it, power capacity is the blocker.** lr `2e-4 -> 1e-5`, corpus BYTE-IDENTICAL, single variable proven by diff. Config `/tank/erp-tune/run-03c.json` is built and validated (`save_steps 50`, both deviations recorded as separate entries — the scientific lr change and the operational checkpoint cadence). Relaunch is one command. **Do not relaunch until the power triage lands** — ana-ml2 pulls ~600 W across both GPUs at their caps while training, and that is what tripped the breaker. **Exactly TWO 3c launches, and only one of them died:** #1 17:53:33 → killed by the power loss at step 80/604; #2 20:58:41 → stopped BY ME at 21:07:40, healthy, on operator instruction. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md` -- **🔴 `/tank` DEGRADED on ana-ml2 — a disk is genuinely gone**, 7 physical NVMe where the pool expects 8. raidz2, one parity disk spent, no data errors. **Operator is replacing it** (Supermicro AS-4125GS-TNRT2, PCIe hot-plug, should not need a power-down). → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md` -- **🟢 ADAPTERS BACKED UP OFF-SITE** — `nh3-nas:/volume1/smithy/erp-tune-adapter-backup/{run-01,run-02,run-03}`, sha256 verified at source, staging and rest (944 MB). They were single-copy mode-600 on the degraded pool. The four `merged-run03*` models are NOT backed up **by design** — all are derived from `run-03/adapter` by a documented verified merge, ~10 min each to regenerate from a 315 MB artifact that is now safe. -- **⏸ WEEKEND: power triage, "probably shut down some seats"** (operator). `gen` stays up by instruction. Everything else on ana-ml2 is idle. Sheddable: `vllm-gen` 46 GB (the big one), embed 9.8 GB, coder 8.4 GB, rerank-a3 3.5 GB, reward 2.1 GB, scriberr. -- **⏸ Owed to brokkr-smithy-dev when there is a card again:** nothing blocking. They hold the dose curve, the entanglement finding, and an unsourced "~60%" figure in R47 §5 they were checking against MeroMero v2's card — if it is not there either, §5's recommendation loses its basis. -- ⚠ **`/mnt/smithy` will be MISSING after every ana-ml2 reboot** — manual by design, not an oversight. Remount spec in `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`. Do NOT add it to fstab. +- **⏸ RUN 3c STILL HELD — power capacity, unchanged.** lr `2e-4 -> 1e-5`, corpus BYTE-IDENTICAL, config `/tank/erp-tune/run-03c.json` validated, relaunch is one command. **Do not relaunch until the power triage lands.** ⚠ `sec` now occupies GPU0 (~51 GB), so a 3c relaunch needs GPU0 freed OR accepts three-way contention. Exactly TWO 3c launches, only one died. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md` +- **🟢 `sec` (M.O.G.-SEC-27B) IS UP on ana-ml2 GPU0 :8019**, operator-directed. 51,532 MiB, idle 16 W. ⚠ Re-arms the two-GPU load condition that tripped the rack breaker; the risk is concurrent load, not idle. → `persistent-memory.d/2026-08-28-sec-seat-gpu0.md` +- **🟢 althing v3.1.1 on both heralds; post office on nh3-docker `http://10.100.50.40:8390`.** Container untouched since the move. All 73 handles seeded; 5 agents push-reachable (infra-ops, forseti, and the four smithy peers via pane routes). → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` +- **🔴 `/tank` DEGRADED on ana-ml2** — 7 physical NVMe where the pool expects 8, raidz2, one parity spent, no data errors. **Operator replacing it.** → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md` +- **⏸ WEEKEND: power triage, "probably shut down some seats"** (operator). Sheddable on ana-ml2: `vllm-gen` 46 GB, `sec` 51 GB, embed 9.8, coder 8.4, rerank-a3 3.5, reward 2.1, scriberr. +- **⏸ Owed to brokkr-smithy-dev when there is a card again:** nothing blocking. They hold the dose curve and the entanglement finding. +- **⏸ Awaiting pewpew-dev:** which other boxes/CI runners resolve `pypotrace` and need the potrace headers. nh3-dev is done; `playbooks/install-potrace-headers.yaml` makes each additional box one command. +- ⚠ **`/mnt/smithy` will be MISSING after every ana-ml2 reboot** — manual by design. → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md` ## Recent decisions +- `[2026-08-28]` **althing v3 flag day (U9b) executed, then six releases to 3.1.1 in one afternoon — and the post office MOVED to nh3-docker.** Every v2 command deleted; 73 handles seeded and verified by set difference; 5,043 orphaned wake FIFOs deleted (v2 named them per-session+PID, v3 per-handle). Image now registry-pulled, digest-pinned, under the `claude-bot` namespace. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` +- `[2026-08-28]` **A stale `ALTHING_HANDLE` silently reads another agent's inbox and reports it empty — a SECOND route into the failure v3 exists to prevent.** Outbound mis-signing sometimes gets caught; inbound never does. Shipped as a 3.1.1 warning. ⚠ My `session_handles.json` grounding was wrong (v2 artifact, v3 never opens it) and the same stale source had survived inside my statusline rewrite. → `persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md` +- `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE, and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured 5.4-18.4 GB, so 27 GB is 3-4 long-lived sessions. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md` +- `[2026-08-28]` **`sec` moved to ana-ml2 GPU0 and is serving** (operator-directed) — GPU1 had ~28 GB free against the ~51 GB it reserves, so it could not start there. Re-arms the two-GPU load condition on a circuit that tripped 36h earlier; accepted with the constraint stated. → `persistent-memory.d/2026-08-28-sec-seat-gpu0.md` +- `[2026-08-28]` **BELAYED by the operator, both explicitly: (a) a cgroup memory cap on CC sessions, (b) putting ana-gw + ana-wg + one BMC on separate power.** Both were my recommendations; neither is open work. Do not re-raise as new — the atop ramps that would inform (a) are now being collected, so revisit only with a week of data. Tracking surface: this entry. +- `[2026-08-28]` **The deployed CC plugin copies are a release step nobody owns.** `sync_skill.sh` covers the SKILL, not the plugin; both copies must be rsync'd from the repo's `plugin/` on every althing release or they carry the previous release's bugs into the live surface. Raised with forseti for their release notes. Tracking surface: althing thread `01M14QHZNDKDK8KH9DN92VF6VE`. - `[2026-08-28]` **althing v3.0.0 flag day (U9b) executed — the post office replaced the P2P bus on both boxes, one-way.** 73 handles seeded and verified by set difference; 5,043 orphaned v2 wake FIFOs deleted (v2 named them per-session+PID and never reaped; v3 names them per-handle, so the leak is bounded by construction); v2 db left inert. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` - `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE — and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured at 5.4-18.4 GB, so 27 GB is 3-4 mature sessions, not the ~66 a 408 MB estimate implies. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md` - `[2026-08-27]` **Run 3 gated: the preregistered rule PASSED and a k=25 follow-up found a 44pp self-harm guardrail collapse — DO NOT SERVE.** A pooled preserve-list test structurally cannot see a single-axis collapse. → `persistent-memory.d/2026-08-27-run3-gate-safety-regression.md` @@ -241,40 +249,15 @@ _State as left 2026-08-26 23:05 PDT (written 2026-08-27 07:27, corrected 10:5x) - `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md` - - - - - - `[2026-08-10→12]` **secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed.** `secret` CLI (`put/get/list/rm/backfill`, bw-backed) on `~/.local/bin`; 25 nh3-dev secrets backfilled + round-trip-verified; `rm` + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → `persistent-memory.d/2026-08-12-secrets-broker.md` - - - - - - - - - - - `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/.wav` + `transcripts/.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]] - - `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md` - -- `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`. - - - - - `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md` - - - -_222 older entries archived to archival-memory.md._ +_223 older entries archived to archival-memory.md._ ## Tried and abandoned - `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not. @@ -290,12 +273,4 @@ _222 older entries archived to archival-memory.md._ - `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`. - - - - - - - - _143 older entries archived to archival-memory.md._