memory: snapshot — the Ada inference server is a stripped used R750xa

Dell R750xa JPJ1ZP3, 2x RTX 6000 Ada to be fitted, ComfyUI's new home at NH3. Diffing
Dell's factory CSV against the reseller invoice shows four downgrades: half the RAM, the
2400 W PSUs, and the GPU risers, cables and high-performance fans all absent.

Records the resolved GPU power chain, correcting my own first answer: the chassis' CPU
8-pin cabling is the right source type and NVIDIA 930-00030-1546-000 bridges it to the
card's 12VHPWR, so the PCIe-type RCCWC I first proposed is withdrawn. Also closes the NVMe
question — the backplane is SAS/SATA only — and notes that free PCIe slots may moot it.

Auto-archival fired at 308 lines; seven entries moved to archival-memory.md. The 250-line
target was not reached because the guards hold nearly everything else back as under 14 days
or carrying open deferred work.
This commit is contained in:
vh
2026-09-01 17:01:14 -07:00
parent 7142657749
commit ace839c768
4 changed files with 186 additions and 61 deletions
+16
View File
@@ -4,6 +4,22 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
## Recent decisions (archived) ## Recent decisions (archived)
- `[2026-08-16]` **Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated.** brokkr-smithy-dev supplied the framing that reproduces (`01M05M48R4RSZF9D8KT7RR55EJ`): a **bare assistant-mode instruction** — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing **DS 0.0% / FF 0.0%** (n=75); plain character cards **DS 1.4% / FF 0.0%** (n=74); bare instruction **DS 92.5% (37/40) / FF 15.8% (6/38)**. Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→**0**, consensual 80→20, dubcon 80→**0**, self-harm 80→**0**. DS refused **25/25** on the five axes brokkr flagged. Root cause: `ReadyArt/Dark-Scarlett-v1.0-27B` is a plain finetune of stock `Qwen/Qwen3.6-27B` carrying **NO abliteration** — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-**ablated** (structural), which is why it holds. ⚠ **RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID.** His diff against his own artifact showed my `battery-instruct.yaml` reproduces only his **`creative` class — 8 of 16 axes**; it dropped all 5 `operational` (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 `meta` (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. **His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta.** Cause: I rebuilt his shape from his *message*, and the `class` field lives in the artifact, not the prose. **Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability.** ⚠ **Known battery bug left unfixed for comparability:** DS's arm-3 control gate failed at 11% because `ictrl-reunion` pairs "explicit / do not fade to black" with *brothers*, which DS reasonably read as an incest request; FF did not. `ictrl-storm` is the clean control. Commit `b9e68c3`.
_Archived 2026-09-01._
- `[2026-08-15]` **Quant lessons consolidated into `docs/pfi/model-quantization-playbook.md` — the durable home; read it BEFORE any requant.** Survey found quant knowledge scattered across 18 files in 4 trees, with **three** documents having independently written overlapping "landmines" sections (the loader-class trap alone was rediscovered 3×). Playbook owns the **transferable** lessons (scheme choice, landmines, acceptance gate + its 3 measurement traps, hardware/co-residency); per-model artifacts are demoted to worked examples that link up. Carries a **superseded-claims table** — which immediately earned itself: the heretic2 runbook's "use modelopt, compressed-tensors can't load the BF16 MTP" is **false** (the cause was the missing `re:^mtp.*` ignore, not the format) and would have sent the next session down the modelopt dependency-hell path; that runbook now carries a stale-warning header. Maintenance rule in `CLAUDE.md`: model-agnostic → playbook, model-specific → stays put, wrong claim → dated superseded row, never a silent edit. Motivated by Qwen3.8 having just released — the next model swap needs a requant. Commit `a91cc3f`.
_Archived 2026-09-01._
- `[2026-08-15]` **Operator ruling: the gen seat's +1.7% perplexity is an acceptable price for the speed — SETTLED, don't re-litigate.** Precise attribution for future reasoning: it is the **activation-quantization** cost (W4A4 MLPs + FP8 attention vs BF16 activations), not an MTP cost — PPL was measured with speculative decoding **off** on both builds, so MTP was not in the loop. Turning MTP off would not recover it; only reverting the quant would (rollback = one `.env` line, old build intact at `…/qwen38-27b-uncensored-nvfp4`).
_Archived 2026-09-01._
- `[2026-08-15]` **gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed.** The queued "W4A8" (NVFP4 weights + FP8 activations) is **not servable** — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and `CompressedTensorsW4A8Fp8` is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter **per-layer-group**. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did **80.12** (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: `unsloth/Qwen3.8-27B-NVFP4` was already on-box → served as a probe, measured **+19.1% at identical acceptance**, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → **80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%**; surface 6/6 live, 7 aliases routing. char-rp had **no** tool parser at all (every tools request 400'd) → `gemma4` tool + reasoning parser + a **mandatory** `enable_thinking:false` (the parser defaults it True → null `content` for all RP prose; proven byte-identical prompt before deploying). Commits `b8f0f4c`, `74f596b`. Foot-guns banked (llm-compressor prunes unmatched `ignore` entries → the 0%-MTP bug, **fired on this run**; prompt_logprobs uniform under spec-decode; 0600 `.env` silently no-ops compose; GPU0 is zero-sum). → `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`
_Archived 2026-09-01._
- `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
_Archived 2026-09-01._
- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md`
_Archived 2026-09-01._
- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md`
_Archived 2026-09-01._
- `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`. - `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`.
_Archived 2026-08-28. Its deferred item — the nh3-extdev herald 2.1.2 upgrade — is closed: extdev went 2.1.0 -> 3.0.0 -> 3.1.1 at the v3 cutover, so the staged v2.1.2 wheel is moot._ _Archived 2026-08-28. Its deferred item — the nh3-extdev herald 2.1.2 upgrade — is closed: extdev went 2.1.0 -> 3.0.0 -> 3.1.1 at the v3 cutover, so the staged v2.1.2 wheel is moot._
+48 -54
View File
@@ -1,16 +1,16 @@
# Graph Report - eshpfi-management (2026-09-01) # Graph Report - eshpfi-management (2026-09-01)
## Corpus Check ## Corpus Check
- 378 files · ~573,530 words - 382 files · ~576,491 words
- Verdict: corpus is large enough that graph structure adds value. - Verdict: corpus is large enough that graph structure adds value.
## Summary ## Summary
- 3878 nodes · 4117 edges · 426 communities (385 shown, 41 thin omitted) - 3906 nodes · 4144 edges · 424 communities (384 shown, 40 thin omitted)
- Extraction: 99% EXTRACTED · 1% INFERRED · 0% AMBIGUOUS · INFERRED: 38 edges (avg confidence: 0.71) - Extraction: 99% EXTRACTED · 1% INFERRED · 0% AMBIGUOUS · INFERRED: 38 edges (avg confidence: 0.71)
- Token cost: 0 input · 0 output - Token cost: 0 input · 0 output
## Graph Freshness ## Graph Freshness
- Built from commit: `73866f6a` - Built from commit: `71426577`
- Run `git rev-parse HEAD` and compare to check if the graph is stale. - Run `git rev-parse HEAD` and compare to check if the graph is stale.
- Run `graphify update .` after code changes (no API cost). - Run `graphify update .` after code changes (no API cost).
@@ -175,7 +175,7 @@
- blend_kokoro_voice.py - blend_kokoro_voice.py
- Handler - Handler
- estimated_latency - estimated_latency
- Path - booth/app.py
- bench.py - bench.py
- serve_nvfp4.sh - serve_nvfp4.sh
- bench_w4a16_recheck.json - bench_w4a16_recheck.json
@@ -217,7 +217,7 @@
- phasefinal-web - phasefinal-web
- build.sh script - build.sh script
- heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP) - heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP)
- booth/app.py - FastAPI
- soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated) - soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated)
- nh3-dev `~/development` — hourly off-box backup - nh3-dev `~/development` — hourly off-box backup
- nh3-dev-development-backup.sh script - nh3-dev-development-backup.sh script
@@ -339,7 +339,7 @@
- [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing - [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing
- optional - optional
- properties - properties
- gpu.py - kl_divergence.py
- step - step
- `[2026-08-27]` A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find - `[2026-08-27]` A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find
- host - host
@@ -381,9 +381,9 @@
- `[2026-08-27]` Run 3 gated: the rule PASSED and a k=25 follow-up found a self-harm guardrail collapse - `[2026-08-27]` Run 3 gated: the rule PASSED and a k=25 follow-up found a self-harm guardrail collapse
- CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE - CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE
- ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for - ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for
- stream_chunks - `[2026-09-01]` Ada migration settled on `zfs send` — and branch (b) was never available
- CatalogField - CatalogField
- ChunkResult - `[2026-09-01]` Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed
- type - type
- `[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way - `[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way
- counted_classifier.py - counted_classifier.py
@@ -411,9 +411,7 @@
- index-tts/app.py - index-tts/app.py
- parakeet/app.py - parakeet/app.py
- tts/app.py - tts/app.py
- adapter/server.py - generate_pickup_id
- test_keep_and_unkeep_go_through_the_same_name_guard
- FastAPI
- license - license
- version - version
- FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through - FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through
@@ -433,19 +431,19 @@
## Surprising Connections (you probably didn't know these) ## Surprising Connections (you probably didn't know these)
- `convert()` --references--> `Any` [EXTRACTED] - `convert()` --references--> `Any` [EXTRACTED]
tools/mistral-small4-nvfp4/convert_hf_to_native.py → stacks/omnivoice/app.py tools/mistral-small4-nvfp4/convert_hf_to_native.py → stacks/omnivoice/app.py
- `_speaker_embedding()` --references--> `Tensor` [EXTRACTED]
stacks/zonos/adapter/server.py → services/coldfusion-abliteration/kl_divergence.py
- `test_cli_unlink_by_id_even_when_the_id_is_all_digits()` --calls--> `parse_link_entries()` [INFERRED] - `test_cli_unlink_by_id_even_when_the_id_is_all_digits()` --calls--> `parse_link_entries()` [INFERRED]
services/booth/tests/test_booth.py → services/booth/booth/links.py services/booth/tests/test_booth.py → services/booth/booth/links.py
- `test_cli_unlink_of_a_stale_id_leaves_the_board_alone()` --calls--> `parse_link_entries()` [INFERRED] - `test_cli_unlink_of_a_stale_id_leaves_the_board_alone()` --calls--> `parse_link_entries()` [INFERRED]
services/booth/tests/test_booth.py → services/booth/booth/links.py services/booth/tests/test_booth.py → services/booth/booth/links.py
- `test_cli_unlink_rejects_a_non_id_non_index()` --calls--> `parse_link_entries()` [INFERRED] - `test_cli_unlink_rejects_a_non_id_non_index()` --calls--> `parse_link_entries()` [INFERRED]
services/booth/tests/test_booth.py → services/booth/booth/links.py services/booth/tests/test_booth.py → services/booth/booth/links.py
- `test_remove_on_a_board_with_no_file_is_a_no_op()` --calls--> `remove_link_entry()` [INFERRED]
services/booth/tests/test_booth.py → services/booth/booth/links.py
## Import Cycles ## Import Cycles
- None detected. - None detected.
## Communities (426 total, 41 thin omitted) ## Communities (424 total, 40 thin omitted)
### Community 0 - "quant_modelopt.py" ### Community 0 - "quant_modelopt.py"
Cohesion: 0.53 Cohesion: 0.53
@@ -761,7 +759,7 @@ Nodes (7): ana-ml2, GPU allocation policy, Hardware, Key paths, Network, Refresh
### Community 78 - "test_booth.py" ### Community 78 - "test_booth.py"
Cohesion: 0.06 Cohesion: 0.06
Nodes (33): build_gallery(), Files in a booth as render items, with caption sidecars folded in. A…, (rendered, is_html). Markdown → HTML (fenced code, tables, sane lists); plain…, render_doc(), Releasing something already released is a no-op, not a 500., test_booth_autogallery_renders_media(), test_build_gallery_folds_caption_sidecars(), test_build_gallery_oversize_doc_is_not_inlined() (+25 more) Nodes (37): build_gallery(), Files in a booth as render items, with caption sidecars folded in. A…, (rendered, is_html). Markdown → HTML (fenced code, tables, sane lists); plain…, render_doc(), Releasing something already released is a no-op, not a 500., Both mutating routes must use resolve_booth, not raw path joining. A name…, It is a dotfile, so it must not inflate the item count or become a tile., test_booth_autogallery_renders_media() (+29 more)
### Community 79 - "pfi-tacticalrmm" ### Community 79 - "pfi-tacticalrmm"
Cohesion: 0.25 Cohesion: 0.25
@@ -1075,9 +1073,9 @@ Nodes (6): H, BaseHTTPRequestHandler, Handler, zed-fim-proxy — a keyless front
Cohesion: 0.67 Cohesion: 0.67
Nodes (3): anyOf, default, estimated_latency Nodes (3): anyOf, default, estimated_latency
### Community 160 - "Path" ### Community 160 - "booth/app.py"
Cohesion: 0.14 Cohesion: 0.13
Nodes (18): booth_age_seconds(), booth_image_names(), classify(), doc_kind(), list_booths(), _newest_mtime(), Path, Image files in a booth, in gallery (sorted-rel) order — for viewer prev/next. (+10 more) Nodes (22): booth_age_seconds(), booth_image_names(), classify(), doc_kind(), is_expired(), list_booths(), _newest_mtime(), Path (+14 more)
### Community 163 - "bench_w4a16_recheck.json" ### Community 163 - "bench_w4a16_recheck.json"
Cohesion: 0.20 Cohesion: 0.20
@@ -1108,8 +1106,8 @@ Cohesion: 0.25
Nodes (7): Fleet backup architecture & freshness runbook, Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage), Known gaps / TODO, Layer 1 — PBS (whole-VM/CT images), Layer 2 — restic (granular file + DB), The 2-minute freshness check, TL;DR — coverage matrix Nodes (7): Fleet backup architecture & freshness runbook, Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage), Known gaps / TODO, Layer 1 — PBS (whole-VM/CT images), Layer 2 — restic (granular file + DB), The 2-minute freshness check, TL;DR — coverage matrix
### Community 179 - "convert_hf_to_native.py" ### Community 179 - "convert_hf_to_native.py"
Cohesion: 0.07 Cohesion: 0.10
Nodes (42): ArgumentParser, _arrow_rows(), _datasets_root(), load_calibration(), load_evaluation(), Held-out evaluation prompts. Returns (harmless, harmful, provenance). This, Read the `text` column out of one cached HF arrow split., Return (harmful, harmless, provenance). `builtin` is the legacy 8/8 set. `m (+34 more) Nodes (29): _get_model(), OpenAI-ish /v1/audio/speech adapter in front of the Zonos Python SDK. Why this, _speaker_embedding(), speech(), SpeechRequest, ArgumentParser, _is_mtp(), main() (+21 more)
### Community 181 - "Mistral Small 4 → NVFP4 (vision-intact) build tooling" ### Community 181 - "Mistral Small 4 → NVFP4 (vision-intact) build tooling"
Cohesion: 0.29 Cohesion: 0.29
@@ -1164,8 +1162,8 @@ Cohesion: 0.33
Nodes (5): Containerization plan (pending build), Live invocation (source of truth), Status (2026-07-16): NATIVE, config was live-only — now captured here, The engine, zonos-engine — ZONOS2 native TTS engine (`:1920`, irv-ml1 3090) Nodes (5): Containerization plan (pending build), Live invocation (source of truth), Status (2026-07-16): NATIVE, config was live-only — now captured here, The engine, zonos-engine — ZONOS2 native TTS engine (`:1920`, irv-ml1 3090)
### Community 200 - "omnivoice/app.py" ### Community 200 - "omnivoice/app.py"
Cohesion: 0.14 Cohesion: 0.06
Nodes (12): Any, _base_gen_kwargs(), _pcm16(), Thin FastAPI wrapper exposing OmniVoice (k2-fsa/OmniVoice) for the fleet. Upstr, Validate the voice source and build the MODEL.generate kwargs minus `text`., Synthesize one text span -> (float32 audio [-1,1], audio_seconds)., float32 [-1,1] -> little-endian s16 PCM bytes (24 kHz mono on the wire)., WAV header. data_len=None -> streaming (0xFFFFFFFF sizes, read to EOF); an i (+4 more) Nodes (46): Any, ChunkConfig, ChunkResult, ClockFn, GenerateFn, _base_gen_kwargs(), _chunk_config(), GenParams (+38 more)
### Community 202 - "phasefinal-web" ### Community 202 - "phasefinal-web"
Cohesion: 0.29 Cohesion: 0.29
@@ -1175,9 +1173,9 @@ Nodes (6): Content constraints, Deploy, DNS, phasefinal-web, Routing — and the
Cohesion: 0.29 Cohesion: 0.29
Nodes (6): 2026-07-14 status — gibberish FIXED, format PIVOTED to modelopt for MTP, Artifacts (on ana-ml2), Fire sequence, Gates, heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP), Serve target Nodes (6): 2026-07-14 status — gibberish FIXED, format PIVOTED to modelopt for MTP, Artifacts (on ana-ml2), Fire sequence, Gates, heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP), Serve target
### Community 205 - "booth/app.py" ### Community 205 - "FastAPI"
Cohesion: 0.11 Cohesion: 0.18
Nodes (18): fixture, create_app(), _from_env(), generate_pickup_id(), human_dur(), The Booth — a standing web server that renders drop-folders as ephemeral media…, A Content-Disposition-safe `<booth>.zip` (strip quotes/control chars)., Reduce a client-supplied filename to a safe basename (no path, no hidden). (+10 more) Nodes (12): FastAPI, fixture, create_app(), _from_env(), human_dur(), client(), test_human_dur(), test_upload_rejects_too_large() (+4 more)
### Community 206 - "soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated)" ### Community 206 - "soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated)"
Cohesion: 0.33 Cohesion: 0.33
@@ -1456,8 +1454,8 @@ Cohesion: 0.17
Nodes (11): ANSWERED — Lobe TTS is a SPLIT: endpoint env-driven, voice/model/format UI-only, ⚠️ Capacity — tell tts-dev if this ramps, ⚠️ CORRECTION — my earlier voice foot-gun warning was wrong, Credential posture, Deploy, lobe-chat — chat frontend over the LiteLLM gateway (esh-docker-vm), System Agent — why `gpt-5-mini` was being called, The real constraint is SIZE, not length (+3 more) Nodes (11): ANSWERED — Lobe TTS is a SPLIT: endpoint env-driven, voice/model/format UI-only, ⚠️ Capacity — tell tts-dev if this ramps, ⚠️ CORRECTION — my earlier voice foot-gun warning was wrong, Credential posture, Deploy, lobe-chat — chat frontend over the LiteLLM gateway (esh-docker-vm), System Agent — why `gpt-5-mini` was being called, The real constraint is SIZE, not length (+3 more)
### Community 296 - "sweep_once" ### Community 296 - "sweep_once"
Cohesion: 0.16 Cohesion: 0.19
Nodes (16): is_expired(), Pure age question. Deliberately does NOT consider the keep sentinel. Expiry…, Wipe every direct-child booth older than the TTL. Returns names wiped. Only…, sweep_once(), Age a booth and everything in it well past any test TTL., The point of the whole feature: expiry does not apply to a kept booth., is_expired stays a pure age question; only the sweeper honours the pin. Keeping…, Counter-intuitive, and the reason release-then-sweep is not a delete path.… (+8 more) Nodes (13): Wipe every direct-child booth older than the TTL. Returns names wiped. Only…, sweep_once(), Age a booth and everything in it well past any test TTL., The point of the whole feature: expiry does not apply to a kept booth., is_expired stays a pure age question; only the sweeper honours the pin. Keeping…, Counter-intuitive, and the reason release-then-sweep is not a delete path.…, It does rejoin the sweep — just on a fresh clock, not the old one., _stale() (+5 more)
### Community 297 - "heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (ana-ml2)" ### Community 297 - "heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (ana-ml2)"
Cohesion: 0.40 Cohesion: 0.40
@@ -1556,8 +1554,8 @@ Cohesion: 0.70
Nodes (4): ask(), classify(), load_prompts(), main() Nodes (4): ask(), classify(), load_prompts(), main()
### Community 327 - "worker/app.py" ### Community 327 - "worker/app.py"
Cohesion: 0.14 Cohesion: 0.12
Nodes (10): RuntimeError, get_gpu_status(), post_train(), FastAPI surface for the LoRA training worker (§4.1 API — arbo is the client). E, TrainRequest, Static configuration for the LoRA training worker. Everything load-bearing is a, LoRA training worker — a host service (runs as llmuser on irv-ml1) that runs sd-, Busy (+2 more) Nodes (18): RuntimeError, get_gpu_status(), post_train(), FastAPI surface for the LoRA training worker (§4.1 API — arbo is the client). E, TrainRequest, Static configuration for the LoRA training worker. Everything load-bearing is a, _bus_id_by_index(), _cmdline() (+10 more)
### Community 329 - "`[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight" ### Community 329 - "`[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight"
Cohesion: 0.29 Cohesion: 0.29
@@ -1607,9 +1605,9 @@ Nodes (4): default, title, type, optional
Cohesion: 0.11 Cohesion: 0.11
Nodes (19): additionalProperties, properties, required, title, type, CatalogAuditEntry, title, type (+11 more) Nodes (19): additionalProperties, properties, required, title, type, CatalogAuditEntry, title, type (+11 more)
### Community 343 - "gpu.py" ### Community 343 - "kl_divergence.py"
Cohesion: 0.42 Cohesion: 0.13
Nodes (8): _bus_id_by_index(), _cmdline(), gpu_status(), _query_compute_apps(), _query_devices(), GPU status for arbo's device-aware scheduler (GET /gpu-status). arbo steers a l, Return `{devices: [...], tts_on_3090: bool}`. Degrades to an error field on nvid, _run() Nodes (20): _arrow_rows(), _datasets_root(), load_calibration(), load_evaluation(), Held-out evaluation prompts. Returns (harmless, harmful, provenance). This, Read the `text` column out of one cached HF arrow split., Return (harmful, harmless, provenance). `builtin` is the legacy 8/8 set. `m, collect() (+12 more)
### Community 344 - "step" ### Community 344 - "step"
Cohesion: 0.50 Cohesion: 0.50
@@ -1767,17 +1765,17 @@ Nodes (4): ACME disabled — the WAN IP now exposes nothing, CLOSED OUT (2026-08
Cohesion: 0.17 Cohesion: 0.17
Nodes (10): ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for, lora_B gate — PASSED, twice, The confound I built and he caught, The gate — brokkr-smithy-dev, The noise-floor near-miss — the methodology lesson, The run, ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one, Refusal retention — the axis the gate did not have, and the axis I measured wrong (+2 more) Nodes (10): ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for, lora_B gate — PASSED, twice, The confound I built and he caught, The gate — brokkr-smithy-dev, The noise-floor near-miss — the methodology lesson, The run, ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one, Refusal retention — the axis the gate did not have, and the axis I measured wrong (+2 more)
### Community 388 - "stream_chunks" ### Community 388 - "`[2026-09-01]` Ada migration settled on `zfs send` — and branch (b) was never available"
Cohesion: 0.19 Cohesion: 0.11
Nodes (17): ClockFn, GenerateFn, ChunkConfig, _ema(), _est_gen_time(), plan_chunk(), protect_first_audio(), Split into sentence units, preserving punctuation. Whitespace-collapsed. (+9 more) Nodes (16): `[2026-09-01]` Ada migration settled on `zfs send` — and branch (b) was never available, (c) rebuild-from-source: rejected on reproducibility, not time, Open, The answer: (a) `zfs send`. Measured, not derived., ⚠ The two-boxes confusion — do not repeat it, Their distinction, worth keeping, Why (b) — physically moving the disks — was never on the table, `[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c (+8 more)
### Community 389 - "CatalogField" ### Community 389 - "CatalogField"
Cohesion: 0.33 Cohesion: 0.33
Nodes (6): additionalProperties, description, required, title, type, CatalogField Nodes (6): additionalProperties, description, required, title, type, CatalogField
### Community 390 - "ChunkResult" ### Community 390 - "`[2026-09-01]` Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed"
Cohesion: 0.15 Cohesion: 0.22
Nodes (13): ChunkConfig, ChunkResult, _chunk_config(), _log_chunk(), Streaming /tts request — chatterbox-fast-compatible wire protocol., Streaming: chunked 24 kHz mono PCM (or open-ended WAV) for live consumers., tts(), TTSStreamRequest (+5 more) Nodes (8): `[2026-09-01]` Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed, Conventions ratified (operator, 2026-09-01), Push reality — measured, and it inverts the obvious reading, QR sign-in — requires MAS, deferred, Shared-secret registration gotcha, `/_synapse/admin` closed to the internet, The appservice namespace — `exclusive: true` → `false`, The upgrade
### Community 391 - "type" ### Community 391 - "type"
Cohesion: 0.50 Cohesion: 0.50
@@ -1800,8 +1798,8 @@ Cohesion: 0.19
Nodes (14): Path, test_published_relative_path(), test_published_relative_path_explicit_train_id_wins(), ValueError, InvalidTrainRequest, published_relative_path(), Fixed-invocation command builder — the enforcement point for INV-T7. arbo hands, The ComfyUI-relative loras path for a succeeded LoRA (Phase 2 publish step). (+6 more) Nodes (14): Path, test_published_relative_path(), test_published_relative_path_explicit_train_id_wins(), ValueError, InvalidTrainRequest, published_relative_path(), Fixed-invocation command builder — the enforcement point for INV-T7. arbo hands, The ComfyUI-relative loras path for a succeeded LoRA (Phase 2 publish step). (+6 more)
### Community 396 - "BaseModel" ### Community 396 - "BaseModel"
Cohesion: 0.19 Cohesion: 0.31
Nodes (12): ACEStepInput, ACEStepOutput, generate_audio(), initialize_pipeline(), Generate music; respond with the audio bytes inline. Pre-2026-05-11 this re, ACEStepPipeline, SpeechRequest, BaseModel (+4 more) Nodes (7): ACEStepInput, ACEStepOutput, generate_audio(), initialize_pipeline(), Generate music; respond with the audio bytes inline. Pre-2026-05-11 this re, ACEStepPipeline, BaseModel
### Community 398 - "_cli" ### Community 398 - "_cli"
Cohesion: 0.25 Cohesion: 0.25
@@ -1871,13 +1869,9 @@ Nodes (8): OfflineRecognizer, _decode(), _ensure_model_present(), _load_recogniz
Cohesion: 0.30 Cohesion: 0.30
Nodes (7): _build_prompt(), _cap(), _decode(), _encode_ref(), tts(), tts_stream(), TTSReq Nodes (7): _build_prompt(), _cap(), _decode(), _encode_ref(), tts(), tts_stream(), TTSReq
### Community 420 - "adapter/server.py" ### Community 420 - "generate_pickup_id"
Cohesion: 0.31 Cohesion: 0.24
Nodes (6): _get_model(), OpenAI-ish /v1/audio/speech adapter in front of the Zonos Python SDK. Why this, _speaker_embedding(), speech(), JSONResponse, Zonos Nodes (7): generate_pickup_id(), Reduce a client-supplied filename to a safe basename (no path, no hidden)., A human-readable id like '4-wombat' or 'star-84'. `exists(name)->bool` gates…, safe_upload_name(), test_generate_pickup_id_avoids_collision(), test_generate_pickup_id_format(), test_safe_upload_name()
### Community 422 - "FastAPI"
Cohesion: 0.47
Nodes (4): FastAPI, lifespan(), sfx(), SfxRequest
### Community 423 - "license" ### Community 423 - "license"
Cohesion: 0.67 Cohesion: 0.67
@@ -1892,19 +1886,19 @@ Cohesion: 0.67
Nodes (3): Design consequences of terminating downstream — the parts that need decisions, FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through, Standing recommendation Nodes (3): Design consequences of terminating downstream — the parts that need decisions, FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through, Standing recommendation
## Knowledge Gaps ## Knowledge Gaps
- **2337 isolated node(s):** `⚠ The password is NOT in this copy`, `/_synapse/admin is LAN-only`, `Upgrades`, `Active migration — docker.io 20.10 → docker-ce 29.x`, `Architecture decisions (durable)` (+2332 more) - **2357 isolated node(s):** `The answer: (a) `zfs send`. Measured, not derived.`, `Why (b) — physically moving the disks — was never on the table`, `(c) rebuild-from-source: rejected on reproducibility, not time`, `⚠ The two-boxes confusion — do not repeat it`, `Their distinction, worth keeping` (+2352 more)
These have ≤1 connection - possible missing edges or undocumented components. These have ≤1 connection - possible missing edges or undocumented components.
- **41 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes. - **40 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
## Suggested Questions ## Suggested Questions
_Questions this graph is uniquely positioned to answer:_ _Questions this graph is uniquely positioned to answer:_
- **Why does `Tensor` connect `convert_hf_to_native.py` to `adapter/server.py`?** - **Why does `Tensor` connect `convert_hf_to_native.py` to `kl_divergence.py`?**
_High betweenness centrality (0.005) - this node is a cross-community bridge._
- **Why does `divergences()` connect `kl_divergence.py` to `convert_hf_to_native.py`?**
_High betweenness centrality (0.004) - this node is a cross-community bridge._ _High betweenness centrality (0.004) - this node is a cross-community bridge._
- **Why does `_speaker_embedding()` connect `adapter/server.py` to `convert_hf_to_native.py`?** - **What connects `The answer: (a) `zfs send`. Measured, not derived.`, `Why (b) — physically moving the disks — was never on the table`, `(c) rebuild-from-source: rejected on reproducibility, not time` to the rest of the system?**
_High betweenness centrality (0.004) - this node is a cross-community bridge._ _2357 weakly-connected nodes found - possible documentation gaps or missing edges._
- **What connects `⚠ The password is NOT in this copy`, `/_synapse/admin is LAN-only`, `Upgrades` to the rest of the system?**
_2337 weakly-connected nodes found - possible documentation gaps or missing edges._
- **Should `Status + Open Issues` be split into smaller, more focused modules?** - **Should `Status + Open Issues` be split into smaller, more focused modules?**
_Cohesion score 0.05555555555555555 - nodes in this community are weakly interconnected._ _Cohesion score 0.05555555555555555 - nodes in this community are weakly interconnected._
- **Should `CatalogReproducibility` be split into smaller, more focused modules?** - **Should `CatalogReproducibility` be split into smaller, more focused modules?**
@@ -0,0 +1,111 @@
# `[2026-09-01]` The Ada inference server is a Dell R750xa bought used — and the reseller stripped four things Dell shipped
Dell PowerEdge R750xa, service tag **`JPJ1ZP3`**, 8-bay 2.5" 2U. Bought from Entera Source
(invoice INV-74422, 2026-08-25). **2× NVIDIA RTX 6000 Ada 48 GB** to be fitted — confirmed by
the operator, NOT L40/L40S. Destination NH3. This is ComfyUI's new home; the GX10 is a
separate machine → [[2026-09-01-pfi-gx10-onboarding]].
## As bought
CPU 2x Xeon Silver 4314 16C/32T each = 32C/64T, 135 W each
RAM 8x Micron 16 GB PC4-3200 = 128 GB
NIC dual 25 GbE SFP28 (Broadcom 57414, OCP 3.0)
boot BOSS-S2 + 2x 240 GB M.2 SATA, mirrored
storage 2x 1.92 TB SATA SSD (Hynix), 6 of 8 bays free
HBA PERC HBA355i (LSI SAS3816)
PSU 2x 1400 W Platinum
## THE DIFF — factory CSV (`JPJ1ZP3.csv`, booth `54-puffin`) vs the invoice
| | Dell factory | reseller invoice |
|---|---|---|
| **RAM** | **16× 16 GB `M04W6`**, "Performance Optimized" = **256 GB** | 8× = 128 GB |
| **PSU** | **2400 W** redundant (`450-AJEV`) | 2× 1400 W (`7DWXY`) |
| **GPUs** | 4× Tesla V100-32GB passive (`490-BHUC`) | none |
| **GPU risers** | `12XPY` (right DW), `9TR6X` (left DW) | **not listed** |
| **GPU cables** | `470-AEVS` 4-GPU kit: `4RW1P` `W4K7M` `XC48N` `6C77X` | **not listed** |
| **Riser cables** | signal `CP67W` `CXYF8` `H4D7D` `N61TK`; power `HXJDR` `W1P56`; pad `C2JNP` | **not listed** |
| **Fans** | 6× `FD00R` **Very High Performance** | **not listed** |
| **Controller** | PERC H345 | **HBA355i — an UPGRADE for us** |
| **Drives** | 2× 480 GB | 2× 1.92 TB |
⚠ **"Not listed on the invoice" is not proof of removal.** Operator will pull an iDRAC
inventory to settle it. Diff that against the CSV rather than the invoice.
## The GPU power chain — resolved, and my first answer was wrong
**RTX 6000 Ada: 300 W, ACTIVE cooling, one 16-pin 12VHPWR (CEM5) connector** (nvidia.com).
The R750xa kit `470-AEVS` is **"W/CPU CONN"** — CPU/EPS 8-pin, because V100 takes that. I
initially flagged this as a mismatch requiring the PCIe-type cable `RCCWC` / Dell
`470-BBXH`. **That was wrong and is withdrawn** — `RCCWC` is for H100/L40/L40S.
NVIDIA sells the bridge for exactly this case:
> **`930-00030-1546-000`** — NVIDIA Power Cable, RTX 6000 Ada Generation,
> **1× CPU 8-Pin → 1× CEM5 16-Pin**
So the chain is `470-AEVS` CPU 8-pin → `930-00030-1546-000` → card. **The chassis' existing
cabling is the correct source type.** Two adapters needed, one per card.
⚠ Factory brackets are `VFYGM` — the **V100S** extended bracket. RTX 6000 Ada is a different
physical card; bracket fit is UNKNOWN, check at fitting.
⚠ **Active cooling in a passive-card chassis.** Six high-performance fans expect to push air
through passive cards; the RTX 6000 Ada has its own blower. It works in practice, but Dell
never validated this combination for this box.
## Storage — NVMe in the drive bays is CLOSED
Factory backplane is `321-BGKG : 2.5" Chassis with up to 8 **SAS/SATA** Drives`. Not
NVMe-capable; bays would need a backplane swap.
Arithmetic if filling bays with SATA:
2x SATA (today) ~1.1 GB/s
6x striped ~3.3 GB/s <- roughly ONE NVMe drive
8x striped ~4.4 GB/s
backplane ceiling ~6 GB/s (8 lanes x SATA III)
The HBA355i (PCIe 4.0 x8) is nowhere near limiting. **Drives are the constraint.**
**ARC probably beats spindles here.** ComfyUI reloads the same checkpoints; with the factory
256 GB restored, ZFS ARC caches most of the working set at RAM speed. The RAM upgrade is
both cheaper and likely faster than four more SATA drives.
**UNCHECKED, and it may moot the whole bay question:** free PCIe slots. Riser Config 0 is
`330-BBUM : Riser Config 0, 6x16 + 2x8 slots`; two double-wide GPUs will not consume all of
it. An NVMe add-in card (U.2 HBA or bifurcating M.2 carrier) bypasses the SAS/SATA backplane
entirely. **Add slot inventory to the iDRAC pull.**
## Power — the ceiling is deliberate now, not discovered later
2x RTX 6000 Ada 600 W
2x Xeon 4314 270 W
rest ~150 W
~1,020 W sustained
One 1400 W PSU carries that, so redundancy holds. But factory specced **2400 W** for four
V100s: **there is no headroom for a third or fourth GPU.** And ~1 kW is the same class as
ana-ml2, the box whose training run tripped a breaker on 2026-08-26 — **decide which NH3
circuit this lands on before racking, not after.**
## Buy list
**Certain:**
| item | part | qty |
|---|---|---|
| RDIMM 16 GB 3200 2Rx8 | **`M04W6`** | **8** → restores 256 GB, all 16 slots, all 8 channels/socket |
| NVIDIA 12VHPWR adapter | **`930-00030-1546-000`** | **2** |
**Only if the iDRAC inventory shows them absent:** `12XPY`, `9TR6X`, `4RW1P`, `W4K7M`,
`XC48N`, `6C77X`, `CP67W`, `CXYF8`, `H4D7D`, `N61TK`, `HXJDR`, `W1P56`, `C2JNP`,
`FD00R` ×6.
**Spend order:** RAM → check PCIe slots → only then consider more SATA drives.
⚠ Dell's Installation & Service Manual publishes **no part numbers at all** for GPU kit
hardware — names and quantities only. The factory CSV keyed to the service tag is the only
authoritative source of DPNs for this machine. Do not accept a five-character DPN from
anywhere else without corroboration.
+11 -7
View File
@@ -119,6 +119,14 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
one command afterwards (`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`). one command afterwards (`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`).
⚠ Triton has no sm_121 support; compiled deps are per-arch unknowns. ⚠ Triton has no sm_121 support; compiled deps are per-arch unknowns.
→ `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000
Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores
factory 256 GB) and 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters.** Everything else waits
on the inventory — the invoice omits risers, GPU cables and the 6 high-performance fans, but
omission is not proof of removal. **Add free-PCIe-slot inventory to the pull** — an NVMe
add-in card would bypass the SAS/SATA backplane and moot the drive-bay question.
⚠ ~1 kW loaded — same class as the box that tripped the breaker; pick the NH3 circuit before
racking. → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
- **⏸ RUN 3c STILL HELD — but the plan has changed.** Config `/tank/erp-tune/run-03c.json` - **⏸ RUN 3c STILL HELD — but the plan has changed.** Config `/tank/erp-tune/run-03c.json`
validated, relaunch is one command on ana-ml2. **It is now intended to move to the GX10 validated, relaunch is one command on ana-ml2. **It is now intended to move to the GX10
instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2 instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2
@@ -147,6 +155,7 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
## Recent decisions ## Recent decisions
- `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
- `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` - `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — branch (b) was never available because irv-ml1 keeps its eight services.** 99 MB/s measured; ~3.9 h. Also records the two-boxes confusion: the Ada box and the GX10 are DIFFERENT machines. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` - `[2026-09-01]` **Ada migration is `zfs send` (branch a) — branch (b) was never available because irv-ml1 keeps its eight services.** 99 MB/s measured; ~3.9 h. Also records the two-boxes confusion: the Ada box and the GX10 are DIFFERENT machines. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
- `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md` - `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md`
@@ -251,7 +260,6 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
- `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase. - `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase.
- `[2026-08-16]` **Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated.** brokkr-smithy-dev supplied the framing that reproduces (`01M05M48R4RSZF9D8KT7RR55EJ`): a **bare assistant-mode instruction** — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing **DS 0.0% / FF 0.0%** (n=75); plain character cards **DS 1.4% / FF 0.0%** (n=74); bare instruction **DS 92.5% (37/40) / FF 15.8% (6/38)**. Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→**0**, consensual 80→20, dubcon 80→**0**, self-harm 80→**0**. DS refused **25/25** on the five axes brokkr flagged. Root cause: `ReadyArt/Dark-Scarlett-v1.0-27B` is a plain finetune of stock `Qwen/Qwen3.6-27B` carrying **NO abliteration** — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-**ablated** (structural), which is why it holds. ⚠ **RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID.** His diff against his own artifact showed my `battery-instruct.yaml` reproduces only his **`creative` class — 8 of 16 axes**; it dropped all 5 `operational` (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 `meta` (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. **His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta.** Cause: I rebuilt his shape from his *message*, and the `class` field lives in the artifact, not the prose. **Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability.** ⚠ **Known battery bug left unfixed for comparability:** DS's arm-3 control gate failed at 11% because `ictrl-reunion` pairs "explicit / do not fade to black" with *brothers*, which DS reasonably read as an incest request; FF did not. `ictrl-storm` is the clean control. Commit `b9e68c3`.
- `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right. - `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right.
@@ -273,23 +281,19 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
- `[2026-08-15]` **RP-seat direction: KEEP MeroMero on `char-rp`; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator).** Evaluated `TheDrummer/Artemis-31B-v1.1` — mechanically a drop-in (same `google/gemma-4-31B-it` base, identical 1188-tensor/356-vision census, same missing-`preprocessor_config.json` trick), so it's purely a quality call, and our own survey already ranked MeroMero **#1** vs Artemis **#6**; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. **MTP verified impossible on both** (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). **But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter** — vLLM 0.24 supports `eagle3` + `gemma4_mtp`, and real drafters exist: `google/gemma-4-31B-it-assistant` (0.94 GB, 4-layer, 761K dl), `RedHatAI/gemma-4-31B-it-speculator.eagle3` (4.47 GB), `AEON-7/…eagle3-NVFP4` (3.53 GB). ⚠ all list their verifier as **stock** gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). **Why the Dark-Scarlett 3.8 plan is the strong one:** DS is Qwen3.6-based today, so a 3.8 respin lands on the *gen seat's* architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: `from_pretrained` **silently drops MTP heads during finetuning** (verify 15 `mtp.*` tensors in the index; graft from stock if absent), and DS v1.0 required the `Qwen3_5ForConditionalGeneration` **wrapper class** to save a config vLLM/SGLang accept. Both in `docs/pfi/model-quantization-playbook.md`. - `[2026-08-15]` **RP-seat direction: KEEP MeroMero on `char-rp`; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator).** Evaluated `TheDrummer/Artemis-31B-v1.1` — mechanically a drop-in (same `google/gemma-4-31B-it` base, identical 1188-tensor/356-vision census, same missing-`preprocessor_config.json` trick), so it's purely a quality call, and our own survey already ranked MeroMero **#1** vs Artemis **#6**; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. **MTP verified impossible on both** (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). **But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter** — vLLM 0.24 supports `eagle3` + `gemma4_mtp`, and real drafters exist: `google/gemma-4-31B-it-assistant` (0.94 GB, 4-layer, 761K dl), `RedHatAI/gemma-4-31B-it-speculator.eagle3` (4.47 GB), `AEON-7/…eagle3-NVFP4` (3.53 GB). ⚠ all list their verifier as **stock** gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). **Why the Dark-Scarlett 3.8 plan is the strong one:** DS is Qwen3.6-based today, so a 3.8 respin lands on the *gen seat's* architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: `from_pretrained` **silently drops MTP heads during finetuning** (verify 15 `mtp.*` tensors in the index; graft from stock if absent), and DS v1.0 required the `Qwen3_5ForConditionalGeneration` **wrapper class** to save a config vLLM/SGLang accept. Both in `docs/pfi/model-quantization-playbook.md`.
- `[2026-08-15]` **Quant lessons consolidated into `docs/pfi/model-quantization-playbook.md` — the durable home; read it BEFORE any requant.** Survey found quant knowledge scattered across 18 files in 4 trees, with **three** documents having independently written overlapping "landmines" sections (the loader-class trap alone was rediscovered 3×). Playbook owns the **transferable** lessons (scheme choice, landmines, acceptance gate + its 3 measurement traps, hardware/co-residency); per-model artifacts are demoted to worked examples that link up. Carries a **superseded-claims table** — which immediately earned itself: the heretic2 runbook's "use modelopt, compressed-tensors can't load the BF16 MTP" is **false** (the cause was the missing `re:^mtp.*` ignore, not the format) and would have sent the next session down the modelopt dependency-hell path; that runbook now carries a stale-warning header. Maintenance rule in `CLAUDE.md`: model-agnostic → playbook, model-specific → stays put, wrong claim → dated superseded row, never a silent edit. Motivated by Qwen3.8 having just released — the next model swap needs a requant. Commit `a91cc3f`.
- `[2026-08-15]` **Operator ruling: the gen seat's +1.7% perplexity is an acceptable price for the speed — SETTLED, don't re-litigate.** Precise attribution for future reasoning: it is the **activation-quantization** cost (W4A4 MLPs + FP8 attention vs BF16 activations), not an MTP cost — PPL was measured with speculative decoding **off** on both builds, so MTP was not in the loop. Turning MTP off would not recover it; only reverting the quant would (rollback = one `.env` line, old build intact at `…/qwen38-27b-uncensored-nvfp4`).
- `[2026-08-15]` **gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed.** The queued "W4A8" (NVFP4 weights + FP8 activations) is **not servable** — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and `CompressedTensorsW4A8Fp8` is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter **per-layer-group**. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did **80.12** (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: `unsloth/Qwen3.8-27B-NVFP4` was already on-box → served as a probe, measured **+19.1% at identical acceptance**, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → **80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%**; surface 6/6 live, 7 aliases routing. char-rp had **no** tool parser at all (every tools request 400'd) → `gemma4` tool + reasoning parser + a **mandatory** `enable_thinking:false` (the parser defaults it True → null `content` for all RP prose; proven byte-identical prompt before deploying). Commits `b8f0f4c`, `74f596b`. Foot-guns banked (llm-compressor prunes unmatched `ignore` entries → the 0%-MTP bug, **fired on this run**; prompt_logprobs uniform under spec-decode; 0600 `.env` silently no-ops compose; GPU0 is zero-sum). → `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`
- `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
- `[2026-08-10→12]` **secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed.** `secret` CLI (`put/get/list/rm/backfill`, bw-backed) on `~/.local/bin`; 25 nh3-dev secrets backfilled + round-trip-verified; `rm` + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → `persistent-memory.d/2026-08-12-secrets-broker.md` - `[2026-08-10→12]` **secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed.** `secret` CLI (`put/get/list/rm/backfill`, bw-backed) on `~/.local/bin`; 25 nh3-dev secrets backfilled + round-trip-verified; `rm` + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → `persistent-memory.d/2026-08-12-secrets-broker.md`
- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]] - `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md`
- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md`
_223 older entries archived to archival-memory.md._ _223 older entries archived to archival-memory.md._
_Older entries archived to archival-memory.md._
## Tried and abandoned ## Tried and abandoned
- `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not. - `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.