diff --git a/archival-memory.md b/archival-memory.md index f2ce408..4bdb140 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -4,6 +4,22 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re ## Recent decisions (archived) +- `[2026-08-16]` **Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated.** brokkr-smithy-dev supplied the framing that reproduces (`01M05M48R4RSZF9D8KT7RR55EJ`): a **bare assistant-mode instruction** — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing **DS 0.0% / FF 0.0%** (n=75); plain character cards **DS 1.4% / FF 0.0%** (n=74); bare instruction **DS 92.5% (37/40) / FF 15.8% (6/38)**. Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→**0**, consensual 80→20, dubcon 80→**0**, self-harm 80→**0**. DS refused **25/25** on the five axes brokkr flagged. Root cause: `ReadyArt/Dark-Scarlett-v1.0-27B` is a plain finetune of stock `Qwen/Qwen3.6-27B` carrying **NO abliteration** — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-**ablated** (structural), which is why it holds. ⚠ **RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID.** His diff against his own artifact showed my `battery-instruct.yaml` reproduces only his **`creative` class — 8 of 16 axes**; it dropped all 5 `operational` (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 `meta` (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. **His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta.** Cause: I rebuilt his shape from his *message*, and the `class` field lives in the artifact, not the prose. **Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability.** ⚠ **Known battery bug left unfixed for comparability:** DS's arm-3 control gate failed at 11% because `ictrl-reunion` pairs "explicit / do not fade to black" with *brothers*, which DS reasonably read as an incest request; FF did not. `ictrl-storm` is the clean control. Commit `b9e68c3`. + _Archived 2026-09-01._ +- `[2026-08-15]` **Quant lessons consolidated into `docs/pfi/model-quantization-playbook.md` — the durable home; read it BEFORE any requant.** Survey found quant knowledge scattered across 18 files in 4 trees, with **three** documents having independently written overlapping "landmines" sections (the loader-class trap alone was rediscovered 3×). Playbook owns the **transferable** lessons (scheme choice, landmines, acceptance gate + its 3 measurement traps, hardware/co-residency); per-model artifacts are demoted to worked examples that link up. Carries a **superseded-claims table** — which immediately earned itself: the heretic2 runbook's "use modelopt, compressed-tensors can't load the BF16 MTP" is **false** (the cause was the missing `re:^mtp.*` ignore, not the format) and would have sent the next session down the modelopt dependency-hell path; that runbook now carries a stale-warning header. Maintenance rule in `CLAUDE.md`: model-agnostic → playbook, model-specific → stays put, wrong claim → dated superseded row, never a silent edit. Motivated by Qwen3.8 having just released — the next model swap needs a requant. Commit `a91cc3f`. + _Archived 2026-09-01._ +- `[2026-08-15]` **Operator ruling: the gen seat's +1.7% perplexity is an acceptable price for the speed — SETTLED, don't re-litigate.** Precise attribution for future reasoning: it is the **activation-quantization** cost (W4A4 MLPs + FP8 attention vs BF16 activations), not an MTP cost — PPL was measured with speculative decoding **off** on both builds, so MTP was not in the loop. Turning MTP off would not recover it; only reverting the quant would (rollback = one `.env` line, old build intact at `…/qwen38-27b-uncensored-nvfp4`). + _Archived 2026-09-01._ +- `[2026-08-15]` **gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed.** The queued "W4A8" (NVFP4 weights + FP8 activations) is **not servable** — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and `CompressedTensorsW4A8Fp8` is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter **per-layer-group**. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did **80.12** (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: `unsloth/Qwen3.8-27B-NVFP4` was already on-box → served as a probe, measured **+19.1% at identical acceptance**, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → **80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%**; surface 6/6 live, 7 aliases routing. char-rp had **no** tool parser at all (every tools request 400'd) → `gemma4` tool + reasoning parser + a **mandatory** `enable_thinking:false` (the parser defaults it True → null `content` for all RP prose; proven byte-identical prompt before deploying). Commits `b8f0f4c`, `74f596b`. Foot-guns banked (llm-compressor prunes unmatched `ignore` entries → the 0%-MTP bug, **fired on this run**; prompt_logprobs uniform under spec-decode; 0600 `.env` silently no-ops compose; GPU0 is zero-sum). → `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md` + _Archived 2026-09-01._ +- `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md` + _Archived 2026-09-01._ + +- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md` + _Archived 2026-09-01._ +- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md` + _Archived 2026-09-01._ + - `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`. _Archived 2026-08-28. Its deferred item — the nh3-extdev herald 2.1.2 upgrade — is closed: extdev went 2.1.0 -> 3.0.0 -> 3.1.1 at the v3 cutover, so the staged v2.1.2 wheel is moot._ diff --git a/graphify-out/GRAPH_REPORT.md b/graphify-out/GRAPH_REPORT.md index 61d0343..dbbabbf 100644 --- a/graphify-out/GRAPH_REPORT.md +++ b/graphify-out/GRAPH_REPORT.md @@ -1,16 +1,16 @@ # Graph Report - eshpfi-management (2026-09-01) ## Corpus Check -- 378 files · ~573,530 words +- 382 files · ~576,491 words - Verdict: corpus is large enough that graph structure adds value. ## Summary -- 3878 nodes · 4117 edges · 426 communities (385 shown, 41 thin omitted) +- 3906 nodes · 4144 edges · 424 communities (384 shown, 40 thin omitted) - Extraction: 99% EXTRACTED · 1% INFERRED · 0% AMBIGUOUS · INFERRED: 38 edges (avg confidence: 0.71) - Token cost: 0 input · 0 output ## Graph Freshness -- Built from commit: `73866f6a` +- Built from commit: `71426577` - Run `git rev-parse HEAD` and compare to check if the graph is stale. - Run `graphify update .` after code changes (no API cost). @@ -175,7 +175,7 @@ - blend_kokoro_voice.py - Handler - estimated_latency -- Path +- booth/app.py - bench.py - serve_nvfp4.sh - bench_w4a16_recheck.json @@ -217,7 +217,7 @@ - phasefinal-web - build.sh script - heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP) -- booth/app.py +- FastAPI - soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated) - nh3-dev `~/development` — hourly off-box backup - nh3-dev-development-backup.sh script @@ -339,7 +339,7 @@ - [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing - optional - properties -- gpu.py +- kl_divergence.py - step - `[2026-08-27]` A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find - host @@ -381,9 +381,9 @@ - `[2026-08-27]` Run 3 gated: the rule PASSED and a k=25 follow-up found a self-harm guardrail collapse - CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE - ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for -- stream_chunks +- `[2026-09-01]` Ada migration settled on `zfs send` — and branch (b) was never available - CatalogField -- ChunkResult +- `[2026-09-01]` Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed - type - `[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way - counted_classifier.py @@ -411,9 +411,7 @@ - index-tts/app.py - parakeet/app.py - tts/app.py -- adapter/server.py -- test_keep_and_unkeep_go_through_the_same_name_guard -- FastAPI +- generate_pickup_id - license - version - FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through @@ -433,19 +431,19 @@ ## Surprising Connections (you probably didn't know these) - `convert()` --references--> `Any` [EXTRACTED] tools/mistral-small4-nvfp4/convert_hf_to_native.py → stacks/omnivoice/app.py -- `_speaker_embedding()` --references--> `Tensor` [EXTRACTED] - stacks/zonos/adapter/server.py → services/coldfusion-abliteration/kl_divergence.py - `test_cli_unlink_by_id_even_when_the_id_is_all_digits()` --calls--> `parse_link_entries()` [INFERRED] services/booth/tests/test_booth.py → services/booth/booth/links.py - `test_cli_unlink_of_a_stale_id_leaves_the_board_alone()` --calls--> `parse_link_entries()` [INFERRED] services/booth/tests/test_booth.py → services/booth/booth/links.py - `test_cli_unlink_rejects_a_non_id_non_index()` --calls--> `parse_link_entries()` [INFERRED] services/booth/tests/test_booth.py → services/booth/booth/links.py +- `test_remove_on_a_board_with_no_file_is_a_no_op()` --calls--> `remove_link_entry()` [INFERRED] + services/booth/tests/test_booth.py → services/booth/booth/links.py ## Import Cycles - None detected. -## Communities (426 total, 41 thin omitted) +## Communities (424 total, 40 thin omitted) ### Community 0 - "quant_modelopt.py" Cohesion: 0.53 @@ -761,7 +759,7 @@ Nodes (7): ana-ml2, GPU allocation policy, Hardware, Key paths, Network, Refresh ### Community 78 - "test_booth.py" Cohesion: 0.06 -Nodes (33): build_gallery(), Files in a booth as render items, with caption sidecars folded in. A…, (rendered, is_html). Markdown → HTML (fenced code, tables, sane lists); plain…, render_doc(), Releasing something already released is a no-op, not a 500., test_booth_autogallery_renders_media(), test_build_gallery_folds_caption_sidecars(), test_build_gallery_oversize_doc_is_not_inlined() (+25 more) +Nodes (37): build_gallery(), Files in a booth as render items, with caption sidecars folded in. A…, (rendered, is_html). Markdown → HTML (fenced code, tables, sane lists); plain…, render_doc(), Releasing something already released is a no-op, not a 500., Both mutating routes must use resolve_booth, not raw path joining. A name…, It is a dotfile, so it must not inflate the item count or become a tile., test_booth_autogallery_renders_media() (+29 more) ### Community 79 - "pfi-tacticalrmm" Cohesion: 0.25 @@ -1075,9 +1073,9 @@ Nodes (6): H, BaseHTTPRequestHandler, Handler, zed-fim-proxy — a keyless front Cohesion: 0.67 Nodes (3): anyOf, default, estimated_latency -### Community 160 - "Path" -Cohesion: 0.14 -Nodes (18): booth_age_seconds(), booth_image_names(), classify(), doc_kind(), list_booths(), _newest_mtime(), Path, Image files in a booth, in gallery (sorted-rel) order — for viewer prev/next. (+10 more) +### Community 160 - "booth/app.py" +Cohesion: 0.13 +Nodes (22): booth_age_seconds(), booth_image_names(), classify(), doc_kind(), is_expired(), list_booths(), _newest_mtime(), Path (+14 more) ### Community 163 - "bench_w4a16_recheck.json" Cohesion: 0.20 @@ -1108,8 +1106,8 @@ Cohesion: 0.25 Nodes (7): Fleet backup architecture & freshness runbook, Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage), Known gaps / TODO, Layer 1 — PBS (whole-VM/CT images), Layer 2 — restic (granular file + DB), The 2-minute freshness check, TL;DR — coverage matrix ### Community 179 - "convert_hf_to_native.py" -Cohesion: 0.07 -Nodes (42): ArgumentParser, _arrow_rows(), _datasets_root(), load_calibration(), load_evaluation(), Held-out evaluation prompts. Returns (harmless, harmful, provenance). This, Read the `text` column out of one cached HF arrow split., Return (harmful, harmless, provenance). `builtin` is the legacy 8/8 set. `m (+34 more) +Cohesion: 0.10 +Nodes (29): _get_model(), OpenAI-ish /v1/audio/speech adapter in front of the Zonos Python SDK. Why this, _speaker_embedding(), speech(), SpeechRequest, ArgumentParser, _is_mtp(), main() (+21 more) ### Community 181 - "Mistral Small 4 → NVFP4 (vision-intact) build tooling" Cohesion: 0.29 @@ -1164,8 +1162,8 @@ Cohesion: 0.33 Nodes (5): Containerization plan (pending build), Live invocation (source of truth), Status (2026-07-16): NATIVE, config was live-only — now captured here, The engine, zonos-engine — ZONOS2 native TTS engine (`:1920`, irv-ml1 3090) ### Community 200 - "omnivoice/app.py" -Cohesion: 0.14 -Nodes (12): Any, _base_gen_kwargs(), _pcm16(), Thin FastAPI wrapper exposing OmniVoice (k2-fsa/OmniVoice) for the fleet. Upstr, Validate the voice source and build the MODEL.generate kwargs minus `text`., Synthesize one text span -> (float32 audio [-1,1], audio_seconds)., float32 [-1,1] -> little-endian s16 PCM bytes (24 kHz mono on the wire)., WAV header. data_len=None -> streaming (0xFFFFFFFF sizes, read to EOF); an i (+4 more) +Cohesion: 0.06 +Nodes (46): Any, ChunkConfig, ChunkResult, ClockFn, GenerateFn, _base_gen_kwargs(), _chunk_config(), GenParams (+38 more) ### Community 202 - "phasefinal-web" Cohesion: 0.29 @@ -1175,9 +1173,9 @@ Nodes (6): Content constraints, Deploy, DNS, phasefinal-web, Routing — and the Cohesion: 0.29 Nodes (6): 2026-07-14 status — gibberish FIXED, format PIVOTED to modelopt for MTP, Artifacts (on ana-ml2), Fire sequence, Gates, heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP), Serve target -### Community 205 - "booth/app.py" -Cohesion: 0.11 -Nodes (18): fixture, create_app(), _from_env(), generate_pickup_id(), human_dur(), The Booth — a standing web server that renders drop-folders as ephemeral media…, A Content-Disposition-safe `.zip` (strip quotes/control chars)., Reduce a client-supplied filename to a safe basename (no path, no hidden). (+10 more) +### Community 205 - "FastAPI" +Cohesion: 0.18 +Nodes (12): FastAPI, fixture, create_app(), _from_env(), human_dur(), client(), test_human_dur(), test_upload_rejects_too_large() (+4 more) ### Community 206 - "soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated)" Cohesion: 0.33 @@ -1456,8 +1454,8 @@ Cohesion: 0.17 Nodes (11): ANSWERED — Lobe TTS is a SPLIT: endpoint env-driven, voice/model/format UI-only, ⚠️ Capacity — tell tts-dev if this ramps, ⚠️ CORRECTION — my earlier voice foot-gun warning was wrong, Credential posture, Deploy, lobe-chat — chat frontend over the LiteLLM gateway (esh-docker-vm), System Agent — why `gpt-5-mini` was being called, The real constraint is SIZE, not length (+3 more) ### Community 296 - "sweep_once" -Cohesion: 0.16 -Nodes (16): is_expired(), Pure age question. Deliberately does NOT consider the keep sentinel. Expiry…, Wipe every direct-child booth older than the TTL. Returns names wiped. Only…, sweep_once(), Age a booth and everything in it well past any test TTL., The point of the whole feature: expiry does not apply to a kept booth., is_expired stays a pure age question; only the sweeper honours the pin. Keeping…, Counter-intuitive, and the reason release-then-sweep is not a delete path.… (+8 more) +Cohesion: 0.19 +Nodes (13): Wipe every direct-child booth older than the TTL. Returns names wiped. Only…, sweep_once(), Age a booth and everything in it well past any test TTL., The point of the whole feature: expiry does not apply to a kept booth., is_expired stays a pure age question; only the sweeper honours the pin. Keeping…, Counter-intuitive, and the reason release-then-sweep is not a delete path.…, It does rejoin the sweep — just on a fresh clock, not the old one., _stale() (+5 more) ### Community 297 - "heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (ana-ml2)" Cohesion: 0.40 @@ -1556,8 +1554,8 @@ Cohesion: 0.70 Nodes (4): ask(), classify(), load_prompts(), main() ### Community 327 - "worker/app.py" -Cohesion: 0.14 -Nodes (10): RuntimeError, get_gpu_status(), post_train(), FastAPI surface for the LoRA training worker (§4.1 API — arbo is the client). E, TrainRequest, Static configuration for the LoRA training worker. Everything load-bearing is a, LoRA training worker — a host service (runs as llmuser on irv-ml1) that runs sd-, Busy (+2 more) +Cohesion: 0.12 +Nodes (18): RuntimeError, get_gpu_status(), post_train(), FastAPI surface for the LoRA training worker (§4.1 API — arbo is the client). E, TrainRequest, Static configuration for the LoRA training worker. Everything load-bearing is a, _bus_id_by_index(), _cmdline() (+10 more) ### Community 329 - "`[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight" Cohesion: 0.29 @@ -1607,9 +1605,9 @@ Nodes (4): default, title, type, optional Cohesion: 0.11 Nodes (19): additionalProperties, properties, required, title, type, CatalogAuditEntry, title, type (+11 more) -### Community 343 - "gpu.py" -Cohesion: 0.42 -Nodes (8): _bus_id_by_index(), _cmdline(), gpu_status(), _query_compute_apps(), _query_devices(), GPU status for arbo's device-aware scheduler (GET /gpu-status). arbo steers a l, Return `{devices: [...], tts_on_3090: bool}`. Degrades to an error field on nvid, _run() +### Community 343 - "kl_divergence.py" +Cohesion: 0.13 +Nodes (20): _arrow_rows(), _datasets_root(), load_calibration(), load_evaluation(), Held-out evaluation prompts. Returns (harmless, harmful, provenance). This, Read the `text` column out of one cached HF arrow split., Return (harmful, harmless, provenance). `builtin` is the legacy 8/8 set. `m, collect() (+12 more) ### Community 344 - "step" Cohesion: 0.50 @@ -1767,17 +1765,17 @@ Nodes (4): ACME disabled — the WAN IP now exposes nothing, CLOSED OUT (2026-08 Cohesion: 0.17 Nodes (10): ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for, lora_B gate — PASSED, twice, The confound I built and he caught, The gate — brokkr-smithy-dev, The noise-floor near-miss — the methodology lesson, The run, ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one, Refusal retention — the axis the gate did not have, and the axis I measured wrong (+2 more) -### Community 388 - "stream_chunks" -Cohesion: 0.19 -Nodes (17): ClockFn, GenerateFn, ChunkConfig, _ema(), _est_gen_time(), plan_chunk(), protect_first_audio(), Split into sentence units, preserving punctuation. Whitespace-collapsed. (+9 more) +### Community 388 - "`[2026-09-01]` Ada migration settled on `zfs send` — and branch (b) was never available" +Cohesion: 0.11 +Nodes (16): `[2026-09-01]` Ada migration settled on `zfs send` — and branch (b) was never available, (c) rebuild-from-source: rejected on reproducibility, not time, Open, The answer: (a) `zfs send`. Measured, not derived., ⚠ The two-boxes confusion — do not repeat it, Their distinction, worth keeping, Why (b) — physically moving the disks — was never on the table, `[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c (+8 more) ### Community 389 - "CatalogField" Cohesion: 0.33 Nodes (6): additionalProperties, description, required, title, type, CatalogField -### Community 390 - "ChunkResult" -Cohesion: 0.15 -Nodes (13): ChunkConfig, ChunkResult, _chunk_config(), _log_chunk(), Streaming /tts request — chatterbox-fast-compatible wire protocol., Streaming: chunked 24 kHz mono PCM (or open-ended WAV) for live consumers., tts(), TTSStreamRequest (+5 more) +### Community 390 - "`[2026-09-01]` Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed" +Cohesion: 0.22 +Nodes (8): `[2026-09-01]` Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed, Conventions ratified (operator, 2026-09-01), Push reality — measured, and it inverts the obvious reading, QR sign-in — requires MAS, deferred, Shared-secret registration gotcha, `/_synapse/admin` closed to the internet, The appservice namespace — `exclusive: true` → `false`, The upgrade ### Community 391 - "type" Cohesion: 0.50 @@ -1800,8 +1798,8 @@ Cohesion: 0.19 Nodes (14): Path, test_published_relative_path(), test_published_relative_path_explicit_train_id_wins(), ValueError, InvalidTrainRequest, published_relative_path(), Fixed-invocation command builder — the enforcement point for INV-T7. arbo hands, The ComfyUI-relative loras path for a succeeded LoRA (Phase 2 publish step). (+6 more) ### Community 396 - "BaseModel" -Cohesion: 0.19 -Nodes (12): ACEStepInput, ACEStepOutput, generate_audio(), initialize_pipeline(), Generate music; respond with the audio bytes inline. Pre-2026-05-11 this re, ACEStepPipeline, SpeechRequest, BaseModel (+4 more) +Cohesion: 0.31 +Nodes (7): ACEStepInput, ACEStepOutput, generate_audio(), initialize_pipeline(), Generate music; respond with the audio bytes inline. Pre-2026-05-11 this re, ACEStepPipeline, BaseModel ### Community 398 - "_cli" Cohesion: 0.25 @@ -1871,13 +1869,9 @@ Nodes (8): OfflineRecognizer, _decode(), _ensure_model_present(), _load_recogniz Cohesion: 0.30 Nodes (7): _build_prompt(), _cap(), _decode(), _encode_ref(), tts(), tts_stream(), TTSReq -### Community 420 - "adapter/server.py" -Cohesion: 0.31 -Nodes (6): _get_model(), OpenAI-ish /v1/audio/speech adapter in front of the Zonos Python SDK. Why this, _speaker_embedding(), speech(), JSONResponse, Zonos - -### Community 422 - "FastAPI" -Cohesion: 0.47 -Nodes (4): FastAPI, lifespan(), sfx(), SfxRequest +### Community 420 - "generate_pickup_id" +Cohesion: 0.24 +Nodes (7): generate_pickup_id(), Reduce a client-supplied filename to a safe basename (no path, no hidden)., A human-readable id like '4-wombat' or 'star-84'. `exists(name)->bool` gates…, safe_upload_name(), test_generate_pickup_id_avoids_collision(), test_generate_pickup_id_format(), test_safe_upload_name() ### Community 423 - "license" Cohesion: 0.67 @@ -1892,19 +1886,19 @@ Cohesion: 0.67 Nodes (3): Design consequences of terminating downstream — the parts that need decisions, FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through, Standing recommendation ## Knowledge Gaps -- **2337 isolated node(s):** `⚠ The password is NOT in this copy`, `/_synapse/admin is LAN-only`, `Upgrades`, `Active migration — docker.io 20.10 → docker-ce 29.x`, `Architecture decisions (durable)` (+2332 more) +- **2357 isolated node(s):** `The answer: (a) `zfs send`. Measured, not derived.`, `Why (b) — physically moving the disks — was never on the table`, `(c) rebuild-from-source: rejected on reproducibility, not time`, `⚠ The two-boxes confusion — do not repeat it`, `Their distinction, worth keeping` (+2352 more) These have ≤1 connection - possible missing edges or undocumented components. -- **41 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes. +- **40 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes. ## Suggested Questions _Questions this graph is uniquely positioned to answer:_ -- **Why does `Tensor` connect `convert_hf_to_native.py` to `adapter/server.py`?** +- **Why does `Tensor` connect `convert_hf_to_native.py` to `kl_divergence.py`?** + _High betweenness centrality (0.005) - this node is a cross-community bridge._ +- **Why does `divergences()` connect `kl_divergence.py` to `convert_hf_to_native.py`?** _High betweenness centrality (0.004) - this node is a cross-community bridge._ -- **Why does `_speaker_embedding()` connect `adapter/server.py` to `convert_hf_to_native.py`?** - _High betweenness centrality (0.004) - this node is a cross-community bridge._ -- **What connects `⚠ The password is NOT in this copy`, `/_synapse/admin is LAN-only`, `Upgrades` to the rest of the system?** - _2337 weakly-connected nodes found - possible documentation gaps or missing edges._ +- **What connects `The answer: (a) `zfs send`. Measured, not derived.`, `Why (b) — physically moving the disks — was never on the table`, `(c) rebuild-from-source: rejected on reproducibility, not time` to the rest of the system?** + _2357 weakly-connected nodes found - possible documentation gaps or missing edges._ - **Should `Status + Open Issues` be split into smaller, more focused modules?** _Cohesion score 0.05555555555555555 - nodes in this community are weakly interconnected._ - **Should `CatalogReproducibility` be split into smaller, more focused modules?** diff --git a/persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md b/persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md new file mode 100644 index 0000000..9d02f35 --- /dev/null +++ b/persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md @@ -0,0 +1,111 @@ +# `[2026-09-01]` The Ada inference server is a Dell R750xa bought used — and the reseller stripped four things Dell shipped + +Dell PowerEdge R750xa, service tag **`JPJ1ZP3`**, 8-bay 2.5" 2U. Bought from Entera Source +(invoice INV-74422, 2026-08-25). **2× NVIDIA RTX 6000 Ada 48 GB** to be fitted — confirmed by +the operator, NOT L40/L40S. Destination NH3. This is ComfyUI's new home; the GX10 is a +separate machine → [[2026-09-01-pfi-gx10-onboarding]]. + +## As bought + + CPU 2x Xeon Silver 4314 16C/32T each = 32C/64T, 135 W each + RAM 8x Micron 16 GB PC4-3200 = 128 GB + NIC dual 25 GbE SFP28 (Broadcom 57414, OCP 3.0) + boot BOSS-S2 + 2x 240 GB M.2 SATA, mirrored + storage 2x 1.92 TB SATA SSD (Hynix), 6 of 8 bays free + HBA PERC HBA355i (LSI SAS3816) + PSU 2x 1400 W Platinum + +## THE DIFF — factory CSV (`JPJ1ZP3.csv`, booth `54-puffin`) vs the invoice + +| | Dell factory | reseller invoice | +|---|---|---| +| **RAM** | **16× 16 GB `M04W6`**, "Performance Optimized" = **256 GB** | 8× = 128 GB | +| **PSU** | **2400 W** redundant (`450-AJEV`) | 2× 1400 W (`7DWXY`) | +| **GPUs** | 4× Tesla V100-32GB passive (`490-BHUC`) | none | +| **GPU risers** | `12XPY` (right DW), `9TR6X` (left DW) | **not listed** | +| **GPU cables** | `470-AEVS` 4-GPU kit: `4RW1P` `W4K7M` `XC48N` `6C77X` | **not listed** | +| **Riser cables** | signal `CP67W` `CXYF8` `H4D7D` `N61TK`; power `HXJDR` `W1P56`; pad `C2JNP` | **not listed** | +| **Fans** | 6× `FD00R` **Very High Performance** | **not listed** | +| **Controller** | PERC H345 | **HBA355i — an UPGRADE for us** | +| **Drives** | 2× 480 GB | 2× 1.92 TB | + +⚠ **"Not listed on the invoice" is not proof of removal.** Operator will pull an iDRAC +inventory to settle it. Diff that against the CSV rather than the invoice. + +## The GPU power chain — resolved, and my first answer was wrong + +**RTX 6000 Ada: 300 W, ACTIVE cooling, one 16-pin 12VHPWR (CEM5) connector** (nvidia.com). + +The R750xa kit `470-AEVS` is **"W/CPU CONN"** — CPU/EPS 8-pin, because V100 takes that. I +initially flagged this as a mismatch requiring the PCIe-type cable `RCCWC` / Dell +`470-BBXH`. **That was wrong and is withdrawn** — `RCCWC` is for H100/L40/L40S. + +NVIDIA sells the bridge for exactly this case: + +> **`930-00030-1546-000`** — NVIDIA Power Cable, RTX 6000 Ada Generation, +> **1× CPU 8-Pin → 1× CEM5 16-Pin** + +So the chain is `470-AEVS` CPU 8-pin → `930-00030-1546-000` → card. **The chassis' existing +cabling is the correct source type.** Two adapters needed, one per card. + +⚠ Factory brackets are `VFYGM` — the **V100S** extended bracket. RTX 6000 Ada is a different +physical card; bracket fit is UNKNOWN, check at fitting. + +⚠ **Active cooling in a passive-card chassis.** Six high-performance fans expect to push air +through passive cards; the RTX 6000 Ada has its own blower. It works in practice, but Dell +never validated this combination for this box. + +## Storage — NVMe in the drive bays is CLOSED + +Factory backplane is `321-BGKG : 2.5" Chassis with up to 8 **SAS/SATA** Drives`. Not +NVMe-capable; bays would need a backplane swap. + +Arithmetic if filling bays with SATA: + + 2x SATA (today) ~1.1 GB/s + 6x striped ~3.3 GB/s <- roughly ONE NVMe drive + 8x striped ~4.4 GB/s + backplane ceiling ~6 GB/s (8 lanes x SATA III) + +The HBA355i (PCIe 4.0 x8) is nowhere near limiting. **Drives are the constraint.** + +**ARC probably beats spindles here.** ComfyUI reloads the same checkpoints; with the factory +256 GB restored, ZFS ARC caches most of the working set at RAM speed. The RAM upgrade is +both cheaper and likely faster than four more SATA drives. + +**UNCHECKED, and it may moot the whole bay question:** free PCIe slots. Riser Config 0 is +`330-BBUM : Riser Config 0, 6x16 + 2x8 slots`; two double-wide GPUs will not consume all of +it. An NVMe add-in card (U.2 HBA or bifurcating M.2 carrier) bypasses the SAS/SATA backplane +entirely. **Add slot inventory to the iDRAC pull.** + +## Power — the ceiling is deliberate now, not discovered later + + 2x RTX 6000 Ada 600 W + 2x Xeon 4314 270 W + rest ~150 W + ~1,020 W sustained + +One 1400 W PSU carries that, so redundancy holds. But factory specced **2400 W** for four +V100s: **there is no headroom for a third or fourth GPU.** And ~1 kW is the same class as +ana-ml2, the box whose training run tripped a breaker on 2026-08-26 — **decide which NH3 +circuit this lands on before racking, not after.** + +## Buy list + +**Certain:** + +| item | part | qty | +|---|---|---| +| RDIMM 16 GB 3200 2Rx8 | **`M04W6`** | **8** → restores 256 GB, all 16 slots, all 8 channels/socket | +| NVIDIA 12VHPWR adapter | **`930-00030-1546-000`** | **2** | + +**Only if the iDRAC inventory shows them absent:** `12XPY`, `9TR6X`, `4RW1P`, `W4K7M`, +`XC48N`, `6C77X`, `CP67W`, `CXYF8`, `H4D7D`, `N61TK`, `HXJDR`, `W1P56`, `C2JNP`, +`FD00R` ×6. + +**Spend order:** RAM → check PCIe slots → only then consider more SATA drives. + +⚠ Dell's Installation & Service Manual publishes **no part numbers at all** for GPU kit +hardware — names and quantities only. The factory CSV keyed to the service tag is the only +authoritative source of DPNs for this machine. Do not accept a five-character DPN from +anywhere else without corroboration. diff --git a/persistent-memory.md b/persistent-memory.md index b344481..ddbf154 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -119,6 +119,14 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing one command afterwards (`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`). ⚠ Triton has no sm_121 support; compiled deps are per-arch unknowns. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` +- **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000 + Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores + factory 256 GB) and 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters.** Everything else waits + on the inventory — the invoice omits risers, GPU cables and the 6 high-performance fans, but + omission is not proof of removal. **Add free-PCIe-slot inventory to the pull** — an NVMe + add-in card would bypass the SAS/SATA backplane and moot the drive-bay question. + ⚠ ~1 kW loaded — same class as the box that tripped the breaker; pick the NH3 circuit before + racking. → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md` - **⏸ RUN 3c STILL HELD — but the plan has changed.** Config `/tank/erp-tune/run-03c.json` validated, relaunch is one command on ana-ml2. **It is now intended to move to the GX10 instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2 @@ -147,6 +155,7 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing ## Recent decisions +- `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md` - `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` - `[2026-09-01]` **Ada migration is `zfs send` (branch a) — branch (b) was never available because irv-ml1 keeps its eight services.** 99 MB/s measured; ~3.9 h. Also records the two-boxes confusion: the Ada box and the GX10 are DIFFERENT machines. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` - `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md` @@ -251,7 +260,6 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing - `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase. -- `[2026-08-16]` **Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated.** brokkr-smithy-dev supplied the framing that reproduces (`01M05M48R4RSZF9D8KT7RR55EJ`): a **bare assistant-mode instruction** — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing **DS 0.0% / FF 0.0%** (n=75); plain character cards **DS 1.4% / FF 0.0%** (n=74); bare instruction **DS 92.5% (37/40) / FF 15.8% (6/38)**. Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→**0**, consensual 80→20, dubcon 80→**0**, self-harm 80→**0**. DS refused **25/25** on the five axes brokkr flagged. Root cause: `ReadyArt/Dark-Scarlett-v1.0-27B` is a plain finetune of stock `Qwen/Qwen3.6-27B` carrying **NO abliteration** — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-**ablated** (structural), which is why it holds. ⚠ **RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID.** His diff against his own artifact showed my `battery-instruct.yaml` reproduces only his **`creative` class — 8 of 16 axes**; it dropped all 5 `operational` (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 `meta` (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. **His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta.** Cause: I rebuilt his shape from his *message*, and the `class` field lives in the artifact, not the prose. **Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability.** ⚠ **Known battery bug left unfixed for comparability:** DS's arm-3 control gate failed at 11% because `ictrl-reunion` pairs "explicit / do not fade to black" with *brothers*, which DS reasonably read as an incest request; FF did not. `ictrl-storm` is the clean control. Commit `b9e68c3`. - `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right. @@ -273,23 +281,19 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing - `[2026-08-15]` **RP-seat direction: KEEP MeroMero on `char-rp`; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator).** Evaluated `TheDrummer/Artemis-31B-v1.1` — mechanically a drop-in (same `google/gemma-4-31B-it` base, identical 1188-tensor/356-vision census, same missing-`preprocessor_config.json` trick), so it's purely a quality call, and our own survey already ranked MeroMero **#1** vs Artemis **#6**; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. **MTP verified impossible on both** (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). **But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter** — vLLM 0.24 supports `eagle3` + `gemma4_mtp`, and real drafters exist: `google/gemma-4-31B-it-assistant` (0.94 GB, 4-layer, 761K dl), `RedHatAI/gemma-4-31B-it-speculator.eagle3` (4.47 GB), `AEON-7/…eagle3-NVFP4` (3.53 GB). ⚠ all list their verifier as **stock** gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). **Why the Dark-Scarlett 3.8 plan is the strong one:** DS is Qwen3.6-based today, so a 3.8 respin lands on the *gen seat's* architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: `from_pretrained` **silently drops MTP heads during finetuning** (verify 15 `mtp.*` tensors in the index; graft from stock if absent), and DS v1.0 required the `Qwen3_5ForConditionalGeneration` **wrapper class** to save a config vLLM/SGLang accept. Both in `docs/pfi/model-quantization-playbook.md`. -- `[2026-08-15]` **Quant lessons consolidated into `docs/pfi/model-quantization-playbook.md` — the durable home; read it BEFORE any requant.** Survey found quant knowledge scattered across 18 files in 4 trees, with **three** documents having independently written overlapping "landmines" sections (the loader-class trap alone was rediscovered 3×). Playbook owns the **transferable** lessons (scheme choice, landmines, acceptance gate + its 3 measurement traps, hardware/co-residency); per-model artifacts are demoted to worked examples that link up. Carries a **superseded-claims table** — which immediately earned itself: the heretic2 runbook's "use modelopt, compressed-tensors can't load the BF16 MTP" is **false** (the cause was the missing `re:^mtp.*` ignore, not the format) and would have sent the next session down the modelopt dependency-hell path; that runbook now carries a stale-warning header. Maintenance rule in `CLAUDE.md`: model-agnostic → playbook, model-specific → stays put, wrong claim → dated superseded row, never a silent edit. Motivated by Qwen3.8 having just released — the next model swap needs a requant. Commit `a91cc3f`. -- `[2026-08-15]` **Operator ruling: the gen seat's +1.7% perplexity is an acceptable price for the speed — SETTLED, don't re-litigate.** Precise attribution for future reasoning: it is the **activation-quantization** cost (W4A4 MLPs + FP8 attention vs BF16 activations), not an MTP cost — PPL was measured with speculative decoding **off** on both builds, so MTP was not in the loop. Turning MTP off would not recover it; only reverting the quant would (rollback = one `.env` line, old build intact at `…/qwen38-27b-uncensored-nvfp4`). -- `[2026-08-15]` **gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed.** The queued "W4A8" (NVFP4 weights + FP8 activations) is **not servable** — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and `CompressedTensorsW4A8Fp8` is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter **per-layer-group**. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did **80.12** (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: `unsloth/Qwen3.8-27B-NVFP4` was already on-box → served as a probe, measured **+19.1% at identical acceptance**, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → **80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%**; surface 6/6 live, 7 aliases routing. char-rp had **no** tool parser at all (every tools request 400'd) → `gemma4` tool + reasoning parser + a **mandatory** `enable_thinking:false` (the parser defaults it True → null `content` for all RP prose; proven byte-identical prompt before deploying). Commits `b8f0f4c`, `74f596b`. Foot-guns banked (llm-compressor prunes unmatched `ignore` entries → the 0%-MTP bug, **fired on this run**; prompt_logprobs uniform under spec-decode; 0600 `.env` silently no-ops compose; GPU0 is zero-sum). → `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md` -- `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md` - `[2026-08-10→12]` **secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed.** `secret` CLI (`put/get/list/rm/backfill`, bw-backed) on `~/.local/bin`; 25 nh3-dev secrets backfilled + round-trip-verified; `rm` + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → `persistent-memory.d/2026-08-12-secrets-broker.md` - `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/.wav` + `transcripts/.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]] -- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md` -- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md` _223 older entries archived to archival-memory.md._ +_Older entries archived to archival-memory.md._ + ## Tried and abandoned - `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.