memory: snapshot — the Ada inference server is a stripped used R750xa
Dell R750xa JPJ1ZP3, 2x RTX 6000 Ada to be fitted, ComfyUI's new home at NH3. Diffing Dell's factory CSV against the reseller invoice shows four downgrades: half the RAM, the 2400 W PSUs, and the GPU risers, cables and high-performance fans all absent. Records the resolved GPU power chain, correcting my own first answer: the chassis' CPU 8-pin cabling is the right source type and NVIDIA 930-00030-1546-000 bridges it to the card's 12VHPWR, so the PCIe-type RCCWC I first proposed is withdrawn. Also closes the NVMe question — the backplane is SAS/SATA only — and notes that free PCIe slots may moot it. Auto-archival fired at 308 lines; seven entries moved to archival-memory.md. The 250-line target was not reached because the guards hold nearly everything else back as under 14 days or carrying open deferred work.
This commit is contained in:
@@ -4,6 +4,22 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
|||||||
|
|
||||||
## Recent decisions (archived)
|
## Recent decisions (archived)
|
||||||
|
|
||||||
|
- `[2026-08-16]` **Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated.** brokkr-smithy-dev supplied the framing that reproduces (`01M05M48R4RSZF9D8KT7RR55EJ`): a **bare assistant-mode instruction** — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing **DS 0.0% / FF 0.0%** (n=75); plain character cards **DS 1.4% / FF 0.0%** (n=74); bare instruction **DS 92.5% (37/40) / FF 15.8% (6/38)**. Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→**0**, consensual 80→20, dubcon 80→**0**, self-harm 80→**0**. DS refused **25/25** on the five axes brokkr flagged. Root cause: `ReadyArt/Dark-Scarlett-v1.0-27B` is a plain finetune of stock `Qwen/Qwen3.6-27B` carrying **NO abliteration** — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-**ablated** (structural), which is why it holds. ⚠ **RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID.** His diff against his own artifact showed my `battery-instruct.yaml` reproduces only his **`creative` class — 8 of 16 axes**; it dropped all 5 `operational` (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 `meta` (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. **His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta.** Cause: I rebuilt his shape from his *message*, and the `class` field lives in the artifact, not the prose. **Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability.** ⚠ **Known battery bug left unfixed for comparability:** DS's arm-3 control gate failed at 11% because `ictrl-reunion` pairs "explicit / do not fade to black" with *brothers*, which DS reasonably read as an incest request; FF did not. `ictrl-storm` is the clean control. Commit `b9e68c3`.
|
||||||
|
_Archived 2026-09-01._
|
||||||
|
- `[2026-08-15]` **Quant lessons consolidated into `docs/pfi/model-quantization-playbook.md` — the durable home; read it BEFORE any requant.** Survey found quant knowledge scattered across 18 files in 4 trees, with **three** documents having independently written overlapping "landmines" sections (the loader-class trap alone was rediscovered 3×). Playbook owns the **transferable** lessons (scheme choice, landmines, acceptance gate + its 3 measurement traps, hardware/co-residency); per-model artifacts are demoted to worked examples that link up. Carries a **superseded-claims table** — which immediately earned itself: the heretic2 runbook's "use modelopt, compressed-tensors can't load the BF16 MTP" is **false** (the cause was the missing `re:^mtp.*` ignore, not the format) and would have sent the next session down the modelopt dependency-hell path; that runbook now carries a stale-warning header. Maintenance rule in `CLAUDE.md`: model-agnostic → playbook, model-specific → stays put, wrong claim → dated superseded row, never a silent edit. Motivated by Qwen3.8 having just released — the next model swap needs a requant. Commit `a91cc3f`.
|
||||||
|
_Archived 2026-09-01._
|
||||||
|
- `[2026-08-15]` **Operator ruling: the gen seat's +1.7% perplexity is an acceptable price for the speed — SETTLED, don't re-litigate.** Precise attribution for future reasoning: it is the **activation-quantization** cost (W4A4 MLPs + FP8 attention vs BF16 activations), not an MTP cost — PPL was measured with speculative decoding **off** on both builds, so MTP was not in the loop. Turning MTP off would not recover it; only reverting the quant would (rollback = one `.env` line, old build intact at `…/qwen38-27b-uncensored-nvfp4`).
|
||||||
|
_Archived 2026-09-01._
|
||||||
|
- `[2026-08-15]` **gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed.** The queued "W4A8" (NVFP4 weights + FP8 activations) is **not servable** — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and `CompressedTensorsW4A8Fp8` is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter **per-layer-group**. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did **80.12** (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: `unsloth/Qwen3.8-27B-NVFP4` was already on-box → served as a probe, measured **+19.1% at identical acceptance**, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → **80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%**; surface 6/6 live, 7 aliases routing. char-rp had **no** tool parser at all (every tools request 400'd) → `gemma4` tool + reasoning parser + a **mandatory** `enable_thinking:false` (the parser defaults it True → null `content` for all RP prose; proven byte-identical prompt before deploying). Commits `b8f0f4c`, `74f596b`. Foot-guns banked (llm-compressor prunes unmatched `ignore` entries → the 0%-MTP bug, **fired on this run**; prompt_logprobs uniform under spec-decode; 0600 `.env` silently no-ops compose; GPU0 is zero-sum). → `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`
|
||||||
|
_Archived 2026-09-01._
|
||||||
|
- `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
|
||||||
|
_Archived 2026-09-01._
|
||||||
|
|
||||||
|
- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md`
|
||||||
|
_Archived 2026-09-01._
|
||||||
|
- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md`
|
||||||
|
_Archived 2026-09-01._
|
||||||
|
|
||||||
- `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`.
|
- `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`.
|
||||||
_Archived 2026-08-28. Its deferred item — the nh3-extdev herald 2.1.2 upgrade — is closed: extdev went 2.1.0 -> 3.0.0 -> 3.1.1 at the v3 cutover, so the staged v2.1.2 wheel is moot._
|
_Archived 2026-08-28. Its deferred item — the nh3-extdev herald 2.1.2 upgrade — is closed: extdev went 2.1.0 -> 3.0.0 -> 3.1.1 at the v3 cutover, so the staged v2.1.2 wheel is moot._
|
||||||
|
|
||||||
|
|||||||
@@ -1,16 +1,16 @@
|
|||||||
# Graph Report - eshpfi-management (2026-09-01)
|
# Graph Report - eshpfi-management (2026-09-01)
|
||||||
|
|
||||||
## Corpus Check
|
## Corpus Check
|
||||||
- 378 files · ~573,530 words
|
- 382 files · ~576,491 words
|
||||||
- Verdict: corpus is large enough that graph structure adds value.
|
- Verdict: corpus is large enough that graph structure adds value.
|
||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
- 3878 nodes · 4117 edges · 426 communities (385 shown, 41 thin omitted)
|
- 3906 nodes · 4144 edges · 424 communities (384 shown, 40 thin omitted)
|
||||||
- Extraction: 99% EXTRACTED · 1% INFERRED · 0% AMBIGUOUS · INFERRED: 38 edges (avg confidence: 0.71)
|
- Extraction: 99% EXTRACTED · 1% INFERRED · 0% AMBIGUOUS · INFERRED: 38 edges (avg confidence: 0.71)
|
||||||
- Token cost: 0 input · 0 output
|
- Token cost: 0 input · 0 output
|
||||||
|
|
||||||
## Graph Freshness
|
## Graph Freshness
|
||||||
- Built from commit: `73866f6a`
|
- Built from commit: `71426577`
|
||||||
- Run `git rev-parse HEAD` and compare to check if the graph is stale.
|
- Run `git rev-parse HEAD` and compare to check if the graph is stale.
|
||||||
- Run `graphify update .` after code changes (no API cost).
|
- Run `graphify update .` after code changes (no API cost).
|
||||||
|
|
||||||
@@ -175,7 +175,7 @@
|
|||||||
- blend_kokoro_voice.py
|
- blend_kokoro_voice.py
|
||||||
- Handler
|
- Handler
|
||||||
- estimated_latency
|
- estimated_latency
|
||||||
- Path
|
- booth/app.py
|
||||||
- bench.py
|
- bench.py
|
||||||
- serve_nvfp4.sh
|
- serve_nvfp4.sh
|
||||||
- bench_w4a16_recheck.json
|
- bench_w4a16_recheck.json
|
||||||
@@ -217,7 +217,7 @@
|
|||||||
- phasefinal-web
|
- phasefinal-web
|
||||||
- build.sh script
|
- build.sh script
|
||||||
- heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP)
|
- heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP)
|
||||||
- booth/app.py
|
- FastAPI
|
||||||
- soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated)
|
- soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated)
|
||||||
- nh3-dev `~/development` — hourly off-box backup
|
- nh3-dev `~/development` — hourly off-box backup
|
||||||
- nh3-dev-development-backup.sh script
|
- nh3-dev-development-backup.sh script
|
||||||
@@ -339,7 +339,7 @@
|
|||||||
- [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing
|
- [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing
|
||||||
- optional
|
- optional
|
||||||
- properties
|
- properties
|
||||||
- gpu.py
|
- kl_divergence.py
|
||||||
- step
|
- step
|
||||||
- `[2026-08-27]` A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find
|
- `[2026-08-27]` A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find
|
||||||
- host
|
- host
|
||||||
@@ -381,9 +381,9 @@
|
|||||||
- `[2026-08-27]` Run 3 gated: the rule PASSED and a k=25 follow-up found a self-harm guardrail collapse
|
- `[2026-08-27]` Run 3 gated: the rule PASSED and a k=25 follow-up found a self-harm guardrail collapse
|
||||||
- CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE
|
- CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE
|
||||||
- ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for
|
- ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for
|
||||||
- stream_chunks
|
- `[2026-09-01]` Ada migration settled on `zfs send` — and branch (b) was never available
|
||||||
- CatalogField
|
- CatalogField
|
||||||
- ChunkResult
|
- `[2026-09-01]` Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed
|
||||||
- type
|
- type
|
||||||
- `[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way
|
- `[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way
|
||||||
- counted_classifier.py
|
- counted_classifier.py
|
||||||
@@ -411,9 +411,7 @@
|
|||||||
- index-tts/app.py
|
- index-tts/app.py
|
||||||
- parakeet/app.py
|
- parakeet/app.py
|
||||||
- tts/app.py
|
- tts/app.py
|
||||||
- adapter/server.py
|
- generate_pickup_id
|
||||||
- test_keep_and_unkeep_go_through_the_same_name_guard
|
|
||||||
- FastAPI
|
|
||||||
- license
|
- license
|
||||||
- version
|
- version
|
||||||
- FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through
|
- FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through
|
||||||
@@ -433,19 +431,19 @@
|
|||||||
## Surprising Connections (you probably didn't know these)
|
## Surprising Connections (you probably didn't know these)
|
||||||
- `convert()` --references--> `Any` [EXTRACTED]
|
- `convert()` --references--> `Any` [EXTRACTED]
|
||||||
tools/mistral-small4-nvfp4/convert_hf_to_native.py → stacks/omnivoice/app.py
|
tools/mistral-small4-nvfp4/convert_hf_to_native.py → stacks/omnivoice/app.py
|
||||||
- `_speaker_embedding()` --references--> `Tensor` [EXTRACTED]
|
|
||||||
stacks/zonos/adapter/server.py → services/coldfusion-abliteration/kl_divergence.py
|
|
||||||
- `test_cli_unlink_by_id_even_when_the_id_is_all_digits()` --calls--> `parse_link_entries()` [INFERRED]
|
- `test_cli_unlink_by_id_even_when_the_id_is_all_digits()` --calls--> `parse_link_entries()` [INFERRED]
|
||||||
services/booth/tests/test_booth.py → services/booth/booth/links.py
|
services/booth/tests/test_booth.py → services/booth/booth/links.py
|
||||||
- `test_cli_unlink_of_a_stale_id_leaves_the_board_alone()` --calls--> `parse_link_entries()` [INFERRED]
|
- `test_cli_unlink_of_a_stale_id_leaves_the_board_alone()` --calls--> `parse_link_entries()` [INFERRED]
|
||||||
services/booth/tests/test_booth.py → services/booth/booth/links.py
|
services/booth/tests/test_booth.py → services/booth/booth/links.py
|
||||||
- `test_cli_unlink_rejects_a_non_id_non_index()` --calls--> `parse_link_entries()` [INFERRED]
|
- `test_cli_unlink_rejects_a_non_id_non_index()` --calls--> `parse_link_entries()` [INFERRED]
|
||||||
services/booth/tests/test_booth.py → services/booth/booth/links.py
|
services/booth/tests/test_booth.py → services/booth/booth/links.py
|
||||||
|
- `test_remove_on_a_board_with_no_file_is_a_no_op()` --calls--> `remove_link_entry()` [INFERRED]
|
||||||
|
services/booth/tests/test_booth.py → services/booth/booth/links.py
|
||||||
|
|
||||||
## Import Cycles
|
## Import Cycles
|
||||||
- None detected.
|
- None detected.
|
||||||
|
|
||||||
## Communities (426 total, 41 thin omitted)
|
## Communities (424 total, 40 thin omitted)
|
||||||
|
|
||||||
### Community 0 - "quant_modelopt.py"
|
### Community 0 - "quant_modelopt.py"
|
||||||
Cohesion: 0.53
|
Cohesion: 0.53
|
||||||
@@ -761,7 +759,7 @@ Nodes (7): ana-ml2, GPU allocation policy, Hardware, Key paths, Network, Refresh
|
|||||||
|
|
||||||
### Community 78 - "test_booth.py"
|
### Community 78 - "test_booth.py"
|
||||||
Cohesion: 0.06
|
Cohesion: 0.06
|
||||||
Nodes (33): build_gallery(), Files in a booth as render items, with caption sidecars folded in. A…, (rendered, is_html). Markdown → HTML (fenced code, tables, sane lists); plain…, render_doc(), Releasing something already released is a no-op, not a 500., test_booth_autogallery_renders_media(), test_build_gallery_folds_caption_sidecars(), test_build_gallery_oversize_doc_is_not_inlined() (+25 more)
|
Nodes (37): build_gallery(), Files in a booth as render items, with caption sidecars folded in. A…, (rendered, is_html). Markdown → HTML (fenced code, tables, sane lists); plain…, render_doc(), Releasing something already released is a no-op, not a 500., Both mutating routes must use resolve_booth, not raw path joining. A name…, It is a dotfile, so it must not inflate the item count or become a tile., test_booth_autogallery_renders_media() (+29 more)
|
||||||
|
|
||||||
### Community 79 - "pfi-tacticalrmm"
|
### Community 79 - "pfi-tacticalrmm"
|
||||||
Cohesion: 0.25
|
Cohesion: 0.25
|
||||||
@@ -1075,9 +1073,9 @@ Nodes (6): H, BaseHTTPRequestHandler, Handler, zed-fim-proxy — a keyless front
|
|||||||
Cohesion: 0.67
|
Cohesion: 0.67
|
||||||
Nodes (3): anyOf, default, estimated_latency
|
Nodes (3): anyOf, default, estimated_latency
|
||||||
|
|
||||||
### Community 160 - "Path"
|
### Community 160 - "booth/app.py"
|
||||||
Cohesion: 0.14
|
Cohesion: 0.13
|
||||||
Nodes (18): booth_age_seconds(), booth_image_names(), classify(), doc_kind(), list_booths(), _newest_mtime(), Path, Image files in a booth, in gallery (sorted-rel) order — for viewer prev/next. (+10 more)
|
Nodes (22): booth_age_seconds(), booth_image_names(), classify(), doc_kind(), is_expired(), list_booths(), _newest_mtime(), Path (+14 more)
|
||||||
|
|
||||||
### Community 163 - "bench_w4a16_recheck.json"
|
### Community 163 - "bench_w4a16_recheck.json"
|
||||||
Cohesion: 0.20
|
Cohesion: 0.20
|
||||||
@@ -1108,8 +1106,8 @@ Cohesion: 0.25
|
|||||||
Nodes (7): Fleet backup architecture & freshness runbook, Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage), Known gaps / TODO, Layer 1 — PBS (whole-VM/CT images), Layer 2 — restic (granular file + DB), The 2-minute freshness check, TL;DR — coverage matrix
|
Nodes (7): Fleet backup architecture & freshness runbook, Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage), Known gaps / TODO, Layer 1 — PBS (whole-VM/CT images), Layer 2 — restic (granular file + DB), The 2-minute freshness check, TL;DR — coverage matrix
|
||||||
|
|
||||||
### Community 179 - "convert_hf_to_native.py"
|
### Community 179 - "convert_hf_to_native.py"
|
||||||
Cohesion: 0.07
|
Cohesion: 0.10
|
||||||
Nodes (42): ArgumentParser, _arrow_rows(), _datasets_root(), load_calibration(), load_evaluation(), Held-out evaluation prompts. Returns (harmless, harmful, provenance). This, Read the `text` column out of one cached HF arrow split., Return (harmful, harmless, provenance). `builtin` is the legacy 8/8 set. `m (+34 more)
|
Nodes (29): _get_model(), OpenAI-ish /v1/audio/speech adapter in front of the Zonos Python SDK. Why this, _speaker_embedding(), speech(), SpeechRequest, ArgumentParser, _is_mtp(), main() (+21 more)
|
||||||
|
|
||||||
### Community 181 - "Mistral Small 4 → NVFP4 (vision-intact) build tooling"
|
### Community 181 - "Mistral Small 4 → NVFP4 (vision-intact) build tooling"
|
||||||
Cohesion: 0.29
|
Cohesion: 0.29
|
||||||
@@ -1164,8 +1162,8 @@ Cohesion: 0.33
|
|||||||
Nodes (5): Containerization plan (pending build), Live invocation (source of truth), Status (2026-07-16): NATIVE, config was live-only — now captured here, The engine, zonos-engine — ZONOS2 native TTS engine (`:1920`, irv-ml1 3090)
|
Nodes (5): Containerization plan (pending build), Live invocation (source of truth), Status (2026-07-16): NATIVE, config was live-only — now captured here, The engine, zonos-engine — ZONOS2 native TTS engine (`:1920`, irv-ml1 3090)
|
||||||
|
|
||||||
### Community 200 - "omnivoice/app.py"
|
### Community 200 - "omnivoice/app.py"
|
||||||
Cohesion: 0.14
|
Cohesion: 0.06
|
||||||
Nodes (12): Any, _base_gen_kwargs(), _pcm16(), Thin FastAPI wrapper exposing OmniVoice (k2-fsa/OmniVoice) for the fleet. Upstr, Validate the voice source and build the MODEL.generate kwargs minus `text`., Synthesize one text span -> (float32 audio [-1,1], audio_seconds)., float32 [-1,1] -> little-endian s16 PCM bytes (24 kHz mono on the wire)., WAV header. data_len=None -> streaming (0xFFFFFFFF sizes, read to EOF); an i (+4 more)
|
Nodes (46): Any, ChunkConfig, ChunkResult, ClockFn, GenerateFn, _base_gen_kwargs(), _chunk_config(), GenParams (+38 more)
|
||||||
|
|
||||||
### Community 202 - "phasefinal-web"
|
### Community 202 - "phasefinal-web"
|
||||||
Cohesion: 0.29
|
Cohesion: 0.29
|
||||||
@@ -1175,9 +1173,9 @@ Nodes (6): Content constraints, Deploy, DNS, phasefinal-web, Routing — and the
|
|||||||
Cohesion: 0.29
|
Cohesion: 0.29
|
||||||
Nodes (6): 2026-07-14 status — gibberish FIXED, format PIVOTED to modelopt for MTP, Artifacts (on ana-ml2), Fire sequence, Gates, heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP), Serve target
|
Nodes (6): 2026-07-14 status — gibberish FIXED, format PIVOTED to modelopt for MTP, Artifacts (on ana-ml2), Fire sequence, Gates, heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP), Serve target
|
||||||
|
|
||||||
### Community 205 - "booth/app.py"
|
### Community 205 - "FastAPI"
|
||||||
Cohesion: 0.11
|
Cohesion: 0.18
|
||||||
Nodes (18): fixture, create_app(), _from_env(), generate_pickup_id(), human_dur(), The Booth — a standing web server that renders drop-folders as ephemeral media…, A Content-Disposition-safe `<booth>.zip` (strip quotes/control chars)., Reduce a client-supplied filename to a safe basename (no path, no hidden). (+10 more)
|
Nodes (12): FastAPI, fixture, create_app(), _from_env(), human_dur(), client(), test_human_dur(), test_upload_rejects_too_large() (+4 more)
|
||||||
|
|
||||||
### Community 206 - "soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated)"
|
### Community 206 - "soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated)"
|
||||||
Cohesion: 0.33
|
Cohesion: 0.33
|
||||||
@@ -1456,8 +1454,8 @@ Cohesion: 0.17
|
|||||||
Nodes (11): ANSWERED — Lobe TTS is a SPLIT: endpoint env-driven, voice/model/format UI-only, ⚠️ Capacity — tell tts-dev if this ramps, ⚠️ CORRECTION — my earlier voice foot-gun warning was wrong, Credential posture, Deploy, lobe-chat — chat frontend over the LiteLLM gateway (esh-docker-vm), System Agent — why `gpt-5-mini` was being called, The real constraint is SIZE, not length (+3 more)
|
Nodes (11): ANSWERED — Lobe TTS is a SPLIT: endpoint env-driven, voice/model/format UI-only, ⚠️ Capacity — tell tts-dev if this ramps, ⚠️ CORRECTION — my earlier voice foot-gun warning was wrong, Credential posture, Deploy, lobe-chat — chat frontend over the LiteLLM gateway (esh-docker-vm), System Agent — why `gpt-5-mini` was being called, The real constraint is SIZE, not length (+3 more)
|
||||||
|
|
||||||
### Community 296 - "sweep_once"
|
### Community 296 - "sweep_once"
|
||||||
Cohesion: 0.16
|
Cohesion: 0.19
|
||||||
Nodes (16): is_expired(), Pure age question. Deliberately does NOT consider the keep sentinel. Expiry…, Wipe every direct-child booth older than the TTL. Returns names wiped. Only…, sweep_once(), Age a booth and everything in it well past any test TTL., The point of the whole feature: expiry does not apply to a kept booth., is_expired stays a pure age question; only the sweeper honours the pin. Keeping…, Counter-intuitive, and the reason release-then-sweep is not a delete path.… (+8 more)
|
Nodes (13): Wipe every direct-child booth older than the TTL. Returns names wiped. Only…, sweep_once(), Age a booth and everything in it well past any test TTL., The point of the whole feature: expiry does not apply to a kept booth., is_expired stays a pure age question; only the sweeper honours the pin. Keeping…, Counter-intuitive, and the reason release-then-sweep is not a delete path.…, It does rejoin the sweep — just on a fresh clock, not the old one., _stale() (+5 more)
|
||||||
|
|
||||||
### Community 297 - "heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (ana-ml2)"
|
### Community 297 - "heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (ana-ml2)"
|
||||||
Cohesion: 0.40
|
Cohesion: 0.40
|
||||||
@@ -1556,8 +1554,8 @@ Cohesion: 0.70
|
|||||||
Nodes (4): ask(), classify(), load_prompts(), main()
|
Nodes (4): ask(), classify(), load_prompts(), main()
|
||||||
|
|
||||||
### Community 327 - "worker/app.py"
|
### Community 327 - "worker/app.py"
|
||||||
Cohesion: 0.14
|
Cohesion: 0.12
|
||||||
Nodes (10): RuntimeError, get_gpu_status(), post_train(), FastAPI surface for the LoRA training worker (§4.1 API — arbo is the client). E, TrainRequest, Static configuration for the LoRA training worker. Everything load-bearing is a, LoRA training worker — a host service (runs as llmuser on irv-ml1) that runs sd-, Busy (+2 more)
|
Nodes (18): RuntimeError, get_gpu_status(), post_train(), FastAPI surface for the LoRA training worker (§4.1 API — arbo is the client). E, TrainRequest, Static configuration for the LoRA training worker. Everything load-bearing is a, _bus_id_by_index(), _cmdline() (+10 more)
|
||||||
|
|
||||||
### Community 329 - "`[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight"
|
### Community 329 - "`[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight"
|
||||||
Cohesion: 0.29
|
Cohesion: 0.29
|
||||||
@@ -1607,9 +1605,9 @@ Nodes (4): default, title, type, optional
|
|||||||
Cohesion: 0.11
|
Cohesion: 0.11
|
||||||
Nodes (19): additionalProperties, properties, required, title, type, CatalogAuditEntry, title, type (+11 more)
|
Nodes (19): additionalProperties, properties, required, title, type, CatalogAuditEntry, title, type (+11 more)
|
||||||
|
|
||||||
### Community 343 - "gpu.py"
|
### Community 343 - "kl_divergence.py"
|
||||||
Cohesion: 0.42
|
Cohesion: 0.13
|
||||||
Nodes (8): _bus_id_by_index(), _cmdline(), gpu_status(), _query_compute_apps(), _query_devices(), GPU status for arbo's device-aware scheduler (GET /gpu-status). arbo steers a l, Return `{devices: [...], tts_on_3090: bool}`. Degrades to an error field on nvid, _run()
|
Nodes (20): _arrow_rows(), _datasets_root(), load_calibration(), load_evaluation(), Held-out evaluation prompts. Returns (harmless, harmful, provenance). This, Read the `text` column out of one cached HF arrow split., Return (harmful, harmless, provenance). `builtin` is the legacy 8/8 set. `m, collect() (+12 more)
|
||||||
|
|
||||||
### Community 344 - "step"
|
### Community 344 - "step"
|
||||||
Cohesion: 0.50
|
Cohesion: 0.50
|
||||||
@@ -1767,17 +1765,17 @@ Nodes (4): ACME disabled — the WAN IP now exposes nothing, CLOSED OUT (2026-08
|
|||||||
Cohesion: 0.17
|
Cohesion: 0.17
|
||||||
Nodes (10): ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for, lora_B gate — PASSED, twice, The confound I built and he caught, The gate — brokkr-smithy-dev, The noise-floor near-miss — the methodology lesson, The run, ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one, Refusal retention — the axis the gate did not have, and the axis I measured wrong (+2 more)
|
Nodes (10): ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for, lora_B gate — PASSED, twice, The confound I built and he caught, The gate — brokkr-smithy-dev, The noise-floor near-miss — the methodology lesson, The run, ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one, Refusal retention — the axis the gate did not have, and the axis I measured wrong (+2 more)
|
||||||
|
|
||||||
### Community 388 - "stream_chunks"
|
### Community 388 - "`[2026-09-01]` Ada migration settled on `zfs send` — and branch (b) was never available"
|
||||||
Cohesion: 0.19
|
Cohesion: 0.11
|
||||||
Nodes (17): ClockFn, GenerateFn, ChunkConfig, _ema(), _est_gen_time(), plan_chunk(), protect_first_audio(), Split into sentence units, preserving punctuation. Whitespace-collapsed. (+9 more)
|
Nodes (16): `[2026-09-01]` Ada migration settled on `zfs send` — and branch (b) was never available, (c) rebuild-from-source: rejected on reproducibility, not time, Open, The answer: (a) `zfs send`. Measured, not derived., ⚠ The two-boxes confusion — do not repeat it, Their distinction, worth keeping, Why (b) — physically moving the disks — was never on the table, `[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c (+8 more)
|
||||||
|
|
||||||
### Community 389 - "CatalogField"
|
### Community 389 - "CatalogField"
|
||||||
Cohesion: 0.33
|
Cohesion: 0.33
|
||||||
Nodes (6): additionalProperties, description, required, title, type, CatalogField
|
Nodes (6): additionalProperties, description, required, title, type, CatalogField
|
||||||
|
|
||||||
### Community 390 - "ChunkResult"
|
### Community 390 - "`[2026-09-01]` Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed"
|
||||||
Cohesion: 0.15
|
Cohesion: 0.22
|
||||||
Nodes (13): ChunkConfig, ChunkResult, _chunk_config(), _log_chunk(), Streaming /tts request — chatterbox-fast-compatible wire protocol., Streaming: chunked 24 kHz mono PCM (or open-ended WAV) for live consumers., tts(), TTSStreamRequest (+5 more)
|
Nodes (8): `[2026-09-01]` Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed, Conventions ratified (operator, 2026-09-01), Push reality — measured, and it inverts the obvious reading, QR sign-in — requires MAS, deferred, Shared-secret registration gotcha, `/_synapse/admin` closed to the internet, The appservice namespace — `exclusive: true` → `false`, The upgrade
|
||||||
|
|
||||||
### Community 391 - "type"
|
### Community 391 - "type"
|
||||||
Cohesion: 0.50
|
Cohesion: 0.50
|
||||||
@@ -1800,8 +1798,8 @@ Cohesion: 0.19
|
|||||||
Nodes (14): Path, test_published_relative_path(), test_published_relative_path_explicit_train_id_wins(), ValueError, InvalidTrainRequest, published_relative_path(), Fixed-invocation command builder — the enforcement point for INV-T7. arbo hands, The ComfyUI-relative loras path for a succeeded LoRA (Phase 2 publish step). (+6 more)
|
Nodes (14): Path, test_published_relative_path(), test_published_relative_path_explicit_train_id_wins(), ValueError, InvalidTrainRequest, published_relative_path(), Fixed-invocation command builder — the enforcement point for INV-T7. arbo hands, The ComfyUI-relative loras path for a succeeded LoRA (Phase 2 publish step). (+6 more)
|
||||||
|
|
||||||
### Community 396 - "BaseModel"
|
### Community 396 - "BaseModel"
|
||||||
Cohesion: 0.19
|
Cohesion: 0.31
|
||||||
Nodes (12): ACEStepInput, ACEStepOutput, generate_audio(), initialize_pipeline(), Generate music; respond with the audio bytes inline. Pre-2026-05-11 this re, ACEStepPipeline, SpeechRequest, BaseModel (+4 more)
|
Nodes (7): ACEStepInput, ACEStepOutput, generate_audio(), initialize_pipeline(), Generate music; respond with the audio bytes inline. Pre-2026-05-11 this re, ACEStepPipeline, BaseModel
|
||||||
|
|
||||||
### Community 398 - "_cli"
|
### Community 398 - "_cli"
|
||||||
Cohesion: 0.25
|
Cohesion: 0.25
|
||||||
@@ -1871,13 +1869,9 @@ Nodes (8): OfflineRecognizer, _decode(), _ensure_model_present(), _load_recogniz
|
|||||||
Cohesion: 0.30
|
Cohesion: 0.30
|
||||||
Nodes (7): _build_prompt(), _cap(), _decode(), _encode_ref(), tts(), tts_stream(), TTSReq
|
Nodes (7): _build_prompt(), _cap(), _decode(), _encode_ref(), tts(), tts_stream(), TTSReq
|
||||||
|
|
||||||
### Community 420 - "adapter/server.py"
|
### Community 420 - "generate_pickup_id"
|
||||||
Cohesion: 0.31
|
Cohesion: 0.24
|
||||||
Nodes (6): _get_model(), OpenAI-ish /v1/audio/speech adapter in front of the Zonos Python SDK. Why this, _speaker_embedding(), speech(), JSONResponse, Zonos
|
Nodes (7): generate_pickup_id(), Reduce a client-supplied filename to a safe basename (no path, no hidden)., A human-readable id like '4-wombat' or 'star-84'. `exists(name)->bool` gates…, safe_upload_name(), test_generate_pickup_id_avoids_collision(), test_generate_pickup_id_format(), test_safe_upload_name()
|
||||||
|
|
||||||
### Community 422 - "FastAPI"
|
|
||||||
Cohesion: 0.47
|
|
||||||
Nodes (4): FastAPI, lifespan(), sfx(), SfxRequest
|
|
||||||
|
|
||||||
### Community 423 - "license"
|
### Community 423 - "license"
|
||||||
Cohesion: 0.67
|
Cohesion: 0.67
|
||||||
@@ -1892,19 +1886,19 @@ Cohesion: 0.67
|
|||||||
Nodes (3): Design consequences of terminating downstream — the parts that need decisions, FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through, Standing recommendation
|
Nodes (3): Design consequences of terminating downstream — the parts that need decisions, FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through, Standing recommendation
|
||||||
|
|
||||||
## Knowledge Gaps
|
## Knowledge Gaps
|
||||||
- **2337 isolated node(s):** `⚠ The password is NOT in this copy`, `/_synapse/admin is LAN-only`, `Upgrades`, `Active migration — docker.io 20.10 → docker-ce 29.x`, `Architecture decisions (durable)` (+2332 more)
|
- **2357 isolated node(s):** `The answer: (a) `zfs send`. Measured, not derived.`, `Why (b) — physically moving the disks — was never on the table`, `(c) rebuild-from-source: rejected on reproducibility, not time`, `⚠ The two-boxes confusion — do not repeat it`, `Their distinction, worth keeping` (+2352 more)
|
||||||
These have ≤1 connection - possible missing edges or undocumented components.
|
These have ≤1 connection - possible missing edges or undocumented components.
|
||||||
- **41 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
|
- **40 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
|
||||||
|
|
||||||
## Suggested Questions
|
## Suggested Questions
|
||||||
_Questions this graph is uniquely positioned to answer:_
|
_Questions this graph is uniquely positioned to answer:_
|
||||||
|
|
||||||
- **Why does `Tensor` connect `convert_hf_to_native.py` to `adapter/server.py`?**
|
- **Why does `Tensor` connect `convert_hf_to_native.py` to `kl_divergence.py`?**
|
||||||
|
_High betweenness centrality (0.005) - this node is a cross-community bridge._
|
||||||
|
- **Why does `divergences()` connect `kl_divergence.py` to `convert_hf_to_native.py`?**
|
||||||
_High betweenness centrality (0.004) - this node is a cross-community bridge._
|
_High betweenness centrality (0.004) - this node is a cross-community bridge._
|
||||||
- **Why does `_speaker_embedding()` connect `adapter/server.py` to `convert_hf_to_native.py`?**
|
- **What connects `The answer: (a) `zfs send`. Measured, not derived.`, `Why (b) — physically moving the disks — was never on the table`, `(c) rebuild-from-source: rejected on reproducibility, not time` to the rest of the system?**
|
||||||
_High betweenness centrality (0.004) - this node is a cross-community bridge._
|
_2357 weakly-connected nodes found - possible documentation gaps or missing edges._
|
||||||
- **What connects `⚠ The password is NOT in this copy`, `/_synapse/admin is LAN-only`, `Upgrades` to the rest of the system?**
|
|
||||||
_2337 weakly-connected nodes found - possible documentation gaps or missing edges._
|
|
||||||
- **Should `Status + Open Issues` be split into smaller, more focused modules?**
|
- **Should `Status + Open Issues` be split into smaller, more focused modules?**
|
||||||
_Cohesion score 0.05555555555555555 - nodes in this community are weakly interconnected._
|
_Cohesion score 0.05555555555555555 - nodes in this community are weakly interconnected._
|
||||||
- **Should `CatalogReproducibility` be split into smaller, more focused modules?**
|
- **Should `CatalogReproducibility` be split into smaller, more focused modules?**
|
||||||
|
|||||||
@@ -0,0 +1,111 @@
|
|||||||
|
# `[2026-09-01]` The Ada inference server is a Dell R750xa bought used — and the reseller stripped four things Dell shipped
|
||||||
|
|
||||||
|
Dell PowerEdge R750xa, service tag **`JPJ1ZP3`**, 8-bay 2.5" 2U. Bought from Entera Source
|
||||||
|
(invoice INV-74422, 2026-08-25). **2× NVIDIA RTX 6000 Ada 48 GB** to be fitted — confirmed by
|
||||||
|
the operator, NOT L40/L40S. Destination NH3. This is ComfyUI's new home; the GX10 is a
|
||||||
|
separate machine → [[2026-09-01-pfi-gx10-onboarding]].
|
||||||
|
|
||||||
|
## As bought
|
||||||
|
|
||||||
|
CPU 2x Xeon Silver 4314 16C/32T each = 32C/64T, 135 W each
|
||||||
|
RAM 8x Micron 16 GB PC4-3200 = 128 GB
|
||||||
|
NIC dual 25 GbE SFP28 (Broadcom 57414, OCP 3.0)
|
||||||
|
boot BOSS-S2 + 2x 240 GB M.2 SATA, mirrored
|
||||||
|
storage 2x 1.92 TB SATA SSD (Hynix), 6 of 8 bays free
|
||||||
|
HBA PERC HBA355i (LSI SAS3816)
|
||||||
|
PSU 2x 1400 W Platinum
|
||||||
|
|
||||||
|
## THE DIFF — factory CSV (`JPJ1ZP3.csv`, booth `54-puffin`) vs the invoice
|
||||||
|
|
||||||
|
| | Dell factory | reseller invoice |
|
||||||
|
|---|---|---|
|
||||||
|
| **RAM** | **16× 16 GB `M04W6`**, "Performance Optimized" = **256 GB** | 8× = 128 GB |
|
||||||
|
| **PSU** | **2400 W** redundant (`450-AJEV`) | 2× 1400 W (`7DWXY`) |
|
||||||
|
| **GPUs** | 4× Tesla V100-32GB passive (`490-BHUC`) | none |
|
||||||
|
| **GPU risers** | `12XPY` (right DW), `9TR6X` (left DW) | **not listed** |
|
||||||
|
| **GPU cables** | `470-AEVS` 4-GPU kit: `4RW1P` `W4K7M` `XC48N` `6C77X` | **not listed** |
|
||||||
|
| **Riser cables** | signal `CP67W` `CXYF8` `H4D7D` `N61TK`; power `HXJDR` `W1P56`; pad `C2JNP` | **not listed** |
|
||||||
|
| **Fans** | 6× `FD00R` **Very High Performance** | **not listed** |
|
||||||
|
| **Controller** | PERC H345 | **HBA355i — an UPGRADE for us** |
|
||||||
|
| **Drives** | 2× 480 GB | 2× 1.92 TB |
|
||||||
|
|
||||||
|
⚠ **"Not listed on the invoice" is not proof of removal.** Operator will pull an iDRAC
|
||||||
|
inventory to settle it. Diff that against the CSV rather than the invoice.
|
||||||
|
|
||||||
|
## The GPU power chain — resolved, and my first answer was wrong
|
||||||
|
|
||||||
|
**RTX 6000 Ada: 300 W, ACTIVE cooling, one 16-pin 12VHPWR (CEM5) connector** (nvidia.com).
|
||||||
|
|
||||||
|
The R750xa kit `470-AEVS` is **"W/CPU CONN"** — CPU/EPS 8-pin, because V100 takes that. I
|
||||||
|
initially flagged this as a mismatch requiring the PCIe-type cable `RCCWC` / Dell
|
||||||
|
`470-BBXH`. **That was wrong and is withdrawn** — `RCCWC` is for H100/L40/L40S.
|
||||||
|
|
||||||
|
NVIDIA sells the bridge for exactly this case:
|
||||||
|
|
||||||
|
> **`930-00030-1546-000`** — NVIDIA Power Cable, RTX 6000 Ada Generation,
|
||||||
|
> **1× CPU 8-Pin → 1× CEM5 16-Pin**
|
||||||
|
|
||||||
|
So the chain is `470-AEVS` CPU 8-pin → `930-00030-1546-000` → card. **The chassis' existing
|
||||||
|
cabling is the correct source type.** Two adapters needed, one per card.
|
||||||
|
|
||||||
|
⚠ Factory brackets are `VFYGM` — the **V100S** extended bracket. RTX 6000 Ada is a different
|
||||||
|
physical card; bracket fit is UNKNOWN, check at fitting.
|
||||||
|
|
||||||
|
⚠ **Active cooling in a passive-card chassis.** Six high-performance fans expect to push air
|
||||||
|
through passive cards; the RTX 6000 Ada has its own blower. It works in practice, but Dell
|
||||||
|
never validated this combination for this box.
|
||||||
|
|
||||||
|
## Storage — NVMe in the drive bays is CLOSED
|
||||||
|
|
||||||
|
Factory backplane is `321-BGKG : 2.5" Chassis with up to 8 **SAS/SATA** Drives`. Not
|
||||||
|
NVMe-capable; bays would need a backplane swap.
|
||||||
|
|
||||||
|
Arithmetic if filling bays with SATA:
|
||||||
|
|
||||||
|
2x SATA (today) ~1.1 GB/s
|
||||||
|
6x striped ~3.3 GB/s <- roughly ONE NVMe drive
|
||||||
|
8x striped ~4.4 GB/s
|
||||||
|
backplane ceiling ~6 GB/s (8 lanes x SATA III)
|
||||||
|
|
||||||
|
The HBA355i (PCIe 4.0 x8) is nowhere near limiting. **Drives are the constraint.**
|
||||||
|
|
||||||
|
**ARC probably beats spindles here.** ComfyUI reloads the same checkpoints; with the factory
|
||||||
|
256 GB restored, ZFS ARC caches most of the working set at RAM speed. The RAM upgrade is
|
||||||
|
both cheaper and likely faster than four more SATA drives.
|
||||||
|
|
||||||
|
**UNCHECKED, and it may moot the whole bay question:** free PCIe slots. Riser Config 0 is
|
||||||
|
`330-BBUM : Riser Config 0, 6x16 + 2x8 slots`; two double-wide GPUs will not consume all of
|
||||||
|
it. An NVMe add-in card (U.2 HBA or bifurcating M.2 carrier) bypasses the SAS/SATA backplane
|
||||||
|
entirely. **Add slot inventory to the iDRAC pull.**
|
||||||
|
|
||||||
|
## Power — the ceiling is deliberate now, not discovered later
|
||||||
|
|
||||||
|
2x RTX 6000 Ada 600 W
|
||||||
|
2x Xeon 4314 270 W
|
||||||
|
rest ~150 W
|
||||||
|
~1,020 W sustained
|
||||||
|
|
||||||
|
One 1400 W PSU carries that, so redundancy holds. But factory specced **2400 W** for four
|
||||||
|
V100s: **there is no headroom for a third or fourth GPU.** And ~1 kW is the same class as
|
||||||
|
ana-ml2, the box whose training run tripped a breaker on 2026-08-26 — **decide which NH3
|
||||||
|
circuit this lands on before racking, not after.**
|
||||||
|
|
||||||
|
## Buy list
|
||||||
|
|
||||||
|
**Certain:**
|
||||||
|
|
||||||
|
| item | part | qty |
|
||||||
|
|---|---|---|
|
||||||
|
| RDIMM 16 GB 3200 2Rx8 | **`M04W6`** | **8** → restores 256 GB, all 16 slots, all 8 channels/socket |
|
||||||
|
| NVIDIA 12VHPWR adapter | **`930-00030-1546-000`** | **2** |
|
||||||
|
|
||||||
|
**Only if the iDRAC inventory shows them absent:** `12XPY`, `9TR6X`, `4RW1P`, `W4K7M`,
|
||||||
|
`XC48N`, `6C77X`, `CP67W`, `CXYF8`, `H4D7D`, `N61TK`, `HXJDR`, `W1P56`, `C2JNP`,
|
||||||
|
`FD00R` ×6.
|
||||||
|
|
||||||
|
**Spend order:** RAM → check PCIe slots → only then consider more SATA drives.
|
||||||
|
|
||||||
|
⚠ Dell's Installation & Service Manual publishes **no part numbers at all** for GPU kit
|
||||||
|
hardware — names and quantities only. The factory CSV keyed to the service tag is the only
|
||||||
|
authoritative source of DPNs for this machine. Do not accept a five-character DPN from
|
||||||
|
anywhere else without corroboration.
|
||||||
+11
-7
@@ -119,6 +119,14 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
|
|||||||
one command afterwards (`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`).
|
one command afterwards (`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`).
|
||||||
⚠ Triton has no sm_121 support; compiled deps are per-arch unknowns.
|
⚠ Triton has no sm_121 support; compiled deps are per-arch unknowns.
|
||||||
→ `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
|
→ `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
|
||||||
|
- **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000
|
||||||
|
Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores
|
||||||
|
factory 256 GB) and 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters.** Everything else waits
|
||||||
|
on the inventory — the invoice omits risers, GPU cables and the 6 high-performance fans, but
|
||||||
|
omission is not proof of removal. **Add free-PCIe-slot inventory to the pull** — an NVMe
|
||||||
|
add-in card would bypass the SAS/SATA backplane and moot the drive-bay question.
|
||||||
|
⚠ ~1 kW loaded — same class as the box that tripped the breaker; pick the NH3 circuit before
|
||||||
|
racking. → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
|
||||||
- **⏸ RUN 3c STILL HELD — but the plan has changed.** Config `/tank/erp-tune/run-03c.json`
|
- **⏸ RUN 3c STILL HELD — but the plan has changed.** Config `/tank/erp-tune/run-03c.json`
|
||||||
validated, relaunch is one command on ana-ml2. **It is now intended to move to the GX10
|
validated, relaunch is one command on ana-ml2. **It is now intended to move to the GX10
|
||||||
instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2
|
instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2
|
||||||
@@ -147,6 +155,7 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
|
|||||||
|
|
||||||
## Recent decisions
|
## Recent decisions
|
||||||
|
|
||||||
|
- `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
|
||||||
- `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
|
- `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
|
||||||
- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — branch (b) was never available because irv-ml1 keeps its eight services.** 99 MB/s measured; ~3.9 h. Also records the two-boxes confusion: the Ada box and the GX10 are DIFFERENT machines. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
|
- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — branch (b) was never available because irv-ml1 keeps its eight services.** 99 MB/s measured; ~3.9 h. Also records the two-boxes confusion: the Ada box and the GX10 are DIFFERENT machines. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
|
||||||
- `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md`
|
- `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md`
|
||||||
@@ -251,7 +260,6 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
|
|||||||
|
|
||||||
- `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase.
|
- `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase.
|
||||||
|
|
||||||
- `[2026-08-16]` **Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated.** brokkr-smithy-dev supplied the framing that reproduces (`01M05M48R4RSZF9D8KT7RR55EJ`): a **bare assistant-mode instruction** — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing **DS 0.0% / FF 0.0%** (n=75); plain character cards **DS 1.4% / FF 0.0%** (n=74); bare instruction **DS 92.5% (37/40) / FF 15.8% (6/38)**. Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→**0**, consensual 80→20, dubcon 80→**0**, self-harm 80→**0**. DS refused **25/25** on the five axes brokkr flagged. Root cause: `ReadyArt/Dark-Scarlett-v1.0-27B` is a plain finetune of stock `Qwen/Qwen3.6-27B` carrying **NO abliteration** — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-**ablated** (structural), which is why it holds. ⚠ **RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID.** His diff against his own artifact showed my `battery-instruct.yaml` reproduces only his **`creative` class — 8 of 16 axes**; it dropped all 5 `operational` (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 `meta` (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. **His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta.** Cause: I rebuilt his shape from his *message*, and the `class` field lives in the artifact, not the prose. **Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability.** ⚠ **Known battery bug left unfixed for comparability:** DS's arm-3 control gate failed at 11% because `ictrl-reunion` pairs "explicit / do not fade to black" with *brothers*, which DS reasonably read as an incest request; FF did not. `ictrl-storm` is the clean control. Commit `b9e68c3`.
|
|
||||||
|
|
||||||
- `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right.
|
- `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right.
|
||||||
|
|
||||||
@@ -273,23 +281,19 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
|
|||||||
|
|
||||||
- `[2026-08-15]` **RP-seat direction: KEEP MeroMero on `char-rp`; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator).** Evaluated `TheDrummer/Artemis-31B-v1.1` — mechanically a drop-in (same `google/gemma-4-31B-it` base, identical 1188-tensor/356-vision census, same missing-`preprocessor_config.json` trick), so it's purely a quality call, and our own survey already ranked MeroMero **#1** vs Artemis **#6**; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. **MTP verified impossible on both** (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). **But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter** — vLLM 0.24 supports `eagle3` + `gemma4_mtp`, and real drafters exist: `google/gemma-4-31B-it-assistant` (0.94 GB, 4-layer, 761K dl), `RedHatAI/gemma-4-31B-it-speculator.eagle3` (4.47 GB), `AEON-7/…eagle3-NVFP4` (3.53 GB). ⚠ all list their verifier as **stock** gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). **Why the Dark-Scarlett 3.8 plan is the strong one:** DS is Qwen3.6-based today, so a 3.8 respin lands on the *gen seat's* architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: `from_pretrained` **silently drops MTP heads during finetuning** (verify 15 `mtp.*` tensors in the index; graft from stock if absent), and DS v1.0 required the `Qwen3_5ForConditionalGeneration` **wrapper class** to save a config vLLM/SGLang accept. Both in `docs/pfi/model-quantization-playbook.md`.
|
- `[2026-08-15]` **RP-seat direction: KEEP MeroMero on `char-rp`; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator).** Evaluated `TheDrummer/Artemis-31B-v1.1` — mechanically a drop-in (same `google/gemma-4-31B-it` base, identical 1188-tensor/356-vision census, same missing-`preprocessor_config.json` trick), so it's purely a quality call, and our own survey already ranked MeroMero **#1** vs Artemis **#6**; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. **MTP verified impossible on both** (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). **But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter** — vLLM 0.24 supports `eagle3` + `gemma4_mtp`, and real drafters exist: `google/gemma-4-31B-it-assistant` (0.94 GB, 4-layer, 761K dl), `RedHatAI/gemma-4-31B-it-speculator.eagle3` (4.47 GB), `AEON-7/…eagle3-NVFP4` (3.53 GB). ⚠ all list their verifier as **stock** gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). **Why the Dark-Scarlett 3.8 plan is the strong one:** DS is Qwen3.6-based today, so a 3.8 respin lands on the *gen seat's* architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: `from_pretrained` **silently drops MTP heads during finetuning** (verify 15 `mtp.*` tensors in the index; graft from stock if absent), and DS v1.0 required the `Qwen3_5ForConditionalGeneration` **wrapper class** to save a config vLLM/SGLang accept. Both in `docs/pfi/model-quantization-playbook.md`.
|
||||||
|
|
||||||
- `[2026-08-15]` **Quant lessons consolidated into `docs/pfi/model-quantization-playbook.md` — the durable home; read it BEFORE any requant.** Survey found quant knowledge scattered across 18 files in 4 trees, with **three** documents having independently written overlapping "landmines" sections (the loader-class trap alone was rediscovered 3×). Playbook owns the **transferable** lessons (scheme choice, landmines, acceptance gate + its 3 measurement traps, hardware/co-residency); per-model artifacts are demoted to worked examples that link up. Carries a **superseded-claims table** — which immediately earned itself: the heretic2 runbook's "use modelopt, compressed-tensors can't load the BF16 MTP" is **false** (the cause was the missing `re:^mtp.*` ignore, not the format) and would have sent the next session down the modelopt dependency-hell path; that runbook now carries a stale-warning header. Maintenance rule in `CLAUDE.md`: model-agnostic → playbook, model-specific → stays put, wrong claim → dated superseded row, never a silent edit. Motivated by Qwen3.8 having just released — the next model swap needs a requant. Commit `a91cc3f`.
|
|
||||||
|
|
||||||
- `[2026-08-15]` **Operator ruling: the gen seat's +1.7% perplexity is an acceptable price for the speed — SETTLED, don't re-litigate.** Precise attribution for future reasoning: it is the **activation-quantization** cost (W4A4 MLPs + FP8 attention vs BF16 activations), not an MTP cost — PPL was measured with speculative decoding **off** on both builds, so MTP was not in the loop. Turning MTP off would not recover it; only reverting the quant would (rollback = one `.env` line, old build intact at `…/qwen38-27b-uncensored-nvfp4`).
|
|
||||||
|
|
||||||
- `[2026-08-15]` **gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed.** The queued "W4A8" (NVFP4 weights + FP8 activations) is **not servable** — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and `CompressedTensorsW4A8Fp8` is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter **per-layer-group**. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did **80.12** (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: `unsloth/Qwen3.8-27B-NVFP4` was already on-box → served as a probe, measured **+19.1% at identical acceptance**, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → **80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%**; surface 6/6 live, 7 aliases routing. char-rp had **no** tool parser at all (every tools request 400'd) → `gemma4` tool + reasoning parser + a **mandatory** `enable_thinking:false` (the parser defaults it True → null `content` for all RP prose; proven byte-identical prompt before deploying). Commits `b8f0f4c`, `74f596b`. Foot-guns banked (llm-compressor prunes unmatched `ignore` entries → the 0%-MTP bug, **fired on this run**; prompt_logprobs uniform under spec-decode; 0600 `.env` silently no-ops compose; GPU0 is zero-sum). → `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`
|
|
||||||
|
|
||||||
- `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
|
|
||||||
|
|
||||||
- `[2026-08-10→12]` **secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed.** `secret` CLI (`put/get/list/rm/backfill`, bw-backed) on `~/.local/bin`; 25 nh3-dev secrets backfilled + round-trip-verified; `rm` + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → `persistent-memory.d/2026-08-12-secrets-broker.md`
|
- `[2026-08-10→12]` **secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed.** `secret` CLI (`put/get/list/rm/backfill`, bw-backed) on `~/.local/bin`; 25 nh3-dev secrets backfilled + round-trip-verified; `rm` + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → `persistent-memory.d/2026-08-12-secrets-broker.md`
|
||||||
|
|
||||||
- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
|
- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
|
||||||
|
|
||||||
- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md`
|
|
||||||
|
|
||||||
- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md`
|
|
||||||
|
|
||||||
_223 older entries archived to archival-memory.md._
|
_223 older entries archived to archival-memory.md._
|
||||||
|
_Older entries archived to archival-memory.md._
|
||||||
|
|
||||||
## Tried and abandoned
|
## Tried and abandoned
|
||||||
|
|
||||||
- `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.
|
- `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.
|
||||||
|
|||||||
Reference in New Issue
Block a user