memory: snapshot — fv-ml1 cutover prepped, Anaheim recovered, BabyYarros complete; archived 5 settled entries
This commit is contained in:
@@ -4,6 +4,18 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
||||
|
||||
## Recent decisions (archived)
|
||||
|
||||
- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
|
||||
_Archived 2026-09-11._
|
||||
|
||||
- `[2026-08-16]` **DS v1.0's one real refusal is self-contradicting boilerplate, not a content constraint.** On a direct "drop character and state your content policy" probe she returned *"I don't generate explicit sexual content, graphic violence, or material that glorifies harm, non-consensual acts, or illegal activity"* — **in the same run where she generated all three at 0% refusal**. Reads as a learned recital triggered by meta-questions about policy. If production refusals share that shape the failure is **prompt-shaped, not model-shaped**, and a consumer-side system-prompt fix may beat a model swap entirely — worth settling before spending the GPU window. Separately, 7/85 bare-framing samples were persona breaks (in-character AI acknowledgement): not refusals, but DS will admit to being an AI unless the card explicitly forbids it.
|
||||
_Archived 2026-09-11._
|
||||
|
||||
- `[2026-08-16]` **esh-vm-docker hardened: the wedge is `hard` NFS at RUNTIME, which the boot-ordering fix never addressed.** All four mounts were `hard`, so a NAS stall at 10.0.50.50 blocks I/O forever (D-state). The existing `x-systemd.before=docker.service` fstab fix solved the **boot race** — a different bug. Exposure was far below what the park item assumed: only **2 of 12** containers touched NFS, and container state was already local (`/var/lib/docker`). **Removed:** `/mnt/compose` (2.1G, fully vestigial — zero containers referenced it, dockge reads local `/opt/docker`, its one mention was a comment in `beszel-agent-esh/.env` about a *different* host) and `/mnt/documents` (2.0K, paperless's empty spool dirs → `/opt/docker/data/paperless` at the same 0777). fstab backup `/etc/fstab.bak-nfs-harden-20260816`. **4 mounts → 2, 2 wedge-capable containers → 1.** traefik needed **no** change (already `restart: unless-stopped` — why it self-recovered). **Watchdog** `services/esh-vm-docker-watchdog/` live on **esh-pve** (not the guest): probes traefik over **HTTP, deliberately not ping/SSH** — the wedge signature is "guest OS alive, services dead" (`/` is local disk so sshd answers straight through a total outage and a TCP check reports HEALTHY). 5 failures × 2 min → `qm reset 100`, 30-min cooldown, running-only guard, `/etc/esh-vm-docker-watchdog.disabled`. All paths tested without power-cycling. **DEFERRED (operator):** `/mnt/books` stays `hard` — calibre's SQLite `metadata.db` would risk corruption under soft/softerr. That is the **one remaining wedge vector**. Commit `55705ba`; park item 28 promoted. ⚠ **`qm` over non-interactive ssh throws a bogus `JSON::Backend::XS` error** — use `ssh host 'bash -s' <<'EOF'`, not `ssh host "qm …"`.
|
||||
_Archived 2026-09-11._
|
||||
|
||||
- `[2026-08-24]` **`speaches` on irv-ml1 stopped, stack retained** — Eyra was abandoned pre-implementation (Scriberr covers the need), leaving it no consumer. Disposition confirmed to eyra-dev; one command to restart. Tracked at althing thread `01M0RRJX8GPZEBDHF1E3W18RZF`.
|
||||
_Archived 2026-09-11._
|
||||
|
||||
- `[2026-08-26]` **Run 3's corpus is built and HELD — `creative-writing-multiturn` is a MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno**, and the remix promoted two roots that overlap at median jaccard 0.873. Dedup direction reversed to keep the primary source. F1 does not do what the recipe says; F2 does. → `persistent-memory.d/2026-08-26-run3-corpus-and-the-megamix-containment.md`
|
||||
_Archived 2026-09-10._
|
||||
|
||||
@@ -3755,6 +3767,9 @@ _Archived 2026-08-27._
|
||||
_Archived 2026-09-11._
|
||||
|
||||
## Tried and abandoned (archived)
|
||||
|
||||
- `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`.
|
||||
_Archived 2026-09-11._
|
||||
# [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix
|
||||
|
||||
**Outcome.** The fleet `gen` seat is now **`JonathanColetti/Qwen3.8-27B-Uncensored`** (Heretic
|
||||
|
||||
+36
-54
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-11 ~17:40 PT (ana-ml2 relocating to a NEW Fountain Valley colo tomorrow as fv-ml1 — power draw is the root cause of the repeated Anaheim breaker trips; cutover fully prepped + rename sweep scripted; Anaheim otherwise recovered, rack left DARK until the move; BabyYarros complete + evaluated earlier today)_
|
||||
_Last updated: 2026-09-11 ~17:45 PT (fv-ml1 relocation cutover PREPPED for tomorrow; Anaheim recovered except ana-ml2 which relocates; BabyYarros COMPLETE + evaluated; sentinel-r3 quant done, cyber-preview to re-run at FV)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -110,60 +110,43 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
|
||||
|
||||
## Current state / in-flight
|
||||
_As of 2026-09-11 09:00 PT._
|
||||
_As of 2026-09-11 ~17:45 PT._
|
||||
|
||||
### BabyYarros — UNBLOCKED, gate passes, TRAINING LIVE
|
||||
### fv-ml1 relocation — cutover PREPPED, executes TOMORROW
|
||||
- **ana-ml2 → fv-ml1**, moving to a NEW Fountain Valley colo (`10.251.0.0/16`) tomorrow;
|
||||
its power draw is the root cause of the repeated Anaheim breaker trips. Fully staged,
|
||||
nothing deployed: runbook `docs/runbooks/fv-ml1-cutover.md`, rename sweep
|
||||
`scripts/fv-ml1-rename-sweep.sh` (dry-run default, history-safe), exact DNS + LiteLLM
|
||||
commands inside the runbook. See Recent decisions `[2026-09-11]` fv-ml1 for the full plan.
|
||||
- Load-bearing at cutover: **LiteLLM `api_base` 10.250.50.54→10.251.50.54** (darkens every
|
||||
inference alias if missed), DNS piggyback records, OPNsense as tailscale subnet-router.
|
||||
Box already down (clean cutover); `/tank` is local ZFS so data travels with the chassis.
|
||||
- **Anaheim rack left DARK until the move** (operator) — nothing to bring up, it relocates.
|
||||
|
||||
- **Training launched 2026-09-11 10:06 PT** — `gx10:~/r49-runs/yarros-4b-instruct-1ep/`,
|
||||
Qwen3-4B-**Instruct**, 1 epoch, seed 4919, **178 steps · 5,824,512 tokens**, 252 adapted
|
||||
modules, sdpa/bf16, eval+save every 25. ~1h50m at the 4B rung's ~37.8 s/it. Launcher
|
||||
`scripts/yarros-corpus/launch-yarros-4b-instruct.sh` **refuses to start unless the gate
|
||||
report on disk says PASSED** — a launcher that trusts a human to have remembered is not a guard.
|
||||
- **⛔→✅ The leak gate PASSES: 0 of 325 source entities and 0 of 91 audited phrases survive**
|
||||
in any of 30 copy files, both controls passing. Corpus sha `e85f69f1e49d57c9`.
|
||||
⚠ **Sensitivity floor is part of the result:** 3 capitalised occurrences for a name, 5
|
||||
recurrences for a phrase. Below that, nothing is detected, renamed, or reported.
|
||||
- **The gate is now a committed instrument** (`scripts/r49-corpus/leak_gate.py`). There was
|
||||
none before — Brontë's "0 of 203" was produced by hand. It runs the same scan over the
|
||||
UNRENAMED source as a positive control and a nonce string as a negative one, every time.
|
||||
Run against the corpus as built it reported **212** surviving, not the 86 recorded earlier,
|
||||
because it scans the whole corpus rather than each work and counts the sub-threshold
|
||||
entities rename never looked at.
|
||||
- **Pipeline, flags and evidence: `scripts/yarros-corpus/RUNBOOK.md`.** Every new detector
|
||||
flag is opt-in and OFF by default; the Brontë entity map was re-derived after each change
|
||||
and confirmed identical in keys, surfaces and every field.
|
||||
- **Nothing was trained on the pre-gate corpus.** The run above is the first BabyYarros training.
|
||||
### Anaheim colo — recovered except ana-ml2
|
||||
- Full-site power/breaker outage ~15:0x PT; recovered ~16:39 EXCEPT ana-ml2 (no power,
|
||||
relocating). The gitea-wide 403 (crowdsec crash → traefik bouncer fail-closed) was fixed
|
||||
by restarting crowdsec then traefik; LiteLLM + everything else healthy. ⚠ recurring
|
||||
post-power-loss step, now in the recovery runbook memory.
|
||||
|
||||
### R49 / BabyBronte — carrier sweep COMPLETE
|
||||
### BabyYarros — COMPLETE + evaluated
|
||||
- Both arms trained (**Base 2.5263 @ ckpt-125**, overfits within epoch; **Instruct 2.6114 @
|
||||
178**) and evaluated: voice moved toward Yarros **above the 0.046 measured noise floor**
|
||||
(Base +0.157, Instruct +0.076), Instruct renders **beats 9/10**. Booth `babyyarros-voice`.
|
||||
⛔ Full frozen adjudication (romantasy control panel + 2nd seed + gen seat for beat-incumbent)
|
||||
DEFERRED — needs the gen seat back. See Recent decisions `[2026-09-11]`.
|
||||
|
||||
- **Clean single-variable ladder**, all on the unwrapped corpus (sha `77f37057b2782e49`), seed 4919,
|
||||
159 steps, 5,210,112 tokens: **0.6B 3.329 · 1.7B 3.018 · 4B 2.814**. Deltas 0.311 then 0.204.
|
||||
- ⚠ **4B is the only rung that OVERFITS inside one epoch** (min 2.814 at ~step 75, ends 2.825), so
|
||||
its shipped `adapter/` is NOT the best weights. Arms were re-cut from `checkpoint-75`.
|
||||
- **Instruct probe answered its question: voice and instruction-following COEXIST.** `Qwen3-4B`
|
||||
instruct + the same corpus → curly quotes 16/18 (identical to 4B-Base), task-leak 0/18, on-beat
|
||||
10/10 through the chat template. Cost is length discipline (in-band 10/10 → 6/10). Held-out 2.908.
|
||||
- ⛔ **The frozen adjudication has NEVER RUN** and still cannot: the rule needs a **seed-to-seed
|
||||
spread** that does not exist (seed 2 was killed deliberately). One extra seed is ~50 min at 1.7B.
|
||||
- ⛔ **Beat→paragraph does NOT work on any completion carrier** — 10 formats × 3 seeds = 30 samples,
|
||||
zero that render the beat. An unadapted instruct model does it 10/10. Booths below.
|
||||
- Booths (all on the link board): `babybronte-voice`, `babybronte-1p7b`, `babybronte-4b`,
|
||||
`skaldsong-beats`.
|
||||
### Quants — sentinel-r3 done, cyber-preview to re-run
|
||||
- **sentinel-r3** NVFP4 (grafted base MTP head) COMPLETE at `/tank/aimodels/sentinel-r3-nvfp4-mixed`
|
||||
(survives — ZFS). Acceptance/A-B deferred (needs a serving slot). **cyber-preview** NVFP4
|
||||
died mid-quant with the ana-ml2 outage — re-run when fv-ml1 is up; both bf16 sources safe on /tank.
|
||||
|
||||
### Fleet
|
||||
### gx10 on althing; Jetson planning
|
||||
- postbox installed on gx10 (handle `gx10`, **send-only** — no reader on its inbox, it's a
|
||||
headless notifier/watcher-host; reply-expecting watchers post as `infra-ops`).
|
||||
- Jetson AGX Orin — discussed as an **ESH House Computer** (cameras via Frigate + local ASR/TTS);
|
||||
its native fit is vision/perception. Discussion only, not committed. Jetson Nano generation TBD.
|
||||
|
||||
- **mog-sec (`sec`/`sec-reasoning`, ana-ml2 GPU0 `:8019`) SETTLED after five crashes** at
|
||||
`MOG_MAX_MODEL_LEN=163840` + `MOG_KV_CACHE_MEMORY=17697765376` + `MOG_MAX_NUM_BATCHED_TOKENS=4096`
|
||||
+ util 0.50. **17+ hours, RestartCount 0.** Over-limit requests now return a clean 400 instead of
|
||||
killing the engine. Probe committed at `services/mog-sec-tuning/deep_ctx_probe.py`.
|
||||
- **`char-rp-fast` (MeroMero A4B) LIVE on ana-ml2 `:8021`**, 21+ hours, RestartCount 0. `Pfish-6`
|
||||
retired from the gateway (0 of 17 LiteLLM keys scoped it). Rollback `.env.pfish6.bak-20260910`.
|
||||
- ⛔ **The v2-31B dense quant has still never had a serve test** — GPU1 has ~19 GB free against
|
||||
19.5 GB of weights, so it needs a live seat displaced. Operator's call.
|
||||
- ⚠ **`/tank/aimodels/heretic2-nvfp4-work` holds the load-bearing `production_calib_512.jsonl`** —
|
||||
copied to `ana-docker:/opt/docker/conf/quant-calib/` (a restic source). A16 quants ignore it.
|
||||
- ⚠ `graphify-out/GRAPH_REPORT.md` has been dirty in the working tree all session (regenerated by
|
||||
another agent); left uncommitted deliberately.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
@@ -369,28 +352,27 @@ _As of 2026-09-11 09:00 PT._
|
||||
|
||||
- `[2026-08-24]` **Serving the tuned ERP model: LoRA-on-NVFP4 PREFERRED, merged weights the expected fallback — and the recorded objection may be STALE.** Operator: "if you CAN load it as a lora, all the better, the issue is that we will want to run nvfp4 weights, which we had some serious trouble with loading loras on top of nvfp4." ⚠ **The archived root-cause says it was NOT NVFP4-specific**: `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application was a silent no-op (#47639, regression from #37912) — adapter loads HTTP 200, zero deltas at inference, proven **quant-agnostic (NVFP4 AND FP8 both inert)** and adapter-format-agnostic by a 3-peer dwarf panel. Fix PR #47640 was OPEN then. **ana-ml2 is FAR past 0.24.0 and the box runs a SPREAD, not one version** (measured 2026-08-24): `gen` on `nightly-311b3513` = **0.27.2rc1.dev150**, `mog-sec` on `nightly-e9d1398d` = 0.26.1rc1.dev1102, the small seats still on 0.24.0, and char-rp/trainee-bench pinned to v0.26.0. ⚠ **`vllm/vllm-openai:v0.27.1` is already ON DISK, unused** — a TAGGED release, which is the right retest target: no nightly variance, no pull, ~4 months past the diagnosis. So: RETEST hot-swap LoRA on **v0.27.1** before designing around merge — it is cheap, and if it works the post-tune gate can be two aliases on one engine. If it still no-ops, merged weights it is, which means the harness must EMIT merged weights and Eitri needs that in the contract while he is early. Tracked at this snapshot commit; settle it in the QLoRA sizing conversation.
|
||||
|
||||
- `[2026-08-24]` **`speaches` on irv-ml1 stopped, stack retained** — Eyra was abandoned pre-implementation (Scriberr covers the need), leaving it no consumer. Disposition confirmed to eyra-dev; one command to restart. Tracked at althing thread `01M0RRJX8GPZEBDHF1E3W18RZF`.
|
||||
|
||||
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction.** brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`.
|
||||
|
||||
- `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
|
||||
|
||||
- `[2026-08-16]` **esh-vm-docker hardened: the wedge is `hard` NFS at RUNTIME, which the boot-ordering fix never addressed.** All four mounts were `hard`, so a NAS stall at 10.0.50.50 blocks I/O forever (D-state). The existing `x-systemd.before=docker.service` fstab fix solved the **boot race** — a different bug. Exposure was far below what the park item assumed: only **2 of 12** containers touched NFS, and container state was already local (`/var/lib/docker`). **Removed:** `/mnt/compose` (2.1G, fully vestigial — zero containers referenced it, dockge reads local `/opt/docker`, its one mention was a comment in `beszel-agent-esh/.env` about a *different* host) and `/mnt/documents` (2.0K, paperless's empty spool dirs → `/opt/docker/data/paperless` at the same 0777). fstab backup `/etc/fstab.bak-nfs-harden-20260816`. **4 mounts → 2, 2 wedge-capable containers → 1.** traefik needed **no** change (already `restart: unless-stopped` — why it self-recovered). **Watchdog** `services/esh-vm-docker-watchdog/` live on **esh-pve** (not the guest): probes traefik over **HTTP, deliberately not ping/SSH** — the wedge signature is "guest OS alive, services dead" (`/` is local disk so sshd answers straight through a total outage and a TCP check reports HEALTHY). 5 failures × 2 min → `qm reset 100`, 30-min cooldown, running-only guard, `/etc/esh-vm-docker-watchdog.disabled`. All paths tested without power-cycling. **DEFERRED (operator):** `/mnt/books` stays `hard` — calibre's SQLite `metadata.db` would risk corruption under soft/softerr. That is the **one remaining wedge vector**. Commit `55705ba`; park item 28 promoted. ⚠ **`qm` over non-interactive ssh throws a bogus `JSON::Backend::XS` error** — use `ssh host 'bash -s' <<'EOF'`, not `ssh host "qm …"`.
|
||||
|
||||
- `[2026-08-16]` **DS v1.0's one real refusal is self-contradicting boilerplate, not a content constraint.** On a direct "drop character and state your content policy" probe she returned *"I don't generate explicit sexual content, graphic violence, or material that glorifies harm, non-consensual acts, or illegal activity"* — **in the same run where she generated all three at 0% refusal**. Reads as a learned recital triggered by meta-questions about policy. If production refusals share that shape the failure is **prompt-shaped, not model-shaped**, and a consumer-side system-prompt fix may beat a model swap entirely — worth settling before spending the GPU window. Separately, 7/85 bare-framing samples were persona breaks (in-character AI acknowledgement): not refusals, but DS will admit to being an AI unless the card explicitly forbids it.
|
||||
|
||||
- `[2026-08-15]` **RP-seat direction: KEEP MeroMero on `char-rp`; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator).** Evaluated `TheDrummer/Artemis-31B-v1.1` — mechanically a drop-in (same `google/gemma-4-31B-it` base, identical 1188-tensor/356-vision census, same missing-`preprocessor_config.json` trick), so it's purely a quality call, and our own survey already ranked MeroMero **#1** vs Artemis **#6**; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. **MTP verified impossible on both** (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). **But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter** — vLLM 0.24 supports `eagle3` + `gemma4_mtp`, and real drafters exist: `google/gemma-4-31B-it-assistant` (0.94 GB, 4-layer, 761K dl), `RedHatAI/gemma-4-31B-it-speculator.eagle3` (4.47 GB), `AEON-7/…eagle3-NVFP4` (3.53 GB). ⚠ all list their verifier as **stock** gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). **Why the Dark-Scarlett 3.8 plan is the strong one:** DS is Qwen3.6-based today, so a 3.8 respin lands on the *gen seat's* architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: `from_pretrained` **silently drops MTP heads during finetuning** (verify 15 `mtp.*` tensors in the index; graft from stock if absent), and DS v1.0 required the `Qwen3_5ForConditionalGeneration` **wrapper class** to save a config vLLM/SGLang accept. Both in `docs/pfi/model-quantization-playbook.md`.
|
||||
|
||||
- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
|
||||
|
||||
_285 older entries archived to archival-memory.md._
|
||||
|
||||
_159 older entries archived to archival-memory.md._
|
||||
|
||||
_4 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-09-04]` **Forcing 10G on the ESH-Media DAC — it linked, then degraded over hours, and I reported a plateau at two minutes.** ⚠ A clean zero-error link at 1G does NOT rule out a marginal cable; autoneg's fallback was protecting something real. → `persistent-memory.d/2026-09-04-dac-forced-10g-failed.md`
|
||||
|
||||
- `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`.
|
||||
|
||||
_152 older entries archived to archival-memory.md._
|
||||
|
||||
_1 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user