memory: snapshot for /clear — dots.tts engine arc + LFM2.5 landed; in-flight refreshed to session end-state

This commit is contained in:
vh
2026-08-10 15:07:18 -07:00
parent 077570167f
commit 944c22a95c
+7 -5
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-08-08_
_Last updated: 2026-08-10_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -109,9 +109,11 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-08-08 — long session; all major arcs LANDED (full detail per arc is in Recent decisions). Genuinely in-flight now is thin:_
_As of 2026-08-10 — long session; the dots.tts TTS-engine + LFM2.5 arcs LANDED (full detail per arc in Recent decisions). Genuinely in-flight is thin:_
- **AWAITING peers (non-blocking):** (1) **ratatoskr-dev** to apply their side of the chatterbox-fast fix — revert sampling knobs to default (top_k 80→1000, temp 0.5→0.8) + send full text (server now caps generations at 250 chars, chunks internally); relayed msg `01KZER9X7S`. _(worldtree-dev #400 CLOSED 2026-08-08 — see Recent decisions; snapshot cleared, loop done.)_
- **dots.tts is the new fleet RP-TTS engine — LIVE + consumer-confirmed.** v2 on irv-ml1:8198 (curly-sanitize + sentence-chunking), ratatoskr migrated + verified clean end-to-end; chatterbox-fast RECLAIMED (:8197 stopped, revive `docker start chatterbox-fast`). Canonical voice corpus in `voices/`. ⚠ WATCH: deployed container GPU creeps ~6→14GB/8h (cache accumulation; a redeploy resets it); the **pinned Dockerfile is the only rebuild path** until upstream fixes the phantom `gradio==6.17.0` constraint. Full arc → Recent decisions.
- **LFM2.5-2.6b bake-off alias for brokkr — DELIVERED, awaiting his bench.** `vllm-lfm25` ana-ml2:8021 + LiteLLM `lfm2.5-2.6b` (deepseek_r1 reasoning-parser, vendor sampling); PRODUCTION-CLEARED (operator <USD 10M ruling) but kept OUT of default/fallback routing per brokkr's measurement gate. Peer-side now.
- **Zonos2 PARKED DOWN on the 3090** (irv-ml1; operator ruled "leave down for now" 2026-08-08) — frees ~17 GB for chatterbox-fast; asset-engine + gateway-chat TTS idle meanwhile. **Manual restore command saved** in the Zonos-down Recent-decisions entry + auto-memory `reference_zonos_tts_stack` (detached native proc; kill the multiprocessing-fork CHILDREN, not just the parent).
@@ -119,13 +121,13 @@ _As of 2026-08-08 — long session; all major arcs LANDED (full detail per arc i
- **EVIDENCE HOLD (partial):** WT #394 index-row half LIFTED + swept (kb 8,230→2,876); the **FILE half STILL STANDS** — do NOT delete the on-disk generation dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`). Also un-actioned worldtree-side: the `reconcile --repair` rc=1 git-staging bug (phantom `docs/specs/saga-taxonomy-redesign-plan.md`) — flagged, their code.
- **OPEN LOOPS (2026-08-10):** (1) **dots-tts ratatoskr cutover — COMPLETE.** Operator ear-confirmed on the live RP surface (:8765); **dots-tts v0.22.2 is ratatoskr's PRIMARY RP voice** (tts.py seam → OpenAI `/v1/audio/speech`, SR→48k, default voice→glados, donut carries; 520 tests green; their one-synth-per-turn lock DEC-5 matches the server's serialized model). ratatoskr RELEASED chatterbox-fast :8197. **chatterbox-fast RECLAIMED (operator go 2026-08-10):** `docker compose stop` on irv-ml1 freed ~7GB on the 3090 (GPU0 16.0→9.1GB); container Exited(0) — `restart: unless-stopped` honors the manual stop so it stays down across reboots; revive with `docker start chatterbox-fast`. dots-tts unaffected. **dots.tts migration FULLY CLOSED.** thread `01KZNZF1…`. (2) **LFM2.5-2.6b bake-off for brokkr — DONE:** `vllm-lfm25` LIVE on ana-ml2 GPU1 :8021 (LiquidAI/LFM2.5-2.6B, Lfm2ForCausalLM/vLLM 0.24.0, BF16, util 0.09 ~8.7GB into GPU1's unreserved slack — no prod reservation touched; max-len 16384; NO `--reasoning-parser` so content non-empty). LiteLLM alias `lfm2.5-2.6b` (vendor sampling temp0.1 + topk50/reppen1.1 via `extra_body` so drop_params doesn't strip them). **PRODUCTION-CLEARED 2026-08-10** (operator ruled they're <USD 10M → LFM Open License permits commercial use; eval-only hold dropped) — but STILL kept OUT of any default/fallback routing chain per brokkr's *measurement* gate (nothing's benched it; lifts when the bake-off says). **`--reasoning-parser deepseek_r1` ADDED** (LFM2.5 is `</think>`-delimited: opening `<think>` injected by chat template into the prompt, output = `reasoning…</think>answer`; deepseek_r1 splits on `</think>`). Re-smoke: content=valid JSON (51 chars, `json.loads` passes) + reasoning_content=824 chars → both halves of brokkr's gate met (scoreable + non-empty). ⚠ model wraps its answer in a ```json fence — a bare `json.loads(content)` chokes; scorer must strip the fence. brokkr notified + clear to run (`01KZP142…`). Backups: litellm `config.yaml.bak-pre-lfm25-20260810`; targeted `up -d vllm-lfm25` (shared-.env → never bare up-d). Repo: `stacks/vllm/` + `stacks/litellm/conf/config.yaml`.
- _(dots.tts migration + LFM2.5 stand-up: both fully landed this session — detail in Recent decisions `[2026-08-09→10]` / `[2026-08-10]`; summarized in the two lead bullets above. Rollbacks: dots `sed DOTS_TAG=v1 + up -d`; LFM2.5 backup `config.yaml.bak-pre-lfm25-20260810`.)_
- **OPEN FOLLOW-UPS:** chatterbox-fast **deployed flat-build-context vs package-repo divergence** (reconcile so a repo rebuild matches deploy — repo commit `6bc7bf0` has cap+norm_loudness in package layout; deployed is flat cap-only). Standing/parked: CI-flip runner-auth research, gitea `REQUIRE_SIGNIN_VIEW=false`, #363 research-wing ingest (no deadline), zonos-gateway CI-wire.
**althing monitor** ARMED (handle `infra-ops`, task `b1uat4njl`; herald up). ⚠️ Re-arm ONLY after a real FIRE (`<task-notification> completed rc0`), NEVER after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, **NEVER chained with `&`/`&&`** (orphans it to init — bit me this session; reclaim with `althing-cli stop-monitor` then re-arm cleanly).
**eshpfi push state:** `origin/main` behind by ~13 commits (prior 7 + this session's 6 memory commits: 7ea8dd3/9147bc9/2e0bb85/2d7eb90/6332f14/6399a5a + this snapshot). **Push is the operator's call.** `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's).
**eshpfi push state:** `origin/main` well behind — this session added ~15 commits (the whole dots.tts engine arc `fca1a54`→`0775701`: voices corpus, dots-tts stack, LFM2.5 vllm+litellm, v2 fix, + memory snapshots) on top of prior unpushed work. **Push is the operator's call.** `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's).
## Recent decisions