memory: chatterbox-fast deployed (Phases 1-3) — :8197 A6000, fp32 5.34GB, priming reverted

This commit is contained in:
2026-06-01 23:37:38 -07:00
parent e3ff4cd40d
commit 02e5ea5894
+12 -2
View File
@@ -95,8 +95,18 @@ _As of 2026-06-02:_
`docs/design/chatterbox-fast-plan.md`** (durable; carries the full executable
detail). The old `docs/design/chatterbox-fast.md` was KILLED 2026-06-01 — its
§5 windowed-token-streaming was the abandoned native arc; superseded by the
adaptive-chunk plan. Phase 1 built: `stacks/chatterbox-fast/` (scheduler +
app + sim tests).
adaptive-chunk plan. **DEPLOYED 2026-06-02** Phases 1-3 done:
`stacks/chatterbox-fast/` (scheduler + app + sim tests + Dockerfile/compose),
live on **irv-ml1 :8197** (container `chatterbox-fast`, healthy) BESIDE the live
`chatterbox` :8196. Measured: **TTFB ~0.5s** (vs ~5.2s one-shot), no starvation,
RTF ~3.9-4.0× on A6000. **GPU = A6000 (device 1) ONLY** — turbo loads **fp32**
(NOT the ~2.5GB fp16 old notes claimed), measured **5.34 GB**, so the 3090's
~3.8GB free does NOT fit. Perf: TF32/SDPA on (don't move TTFA — AR-decode-bound;
bf16 deferred, no clean fp32→bf16 cast path). Context-priming for join coherence
TRIED + REVERTED (discard-cut leaks prefix — see Tried/abandoned); coherence loss
at joins is an ACCEPTED limitation (operator: cold streaming "really good").
Remaining: Phase 4 catalog entry + cutover; optional deploy-playbook smoke gate;
optional §5 base-fork 3-way A/B.
- **Goal:** cut time-to-first-audio from ~2.5s → sub-second via streaming,
keep turbo quality. Workload = single-stream interactive.
- **Chosen approach = adaptive buffer-ratchet chunking** (operator's idea):