memory: chatterbox-fast deployed (Phases 1-3) — :8197 A6000, fp32 5.34GB, priming reverted
This commit is contained in:
+12
-2
@@ -95,8 +95,18 @@ _As of 2026-06-02:_
|
||||
`docs/design/chatterbox-fast-plan.md`** (durable; carries the full executable
|
||||
detail). The old `docs/design/chatterbox-fast.md` was KILLED 2026-06-01 — its
|
||||
§5 windowed-token-streaming was the abandoned native arc; superseded by the
|
||||
adaptive-chunk plan. Phase 1 built: `stacks/chatterbox-fast/` (scheduler +
|
||||
app + sim tests).
|
||||
adaptive-chunk plan. **DEPLOYED 2026-06-02** — Phases 1-3 done:
|
||||
`stacks/chatterbox-fast/` (scheduler + app + sim tests + Dockerfile/compose),
|
||||
live on **irv-ml1 :8197** (container `chatterbox-fast`, healthy) BESIDE the live
|
||||
`chatterbox` :8196. Measured: **TTFB ~0.5s** (vs ~5.2s one-shot), no starvation,
|
||||
RTF ~3.9-4.0× on A6000. **GPU = A6000 (device 1) ONLY** — turbo loads **fp32**
|
||||
(NOT the ~2.5GB fp16 old notes claimed), measured **5.34 GB**, so the 3090's
|
||||
~3.8GB free does NOT fit. Perf: TF32/SDPA on (don't move TTFA — AR-decode-bound;
|
||||
bf16 deferred, no clean fp32→bf16 cast path). Context-priming for join coherence
|
||||
TRIED + REVERTED (discard-cut leaks prefix — see Tried/abandoned); coherence loss
|
||||
at joins is an ACCEPTED limitation (operator: cold streaming "really good").
|
||||
Remaining: Phase 4 catalog entry + cutover; optional deploy-playbook smoke gate;
|
||||
optional §5 base-fork 3-way A/B.
|
||||
- **Goal:** cut time-to-first-audio from ~2.5s → sub-second via streaming,
|
||||
keep turbo quality. Workload = single-stream interactive.
|
||||
- **Chosen approach = adaptive buffer-ratchet chunking** (operator's idea):
|
||||
|
||||
Reference in New Issue
Block a user