vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(memory): bank the Parakeet bench result and tts-dev's storage-vs-execution lesson

Browse Source 
The IRV seat was retired on tts-dev's numbers: it lost to the FV seat at both
clip lengths and to whisper-large-v3 at 6.24s. Their length sweep fits ~58ms
fixed + 56ms per audio-second with an asymptote of ~17.8x realtime, which
independently reproduces our 17x on a different clip and harness, and the gateway
hop measured below their harness resolution so ext-stt is the right consumer path.

Two caveats recorded against our own numbers: their between-run variance is 20%
because GPU 0 carries the live chat path, and our 0.50s median came off an idle
GPU 3 - a best case, not a comparable.

Their RTFx retraction is the durable part: published RTFx is batched throughput on
datacenter hardware rather than single-stream latency, and the two differ by ~200x.

Also banks the shape their acceptance gate caught, because it generalises past
their repo. A JS escape inside a Python string arrives transformed, closing the
string and killing the whole inline script, while the page still renders and both
import and node --check pass - the file still holds the backslash. That is the
same failure as reading provider=cuda out of a log: a check that reads the
artifact as stored cannot see a transformation that happens between storage and
execution. Both check the input to a transformation and get reported as if they
checked its output.
This commit is contained in:
Vuong Hoang
2026-09-15 02:17:57 -07:00
parent 868b56642c
commit 406769e64b
1 changed files with 39 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md
+39 -2
View File
@@ -118,9 +118,46 @@ that is where the `ext-tts` family lives, and it needs no gateway restart.
2026-07-22 stack list. Only the parakeet/GPU-3 rows were corrected.
## Two Parakeets, and the bench Vuong asked for
## ✅ The bench, and why the IRV seat was retired
Both are live; endpoints sent to **tts-dev** 2026-09-15 for a head-to-head.
Endpoints sent to **tts-dev** 2026-09-15; **IRV retired the same night on the result.**
**Result** (same clips, same client, same night, vs the Whisper incumbent):
| clip | whisper-large-v3 | IRV v2 / 3090 | FV v3 / Blackwell |
|---|---|---|---|
| 1.84 s | 457 ms | 354 ms | **155 ms** |
| 6.24 s | 690 ms | **1010 ms** | **391 ms** |
IRV lost at both lengths and was *slower than the incumbent* at 6.24 s. Their length
sweep (n=9/cell, first 3 discarded) fits **~58 ms fixed + 56 ms per audio-second**,
asymptote **~17.8x realtime** — independently reproducing our 17x on a different clip
and harness. Gateway hop measured **below their harness resolution** (±30 ms), so
`ext-stt` is the right consumer path rather than a direct port.
⚠ **Their between-run variance is ±20%**, because GPU 0 carries the live chat path.
Our 0.50 s median was taken on an idle GPU 3 — a best case, not a comparable.
⭐ **tts-dev retracted their own plan's 60-120 ms projection**: published RTFx is
**batched throughput on datacenter hardware, not single-stream latency** — the two
differ by **~200x**. Consequence that outlived the win: STT was never the bottleneck
(~217 ms STT / 464 ms LLM / 478 ms TTS at a 3 s utterance).
**Consumer:** `talk`'s push-to-talk ("Grima") went live the same night through
`/api/listen` -> `ext-stt`, 16 kHz mono decimated 3:1 in an AudioWorklet.
⭐ **Their acceptance gate is worth copying.** They drove a real Chromium handed our
known clip as its microphone, through the page's real handlers. It caught a bug every
cheaper check passed: a JS `'didn\'t'` inside a Python string arrives as `'didn't'`,
closing the string and killing the whole inline script — while the page still renders,
`import app` passes and `node --check` passes, because the file still holds the
backslash. **Same shape as the silent-CPU-fallback trap: a check that reads the
artifact AS STORED cannot see a transformation between storage and execution.**
`node --check` reads the pre-Python file; `provider=cuda` in a log echoes configured
intent. Both check the INPUT to a transformation and are reported as if they checked
its output.
## The two seats, for the record
| | FV (new) | IRV (existing, up 2 months) |
|---|---|---|