docs(parakeet): seat A/B vs parakeet-unified-en-0.6b - latency is the int8-on-CPU runtime; unified wins WER

A/B of the live STT seat (fv-ml1 GPU 0, sherpa-onnx int8 v3) against
nvidia/parakeet-unified-en-0.6b, measured on GPU 3 with the seat's own image,
k2-fsa's published unified int8 export, fp32/fp16 exports made with k2-fsa's
recipe, v2 int8, and NeMo 3.0.0 (fp32, bf16 autocast, bf16 weights).

- Seat int8 graph runs on one CPU thread (cpu/wall 1.00, GPU 2-9%).
- unified-en under NeMo: -121/-234/-530 ms vs the seat at 1-3/3-8/8-20 s
  (paired, n=120/bin; floor <=6 ms; +50 ms positive control reads +52-54).
- unified-en WER lower in every runtime: -0.7 pp clean, -1.5 pp other,
  -3.2 to -4.4 pp AMI (paired CIs exclude 0).
- Seat defects found: hard 400 s input ceiling (HTTP 500), truncation after
  a quiet 1.5 s pause, and severe long-window dropouts (int8 v3 only).
- B-bf16w needs +0.8 to +1.5 GB over the seat's 1,690 MiB on GPU 0.

Raw requests, hypotheses, manifests and the full harness under
services/parakeet-ab-2026-09-30/. No deploy; live seat untouched apart
from 240 light test requests.
This commit is contained in:
vh
2026-09-30 18:51:44 -07:00
parent 11174ffea1
commit a6c1d3c454
64 changed files with 23022 additions and 0 deletions
@@ -0,0 +1,31 @@
#!/usr/bin/env python3
"""Long-form block: each public file WHOLE, in one request, exactly as the seat would receive it, twice per
arm (determinism). Arms run one at a time so each one's long-form memory peak is its own.
Writes out/raw/long/<file>--<arm>--<k>.json {arm, file, k, e2e_ms, server_ms, status, text, t0, t1}.
usage: run_long.py ARM:PORT ...
"""
import json
import os
import sys
import time
AB = "/tank/spikes/parakeet-ab"
sys.path.insert(0, f"{AB}/code")
from bench import post # noqa: E402
FILES = {"wilde": "/tank/spikes/scriberr-slicer/public/wilde.wav", "scotus": "/tank/spikes/scriberr-slicer/public/scotus.wav"}
os.makedirs(f"{AB}/out/raw/long", exist_ok=True)
for spec in sys.argv[1:]:
arm, port = spec.split(":")
for f, path in FILES.items():
wav = open(path, "rb").read()
for k in (1, 2):
t0 = time.time()
try:
res = post(f"http://127.0.0.1:{port}/v1/audio/transcriptions", wav, timeout=3600)
except Exception as e: # noqa: BLE001
res = dict(status=-1, err=repr(e)[:300], e2e_ms=None, server_ms=None, text=None)
t1 = time.time()
json.dump(dict(arm=arm, file=f, k=k, t0=t0, t1=t1, **res), open(f"{AB}/out/raw/long/{f}--{arm}--{k}.json", "w"))
print(arm, f, k, res.get("status"), res.get("e2e_ms"), (res.get("err") or "")[:200], flush=True)
print("LONG DONE", flush=True)