docs(parakeet): seat A/B vs parakeet-unified-en-0.6b - latency is the int8-on-CPU runtime; unified wins WER
A/B of the live STT seat (fv-ml1 GPU 0, sherpa-onnx int8 v3) against nvidia/parakeet-unified-en-0.6b, measured on GPU 3 with the seat's own image, k2-fsa's published unified int8 export, fp32/fp16 exports made with k2-fsa's recipe, v2 int8, and NeMo 3.0.0 (fp32, bf16 autocast, bf16 weights). - Seat int8 graph runs on one CPU thread (cpu/wall 1.00, GPU 2-9%). - unified-en under NeMo: -121/-234/-530 ms vs the seat at 1-3/3-8/8-20 s (paired, n=120/bin; floor <=6 ms; +50 ms positive control reads +52-54). - unified-en WER lower in every runtime: -0.7 pp clean, -1.5 pp other, -3.2 to -4.4 pp AMI (paired CIs exclude 0). - Seat defects found: hard 400 s input ceiling (HTTP 500), truncation after a quiet 1.5 s pause, and severe long-window dropouts (int8 v3 only). - B-bf16w needs +0.8 to +1.5 GB over the seat's 1,690 MiB on GPU 0. Raw requests, hypotheses, manifests and the full harness under services/parakeet-ab-2026-09-30/. No deploy; live seat untouched apart from 240 light test requests.
This commit is contained in:
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Long-form block: each public file WHOLE, in one request, exactly as the seat would receive it, twice per
|
||||
arm (determinism). Arms run one at a time so each one's long-form memory peak is its own.
|
||||
Writes out/raw/long/<file>--<arm>--<k>.json {arm, file, k, e2e_ms, server_ms, status, text, t0, t1}.
|
||||
usage: run_long.py ARM:PORT ...
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
AB = "/tank/spikes/parakeet-ab"
|
||||
sys.path.insert(0, f"{AB}/code")
|
||||
from bench import post # noqa: E402
|
||||
|
||||
FILES = {"wilde": "/tank/spikes/scriberr-slicer/public/wilde.wav", "scotus": "/tank/spikes/scriberr-slicer/public/scotus.wav"}
|
||||
os.makedirs(f"{AB}/out/raw/long", exist_ok=True)
|
||||
for spec in sys.argv[1:]:
|
||||
arm, port = spec.split(":")
|
||||
for f, path in FILES.items():
|
||||
wav = open(path, "rb").read()
|
||||
for k in (1, 2):
|
||||
t0 = time.time()
|
||||
try:
|
||||
res = post(f"http://127.0.0.1:{port}/v1/audio/transcriptions", wav, timeout=3600)
|
||||
except Exception as e: # noqa: BLE001
|
||||
res = dict(status=-1, err=repr(e)[:300], e2e_ms=None, server_ms=None, text=None)
|
||||
t1 = time.time()
|
||||
json.dump(dict(arm=arm, file=f, k=k, t0=t0, t1=t1, **res), open(f"{AB}/out/raw/long/{f}--{arm}--{k}.json", "w"))
|
||||
print(arm, f, k, res.get("status"), res.get("e2e_ms"), (res.get("err") or "")[:200], flush=True)
|
||||
print("LONG DONE", flush=True)
|
||||
Reference in New Issue
Block a user