vh
a6c1d3c454
A/B of the live STT seat (fv-ml1 GPU 0, sherpa-onnx int8 v3) against nvidia/parakeet-unified-en-0.6b, measured on GPU 3 with the seat's own image, k2-fsa's published unified int8 export, fp32/fp16 exports made with k2-fsa's recipe, v2 int8, and NeMo 3.0.0 (fp32, bf16 autocast, bf16 weights). - Seat int8 graph runs on one CPU thread (cpu/wall 1.00, GPU 2-9%). - unified-en under NeMo: -121/-234/-530 ms vs the seat at 1-3/3-8/8-20 s (paired, n=120/bin; floor <=6 ms; +50 ms positive control reads +52-54). - unified-en WER lower in every runtime: -0.7 pp clean, -1.5 pp other, -3.2 to -4.4 pp AMI (paired CIs exclude 0). - Seat defects found: hard 400 s input ceiling (HTTP 500), truncation after a quiet 1.5 s pause, and severe long-window dropouts (int8 v3 only). - B-bf16w needs +0.8 to +1.5 GB over the seat's 1,690 MiB on GPU 0. Raw requests, hypotheses, manifests and the full harness under services/parakeet-ab-2026-09-30/. No deploy; live seat untouched apart from 240 light test requests.
15 lines
3.4 KiB
Plaintext
15 lines
3.4 KiB
Plaintext
{"arm": "ab-a1", "file": "wilde", "windows": 4, "failed": 0, "wer": 48.72, "ref_words": 3580, "dropouts": 8, "dropout_words": 1676, "clean_dropouts": 8, "clean_dropout_words": 1676, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 17935.655}
|
|
{"arm": "ab-a1", "file": "scotus", "windows": 5, "failed": 0, "wer": 9.44, "ref_words": 5772, "dropouts": 12, "dropout_words": 247, "clean_dropouts": 8, "clean_dropout_words": 196, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 17526.651}
|
|
{"arm": "ab-af32", "file": "wilde", "windows": 4, "failed": 0, "wer": 3.99, "ref_words": 3580, "dropouts": 2, "dropout_words": 57, "clean_dropouts": 2, "clean_dropout_words": 57, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 3295.982}
|
|
{"arm": "ab-af32", "file": "scotus", "windows": 5, "failed": 0, "wer": 4.42, "ref_words": 5772, "dropouts": 4, "dropout_words": 53, "clean_dropouts": 0, "clean_dropout_words": 0, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 2525.397}
|
|
{"arm": "ab-b16w", "file": "wilde", "windows": 4, "failed": 0, "wer": 2.29, "ref_words": 3580, "dropouts": 0, "dropout_words": 0, "clean_dropouts": 0, "clean_dropout_words": 0, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 1671.85}
|
|
{"arm": "ab-b16w", "file": "scotus", "windows": 5, "failed": 0, "wer": 5.02, "ref_words": 5772, "dropouts": 4, "dropout_words": 50, "clean_dropouts": 1, "clean_dropout_words": 13, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 1305.858}
|
|
{"arm": "ab-b32", "file": "wilde", "windows": 4, "failed": 0, "wer": 2.29, "ref_words": 3580, "dropouts": 0, "dropout_words": 0, "clean_dropouts": 0, "clean_dropout_words": 0, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 2452.558}
|
|
{"arm": "ab-b32", "file": "scotus", "windows": 5, "failed": 0, "wer": 5.02, "ref_words": 5772, "dropouts": 4, "dropout_words": 50, "clean_dropouts": 1, "clean_dropout_words": 13, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 1635.995}
|
|
{"arm": "ab-c", "file": "wilde", "windows": 4, "failed": 0, "wer": 2.49, "ref_words": 3580, "dropouts": 0, "dropout_words": 0, "clean_dropouts": 0, "clean_dropout_words": 0, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 18471.986}
|
|
{"arm": "ab-c", "file": "scotus", "windows": 5, "failed": 0, "wer": 5.02, "ref_words": 5772, "dropouts": 4, "dropout_words": 50, "clean_dropouts": 1, "clean_dropout_words": 13, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 17939.31}
|
|
{"arm": "ab-cf32", "file": "wilde", "windows": 4, "failed": 0, "wer": 2.4, "ref_words": 3580, "dropouts": 0, "dropout_words": 0, "clean_dropouts": 0, "clean_dropout_words": 0, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 5789.12}
|
|
{"arm": "ab-cf32", "file": "scotus", "windows": 5, "failed": 0, "wer": 5.02, "ref_words": 5772, "dropouts": 4, "dropout_words": 50, "clean_dropouts": 1, "clean_dropout_words": 13, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 1978.251}
|
|
{"arm": "ab-d", "file": "wilde", "windows": 4, "failed": 0, "wer": 6.12, "ref_words": 3580, "dropouts": 3, "dropout_words": 133, "clean_dropouts": 3, "clean_dropout_words": 133, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 18578.754}
|
|
{"arm": "ab-d", "file": "scotus", "windows": 5, "failed": 0, "wer": 4.7, "ref_words": 5772, "dropouts": 2, "dropout_words": 20, "clean_dropouts": 0, "clean_dropout_words": 0, "insertion_runs": 0, "insertion_words": 0, "e2e_ms_median": 17848.764}
|