vh
a6c1d3c454
A/B of the live STT seat (fv-ml1 GPU 0, sherpa-onnx int8 v3) against nvidia/parakeet-unified-en-0.6b, measured on GPU 3 with the seat's own image, k2-fsa's published unified int8 export, fp32/fp16 exports made with k2-fsa's recipe, v2 int8, and NeMo 3.0.0 (fp32, bf16 autocast, bf16 weights). - Seat int8 graph runs on one CPU thread (cpu/wall 1.00, GPU 2-9%). - unified-en under NeMo: -121/-234/-530 ms vs the seat at 1-3/3-8/8-20 s (paired, n=120/bin; floor <=6 ms; +50 ms positive control reads +52-54). - unified-en WER lower in every runtime: -0.7 pp clean, -1.5 pp other, -3.2 to -4.4 pp AMI (paired CIs exclude 0). - Seat defects found: hard 400 s input ceiling (HTTP 500), truncation after a quiet 1.5 s pause, and severe long-window dropouts (int8 v3 only). - B-bf16w needs +0.8 to +1.5 GB over the seat's 1,690 MiB on GPU 0. Raw requests, hypotheses, manifests and the full harness under services/parakeet-ab-2026-09-30/. No deploy; live seat untouched apart from 240 light test requests.
1 line
844 B
JSON
1 line
844 B
JSON
{"ab-b16": [3242482, 3243244, 3243408, 3243410, 3243412, 3243414, 3243416, 3243418, 3243419, 3243422, 3243424, 3243425, 3243427, 3243429, 3243431, 3243434, 3243436, 3243438, 3243440, 3243442, 3243444, 3243446, 3243448, 3243450, 3243452, 3243454, 3243456, 3243458, 3243460, 3243462, 3243464, 3243466, 3243468, 3243470], "ab-b32": [3240981, 3241692, 3241883, 3241890, 3241896, 3241898, 3241901, 3241903, 3241905, 3241907, 3241909, 3241910, 3241913, 3241915, 3241917, 3241919, 3241921, 3241923, 3241925, 3241927, 3241929, 3241931, 3241933, 3241935, 3241937, 3241939, 3241941, 3241943, 3241945, 3241947, 3241949, 3241951, 3241953, 3241955], "ab-af32": [3237908], "ab-cf16": [3234656], "ab-cf32": [3231568], "ab-d": [3228280], "ab-c": [3225020], "ab-at16": [3221843], "ab-apl": [3218510], "ab-a50": [3215242], "ab-a2": [3211852], "ab-a1": [3208587]} |