vh
a6c1d3c454
A/B of the live STT seat (fv-ml1 GPU 0, sherpa-onnx int8 v3) against nvidia/parakeet-unified-en-0.6b, measured on GPU 3 with the seat's own image, k2-fsa's published unified int8 export, fp32/fp16 exports made with k2-fsa's recipe, v2 int8, and NeMo 3.0.0 (fp32, bf16 autocast, bf16 weights). - Seat int8 graph runs on one CPU thread (cpu/wall 1.00, GPU 2-9%). - unified-en under NeMo: -121/-234/-530 ms vs the seat at 1-3/3-8/8-20 s (paired, n=120/bin; floor <=6 ms; +50 ms positive control reads +52-54). - unified-en WER lower in every runtime: -0.7 pp clean, -1.5 pp other, -3.2 to -4.4 pp AMI (paired CIs exclude 0). - Seat defects found: hard 400 s input ceiling (HTTP 500), truncation after a quiet 1.5 s pause, and severe long-window dropouts (int8 v3 only). - B-bf16w needs +0.8 to +1.5 GB over the seat's 1,690 MiB on GPU 0. Raw requests, hypotheses, manifests and the full harness under services/parakeet-ab-2026-09-30/. No deploy; live seat untouched apart from 240 light test requests.
13 lines
2.9 KiB
JSON
13 lines
2.9 KiB
JSON
{"arm": "ab-a1", "desc": "A' #1: seat image, v3 int8, 1 thread", "port": 18301, "args": ["sherpa", "/tank/parakeet/models", "1"], "pid": 3208587, "t_started": 1790811521.2825475, "t_ready": 1790811569.5255, "cold_s": 48.2}
|
|
{"arm": "ab-a2", "desc": "A' #2: identical second instance (floor)", "port": 18302, "args": ["sherpa", "/tank/parakeet/models", "1"], "pid": 3211852, "t_started": 1790811576.796222, "t_ready": 1790811624.5894935, "cold_s": 47.8}
|
|
{"arm": "ab-a50", "desc": "A' +50 ms injected (positive control)", "port": 18303, "args": ["sherpa", "/tank/parakeet/models", "1", "int8.onnx", "50"], "pid": 3215242, "t_started": 1790811639.0406506, "t_ready": 1790811686.5703068, "cold_s": 47.5}
|
|
{"arm": "ab-apl", "desc": "A' with the image's app untouched (null)", "port": 18309, "args": ["sherpa", "/tank/parakeet/models", "1", "int8.onnx", "0", "1"], "pid": 3218510, "t_started": 1790811700.479341, "t_ready": 1790811748.4358485, "cold_s": 48.0}
|
|
{"arm": "ab-at16", "desc": "A' with NUM_THREADS=16", "port": 18326, "args": ["sherpa", "/tank/parakeet/models", "16"], "pid": 3221843, "t_started": 1790811761.932911, "t_ready": 1790811809.6139424, "cold_s": 47.7}
|
|
{"arm": "ab-c", "desc": "C: unified-en int8 (k2-fsa), seat runtime", "port": 18304, "args": ["sherpa", "/tank/spikes/parakeet-ab/models/unified-int8", "1"], "pid": 3225020, "t_started": 1790811814.8874962, "t_ready": 1790811863.2119331, "cold_s": 48.3}
|
|
{"arm": "ab-d", "desc": "D: v2 int8 (k2-fsa), seat runtime", "port": 18305, "args": ["sherpa", "/tank/spikes/parakeet-ab/models/v2-int8", "1"], "pid": 3228280, "t_started": 1790811870.555265, "t_ready": 1790811918.3397377, "cold_s": 47.8}
|
|
{"arm": "ab-cf32", "desc": "C-fp32: unified-en fp32 ONNX (k2-fsa export recipe), seat runtime", "port": 18312, "args": ["sherpa", "/tank/spikes/parakeet-ab/models/unified-f32", "1", "onnx"], "pid": 3231568, "t_started": 1790811925.3162308, "t_ready": 1790811970.5963125, "cold_s": 45.3}
|
|
{"arm": "ab-cf16", "desc": "C-fp16: unified-en fp16 ONNX (pre_encode kept fp32), seat runtime", "port": 18310, "args": ["sherpa", "/tank/spikes/parakeet-ab/models/unified-f16", "1", "fp16.onnx"], "pid": 3234656, "t_started": 1790811976.9534578, "t_ready": 1790812022.51367, "cold_s": 45.6}
|
|
{"arm": "ab-af32", "desc": "A-fp32: the seat's v3 as fp32 ONNX (k2-fsa export recipe), seat runtime", "port": 18311, "args": ["sherpa", "/tank/spikes/parakeet-ab/models/v3-f32", "1", "onnx"], "pid": 3237908, "t_started": 1790812033.2831473, "t_ready": 1790812077.8861537, "cold_s": 44.6}
|
|
{"arm": "ab-b32", "desc": "B-fp32: unified-en, NeMo 3.0.0 torch", "port": 18306, "args": ["nemo", "fp32", "direct", "1"], "pid": 3241955, "t_started": 1790812084.4791186, "t_ready": 1790812096.572915, "cold_s": 12.1}
|
|
{"arm": "ab-b16", "desc": "B-bf16: unified-en, NeMo 3.0.0 torch, autocast bf16", "port": 18307, "args": ["nemo", "bf16", "direct", "1"], "pid": 3243470, "t_started": 1790812110.0783296, "t_ready": 1790812121.9070916, "cold_s": 11.8}
|