vh
a6c1d3c454
A/B of the live STT seat (fv-ml1 GPU 0, sherpa-onnx int8 v3) against nvidia/parakeet-unified-en-0.6b, measured on GPU 3 with the seat's own image, k2-fsa's published unified int8 export, fp32/fp16 exports made with k2-fsa's recipe, v2 int8, and NeMo 3.0.0 (fp32, bf16 autocast, bf16 weights). - Seat int8 graph runs on one CPU thread (cpu/wall 1.00, GPU 2-9%). - unified-en under NeMo: -121/-234/-530 ms vs the seat at 1-3/3-8/8-20 s (paired, n=120/bin; floor <=6 ms; +50 ms positive control reads +52-54). - unified-en WER lower in every runtime: -0.7 pp clean, -1.5 pp other, -3.2 to -4.4 pp AMI (paired CIs exclude 0). - Seat defects found: hard 400 s input ceiling (HTTP 500), truncation after a quiet 1.5 s pause, and severe long-window dropouts (int8 v3 only). - B-bf16w needs +0.8 to +1.5 GB over the seat's 1,690 MiB on GPU 0. Raw requests, hypotheses, manifests and the full harness under services/parakeet-ab-2026-09-30/. No deploy; live seat untouched apart from 240 light test requests.
1 line
1.4 KiB
JSON
1 line
1.4 KiB
JSON
{"ab-af32": [3306068], "ab-cf32": [3302918], "ab-b16": [3301181, 3301983, 3302150, 3302152, 3302154, 3302156, 3302158, 3302160, 3302162, 3302164, 3302166, 3302168, 3302171, 3302174, 3302176, 3302183, 3302188, 3302192, 3302194, 3302196, 3302198, 3302200, 3302202, 3302204, 3302206, 3302208, 3302210, 3302212, 3302214, 3302216, 3302218, 3302220, 3302222, 3302224], "ab-b16w": [3299497, 3300176, 3300430, 3300432, 3300434, 3300436, 3300438, 3300440, 3300442, 3300444, 3300446, 3300448, 3300450, 3300453, 3300456, 3300458, 3300465, 3300492, 3300493, 3300496, 3300498, 3300501, 3300504, 3300511, 3300513, 3300515, 3300517, 3300519, 3300521, 3300522, 3300525, 3300527, 3300529, 3300531], "ab-b32c": [3298073, 3298755, 3298916, 3298918, 3298920, 3298922, 3298924, 3298926, 3298929, 3298931, 3298934, 3298941, 3298944, 3298947, 3298950, 3298953, 3298955, 3298957, 3298959, 3298960, 3298963, 3298965, 3298967, 3298969, 3298971, 3298973, 3298975, 3298977, 3298979, 3298981, 3298983, 3298985, 3298987, 3298989], "ab-b32": [3296542, 3297309, 3297500, 3297502, 3297504, 3297506, 3297508, 3297510, 3297512, 3297514, 3297516, 3297517, 3297520, 3297522, 3297524, 3297525, 3297528, 3297530, 3297531, 3297534, 3297536, 3297538, 3297540, 3297542, 3297544, 3297546, 3297548, 3297550, 3297551, 3297554, 3297556, 3297558, 3297567, 3297569], "ab-cf16": [3234656], "ab-d": [3228280], "ab-c": [3225020], "ab-at16": [3221843], "ab-a2": [3211852], "ab-a1": [3208587]} |