vh
a6c1d3c454
A/B of the live STT seat (fv-ml1 GPU 0, sherpa-onnx int8 v3) against nvidia/parakeet-unified-en-0.6b, measured on GPU 3 with the seat's own image, k2-fsa's published unified int8 export, fp32/fp16 exports made with k2-fsa's recipe, v2 int8, and NeMo 3.0.0 (fp32, bf16 autocast, bf16 weights). - Seat int8 graph runs on one CPU thread (cpu/wall 1.00, GPU 2-9%). - unified-en under NeMo: -121/-234/-530 ms vs the seat at 1-3/3-8/8-20 s (paired, n=120/bin; floor <=6 ms; +50 ms positive control reads +52-54). - unified-en WER lower in every runtime: -0.7 pp clean, -1.5 pp other, -3.2 to -4.4 pp AMI (paired CIs exclude 0). - Seat defects found: hard 400 s input ceiling (HTTP 500), truncation after a quiet 1.5 s pause, and severe long-window dropouts (int8 v3 only). - B-bf16w needs +0.8 to +1.5 GB over the seat's 1,690 MiB on GPU 0. Raw requests, hypotheses, manifests and the full harness under services/parakeet-ab-2026-09-30/. No deploy; live seat untouched apart from 240 light test requests.
31 lines
2.0 KiB
Plaintext
31 lines
2.0 KiB
Plaintext
A-live b1_3 n= 40 e2e p50 186.9 p90 221.1 max 232.6
|
|
A-live b3_8 n= 40 e2e p50 307.5 p90 412.5 max 454.0
|
|
A-live b8_20 n= 40 e2e p50 625.7 p90 861.8 max 955.6
|
|
A-gateway b1_3 n= 20 e2e p50 256.8 p90 296.6 max 309.0
|
|
A-gateway b3_8 n= 20 e2e p50 388.0 p90 494.3 max 511.0
|
|
A-gateway b8_20 n= 20 e2e p50 711.3 p90 967.7 max 1094.8
|
|
A-nh3direct b1_3 n= 20 e2e p50 210.8 p90 231.0 max 245.3
|
|
A-nh3direct b3_8 n= 20 e2e p50 333.4 p90 445.4 max 452.4
|
|
A-nh3direct b8_20 n= 20 e2e p50 649.2 p90 880.7 max 974.8
|
|
paired: median diff [95% CI]
|
|
A-live - ab-a1 b1_3 pairs= 40 +36.6 [ +34.2, +37.7]
|
|
A-live - ab-a1 b3_8 pairs= 40 +39.6 [ +38.2, +40.7]
|
|
A-live - ab-a1 b8_20 pairs= 40 +42.2 [ +35.5, +47.2]
|
|
A-gateway - A-live b1_3 pairs= 20 +56.1 [ +49.6, +75.8]
|
|
A-gateway - A-live b3_8 pairs= 20 +76.1 [ +59.1, +79.1]
|
|
A-gateway - A-live b8_20 pairs= 20 +86.0 [ +78.6, +97.2]
|
|
A-nh3direct - A-live b1_3 pairs= 20 +24.0 [ +20.6, +26.5]
|
|
A-nh3direct - A-live b3_8 pairs= 20 +29.4 [ +24.0, +33.0]
|
|
A-nh3direct - A-live b8_20 pairs= 20 +26.7 [ +14.7, +34.4]
|
|
A-gateway - A-nh3direct b1_3 pairs= 20 +36.1 [ +29.1, +60.5]
|
|
A-gateway - A-nh3direct b3_8 pairs= 20 +46.3 [ +38.8, +52.9]
|
|
A-gateway - A-nh3direct b8_20 pairs= 20 +61.2 [ +54.1, +64.3]
|
|
A-gateway - ab-b32 b1_3 pairs= 20 +233.6 [ +214.5, +247.4]
|
|
A-gateway - ab-b32 b3_8 pairs= 20 +360.9 [ +321.0, +399.6]
|
|
A-gateway - ab-b32 b8_20 pairs= 20 +674.5 [ +595.8, +773.3]
|
|
A-live - ab-b32 b1_3 pairs= 40 +163.4 [ +154.9, +174.2]
|
|
A-live - ab-b32 b3_8 pairs= 40 +280.5 [ +250.0, +301.0]
|
|
A-live - ab-b32 b8_20 pairs= 40 +589.0 [ +515.7, +642.3]
|
|
A-live vs ab-a1 identical text: 120 / 120
|
|
A-gateway vs A-live identical text: 60 / 60
|