vh
a6c1d3c454
A/B of the live STT seat (fv-ml1 GPU 0, sherpa-onnx int8 v3) against nvidia/parakeet-unified-en-0.6b, measured on GPU 3 with the seat's own image, k2-fsa's published unified int8 export, fp32/fp16 exports made with k2-fsa's recipe, v2 int8, and NeMo 3.0.0 (fp32, bf16 autocast, bf16 weights). - Seat int8 graph runs on one CPU thread (cpu/wall 1.00, GPU 2-9%). - unified-en under NeMo: -121/-234/-530 ms vs the seat at 1-3/3-8/8-20 s (paired, n=120/bin; floor <=6 ms; +50 ms positive control reads +52-54). - unified-en WER lower in every runtime: -0.7 pp clean, -1.5 pp other, -3.2 to -4.4 pp AMI (paired CIs exclude 0). - Seat defects found: hard 400 s input ceiling (HTTP 500), truncation after a quiet 1.5 s pause, and severe long-window dropouts (int8 v3 only). - B-bf16w needs +0.8 to +1.5 GB over the seat's 1,690 MiB on GPU 0. Raw requests, hypotheses, manifests and the full harness under services/parakeet-ab-2026-09-30/. No deploy; live seat untouched apart from 240 light test requests.
25 lines
3.8 KiB
JSON
25 lines
3.8 KiB
JSON
{"arm": "ab-b32", "id": "b1_3_10", "bin": "b1_3", "dur": 2.555, "t_wall": 1790816221.013521, "e2e_ms": 59.851, "server_ms": 54.094, "status": 200, "err": null}
|
|
{"arm": "ab-b32", "id": "b3_8_10", "bin": "b3_8", "dur": 5.05, "t_wall": 1790816221.0585623, "e2e_ms": 43.834, "server_ms": 41.805, "status": 200, "err": null}
|
|
{"arm": "ab-b32", "id": "b8_20_10", "bin": "b8_20", "dur": 11.64, "t_wall": 1790816221.0981514, "e2e_ms": 37.795, "server_ms": 35.555, "status": 200, "err": null}
|
|
{"arm": "ab-b32", "id": "b20_60_10", "bin": "b20_60", "dur": 54.9, "t_wall": 1790816221.522522, "e2e_ms": 418.164, "server_ms": 411.974, "status": 200, "err": null}
|
|
{"arm": "ab-b32c", "id": "b1_3_10", "bin": "b1_3", "dur": 2.555, "t_wall": 1790816241.7849388, "e2e_ms": 59.458, "server_ms": 55.307, "status": 200, "err": null}
|
|
{"arm": "ab-b32c", "id": "b3_8_10", "bin": "b3_8", "dur": 5.05, "t_wall": 1790816241.8273792, "e2e_ms": 41.581, "server_ms": 39.885, "status": 200, "err": null}
|
|
{"arm": "ab-b32c", "id": "b8_20_10", "bin": "b8_20", "dur": 11.64, "t_wall": 1790816241.8661036, "e2e_ms": 37.455, "server_ms": 35.537, "status": 200, "err": null}
|
|
{"arm": "ab-b32c", "id": "b20_60_10", "bin": "b20_60", "dur": 54.9, "t_wall": 1790816242.2739592, "e2e_ms": 402.225, "server_ms": 397.094, "status": 200, "err": null}
|
|
{"arm": "ab-b16w", "id": "b1_3_10", "bin": "b1_3", "dur": 2.555, "t_wall": 1790816263.127731, "e2e_ms": 63.82, "server_ms": 60.048, "status": 200, "err": null}
|
|
{"arm": "ab-b16w", "id": "b3_8_10", "bin": "b3_8", "dur": 5.05, "t_wall": 1790816263.1586285, "e2e_ms": 30.025, "server_ms": 28.023, "status": 200, "err": null}
|
|
{"arm": "ab-b16w", "id": "b8_20_10", "bin": "b8_20", "dur": 11.64, "t_wall": 1790816263.1953354, "e2e_ms": 35.514, "server_ms": 33.704, "status": 200, "err": null}
|
|
{"arm": "ab-b16w", "id": "b20_60_10", "bin": "b20_60", "dur": 54.9, "t_wall": 1790816263.6265936, "e2e_ms": 425.701, "server_ms": 420.002, "status": 200, "err": null}
|
|
{"arm": "ab-b16", "id": "b1_3_10", "bin": "b1_3", "dur": 2.555, "t_wall": 1790816287.585979, "e2e_ms": 66.902, "server_ms": 62.627, "status": 200, "err": null}
|
|
{"arm": "ab-b16", "id": "b3_8_10", "bin": "b3_8", "dur": 5.05, "t_wall": 1790816287.6214647, "e2e_ms": 34.6, "server_ms": 32.956, "status": 200, "err": null}
|
|
{"arm": "ab-b16", "id": "b8_20_10", "bin": "b8_20", "dur": 11.64, "t_wall": 1790816287.6683974, "e2e_ms": 45.725, "server_ms": 43.718, "status": 200, "err": null}
|
|
{"arm": "ab-b16", "id": "b20_60_10", "bin": "b20_60", "dur": 54.9, "t_wall": 1790816288.0451334, "e2e_ms": 371.06, "server_ms": 365.392, "status": 200, "err": null}
|
|
{"arm": "ab-cf32", "id": "b1_3_10", "bin": "b1_3", "dur": 2.555, "t_wall": 1790816351.6597226, "e2e_ms": 82.221, "server_ms": 69.437, "status": 200, "err": null}
|
|
{"arm": "ab-cf32", "id": "b3_8_10", "bin": "b3_8", "dur": 5.05, "t_wall": 1790816351.7125669, "e2e_ms": 52.004, "server_ms": 50.127, "status": 200, "err": null}
|
|
{"arm": "ab-cf32", "id": "b8_20_10", "bin": "b8_20", "dur": 11.64, "t_wall": 1790816351.7899551, "e2e_ms": 76.074, "server_ms": 73.933, "status": 200, "err": null}
|
|
{"arm": "ab-cf32", "id": "b20_60_10", "bin": "b20_60", "dur": 54.9, "t_wall": 1790816352.0501974, "e2e_ms": 254.323, "server_ms": 245.954, "status": 200, "err": null}
|
|
{"arm": "ab-af32", "id": "b1_3_10", "bin": "b1_3", "dur": 2.555, "t_wall": 1790816412.624493, "e2e_ms": 80.051, "server_ms": 75.117, "status": 200, "err": null}
|
|
{"arm": "ab-af32", "id": "b3_8_10", "bin": "b3_8", "dur": 5.05, "t_wall": 1790816412.6803863, "e2e_ms": 55.033, "server_ms": 53.184, "status": 200, "err": null}
|
|
{"arm": "ab-af32", "id": "b8_20_10", "bin": "b8_20", "dur": 11.64, "t_wall": 1790816412.753076, "e2e_ms": 71.369, "server_ms": 68.847, "status": 200, "err": null}
|
|
{"arm": "ab-af32", "id": "b20_60_10", "bin": "b20_60", "dur": 54.9, "t_wall": 1790816412.9848804, "e2e_ms": 226.024, "server_ms": 218.215, "status": 200, "err": null}
|