vh
a6c1d3c454
A/B of the live STT seat (fv-ml1 GPU 0, sherpa-onnx int8 v3) against nvidia/parakeet-unified-en-0.6b, measured on GPU 3 with the seat's own image, k2-fsa's published unified int8 export, fp32/fp16 exports made with k2-fsa's recipe, v2 int8, and NeMo 3.0.0 (fp32, bf16 autocast, bf16 weights). - Seat int8 graph runs on one CPU thread (cpu/wall 1.00, GPU 2-9%). - unified-en under NeMo: -121/-234/-530 ms vs the seat at 1-3/3-8/8-20 s (paired, n=120/bin; floor <=6 ms; +50 ms positive control reads +52-54). - unified-en WER lower in every runtime: -0.7 pp clean, -1.5 pp other, -3.2 to -4.4 pp AMI (paired CIs exclude 0). - Seat defects found: hard 400 s input ceiling (HTTP 500), truncation after a quiet 1.5 s pause, and severe long-window dropouts (int8 v3 only). - B-bf16w needs +0.8 to +1.5 GB over the seat's 1,690 MiB on GPU 0. Raw requests, hypotheses, manifests and the full harness under services/parakeet-ab-2026-09-30/. No deploy; live seat untouched apart from 240 light test requests.
50 lines
2.3 KiB
JSON
50 lines
2.3 KiB
JSON
[
|
|
{
|
|
"src": "github:k2-fsa/sherpa-onnx@asr-models/sherpa-onnx-nemo-parakeet-unified-en-0.6b-int8-non-streaming.tar.bz2",
|
|
"sha256": "99f63605b3a85a54c250c0869670a687b7d6598a47bf2421515e1f839a76e150",
|
|
"expected": "99f63605b3a85a54c250c0869670a687b7d6598a47bf2421515e1f839a76e150",
|
|
"ok": true
|
|
},
|
|
{
|
|
"src": "github:k2-fsa/sherpa-onnx@asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2",
|
|
"sha256": "157c157bc51155e03e37d2466522a3a737dd9c72bb25f36eb18912964161e1ad",
|
|
"expected": "157c157bc51155e03e37d2466522a3a737dd9c72bb25f36eb18912964161e1ad",
|
|
"ok": true
|
|
},
|
|
{
|
|
"src": "hf-dataset:openslr/librispeech_asr@71cacbfb7e2354c4226d01e70d77d5fca3d04ba1/all/test.clean/0000.parquet",
|
|
"sha256": "7113aa4c3cf963fb54697145719a7725f984c8836d1c494a554cbb9f1a017df0",
|
|
"expected": "7113aa4c3cf963fb54697145719a7725f984c8836d1c494a554cbb9f1a017df0",
|
|
"ok": true
|
|
},
|
|
{
|
|
"src": "hf-dataset:openslr/librispeech_asr@71cacbfb7e2354c4226d01e70d77d5fca3d04ba1/all/test.other/0000.parquet",
|
|
"sha256": "38e0c86a8104585c577badd707ca4331e20fa2c645f46180af0b7bcdecff9249",
|
|
"expected": "38e0c86a8104585c577badd707ca4331e20fa2c645f46180af0b7bcdecff9249",
|
|
"ok": true
|
|
},
|
|
{
|
|
"src": "hf-dataset:edinburghcstr/ami@46f28f2503e2ec48f8867a84eef356c70476beab/ihm/test-00000-of-00004.parquet",
|
|
"sha256": "d95920dccc6924c15215239461bf5d1152fe07c9c61add073bd12da26dd602e0",
|
|
"expected": "d95920dccc6924c15215239461bf5d1152fe07c9c61add073bd12da26dd602e0",
|
|
"ok": true
|
|
},
|
|
{
|
|
"src": "hf-dataset:edinburghcstr/ami@46f28f2503e2ec48f8867a84eef356c70476beab/ihm/test-00001-of-00004.parquet",
|
|
"sha256": "07a2b1c407cf36bc1e66c1362b0248ed32c897edc2a116549d1dc6b456dc2325",
|
|
"expected": "07a2b1c407cf36bc1e66c1362b0248ed32c897edc2a116549d1dc6b456dc2325",
|
|
"ok": true
|
|
},
|
|
{
|
|
"src": "hf-dataset:edinburghcstr/ami@46f28f2503e2ec48f8867a84eef356c70476beab/ihm/test-00002-of-00004.parquet",
|
|
"sha256": "83cde21e55fc884295585327b0ee334cba47ad53bdb787dd5f97437067aca5f8",
|
|
"expected": "83cde21e55fc884295585327b0ee334cba47ad53bdb787dd5f97437067aca5f8",
|
|
"ok": true
|
|
},
|
|
{
|
|
"src": "hf-dataset:edinburghcstr/ami@46f28f2503e2ec48f8867a84eef356c70476beab/ihm/test-00003-of-00004.parquet",
|
|
"sha256": "3312bb79400ddef62d5a0097e124dfabe86a693b4643e7fef7d72a62588280d7",
|
|
"expected": "3312bb79400ddef62d5a0097e124dfabe86a693b4643e7fef7d72a62588280d7",
|
|
"ok": true
|
|
}
|
|
] |