vh
a6c1d3c454
A/B of the live STT seat (fv-ml1 GPU 0, sherpa-onnx int8 v3) against nvidia/parakeet-unified-en-0.6b, measured on GPU 3 with the seat's own image, k2-fsa's published unified int8 export, fp32/fp16 exports made with k2-fsa's recipe, v2 int8, and NeMo 3.0.0 (fp32, bf16 autocast, bf16 weights). - Seat int8 graph runs on one CPU thread (cpu/wall 1.00, GPU 2-9%). - unified-en under NeMo: -121/-234/-530 ms vs the seat at 1-3/3-8/8-20 s (paired, n=120/bin; floor <=6 ms; +50 ms positive control reads +52-54). - unified-en WER lower in every runtime: -0.7 pp clean, -1.5 pp other, -3.2 to -4.4 pp AMI (paired CIs exclude 0). - Seat defects found: hard 400 s input ceiling (HTTP 500), truncation after a quiet 1.5 s pause, and severe long-window dropouts (int8 v3 only). - B-bf16w needs +0.8 to +1.5 GB over the seat's 1,690 MiB on GPU 0. Raw requests, hypotheses, manifests and the full harness under services/parakeet-ab-2026-09-30/. No deploy; live seat untouched apart from 240 light test requests.
108 lines
9.5 KiB
Plaintext
108 lines
9.5 KiB
Plaintext
## single stream (ms)
|
|
ab-af32 b1_3 n=120 e2e p50 39.3 [38.2,40.3] p90 42.1 p99 62.1 max 71.4 | server p50 37.7 p90 40.3 p99 59.0 | http overhead p50 1.6
|
|
ab-af32 b3_8 n=120 e2e p50 47.8 [47.1,49.0] p90 54.5 p99 59.5 max 72.8 | server p50 46.2 p90 52.4 p99 57.5 | http overhead p50 1.7
|
|
ab-af32 b8_20 n=120 e2e p50 68.3 [66.7,71.9] p90 81.4 p99 87.2 max 88.5 | server p50 66.2 p90 79.2 p99 84.9 | http overhead p50 2.1
|
|
ab-af32 b20_60 n=120 e2e p50 170.0 [160.5,177.9] p90 223.8 p99 239.2 max 242.1 | server p50 165.0 p90 217.6 p99 233.0 | http overhead p50 5.1
|
|
ab-b16 b1_3 n=120 e2e p50 28.5 [28.2,28.8] p90 31.4 p99 58.3 max 70.4 | server p50 27.2 p90 30.0 p99 55.1 | http overhead p50 1.3
|
|
ab-b16 b3_8 n=120 e2e p50 32.2 [31.8,33.0] p90 35.5 p99 40.6 max 64.7 | server p50 30.8 p90 34.0 p99 38.9 | http overhead p50 1.4
|
|
ab-b16 b8_20 n=120 e2e p50 38.4 [37.9,39.2] p90 43.2 p99 45.8 max 71.2 | server p50 36.6 p90 41.2 p99 44.1 | http overhead p50 1.7
|
|
ab-b16 b20_60 n=120 e2e p50 65.7 [62.9,70.3] p90 82.1 p99 302.1 max 454.3 | server p50 61.9 p90 76.7 p99 91.8 | http overhead p50 3.8
|
|
ab-b16w b1_3 n=120 e2e p50 23.0 [22.7,23.2] p90 26.1 p99 50.9 max 60.4 | server p50 21.8 p90 24.9 p99 47.5 | http overhead p50 1.3
|
|
ab-b16w b3_8 n=120 e2e p50 27.0 [26.6,27.9] p90 30.3 p99 43.2 max 63.7 | server p50 25.7 p90 29.0 p99 41.1 | http overhead p50 1.4
|
|
ab-b16w b8_20 n=120 e2e p50 33.2 [32.5,34.1] p90 38.2 p99 207.1 max 240.5 | server p50 31.5 p90 36.2 p99 42.9 | http overhead p50 1.6
|
|
ab-b16w b20_60 n=120 e2e p50 61.8 [59.2,68.1] p90 77.4 p99 268.9 max 442.5 | server p50 57.5 p90 72.6 p99 98.4 | http overhead p50 4.4
|
|
ab-b32 b1_3 n=120 e2e p50 22.6 [22.3,23.0] p90 24.3 p99 33.1 max 46.6 | server p50 21.4 p90 23.1 p99 29.9 | http overhead p50 1.2
|
|
ab-b32 b3_8 n=120 e2e p50 26.3 [25.7,26.8] p90 28.8 p99 31.2 max 57.9 | server p50 25.0 p90 27.4 p99 29.8 | http overhead p50 1.3
|
|
ab-b32 b8_20 n=120 e2e p50 34.0 [33.1,34.9] p90 40.1 p99 44.6 max 65.4 | server p50 32.4 p90 38.3 p99 42.6 | http overhead p50 1.6
|
|
ab-b32 b20_60 n=120 e2e p50 76.1 [71.8,85.4] p90 98.3 p99 306.8 max 408.1 | server p50 72.2 p90 93.1 p99 97.4 | http overhead p50 4.0
|
|
ab-b32c b1_3 n=120 e2e p50 22.7 [22.5,23.1] p90 25.0 p99 49.3 max 55.9 | server p50 21.4 p90 23.5 p99 46.0 | http overhead p50 1.3
|
|
ab-b32c b3_8 n=120 e2e p50 26.3 [25.9,26.6] p90 28.9 p99 37.5 max 56.4 | server p50 24.8 p90 27.4 p99 35.8 | http overhead p50 1.4
|
|
ab-b32c b8_20 n=120 e2e p50 33.9 [32.8,34.6] p90 40.0 p99 44.2 max 62.5 | server p50 32.3 p90 38.0 p99 42.4 | http overhead p50 1.7
|
|
ab-b32c b20_60 n=120 e2e p50 77.9 [72.4,85.1] p90 98.7 p99 273.7 max 422.6 | server p50 73.2 p90 93.4 p99 101.1 | http overhead p50 4.4
|
|
ab-cf32 b1_3 n=120 e2e p50 40.6 [39.9,41.6] p90 43.7 p99 63.8 max 73.2 | server p50 39.1 p90 42.1 p99 60.3 | http overhead p50 1.6
|
|
ab-cf32 b3_8 n=120 e2e p50 51.2 [49.9,52.0] p90 59.5 p99 67.5 max 74.0 | server p50 49.5 p90 57.7 p99 65.4 | http overhead p50 1.8
|
|
ab-cf32 b8_20 n=120 e2e p50 75.5 [73.4,80.5] p90 92.0 p99 99.0 max 99.2 | server p50 73.2 p90 89.6 p99 96.7 | http overhead p50 2.1
|
|
ab-cf32 b20_60 n=120 e2e p50 190.2 [177.9,206.0] p90 254.5 p99 264.7 max 266.5 | server p50 185.1 p90 248.5 p99 258.2 | http overhead p50 5.2
|
|
|
|
## paired median difference vs ab-b32 (e2e ms; 95% bootstrap CI)
|
|
ab-af32 b1_3 pairs=120 diff +16.5 [ +15.4, +17.0]
|
|
ab-af32 b3_8 pairs=120 diff +21.6 [ +20.9, +22.3]
|
|
ab-af32 b8_20 pairs=120 diff +35.0 [ +33.4, +36.6]
|
|
ab-af32 b20_60 pairs=120 diff +92.2 [ +83.5, +98.1]
|
|
ab-b16 b1_3 pairs=120 diff +5.8 [ +5.7, +6.0]
|
|
ab-b16 b3_8 pairs=120 diff +6.0 [ +5.7, +6.4]
|
|
ab-b16 b8_20 pairs=120 diff +4.7 [ +4.5, +5.3]
|
|
ab-b16 b20_60 pairs=120 diff -9.7 [ -11.5, -8.1]
|
|
ab-b16w b1_3 pairs=120 diff +0.5 [ +0.4, +0.6]
|
|
ab-b16w b3_8 pairs=120 diff +0.9 [ +0.4, +1.2]
|
|
ab-b16w b8_20 pairs=120 diff -0.3 [ -0.8, +0.3]
|
|
ab-b16w b20_60 pairs=120 diff -13.7 [ -15.2, -12.2]
|
|
ab-b32c b1_3 pairs=120 diff +0.0 [ -0.0, +0.1]
|
|
ab-b32c b3_8 pairs=120 diff +0.1 [ -0.0, +0.1]
|
|
ab-b32c b8_20 pairs=120 diff -0.1 [ -0.1, -0.1]
|
|
ab-b32c b20_60 pairs=120 diff +1.2 [ +0.9, +1.5]
|
|
ab-cf32 b1_3 pairs=120 diff +18.1 [ +17.4, +18.5]
|
|
ab-cf32 b3_8 pairs=120 diff +24.8 [ +23.9, +25.8]
|
|
ab-cf32 b8_20 pairs=120 diff +42.3 [ +40.1, +45.4]
|
|
ab-cf32 b20_60 pairs=120 diff +111.0 [ +104.0, +120.0]
|
|
|
|
## concurrency 4 (ms)
|
|
ab-af32 b1_3 n=40 err=0 e2e p50 144.1 p90 157.4 p99 174.4 | server p50 37.3 | 26.94 req/s, 60.2x realtime
|
|
ab-af32 b20_60 n=40 err=0 e2e p50 684.2 p90 1047.8 p99 1200.8 | server p50 167.9 | 5.64 req/s, 242.3x realtime
|
|
ab-af32 b3_8 n=40 err=0 e2e p50 187.4 p90 226.0 p99 269.7 | server p50 46.2 | 21.13 req/s, 105.2x realtime
|
|
ab-af32 b8_20 n=40 err=0 e2e p50 273.4 p90 292.1 p99 354.1 | server p50 65.7 | 14.56 req/s, 174.6x realtime
|
|
ab-b16 b1_3 n=40 err=0 e2e p50 105.2 p90 107.3 p99 132.3 | server p50 25.7 | 36.53 req/s, 81.7x realtime
|
|
ab-b16 b20_60 n=40 err=0 e2e p50 258.1 p90 393.5 p99 528.4 | server p50 60.9 | 15.13 req/s, 649.5x realtime
|
|
ab-b16 b3_8 n=40 err=0 e2e p50 117.5 p90 145.5 p99 170.1 | server p50 29.3 | 33.28 req/s, 165.6x realtime
|
|
ab-b16 b8_20 n=40 err=0 e2e p50 145.2 p90 153.3 p99 182.5 | server p50 35.5 | 27.29 req/s, 327.1x realtime
|
|
ab-b16w b1_3 n=40 err=0 e2e p50 84.4 p90 86.1 p99 114.2 | server p50 20.6 | 45.25 req/s, 101.2x realtime
|
|
ab-b16w b20_60 n=40 err=0 e2e p50 236.2 p90 370.4 p99 443.2 | server p50 55.3 | 16.52 req/s, 709.0x realtime
|
|
ab-b16w b3_8 n=40 err=0 e2e p50 97.2 p90 122.1 p99 140.4 | server p50 23.9 | 40.32 req/s, 200.7x realtime
|
|
ab-b16w b8_20 n=40 err=0 e2e p50 124.3 p90 130.7 p99 156.6 | server p50 30.5 | 31.8 req/s, 381.2x realtime
|
|
ab-b32 b1_3 n=40 err=0 e2e p50 83.8 p90 88.4 p99 107.7 | server p50 20.4 | 45.61 req/s, 102.0x realtime
|
|
ab-b32 b20_60 n=40 err=0 e2e p50 314.2 p90 455.5 p99 490.8 | server p50 71.8 | 13.02 req/s, 558.8x realtime
|
|
ab-b32 b3_8 n=40 err=0 e2e p50 96.8 p90 118.8 p99 139.3 | server p50 24.1 | 40.53 req/s, 201.7x realtime
|
|
ab-b32 b8_20 n=40 err=0 e2e p50 129.4 p90 147.6 p99 166.6 | server p50 31.1 | 30.46 req/s, 365.2x realtime
|
|
ab-b32c b1_3 n=40 err=0 e2e p50 86.1 p90 88.5 p99 112.7 | server p50 20.9 | 44.44 req/s, 99.4x realtime
|
|
ab-b32c b20_60 n=40 err=0 e2e p50 302.8 p90 453.7 p99 556.9 | server p50 71.0 | 13.11 req/s, 563.0x realtime
|
|
ab-b32c b3_8 n=40 err=0 e2e p50 96.4 p90 116.4 p99 207.8 | server p50 24.3 | 40.36 req/s, 200.9x realtime
|
|
ab-b32c b8_20 n=40 err=0 e2e p50 129.5 p90 148.6 p99 164.9 | server p50 30.9 | 30.7 req/s, 368.1x realtime
|
|
ab-cf32 b1_3 n=40 err=0 e2e p50 148.5 p90 162.5 p99 180.1 | server p50 38.8 | 26.04 req/s, 58.2x realtime
|
|
ab-cf32 b20_60 n=40 err=0 e2e p50 789.2 p90 1167.6 p99 1283.0 | server p50 188.9 | 5.09 req/s, 218.6x realtime
|
|
ab-cf32 b3_8 n=40 err=0 e2e p50 199.3 p90 255.9 p99 284.7 | server p50 48.8 | 19.98 req/s, 99.4x realtime
|
|
ab-cf32 b8_20 n=40 err=0 e2e p50 295.0 p90 329.7 p99 369.9 | server p50 72.3 | 13.26 req/s, 159.0x realtime
|
|
|
|
## first call per bin after cold start (ms) vs that arm's warm e2e p50
|
|
ab-b32 b1_3 first 59.9 warm p50 22.6
|
|
ab-b32 b3_8 first 43.8 warm p50 26.3
|
|
ab-b32 b8_20 first 37.8 warm p50 34.0
|
|
ab-b32 b20_60 first 418.2 warm p50 76.1
|
|
ab-b32c b1_3 first 59.5 warm p50 22.7
|
|
ab-b32c b3_8 first 41.6 warm p50 26.3
|
|
ab-b32c b8_20 first 37.5 warm p50 33.9
|
|
ab-b32c b20_60 first 402.2 warm p50 77.9
|
|
ab-b16w b1_3 first 63.8 warm p50 23.0
|
|
ab-b16w b3_8 first 30.0 warm p50 27.0
|
|
ab-b16w b8_20 first 35.5 warm p50 33.2
|
|
ab-b16w b20_60 first 425.7 warm p50 61.8
|
|
ab-b16 b1_3 first 66.9 warm p50 28.5
|
|
ab-b16 b3_8 first 34.6 warm p50 32.2
|
|
ab-b16 b8_20 first 45.7 warm p50 38.4
|
|
ab-b16 b20_60 first 371.1 warm p50 65.7
|
|
ab-cf32 b1_3 first 82.2 warm p50 40.6
|
|
ab-cf32 b3_8 first 52.0 warm p50 51.2
|
|
ab-cf32 b8_20 first 76.1 warm p50 75.5
|
|
ab-cf32 b20_60 first 254.3 warm p50 190.2
|
|
ab-af32 b1_3 first 80.1 warm p50 39.3
|
|
ab-af32 b3_8 first 55.0 warm p50 47.8
|
|
ab-af32 b8_20 first 71.4 warm p50 68.3
|
|
ab-af32 b20_60 first 226.0 warm p50 170.0
|
|
|
|
## GPU memory per process (MiB)
|
|
{"arm": "ab-b32", "pid": 3296542, "cold_s": 11.8, "rest_after_warmup_mib": 3836, "peak_utterance_block_mib": 4800, "peak_whole_life_mib": 5620, "last_mib": 4800, "samples": 2211}
|
|
{"arm": "ab-b32c", "pid": 3298073, "cold_s": 12.1, "rest_after_warmup_mib": 3836, "peak_utterance_block_mib": 4800, "peak_whole_life_mib": 4800, "last_mib": 4800, "samples": 2108}
|
|
{"arm": "ab-b16w", "pid": 3299497, "cold_s": 13.0, "rest_after_warmup_mib": 2492, "peak_utterance_block_mib": 2794, "peak_whole_life_mib": 3194, "last_mib": 2794, "samples": 2006}
|
|
{"arm": "ab-b16", "pid": 3301181, "cold_s": 11.8, "rest_after_warmup_mib": 4802, "peak_utterance_block_mib": 5126, "peak_whole_life_mib": 5620, "last_mib": 5126, "samples": 1892}
|
|
{"arm": "ab-cf32", "pid": 3302918, "cold_s": 45.2, "rest_after_warmup_mib": 3874, "peak_utterance_block_mib": 6946, "peak_whole_life_mib": 6946, "last_mib": 6946, "samples": 1789}
|
|
{"arm": "ab-af32", "pid": 3306068, "cold_s": 44.7, "rest_after_warmup_mib": 3922, "peak_utterance_block_mib": 6994, "peak_whole_life_mib": 6994, "last_mib": 6994, "samples": 1494}
|