esh-pfi-infrastructure/docs/pfi/scriberr-slicer-bench-2026-09-30.md

Scriberr Parakeet slicer bench (2026-09-30)

Prime's ruling, 2026-09-30: "build the slicer". This document records how the pause-aware slicer patch (stacks/scriberr/patches/0001-parakeet-pause-aware-slicer.patch) was measured and why the shipped variant was chosen. Metrics only: two of the recordings are Prime's and private, so no transcript text appears here or anywhere in git. Those recordings, and every transcript made from them, stay on fv-ml1 in /tank/spikes/scriberr-slicer/private/ (mode 700).

Recordings (4 files, 118 minutes)

All four were converted to 16 kHz mono WAV with the image's ffmpeg, which is what Scriberr feeds Parakeet.

id	length	what	provenance / licence
p1	35.3 min	Prime's upload, conversational	private
p2	22.3 min	Prime's upload, conversational	private
scotus	30.0 min (first half hour)	U.S. Supreme Court oral argument, Loper Bright Enterprises v. Raimondo, No. 22-451, argued 2024-01-17; spontaneous multi-speaker speech with interruptions	https://www.supremecourt.gov/media/audio/mp3files/22-451.mp3 (sha256 7e704b1f…5dc2); U.S. Government work, public domain (17 U.S.C. §105)
wilde	24.4 min	LibriVox The Trial of Oscar Wilde (dramatic reading), section 1; several readers, courtroom dialogue	https://archive.org/download/trialofoscarwilde_1601_librivox/trialofoscarwilde_01_anon_64kb.mp3 (sha256 492f4c56…3420); public domain (LibriVox, PD Mark 1.0)

Harness

• Image and env: scriberr:local-blackwell (the live image), the live /tank/scriberr/whisperx-env mounted read-only, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, invoked exactly as Scriberr does (uv run --native-tls --project /app/whisperx-env/parakeet python …). It runs as uid 1002 with USER set, not as appuser 10001; that changes file ownership only.
• GPU: transient containers on GPU 3 only (verified: the container sees one card, UUID GPU-186dacf4…). GPU 3 was at 2 MiB before and after.
• Reference: upstream's standard script on the whole file in one pass with local attention (--context-left 255 --context-right 255), which has no cuts. It needs >16 GB, which is why it cannot run on GPU 1.
• Variants run through the real transcribe_buffered(); the only harness change is that the model is loaded once per process instead of once per call. The CLI memory runs below reproduce the harness output byte for byte.
• Placements: every variant at three maximum lengths, 120, 110 and 100 s. Parakeet is deterministic, so a repeat run cannot supply variance; moving the cuts can. Each (file, variant) therefore has n = 3 different cut placements.

Metric

Each variant's words are aligned against the reference, after lowercasing and stripping punctuation (difflib opcodes, then exact Levenshtein inside each non-matching block). Each error gets a time: the hypothesis word's start for a substitution or insertion, the reference word's start for a deletion.

• near-cut / elsewhere word errors (S/I/D): near-cut means within ±3 s of any of that variant's cuts (for overlap variants, the cut is the stitch point at the middle of the overlap, and both chunk edges lie within ±3 s of it).
• dropped / duplicated at cuts: deletions near a cut, and insertions near a cut that repeat an adjacent word.
• error events: errors clustered with gaps ≤ 1 s; rate per minute elsewhere.
• damaged cuts (the decision metric): cuts with at least one error within ±3 s, against damaged phantoms, the same test at points midway between the variant's own cuts (same count, same spacing, as far from any cut as the audio gets). The phantom rate is the background any slicer's cuts are judged against.

Why the decision metric is not raw word counts

The negative control caught it. Word errors elsewhere varied by up to ±50 % between slicers of the same length (p1: 131 to 215). Most of that comes from a few unstable stretches, the same stretches for every variant (p1: 976–992 s, 1291–1298 s, 928–934 s), where the reference and any slicing disagree by 20–60 words depending on context, at arbitrary distances from any cut. That is heavy-tailed noise, not cut damage. Counting events, and asking per cut whether anything near it went wrong, is robust to it; the word counts are still reported.

Variants

name	overlap	pause search	stitch
fixed	0	off	none: upstream's slicer (patched code with both off reproduces it byte for byte)
pause	0	25 s	none needed
overlap	4 s	off	v1: each word kept by the chunk whose cut-to-cut range holds its start time (the brief's rule)
both	4 s	25 s	v1
overlap2	4 s	off	v2: hand over at the nearest word both chunks transcribed alike, within 0.5 s (shipped as v3, identical output)
both2	4 s	25 s	v2
both8	8 s	25 s	v2

v3 is v2 after code review, and it is what ships. Anchors are paired by time first (same text and within 0.5 s, so a longer text match elsewhere in the overlap cannot crowd out the true anchor), punctuation alone never anchors, and the anchor's copy comes from whichever chunk keeps word starts in time order. Re-run on all 12 overlap runs (4 files × 3 placements), v3's output is byte-identical to v2's (and "both3" to "both2"): the reviewer's cases did not occur in this audio, so every v2 number below holds for the shipped code.

v2 exists because v1 failed its own goal. At 26 of 108 "both" stitches the same word sat on both sides of the cut, the two copies' start times 0.00–0.09 s apart (one encoder frame): a word that follows a pause is timestamped anywhere in the pause, and a pause is exactly where a pause-aware cut lands. Start time at the midpoint is the worst possible stitch rule for pause cuts.

Results

Pooled (4 files × 3 placements)

variant	damaged cuts	damaged phantoms	excess	by length 120 / 110 / 100	near-cut words	dropped	duplicated	near-cut events	events elsewhere /min
fixed (upstream)	93/179 = 52 %	34/179 = 19 %	+0.33	0.48 / 0.52 / 0.55	223	56	17	102	2.08
pause	53/201 = 26 %	36/201 = 18 %	+0.085	0.33 / 0.20 / 0.27	113	52	0	59	2.23
overlap (v1)	52/184 = 28 %	33/184 = 18 %	+0.10	0.30 / 0.28 / 0.27	117	25	18	60	2.00
both (v1)	75/207 = 36 %	38/207 = 18 %	+0.18	0.39 / 0.35 / 0.36	126	21	40	87	2.35
overlap2	41/184 = 22 %	33/184 = 18 %	+0.04	0.21 / 0.22 / 0.24	101	24	2	47	2.00
both2	52/207 = 25 %	38/207 = 18 %	+0.07	0.32 / 0.20 / 0.24	89	22	2	54	2.35
both8	48/222 = 22 %	46/222 = 21 %	+0.01	0.23 / 0.23 / 0.19	120	42	0	54	2.18

Per file (summed over the 3 placements)

file	variant	damaged cuts	phantom	near S/I/D	near words	else words/min	drop	dup	events near	events else/min
p1	fixed	27/57	11/57	21/27/9	57	4.16	9	10	29	2.15
p1	pause	19/64	8/64	20/3/25	48	5.62	25	0	20	2.39
p1	overlap	20/59	16/59	16/26/4	46	4.66	4	8	23	2.07
p1	both	29/67	13/67	23/25/4	52	5.70	4	17	35	2.56
p1	overlap2	15/59	16/59	16/19/4	39	4.66	4	0	17	2.07
p1	both2	21/67	13/67	23/8/5	36	5.70	5	0	21	2.56
p1	both8	18/70	15/70	16/2/8	26	6.22	8	0	18	2.40
p2	fixed	14/36	4/36	10/7/14	31	4.45	14	2	16	1.39
p2	pause	10/40	6/40	11/1/1	13	2.71	1	0	10	1.56
p2	overlap	6/36	6/36	4/2/0	6	3.56	0	2	6	1.47
p2	both	16/41	6/41	13/10/0	23	3.11	0	8	18	1.64
p2	overlap2	4/36	6/36	4/0/0	4	3.56	0	0	4	1.47
p2	both2	10/41	6/41	13/2/0	15	3.11	0	0	11	1.64
p2	both8	9/43	7/43	9/1/1	11	3.17	1	0	9	1.58
scotus	fixed	27/47	15/47	14/49/21	84	9.47	21	4	30	3.34
scotus	pause	18/54	17/54	13/9/10	32	8.63	10	0	22	3.36
scotus	overlap	20/49	6/49	16/23/11	50	9.43	11	5	24	3.01
scotus	both	19/55	14/55	9/13/5	27	9.46	5	8	21	3.39
scotus	overlap2	18/49	6/49	15/20/10	45	9.43	10	2	21	3.01
scotus	both2	15/55	14/55	9/7/5	21	9.46	5	2	15	3.39
scotus	both8	14/61	19/61	14/31/10	55	8.19	10	0	19	3.30
wilde	fixed	25/39	4/39	9/30/12	51	6.91	12	1	27	1.03
wilde	pause	6/43	5/43	4/0/16	20	6.95	16	0	7	1.23
wilde	overlap	6/40	5/40	2/3/10	15	6.74	10	3	7	1.13
wilde	both	11/44	5/44	4/8/12	24	9.43	12	7	13	1.43
wilde	overlap2	4/40	5/40	3/0/10	13	6.74	10	0	5	1.13
wilde	both2	6/44	5/44	4/1/12	17	9.43	12	0	7	1.43
wilde	both8	7/48	5/48	5/0/23	28	6.80	23	0	8	1.04

Where the remaining near-cut errors sit

Signed distance from the cut, pooled over files: upstream's errors pile up within ±0.5 s (chopped words; at 100 s, 24 insertions in [−0.5, 0) alone). Pause-only leaves deletions 1–3 s before its cuts, where the left chunk ends (8/9, 3/4, 7/8 in [−3, −1) at the three lengths). overlap2's residue is spread evenly over ±3 s, which is what background looks like. both2 and both8 are clean at the handover but keep some of pause-only's pre-cut deletions.

Controls

• A-vs-A: the same variant twice gives byte-identical words, segments and text (fixed-120, both-120, both2-120 on all 4 files, in one process), and the shipped script run three times as separate CLI processes matches the harness output byte for byte (p1 and scotus). Decoding is deterministic, so run-to-run variance is zero; the variance that matters comes from cut placement.
• Harness validity: patched code with overlap and pause search off equals upstream's unmodified script on all 4 files; v2 without overlap equals v1.
• Positive control: upstream's fixed cutter damages 52 % of its cuts against a 19 % background (+0.33, about 7 standard errors), per file 47–64 % against 11–25 %. The instrument sees cut damage.
• Negative control: the phantom (background) rate is 18–21 % for every variant, and error events away from cuts run at 2.0–2.35 per minute for all of them. Word counts away from cuts do not agree across slicers (see "Parakeet drops stretches" below), which is why they are not the decision metric.
• Sensitivity floor: at ~180–220 cuts per variant the 2-standard-error band on a difference of damaged-cut rates is ±0.08 pooled and ±0.17 for one file. No slicer can be measured below the background (~18–21 %). Word-count differences under ~60 words are unresolvable, because a single dropped stretch (next section) is 10–60 words.

Verdict

Every variant that overlaps with the v2 handover, and pause-only, beats upstream by 0.26–0.30 in damaged-cut rate, far outside the ±0.08 floor. Among them the differences are inside the floor. overlap2 ships: tied best on damaged cuts, fewest near-cut error events (47 vs 102), near-zero duplicates, nothing concentrated at the handover, and the simplest mechanism. Pause search measured neutral once the stitch was fixed, so it stays in the patch as an opt-in --pause-search, off by default.

Memory (GPU 3, production invocation, nvidia-smi every 0.2 s)

run	n	per-process peak
upstream script, 120 s, p1	3	5,496 / 5,496 / 5,496 MiB (reproduces the 2026-09-30 budget figure)
shipped script (v2), 120 s, p1	3	5,496 / 5,496 / 5,496 MiB
shipped script (v3, final), 120 s, p1	3	5,496 / 5,496 / 5,496 MiB; all 3 CLI outputs byte-identical to the harness
shipped script, 120 s, scotus (public)	3	5,496 / 5,496 / 5,496 MiB
positive control: shipped script, 300 s, scotus	1	7,056 MiB (deterministic; 300 s was n=3 in the earlier table)

Zero spread; the peak is set by the 120 s maximum, and the overlap sits inside it. The scotus file reads the same peak as p1, so the rebuild script uses it as its public memory fixture. A spike shorter than the 0.2 s sample period could be missed.

Parakeet drops stretches of speech (separate finding, not the slicer)

Every chunked variant, upstream's included, sometimes skips a run of ≥10 consecutive words in the middle of a chunk, and which runs it skips changes chaotically with the cut placement. Wilde at 120 s loses nothing, at 110 s 171 words, and overlap2 at 110 s loses one 60-second stretch. Across 4 files × 3 placements:

variant	runs of ≥10 words lost	words lost
fixed (upstream)	15	599
pause	17	511
overlap2	12	656
both2	14	719
both8	12	560

The whole-file local-attention reference does the same: runs where every slicer has words the reference lacks (13–16 per 12 comparisons, 340–400 words). Today's production setting (upstream, 120 s) lost 85 words in 2 runs on p2. The slicer neither causes nor cures it; it needs its own investigation (decoder settings, chunk length, or model), which was out of scope here.