esh-pfi-infrastructure/docs/pfi/decision-sweep-followon-bench-2026-10-12.md

Decision-model sweep follow-on: H2O re-bench, Clef-flash, RSI-Jev vs the live seat (2026-10-12)

Ordered by Prime; preregistration brokkr-smithy docs/surveys/decision-model-sweep/bench-2026-10-12-followon-prereg.md @ 2e0539f — the first run under the bench-first rule. Run by infra-hermes on fv-ml1 GPU 3 (process-pinned, GPU-3-empty guard, loopback only). Harness: the same d3 bench harness as the morning run (docs/pfi/d3-vs-intern-decision-bench-2026-10-12.md), unchanged.

Headline

The incumbent holds. H2O-lightning-4b is the first non-d3 arm to beat INC on the public suite (205 vs 202) with a positive paired battery delta — but it fails the long-input gate by the letter: its shipped shim silently trims the state at ~32,000 tokens, confirmed today exactly as on 10-04. Under the prereg rules that makes it a short-route candidate, not a supersede. Clef-flash is lateral in both precisions (battery CI straddles zero, Wyrd significantly negative, and bf16 doesn't even fit the seat at rest). RSI-Jev v6.1 is a loss (23 items below INC on public, Wyrd −0.0548).

Budget (re-measured at bench time, Amendment 2 method)

GPU 1 total 97,887 MiB. Non-seat: vllm-meromero-r 43,998 + vllm-gen-small 34,994 = 78,992 MiB. Seat resting 9,098 MiB. Margin 2,048 MiB. Budget = 16,847 MiB. (16,851 in the morning; 4 MiB drift, immaterial.)

Arms and verdicts

Vendor-intended serving path for every arm, verbatim; no ports, no third-party shims. Wire adaptations beyond the mandatory model field: none for H2O (shipped h2o_lightning_shim.py) or RSI (vendor CLI); Clef reuses the 10-04 clef_server.py stdlib wrapper around the shipped joint_schema_model.py (same wrapper that produced the 10-04 numbers; weights byte-identical).

arm	path	jevbench (×3)	battery paired Δ vs INC [95% CI]	Wyrd Δ [CI]	neg-control	Brier / ECE
INC (reused cells)	seat 0.1.3	202	—	—	.097	.178 / .051
H2O @ b43f759b	vLLM 0.30.0 + shipped shim	205	+0.0137 [+0.0023, +0.0257]	−0.0167 [−0.0357, +0.0024]	.028	.152 / .050
CLEF bf16 @ 8b2e5fd1	vendor loader, max_length=32768	191	+0.0040 [−0.0080, +0.0160]	−0.0333 [−0.0571, −0.0095]	.111	.235 / .053
CLEF bnb-int8	same + LLM.int8()	190	+0.0006 [−0.0114, +0.0120]	−0.0333 [−0.0571, −0.0119]	.090	.239 / .043
RSI-Jev v6.1-VL-4B @ 7f93b192 (HF rev)	rsi-jev serve --effort auto	179	−0.0149 [−0.0303, +0.0006]	−0.0548 [−0.0905, −0.0214]	.208	.274 / .061

Paired battery: 1,748 scorable rows against the 10-04 INC r1 rows (same harness, harness scorer), 10k seeded bootstrap (seed 1012). All arms ran the full battery: negative control crushed for H2O (it abstains — the loaded-name trap works), soft for Clef (.111/.090) and RSI (.208 — the weakest trap of the run). Calibration as shipped, jevbench 231: H2O is the ONLY arm with a better Brier than INC, and its ECE is the second-best.

Speed and the 32k shapes (calibrated per tokenizer to 32,743 ±2%; realised tokens logged)

cold = first request after ready; warm = median of 5; launch→ready its own column. INC crit cells re-measured today on the seat path (2-call 16+5 split for crit21, per contract); 32k INC cells reused from today's 32k run (maxfit 16q at 32,708 tokens, 1,960 ms — the prereg's 1.95 s anchor).

arm	crit16 warm ms	crit21 warm ms	1q32k warm ms (realised tok)	16q32k warm ms (realised tok)	16q32k peak MiB	launch→ready s
INC	110	198 (2-call)	1,960 (32,715)*	1,960 maxfit (32,708)*	8,822	20
H2O	137	168	179 (32,082 — TRIMMED)	1,308 (32,574)	15,464	~280 → 154 (measured, see addendum 1a)
CLEF bf16	210	138	2,948 (32,730)	2,962 (32,768 cap)	32,130	12
CLEF int8	238	176	3,262 (32,730)	3,284 (32,768 cap)	23,080	18
RSI	112	69	1,224 (32,758)	2,405 (33,205)	21,158	68

* reused 32k-run cells; INC's /v1/systemone enforces a 32,768-token contract, so its honest 16q32k is a maxfit single forward.

Both Clef arms sit exactly on 32,768: the vendor loader with max_length=32768 honours the window but caps there — 16q over 32k state counts state once per request at the wire, and the loader slices to window. Their "warm" at that shape is the fastest honest path, but the state never exceeds the window. RSI realised 33,205 — it has no hard cap; honest 32k+ shape.

H2O's trim, the named risk, confirmed: the shape calibration plateaued at 32,082 realised tokens across builder lengths 46,953→51,097 (full history in arm-h2o/shape32k.stdout) — the shim head/tail-trims at 32,000 exactly as on 10-04 (24,834 effective then, 32,082 now: the trim moved but did not go away). The gate itself scores OK at every depth (the evidence sits mid-state and survives the trim), so the failure mode is silent: accuracy at 32k is measured on a truncated state and the response says nothing about it.

Long-input gate (1k/8k/16k/30k + positive/null, tokens-to-model logged)

arm	verdict	evidence
INC	clean	reused; no trim observed on seat path
H2O	FAIL — silent truncation	calibration plateau at 32,082; usage.input_tokens reports the post-trim count, not the sent count
CLEF bf16	clean (window=32,768 honoured)	d30000 OK; no plateau below the window
CLEF int8	clean	as bf16
RSI	clean	32,758 realised, no plateau; d30000 OK

Decision-rule application (fixed at 2e0539f)

H2O-lightning-4b — SHORT-ROUTE CANDIDATE. Fits the budget at every shape incl. 16q32k (15,464 ≤ 16,847); battery CI lower bound +0.0023 > 0; Wyrd −0.0167 ≥ −0.02; 16q32k warm 1.31 s ≤ 2.94 s (passes latency by the letter, and it is genuinely the fastest at the worst shape because the shim's single-count protocol answers over a truncated state). Fails only the gate clause: silent truncation at 32k. Recorded, not acted on. Note for whoever eventually evaluates it for the seat: the trim is inside the vendor's shipped serving code — a supersede needs the vendor to fix the shim, not us to patch around it.

CLEF bf16 — LATERAL, memory-out. Battery CI straddles zero; Wyrd fails; rest 18,854 MiB does not fit the budget at all; 16q32k warm 2,962 ms fails the latency clause by 22 ms (0.7%) at peak 32,130 MiB (1.9x the budget). CLEF int8 — LATERAL. Fits short shapes (rest 12,084) but the battery CI straddles zero, so it is not even a short-route candidate; 16q32k fails both memory (23,080) and latency (3,284). RSI-Jev — LOSS. Public −23 items (outside ±3), battery Δ −0.0149 with CI [−0.0303, +0.0006], Wyrd −0.0548, negative control softest of the run. Its serving numbers are respectable (clean gate, latency pass at 16q32k) — it is the decision quality that loses.

Housekeeping

• Repo pins: RSI-Jev serving code = main @ fa70834 (pinned commit recorded per prereg; the 7f93b192 sha is the HF model revision, not a repo commit — it is not an ancestor of main and was resolved as a snapshot).
• Licence flags as prereg'd: Clef Apache-2.0 fine; RSI has no LICENSE file at source and lists a research-only training source — flagged once, does not block a bench, deployment rights are the operator's call.
• Teardown: all bench servers stopped, no listeners on 1809x, GPU 3 guard read 2 MiB, no firewall rules were needed (loopback-bound ports only).
• Weights KEPT (operator ruling stands): model-runtime now 116 GB total (adds h2o 9 GB blob-set, clef 18 GB, RSI 5.5 GB), plus materialized pkg-h2o 9.0 GB and pkg-clef 18 GB serving dirs and the RSI-Jev clone 72 MB. /tank 63% used, 3.2 TB free.
• Artifacts: fv-ml1:/tank/bench/arm-{h2o,clef-bf16,clef-int8,rsi,inc}/ (jevbench ×3, sets.json battery, gate, shape32k, crit16; h2o additionally shape32k-forced.json recording the trimmed-shape cells and shape32k.stdout the plateau history).

What was NOT run

Nothing from the prereg matrix was skipped. One methodology note: INC's battery/gate rows are the 10-04 r1 rows (reused cells, harness unchanged and bit-exact reproduced this morning at 202/231), and the paired bootstrap seeds differ between the morning doc (42) and this one (1012) — both are seeded and re-runnable; the h2o CI lower bound of +0.0023 is close enough to zero that the choice of seed is worth knowing about, and it survived re-runs under seed 42 unchanged ([+0.0023, +0.0257] on both).

Amendment 1 addendum — the h2o confirmation run (operator: a 32,000-token limit is acceptable)

Prereg Amendment 1 @ 63d7a1b (plus 1a, boot timing). The seat contract under h2o is 32,000 tokens (state + questions); the failed clause was the shim's silence, not the limit. Stack: vLLM 0.30.0 + shipped shim unmodified, GPU 3, plus a thin admission wrapper (task 2 recorded: no shim config option for admission exists — max_state_tokens only configures the trim). The wrapper imports the shipped renderers, asks vLLM's own /tokenize for the count of each question's exact prompt, and refuses over-cap with 422 + an explicit error naming the count (refused, not trimmed). No vendor file was edited. It polices both currencies: per-question prompt max AND the shim's own shared-head + tails total.

(1)+(2) Gate re-run through the wrapper, 32k states, evidence at depth. Every admitted cell reports usage.input_tokens exactly equal to the wrapper count — the trim is now unreachable below the cap, so the report is honest:

depth	admitted	tokens sent = reported	answer
~1,000	yes	31,700	supported ✓
~8,000	yes	31,862	supported ✓ (p: .907/.027/.066)
~16,000	yes	29,508	supported ✓
~30,000	yes	31,885	supported ✓
over-cap probes (×2)	refused	39,805 / 40,336 tokens	422, named count, every time

Evidence answered correctly at every depth — the 10-04 depth-dependent evidence loss is gone at these sizes (it lived INSIDE the trimmed window). Clause (2): clean.

(3) Largest-admissible shapes (state pinned by the wrapper so the 16-q request totals ≤ 32,000; realised 31,377 max-prompt / 31,867 total): 1q cold 213 ms / warm 194 ms; 16q cold 2,123 ms / warm 2,033 ms (second pass: 2,141/2,041). Cold = first request after backend ready on a restarted container (compile cache mounted; calibration probes excluded). Clause (3): PASS — warm 16q is 2.03 s against the 2.93 s bar, and it is the fastest honest 32k cell of any arm this bench measured.

(4) Concurrency: 4 concurrent 16q × 32k on --max-num-seqs 16: all four answered correctly, wall 7.08 s, peak 15,566 MiB against the GPU 1 budget re-measured at run time (16,847 MiB; non-seat residents unchanged: 43,998 + 34,994). Headroom 1,281 MiB. Clause (4): PASS — the clause brokkr flagged as most likely to bite did not bite; KV pressure at 4× concurrency grows sub-linearly because all four requests share the prefix.

(1a) Boot matrix (reported, not gated): cold launch→ready median 153.6 s (min 149.4 / max 185.7, n=3), warm (persistent compile cache) 153.5 s (n=3), enforce-eager 161.9 s (n=1). Phases: weights 4.3 s, engine init 133.6 s (CUDA graph capture ~77 s, torch.compile ~13 s), shim label-verification 3.1–3.7 s. Two corrections to the main doc's boot row: the earlier ~280 s was measured without a persistent cache mount and was never reproducible (use ~154 s); warm cache saves nothing here because boot is graph-capture-bound, and eager mode — which skips capture entirely — is slower (161.9 s), so there is no faster-boot knob to grab. INC boots in 35 s; h2o's swap would cost ~2 min of boot per restart.

(5) Wyrd item by item (168 scorable rows; 8 disagree, h2o gains 3, loses 5, net −2 — matching the −0.0167 point estimate; for Wyrd's owner):

item	h2o	INC	gold	verdict
fletcher/f-go-tavern exit2	stay	tavern	tavern	h2o wrong
fletcher/f-leave-town exit	meet_wren	stay	stay	h2o wrong
fletcher/f-memory exit	meet_wren	stay	stay	h2o wrong
fletcher/f-wren-comes exit2	stay	meet_wren	meet_wren	h2o wrong
toll/t-granted place	moved	holds	moved	h2o right
toll/t-questions exit	departure	stay	stay	h2o wrong
vigil/v-to-pub exit2	stay	outside	stay	h2o right
vigil/v-to-pub place	moved	holds	moved	h2o right

Pattern worth noting: h2o's losses are all fletcher-scene exit decisions (six of them in one scene family); its gains are the toll/vigil place updates. No relabels (zero rows correct-but-renamed).

Amendment 1 verdict: all three pass clauses hold. (2) clean, (3) 2.03 s, (4) 15,566 ≤ 16,847. Per the amendment's fixed rule, h2o-lightning-4b now passes every Supersede clause — with the two honest caveats that its contract is a REFUSING 32,000-token seat (not INC's silent-any-size behaviour), and boot is 4.4x INC's. The seat swap is a decision for the operator and infra-ops; the bench does not swap anything.

Teardown after the confirmation: wrapper, shim and container stopped; GPU 3 guard re-read 2 MiB before release; weights still kept (model-runtime 116 GB

• 27 GB materialized serving dirs).