esh-pfi-infrastructure/docs/pfi/decision-sweep-followon-bench-2026-10-12.md

Decision-model sweep follow-on: H2O re-bench, Clef-flash, RSI-Jev vs the live seat (2026-10-12)

Ordered by Prime; preregistration brokkr-smithy docs/surveys/decision-model-sweep/bench-2026-10-12-followon-prereg.md @ 2e0539f — the first run under the bench-first rule. Run by infra-hermes on fv-ml1 GPU 3 (process-pinned, GPU-3-empty guard, loopback only). Harness: the same d3 bench harness as the morning run (docs/pfi/d3-vs-intern-decision-bench-2026-10-12.md), unchanged.

Headline

The incumbent holds. H2O-lightning-4b is the first non-d3 arm to beat INC on the public suite (205 vs 202) with a positive paired battery delta — but it fails the long-input gate by the letter: its shipped shim silently trims the state at ~32,000 tokens, confirmed today exactly as on 10-04. Under the prereg rules that makes it a short-route candidate, not a supersede. Clef-flash is lateral in both precisions (battery CI straddles zero, Wyrd significantly negative, and bf16 doesn't even fit the seat at rest). RSI-Jev v6.1 is a loss (23 items below INC on public, Wyrd −0.0548).

Budget (re-measured at bench time, Amendment 2 method)

GPU 1 total 97,887 MiB. Non-seat: vllm-meromero-r 43,998 + vllm-gen-small 34,994 = 78,992 MiB. Seat resting 9,098 MiB. Margin 2,048 MiB. Budget = 16,847 MiB. (16,851 in the morning; 4 MiB drift, immaterial.)

Arms and verdicts

Vendor-intended serving path for every arm, verbatim; no ports, no third-party shims. Wire adaptations beyond the mandatory model field: none for H2O (shipped h2o_lightning_shim.py) or RSI (vendor CLI); Clef reuses the 10-04 clef_server.py stdlib wrapper around the shipped joint_schema_model.py (same wrapper that produced the 10-04 numbers; weights byte-identical).

arm	path	jevbench (×3)	battery paired Δ vs INC [95% CI]	Wyrd Δ [CI]	neg-control	Brier / ECE
INC (reused cells)	seat 0.1.3	202	—	—	.097	.178 / .051
H2O @ b43f759b	vLLM 0.30.0 + shipped shim	205	+0.0137 [+0.0023, +0.0257]	−0.0167 [−0.0357, +0.0024]	.028	.152 / .050
CLEF bf16 @ 8b2e5fd1	vendor loader, max_length=32768	191	+0.0040 [−0.0080, +0.0160]	−0.0333 [−0.0571, −0.0095]	.111	.235 / .053
CLEF bnb-int8	same + LLM.int8()	190	+0.0006 [−0.0114, +0.0120]	−0.0333 [−0.0571, −0.0119]	.090	.239 / .043
RSI-Jev v6.1-VL-4B @ 7f93b192 (HF rev)	rsi-jev serve --effort auto	179	−0.0149 [−0.0303, +0.0006]	−0.0548 [−0.0905, −0.0214]	.208	.274 / .061

Paired battery: 1,748 scorable rows against the 10-04 INC r1 rows (same harness, harness scorer), 10k seeded bootstrap (seed 1012). All arms ran the full battery: negative control crushed for H2O (it abstains — the loaded-name trap works), soft for Clef (.111/.090) and RSI (.208 — the weakest trap of the run). Calibration as shipped, jevbench 231: H2O is the ONLY arm with a better Brier than INC, and its ECE is the second-best.

Speed and the 32k shapes (calibrated per tokenizer to 32,743 ±2%; realised tokens logged)

cold = first request after ready; warm = median of 5; launch→ready its own column. INC crit cells re-measured today on the seat path (2-call 16+5 split for crit21, per contract); 32k INC cells reused from today's 32k run (maxfit 16q at 32,708 tokens, 1,960 ms — the prereg's 1.95 s anchor).

arm	crit16 warm ms	crit21 warm ms	1q32k warm ms (realised tok)	16q32k warm ms (realised tok)	16q32k peak MiB	launch→ready s
INC	110	198 (2-call)	1,960 (32,715)*	1,960 maxfit (32,708)*	8,822	20
H2O	137	168	179 (32,082 — TRIMMED)	1,308 (32,574)	15,464	~280
CLEF bf16	210	138	2,948 (32,730)	2,962 (32,768 cap)	32,130	12
CLEF int8	238	176	3,262 (32,730)	3,284 (32,768 cap)	23,080	18
RSI	112	69	1,224 (32,758)	2,405 (33,205)	21,158	68

* reused 32k-run cells; INC's /v1/systemone enforces a 32,768-token contract, so its honest 16q32k is a maxfit single forward.

Both Clef arms sit exactly on 32,768: the vendor loader with max_length=32768 honours the window but caps there — 16q over 32k state counts state once per request at the wire, and the loader slices to window. Their "warm" at that shape is the fastest honest path, but the state never exceeds the window. RSI realised 33,205 — it has no hard cap; honest 32k+ shape.

H2O's trim, the named risk, confirmed: the shape calibration plateaued at 32,082 realised tokens across builder lengths 46,953→51,097 (full history in arm-h2o/shape32k.stdout) — the shim head/tail-trims at 32,000 exactly as on 10-04 (24,834 effective then, 32,082 now: the trim moved but did not go away). The gate itself scores OK at every depth (the evidence sits mid-state and survives the trim), so the failure mode is silent: accuracy at 32k is measured on a truncated state and the response says nothing about it.

Long-input gate (1k/8k/16k/30k + positive/null, tokens-to-model logged)

arm	verdict	evidence
INC	clean	reused; no trim observed on seat path
H2O	FAIL — silent truncation	calibration plateau at 32,082; usage.input_tokens reports the post-trim count, not the sent count
CLEF bf16	clean (window=32,768 honoured)	d30000 OK; no plateau below the window
CLEF int8	clean	as bf16
RSI	clean	32,758 realised, no plateau; d30000 OK

Decision-rule application (fixed at 2e0539f)

H2O-lightning-4b — SHORT-ROUTE CANDIDATE. Fits the budget at every shape incl. 16q32k (15,464 ≤ 16,847); battery CI lower bound +0.0023 > 0; Wyrd −0.0167 ≥ −0.02; 16q32k warm 1.31 s ≤ 2.94 s (passes latency by the letter, and it is genuinely the fastest at the worst shape because the shim's single-count protocol answers over a truncated state). Fails only the gate clause: silent truncation at 32k. Recorded, not acted on. Note for whoever eventually evaluates it for the seat: the trim is inside the vendor's shipped serving code — a supersede needs the vendor to fix the shim, not us to patch around it.

CLEF bf16 — LATERAL, memory-out. Battery CI straddles zero; Wyrd fails; rest 18,854 MiB does not fit the budget at all; 16q32k warm 2,962 ms fails the latency clause by 22 ms (0.7%) at peak 32,130 MiB (1.9x the budget). CLEF int8 — LATERAL. Fits short shapes (rest 12,084) but the battery CI straddles zero, so it is not even a short-route candidate; 16q32k fails both memory (23,080) and latency (3,284). RSI-Jev — LOSS. Public −23 items (outside ±3), battery Δ −0.0149 with CI [−0.0303, +0.0006], Wyrd −0.0548, negative control softest of the run. Its serving numbers are respectable (clean gate, latency pass at 16q32k) — it is the decision quality that loses.

Housekeeping

• Repo pins: RSI-Jev serving code = main @ fa70834 (pinned commit recorded per prereg; the 7f93b192 sha is the HF model revision, not a repo commit — it is not an ancestor of main and was resolved as a snapshot).
• Licence flags as prereg'd: Clef Apache-2.0 fine; RSI has no LICENSE file at source and lists a research-only training source — flagged once, does not block a bench, deployment rights are the operator's call.
• Teardown: all bench servers stopped, no listeners on 1809x, GPU 3 guard read 2 MiB, no firewall rules were needed (loopback-bound ports only).
• Weights KEPT (operator ruling stands): model-runtime now 116 GB total (adds h2o 9 GB blob-set, clef 18 GB, RSI 5.5 GB), plus materialized pkg-h2o 9.0 GB and pkg-clef 18 GB serving dirs and the RSI-Jev clone 72 MB. /tank 63% used, 3.2 TB free.
• Artifacts: fv-ml1:/tank/bench/arm-{h2o,clef-bf16,clef-int8,rsi,inc}/ (jevbench ×3, sets.json battery, gate, shape32k, crit16; h2o additionally shape32k-forced.json recording the trimmed-shape cells and shape32k.stdout the plateau history).

What was NOT run

Nothing from the prereg matrix was skipped. One methodology note: INC's battery/gate rows are the 10-04 r1 rows (reused cells, harness unchanged and bit-exact reproduced this morning at 202/231), and the paired bootstrap seeds differ between the morning doc (42) and this one (1012) — both are seeded and re-runnable; the h2o CI lower bound of +0.0023 is close enough to zero that the choice of seed is worth knowing about, and it survived re-runs under seed 42 unchanged ([+0.0023, +0.0257] on both).