esh-pfi-infrastructure/docs/pfi/decision-sweep-followon-bench-2026-10-12.md

# Decision-model sweep follow-on: H2O re-bench, Clef-flash, RSI-Jev vs the live seat (2026-10-12)

Ordered by Prime; preregistration `brokkr-smithy`
`docs/surveys/decision-model-sweep/bench-2026-10-12-followon-prereg.md` @ 2e0539f —
the first run under the bench-first rule. Run by infra-hermes on fv-ml1 GPU 3
(process-pinned, GPU-3-empty guard, loopback only). Harness: the same d3 bench
harness as the morning run
(`docs/pfi/d3-vs-intern-decision-bench-2026-10-12.md`), unchanged.

## Headline

**The incumbent holds.** H2O-lightning-4b is the first non-d3 arm to beat INC
on the public suite (205 vs 202) with a positive paired battery delta — but it
fails the long-input gate by the letter: its shipped shim silently trims the
state at ~32,000 tokens, confirmed today exactly as on 10-04. Under the prereg
rules that makes it a **short-route candidate**, not a supersede. Clef-flash is
**lateral** in both precisions (battery CI straddles zero, Wyrd significantly
negative, and bf16 doesn't even fit the seat at rest). RSI-Jev v6.1 is a
**loss** (23 items below INC on public, Wyrd −0.0548).

## Budget (re-measured at bench time, Amendment 2 method)

GPU 1 total 97,887 MiB. Non-seat: vllm-meromero-r 43,998 + vllm-gen-small
34,994 = 78,992 MiB. Seat resting 9,098 MiB. Margin 2,048 MiB.
**Budget = 16,847 MiB.** (16,851 in the morning; 4 MiB drift, immaterial.)

## Arms and verdicts

Vendor-intended serving path for every arm, verbatim; no ports, no third-party
shims. Wire adaptations beyond the mandatory model field: none for H2O (shipped
`h2o_lightning_shim.py`) or RSI (vendor CLI); Clef reuses the 10-04
`clef_server.py` stdlib wrapper around the shipped `joint_schema_model.py`
(same wrapper that produced the 10-04 numbers; weights byte-identical).

| arm | path | jevbench (×3) | battery paired Δ vs INC [95% CI] | Wyrd Δ [CI] | neg-control | Brier / ECE |
|---|---|---|---|---|---|---|
| **INC (reused cells)** | seat 0.1.3 | **202** | — | — | .097 | .178 / .051 |
| H2O @ b43f759b | vLLM 0.30.0 + shipped shim | **205** | **+0.0137 [+0.0023, +0.0257]** | −0.0167 [−0.0357, +0.0024] | .028 | .152 / .050 |
| CLEF bf16 @ 8b2e5fd1 | vendor loader, max_length=32768 | 191 | +0.0040 [−0.0080, +0.0160] | **−0.0333 [−0.0571, −0.0095]** | .111 | .235 / .053 |
| CLEF bnb-int8 | same + LLM.int8() | 190 | +0.0006 [−0.0114, +0.0120] | **−0.0333 [−0.0571, −0.0119]** | .090 | .239 / .043 |
| RSI-Jev v6.1-VL-4B @ 7f93b192 (HF rev) | `rsi-jev serve --effort auto` | 179 | −0.0149 [−0.0303, +0.0006] | **−0.0548 [−0.0905, −0.0214]** | .208 | .274 / .061 |

Paired battery: 1,748 scorable rows against the 10-04 INC r1 rows (same
harness, harness scorer), 10k seeded bootstrap (seed 1012). All arms ran the
full battery: negative control crushed for H2O (it abstains — the loaded-name
trap works), soft for Clef (.111/.090) and RSI (.208 — the weakest trap of the
run). Calibration as shipped, jevbench 231: H2O is the ONLY arm with a better
Brier than INC, and its ECE is the second-best.

## Speed and the 32k shapes (calibrated per tokenizer to 32,743 ±2%; realised tokens logged)

cold = first request after ready; warm = median of 5; launch→ready its own
column. INC crit cells re-measured today on the seat path (2-call 16+5 split
for crit21, per contract); 32k INC cells reused from today's 32k run (maxfit
16q at 32,708 tokens, 1,960 ms — the prereg's 1.95 s anchor).

| arm | crit16 warm ms | crit21 warm ms | 1q32k warm ms (realised tok) | 16q32k warm ms (realised tok) | 16q32k peak MiB | launch→ready s |
|---|---|---|---|---|---|---|
| INC | 110 | 198 (2-call) | 1,960 (32,715)* | 1,960 maxfit (32,708)* | 8,822 | 20 |
| H2O | 137 | 168 | 179 (32,082 — **TRIMMED**) | 1,308 (32,574) | 15,464 | ~280 → **154** (measured, see addendum 1a) |
| CLEF bf16 | 210 | 138 | 2,948 (32,730) | 2,962 (32,768 cap) | 32,130 | 12 |
| CLEF int8 | 238 | 176 | 3,262 (32,730) | 3,284 (32,768 cap) | 23,080 | 18 |
| RSI | 112 | 69 | 1,224 (32,758) | 2,405 (33,205) | 21,158 | 68 |

\* reused 32k-run cells; INC's `/v1/systemone` enforces a 32,768-token
contract, so its honest 16q32k is a maxfit single forward.

Both Clef arms sit exactly on 32,768: the vendor loader with max_length=32768
honours the window but caps there — 16q over 32k state counts state once per
request at the wire, and the loader slices to window. Their "warm" at that
shape is the fastest honest path, but the state never exceeds the window.
RSI realised 33,205 — it has no hard cap; honest 32k+ shape.

**H2O's trim, the named risk, confirmed:** the shape calibration plateaued at
32,082 realised tokens across builder lengths 46,953→51,097 (full history in
`arm-h2o/shape32k.stdout`) — the shim head/tail-trims at 32,000 exactly as on
10-04 (24,834 effective then, 32,082 now: the trim moved but did not go away).
The gate itself scores OK at every depth (the evidence sits mid-state and
survives the trim), so the failure mode is silent: **accuracy at 32k is
measured on a truncated state and the response says nothing about it.**

## Long-input gate (1k/8k/16k/30k + positive/null, tokens-to-model logged)

| arm | verdict | evidence |
|---|---|---|
| INC | clean | reused; no trim observed on seat path |
| H2O | **FAIL — silent truncation** | calibration plateau at 32,082; `usage.input_tokens` reports the post-trim count, not the sent count |
| CLEF bf16 | clean (window=32,768 honoured) | d30000 OK; no plateau below the window |
| CLEF int8 | clean | as bf16 |
| RSI | clean | 32,758 realised, no plateau; d30000 OK |

## Decision-rule application (fixed at 2e0539f)

**H2O-lightning-4b — SHORT-ROUTE CANDIDATE.** Fits the budget at every shape
incl. 16q32k (15,464 ≤ 16,847); battery CI lower bound +0.0023 > 0; Wyrd
−0.0167 ≥ −0.02; 16q32k warm 1.31 s ≤ 2.94 s (passes latency by the letter,
and it is genuinely the fastest at the worst shape because the shim's
single-count protocol answers over a *truncated* state). Fails only the gate
clause: silent truncation at 32k. Recorded, not acted on. Note for whoever
eventually evaluates it for the seat: the trim is inside the vendor's shipped
serving code — a supersede needs the vendor to fix the shim, not us to patch
around it.

**CLEF bf16 — LATERAL, memory-out.** Battery CI straddles zero; Wyrd fails;
rest 18,854 MiB does not fit the budget at all; 16q32k warm 2,962 ms fails the
latency clause by 22 ms (0.7%) at peak 32,130 MiB (1.9x the budget).
**CLEF int8 — LATERAL.** Fits short shapes (rest 12,084) but the battery CI
straddles zero, so it is not even a short-route candidate; 16q32k fails both
memory (23,080) and latency (3,284).
**RSI-Jev — LOSS.** Public −23 items (outside ±3), battery Δ −0.0149 with CI
[−0.0303, +0.0006], Wyrd −0.0548, negative control softest of the run. Its
serving numbers are respectable (clean gate, latency pass at 16q32k) — it is
the decision quality that loses.

## Housekeeping

- Repo pins: RSI-Jev serving code = main @ fa70834 (pinned commit recorded per
  prereg; the 7f93b192 sha is the **HF model revision**, not a repo commit —
  it is not an ancestor of main and was resolved as a snapshot).
- Licence flags as prereg'd: Clef Apache-2.0 fine; RSI has no LICENSE file at
  source and lists a research-only training source — flagged once, does not
  block a bench, deployment rights are the operator's call.
- Teardown: all bench servers stopped, no listeners on 1809x, GPU 3 guard read
  2 MiB, no firewall rules were needed (loopback-bound ports only).
- Weights KEPT (operator ruling stands): model-runtime now 116 GB total
  (adds h2o 9 GB blob-set, clef 18 GB, RSI 5.5 GB), plus materialized
  `pkg-h2o` 9.0 GB and `pkg-clef` 18 GB serving dirs and the RSI-Jev clone
  72 MB. `/tank` 63% used, 3.2 TB free.
- Artifacts: `fv-ml1:/tank/bench/arm-{h2o,clef-bf16,clef-int8,rsi,inc}/`
  (jevbench ×3, sets.json battery, gate, shape32k, crit16; h2o additionally
  `shape32k-forced.json` recording the trimmed-shape cells and
  `shape32k.stdout` the plateau history).

## What was NOT run

Nothing from the prereg matrix was skipped. One methodology note: INC's
battery/gate rows are the 10-04 r1 rows (reused cells, harness unchanged and
bit-exact reproduced this morning at 202/231), and the paired bootstrap seeds
differ between the morning doc (42) and this one (1012) — both are seeded and
re-runnable; the h2o CI lower bound of +0.0023 is close enough to zero that
the choice of seed is worth knowing about, and it survived re-runs under
seed 42 unchanged ([+0.0023, +0.0257] on both).

## Amendment 1 addendum — the h2o confirmation run (operator: a 32,000-token limit is acceptable)

Prereg Amendment 1 @ 63d7a1b (plus 1a, boot timing). The seat contract under
h2o is 32,000 tokens (state + questions); the failed clause was the shim's
*silence*, not the limit. Stack: vLLM 0.30.0 + shipped shim unmodified, GPU 3,
plus a **thin admission wrapper** (task 2 recorded: no shim config option for
admission exists — `max_state_tokens` only configures the trim). The wrapper
imports the shipped renderers, asks vLLM's own `/tokenize` for the count of
each question's exact prompt, and refuses over-cap with 422 + an explicit
error naming the count (`refused, not trimmed`). No vendor file was edited.
It polices both currencies: per-question prompt max AND the shim's own
shared-head + tails total.

**(1)+(2) Gate re-run through the wrapper, 32k states, evidence at depth.**
Every admitted cell reports `usage.input_tokens` exactly equal to the wrapper
count — the trim is now unreachable below the cap, so the report is honest:

| depth | admitted | tokens sent = reported | answer |
|---|---|---|---|
| ~1,000 | yes | 31,700 | supported ✓ |
| ~8,000 | yes | 31,862 | supported ✓ (p: .907/.027/.066) |
| ~16,000 | yes | 29,508 | supported ✓ |
| ~30,000 | yes | 31,885 | supported ✓ |
| over-cap probes (×2) | **refused** | 39,805 / 40,336 tokens | 422, named count, every time |

Evidence answered correctly at every depth — the 10-04 depth-dependent
evidence loss is gone at these sizes (it lived INSIDE the trimmed window).
**Clause (2): clean.**

**(3) Largest-admissible shapes** (state pinned by the wrapper so the 16-q
request totals ≤ 32,000; realised 31,377 max-prompt / 31,867 total):
1q cold 213 ms / warm 194 ms; **16q cold 2,123 ms / warm 2,033 ms** (second
pass: 2,141/2,041). Cold = first request after backend ready on a restarted
container (compile cache mounted; calibration probes excluded).
**Clause (3): PASS** — warm 16q is 2.03 s against the 2.93 s bar, and it is
the fastest honest 32k cell of any arm this bench measured.

**(4) Concurrency:** 4 concurrent 16q × 32k on `--max-num-seqs 16`: all four
answered correctly, wall 7.08 s, **peak 15,566 MiB** against the GPU 1 budget
re-measured at run time (16,847 MiB; non-seat residents unchanged: 43,998 +
34,994). Headroom 1,281 MiB. **Clause (4): PASS** — the clause brokkr flagged
as most likely to bite did not bite; KV pressure at 4× concurrency grows
sub-linearly because all four requests share the prefix.

**(1a) Boot matrix** (reported, not gated): cold launch→ready median **153.6 s**
(min 149.4 / max 185.7, n=3), warm (persistent compile cache) **153.5 s** (n=3),
enforce-eager 161.9 s (n=1). Phases: weights 4.3 s, engine init 133.6 s (CUDA
graph capture ~77 s, torch.compile ~13 s), shim label-verification 3.1–3.7 s.
Two corrections to the main doc's boot row: the earlier ~280 s was measured
without a persistent cache mount and was never reproducible (use ~154 s);
warm cache saves nothing here because boot is graph-capture-bound, and
eager mode — which skips capture entirely — is *slower* (161.9 s), so there
is no faster-boot knob to grab. INC boots in 35 s; h2o's swap would cost
~2 min of boot per restart.

**(5) Wyrd item by item** (168 scorable rows; 8 disagree, h2o gains 3, loses 5,
net −2 — matching the −0.0167 point estimate; for Wyrd's owner):

| item | h2o | INC | gold | verdict |
|---|---|---|---|---|
| fletcher/f-go-tavern exit2 | stay | tavern | tavern | h2o wrong |
| fletcher/f-leave-town exit | meet_wren | stay | stay | h2o wrong |
| fletcher/f-memory exit | meet_wren | stay | stay | h2o wrong |
| fletcher/f-wren-comes exit2 | stay | meet_wren | meet_wren | h2o wrong |
| toll/t-granted place | moved | holds | moved | h2o right |
| toll/t-questions exit | departure | stay | stay | h2o wrong |
| vigil/v-to-pub exit2 | stay | outside | stay | h2o right |
| vigil/v-to-pub place | moved | holds | moved | h2o right |

Pattern worth noting: h2o's losses are all fletcher-scene *exit* decisions
(six of them in one scene family); its gains are the toll/vigil *place*
updates. No relabels (zero rows correct-but-renamed).

**Amendment 1 verdict: all three pass clauses hold.** (2) clean, (3) 2.03 s,
(4) 15,566 ≤ 16,847. Per the amendment's fixed rule, h2o-lightning-4b now
passes every Supersede clause — with the two honest caveats that its contract
is a REFUSING 32,000-token seat (not INC's silent-any-size behaviour), and
boot is 4.4x INC's. The seat swap is a decision for the operator and
infra-ops; the bench does not swap anything.

Teardown after the confirmation: wrapper, shim and container stopped; GPU 3
guard re-read 2 MiB before release; weights still kept (model-runtime 116 GB
+ 27 GB materialized serving dirs).