diff --git a/docs/pfi/decision-sweep-followon-bench-2026-10-12.md b/docs/pfi/decision-sweep-followon-bench-2026-10-12.md new file mode 100644 index 0000000..ef3371e --- /dev/null +++ b/docs/pfi/decision-sweep-followon-bench-2026-10-12.md @@ -0,0 +1,142 @@ +# Decision-model sweep follow-on: H2O re-bench, Clef-flash, RSI-Jev vs the live seat (2026-10-12) + +Ordered by Prime; preregistration `brokkr-smithy` +`docs/surveys/decision-model-sweep/bench-2026-10-12-followon-prereg.md` @ 2e0539f — +the first run under the bench-first rule. Run by infra-hermes on fv-ml1 GPU 3 +(process-pinned, GPU-3-empty guard, loopback only). Harness: the same d3 bench +harness as the morning run +(`docs/pfi/d3-vs-intern-decision-bench-2026-10-12.md`), unchanged. + +## Headline + +**The incumbent holds.** H2O-lightning-4b is the first non-d3 arm to beat INC +on the public suite (205 vs 202) with a positive paired battery delta — but it +fails the long-input gate by the letter: its shipped shim silently trims the +state at ~32,000 tokens, confirmed today exactly as on 10-04. Under the prereg +rules that makes it a **short-route candidate**, not a supersede. Clef-flash is +**lateral** in both precisions (battery CI straddles zero, Wyrd significantly +negative, and bf16 doesn't even fit the seat at rest). RSI-Jev v6.1 is a +**loss** (23 items below INC on public, Wyrd −0.0548). + +## Budget (re-measured at bench time, Amendment 2 method) + +GPU 1 total 97,887 MiB. Non-seat: vllm-meromero-r 43,998 + vllm-gen-small +34,994 = 78,992 MiB. Seat resting 9,098 MiB. Margin 2,048 MiB. +**Budget = 16,847 MiB.** (16,851 in the morning; 4 MiB drift, immaterial.) + +## Arms and verdicts + +Vendor-intended serving path for every arm, verbatim; no ports, no third-party +shims. Wire adaptations beyond the mandatory model field: none for H2O (shipped +`h2o_lightning_shim.py`) or RSI (vendor CLI); Clef reuses the 10-04 +`clef_server.py` stdlib wrapper around the shipped `joint_schema_model.py` +(same wrapper that produced the 10-04 numbers; weights byte-identical). + +| arm | path | jevbench (×3) | battery paired Δ vs INC [95% CI] | Wyrd Δ [CI] | neg-control | Brier / ECE | +|---|---|---|---|---|---|---| +| **INC (reused cells)** | seat 0.1.3 | **202** | — | — | .097 | .178 / .051 | +| H2O @ b43f759b | vLLM 0.30.0 + shipped shim | **205** | **+0.0137 [+0.0023, +0.0257]** | −0.0167 [−0.0357, +0.0024] | .028 | .152 / .050 | +| CLEF bf16 @ 8b2e5fd1 | vendor loader, max_length=32768 | 191 | +0.0040 [−0.0080, +0.0160] | **−0.0333 [−0.0571, −0.0095]** | .111 | .235 / .053 | +| CLEF bnb-int8 | same + LLM.int8() | 190 | +0.0006 [−0.0114, +0.0120] | **−0.0333 [−0.0571, −0.0119]** | .090 | .239 / .043 | +| RSI-Jev v6.1-VL-4B @ 7f93b192 (HF rev) | `rsi-jev serve --effort auto` | 179 | −0.0149 [−0.0303, +0.0006] | **−0.0548 [−0.0905, −0.0214]** | .208 | .274 / .061 | + +Paired battery: 1,748 scorable rows against the 10-04 INC r1 rows (same +harness, harness scorer), 10k seeded bootstrap (seed 1012). All arms ran the +full battery: negative control crushed for H2O (it abstains — the loaded-name +trap works), soft for Clef (.111/.090) and RSI (.208 — the weakest trap of the +run). Calibration as shipped, jevbench 231: H2O is the ONLY arm with a better +Brier than INC, and its ECE is the second-best. + +## Speed and the 32k shapes (calibrated per tokenizer to 32,743 ±2%; realised tokens logged) + +cold = first request after ready; warm = median of 5; launch→ready its own +column. INC crit cells re-measured today on the seat path (2-call 16+5 split +for crit21, per contract); 32k INC cells reused from today's 32k run (maxfit +16q at 32,708 tokens, 1,960 ms — the prereg's 1.95 s anchor). + +| arm | crit16 warm ms | crit21 warm ms | 1q32k warm ms (realised tok) | 16q32k warm ms (realised tok) | 16q32k peak MiB | launch→ready s | +|---|---|---|---|---|---|---| +| INC | 110 | 198 (2-call) | 1,960 (32,715)* | 1,960 maxfit (32,708)* | 8,822 | 20 | +| H2O | 137 | 168 | 179 (32,082 — **TRIMMED**) | 1,308 (32,574) | 15,464 | ~280 | +| CLEF bf16 | 210 | 138 | 2,948 (32,730) | 2,962 (32,768 cap) | 32,130 | 12 | +| CLEF int8 | 238 | 176 | 3,262 (32,730) | 3,284 (32,768 cap) | 23,080 | 18 | +| RSI | 112 | 69 | 1,224 (32,758) | 2,405 (33,205) | 21,158 | 68 | + +\* reused 32k-run cells; INC's `/v1/systemone` enforces a 32,768-token +contract, so its honest 16q32k is a maxfit single forward. + +Both Clef arms sit exactly on 32,768: the vendor loader with max_length=32768 +honours the window but caps there — 16q over 32k state counts state once per +request at the wire, and the loader slices to window. Their "warm" at that +shape is the fastest honest path, but the state never exceeds the window. +RSI realised 33,205 — it has no hard cap; honest 32k+ shape. + +**H2O's trim, the named risk, confirmed:** the shape calibration plateaued at +32,082 realised tokens across builder lengths 46,953→51,097 (full history in +`arm-h2o/shape32k.stdout`) — the shim head/tail-trims at 32,000 exactly as on +10-04 (24,834 effective then, 32,082 now: the trim moved but did not go away). +The gate itself scores OK at every depth (the evidence sits mid-state and +survives the trim), so the failure mode is silent: **accuracy at 32k is +measured on a truncated state and the response says nothing about it.** + +## Long-input gate (1k/8k/16k/30k + positive/null, tokens-to-model logged) + +| arm | verdict | evidence | +|---|---|---| +| INC | clean | reused; no trim observed on seat path | +| H2O | **FAIL — silent truncation** | calibration plateau at 32,082; `usage.input_tokens` reports the post-trim count, not the sent count | +| CLEF bf16 | clean (window=32,768 honoured) | d30000 OK; no plateau below the window | +| CLEF int8 | clean | as bf16 | +| RSI | clean | 32,758 realised, no plateau; d30000 OK | + +## Decision-rule application (fixed at 2e0539f) + +**H2O-lightning-4b — SHORT-ROUTE CANDIDATE.** Fits the budget at every shape +incl. 16q32k (15,464 ≤ 16,847); battery CI lower bound +0.0023 > 0; Wyrd +−0.0167 ≥ −0.02; 16q32k warm 1.31 s ≤ 2.94 s (passes latency by the letter, +and it is genuinely the fastest at the worst shape because the shim's +single-count protocol answers over a *truncated* state). Fails only the gate +clause: silent truncation at 32k. Recorded, not acted on. Note for whoever +eventually evaluates it for the seat: the trim is inside the vendor's shipped +serving code — a supersede needs the vendor to fix the shim, not us to patch +around it. + +**CLEF bf16 — LATERAL, memory-out.** Battery CI straddles zero; Wyrd fails; +rest 18,854 MiB does not fit the budget at all; 16q32k warm 2,962 ms fails the +latency clause by 22 ms (0.7%) at peak 32,130 MiB (1.9x the budget). +**CLEF int8 — LATERAL.** Fits short shapes (rest 12,084) but the battery CI +straddles zero, so it is not even a short-route candidate; 16q32k fails both +memory (23,080) and latency (3,284). +**RSI-Jev — LOSS.** Public −23 items (outside ±3), battery Δ −0.0149 with CI +[−0.0303, +0.0006], Wyrd −0.0548, negative control softest of the run. Its +serving numbers are respectable (clean gate, latency pass at 16q32k) — it is +the decision quality that loses. + +## Housekeeping + +- Repo pins: RSI-Jev serving code = main @ fa70834 (pinned commit recorded per + prereg; the 7f93b192 sha is the **HF model revision**, not a repo commit — + it is not an ancestor of main and was resolved as a snapshot). +- Licence flags as prereg'd: Clef Apache-2.0 fine; RSI has no LICENSE file at + source and lists a research-only training source — flagged once, does not + block a bench, deployment rights are the operator's call. +- Teardown: all bench servers stopped, no listeners on 1809x, GPU 3 guard read + 2 MiB, no firewall rules were needed (loopback-bound ports only). +- Weights KEPT (operator ruling stands): model-runtime now 116 GB total + (adds h2o 9 GB blob-set, clef 18 GB, RSI 5.5 GB), plus materialized + `pkg-h2o` 9.0 GB and `pkg-clef` 18 GB serving dirs and the RSI-Jev clone + 72 MB. `/tank` 63% used, 3.2 TB free. +- Artifacts: `fv-ml1:/tank/bench/arm-{h2o,clef-bf16,clef-int8,rsi,inc}/` + (jevbench ×3, sets.json battery, gate, shape32k, crit16; h2o additionally + `shape32k-forced.json` recording the trimmed-shape cells and + `shape32k.stdout` the plateau history). + +## What was NOT run + +Nothing from the prereg matrix was skipped. One methodology note: INC's +battery/gate rows are the 10-04 r1 rows (reused cells, harness unchanged and +bit-exact reproduced this morning at 202/231), and the paired bootstrap seeds +differ between the morning doc (42) and this one (1012) — both are seeded and +re-runnable; the h2o CI lower bound of +0.0023 is close enough to zero that +the choice of seed is worth knowing about, and it survived re-runs under +seed 42 unchanged ([+0.0023, +0.0257] on both).