docs(pfi): decision-sweep follow-on bench — H2O re-bench, Clef-flash, RSI-Jev vs the live seat
First bench-first run (prereg brokkr-smithy 2e0539f). H2O-lightning-4b is a short-route candidate: public 205, battery CI lower bound +0.0023, fits the GPU 1 budget at every shape — but its shipped shim silently trims state at ~32k and fails the long-input gate by the letter. Clef-flash lateral in both precisions (Wyrd -0.0333 in both; bf16 does not fit at rest). RSI-Jev v6.1 loses on decision quality (public -23, Wyrd -0.0548). INC holds the seat. Torn down; weights kept (model-runtime 116 GB).
This commit is contained in:
@@ -0,0 +1,142 @@
|
|||||||
|
# Decision-model sweep follow-on: H2O re-bench, Clef-flash, RSI-Jev vs the live seat (2026-10-12)
|
||||||
|
|
||||||
|
Ordered by Prime; preregistration `brokkr-smithy`
|
||||||
|
`docs/surveys/decision-model-sweep/bench-2026-10-12-followon-prereg.md` @ 2e0539f —
|
||||||
|
the first run under the bench-first rule. Run by infra-hermes on fv-ml1 GPU 3
|
||||||
|
(process-pinned, GPU-3-empty guard, loopback only). Harness: the same d3 bench
|
||||||
|
harness as the morning run
|
||||||
|
(`docs/pfi/d3-vs-intern-decision-bench-2026-10-12.md`), unchanged.
|
||||||
|
|
||||||
|
## Headline
|
||||||
|
|
||||||
|
**The incumbent holds.** H2O-lightning-4b is the first non-d3 arm to beat INC
|
||||||
|
on the public suite (205 vs 202) with a positive paired battery delta — but it
|
||||||
|
fails the long-input gate by the letter: its shipped shim silently trims the
|
||||||
|
state at ~32,000 tokens, confirmed today exactly as on 10-04. Under the prereg
|
||||||
|
rules that makes it a **short-route candidate**, not a supersede. Clef-flash is
|
||||||
|
**lateral** in both precisions (battery CI straddles zero, Wyrd significantly
|
||||||
|
negative, and bf16 doesn't even fit the seat at rest). RSI-Jev v6.1 is a
|
||||||
|
**loss** (23 items below INC on public, Wyrd −0.0548).
|
||||||
|
|
||||||
|
## Budget (re-measured at bench time, Amendment 2 method)
|
||||||
|
|
||||||
|
GPU 1 total 97,887 MiB. Non-seat: vllm-meromero-r 43,998 + vllm-gen-small
|
||||||
|
34,994 = 78,992 MiB. Seat resting 9,098 MiB. Margin 2,048 MiB.
|
||||||
|
**Budget = 16,847 MiB.** (16,851 in the morning; 4 MiB drift, immaterial.)
|
||||||
|
|
||||||
|
## Arms and verdicts
|
||||||
|
|
||||||
|
Vendor-intended serving path for every arm, verbatim; no ports, no third-party
|
||||||
|
shims. Wire adaptations beyond the mandatory model field: none for H2O (shipped
|
||||||
|
`h2o_lightning_shim.py`) or RSI (vendor CLI); Clef reuses the 10-04
|
||||||
|
`clef_server.py` stdlib wrapper around the shipped `joint_schema_model.py`
|
||||||
|
(same wrapper that produced the 10-04 numbers; weights byte-identical).
|
||||||
|
|
||||||
|
| arm | path | jevbench (×3) | battery paired Δ vs INC [95% CI] | Wyrd Δ [CI] | neg-control | Brier / ECE |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| **INC (reused cells)** | seat 0.1.3 | **202** | — | — | .097 | .178 / .051 |
|
||||||
|
| H2O @ b43f759b | vLLM 0.30.0 + shipped shim | **205** | **+0.0137 [+0.0023, +0.0257]** | −0.0167 [−0.0357, +0.0024] | .028 | .152 / .050 |
|
||||||
|
| CLEF bf16 @ 8b2e5fd1 | vendor loader, max_length=32768 | 191 | +0.0040 [−0.0080, +0.0160] | **−0.0333 [−0.0571, −0.0095]** | .111 | .235 / .053 |
|
||||||
|
| CLEF bnb-int8 | same + LLM.int8() | 190 | +0.0006 [−0.0114, +0.0120] | **−0.0333 [−0.0571, −0.0119]** | .090 | .239 / .043 |
|
||||||
|
| RSI-Jev v6.1-VL-4B @ 7f93b192 (HF rev) | `rsi-jev serve --effort auto` | 179 | −0.0149 [−0.0303, +0.0006] | **−0.0548 [−0.0905, −0.0214]** | .208 | .274 / .061 |
|
||||||
|
|
||||||
|
Paired battery: 1,748 scorable rows against the 10-04 INC r1 rows (same
|
||||||
|
harness, harness scorer), 10k seeded bootstrap (seed 1012). All arms ran the
|
||||||
|
full battery: negative control crushed for H2O (it abstains — the loaded-name
|
||||||
|
trap works), soft for Clef (.111/.090) and RSI (.208 — the weakest trap of the
|
||||||
|
run). Calibration as shipped, jevbench 231: H2O is the ONLY arm with a better
|
||||||
|
Brier than INC, and its ECE is the second-best.
|
||||||
|
|
||||||
|
## Speed and the 32k shapes (calibrated per tokenizer to 32,743 ±2%; realised tokens logged)
|
||||||
|
|
||||||
|
cold = first request after ready; warm = median of 5; launch→ready its own
|
||||||
|
column. INC crit cells re-measured today on the seat path (2-call 16+5 split
|
||||||
|
for crit21, per contract); 32k INC cells reused from today's 32k run (maxfit
|
||||||
|
16q at 32,708 tokens, 1,960 ms — the prereg's 1.95 s anchor).
|
||||||
|
|
||||||
|
| arm | crit16 warm ms | crit21 warm ms | 1q32k warm ms (realised tok) | 16q32k warm ms (realised tok) | 16q32k peak MiB | launch→ready s |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| INC | 110 | 198 (2-call) | 1,960 (32,715)* | 1,960 maxfit (32,708)* | 8,822 | 20 |
|
||||||
|
| H2O | 137 | 168 | 179 (32,082 — **TRIMMED**) | 1,308 (32,574) | 15,464 | ~280 |
|
||||||
|
| CLEF bf16 | 210 | 138 | 2,948 (32,730) | 2,962 (32,768 cap) | 32,130 | 12 |
|
||||||
|
| CLEF int8 | 238 | 176 | 3,262 (32,730) | 3,284 (32,768 cap) | 23,080 | 18 |
|
||||||
|
| RSI | 112 | 69 | 1,224 (32,758) | 2,405 (33,205) | 21,158 | 68 |
|
||||||
|
|
||||||
|
\* reused 32k-run cells; INC's `/v1/systemone` enforces a 32,768-token
|
||||||
|
contract, so its honest 16q32k is a maxfit single forward.
|
||||||
|
|
||||||
|
Both Clef arms sit exactly on 32,768: the vendor loader with max_length=32768
|
||||||
|
honours the window but caps there — 16q over 32k state counts state once per
|
||||||
|
request at the wire, and the loader slices to window. Their "warm" at that
|
||||||
|
shape is the fastest honest path, but the state never exceeds the window.
|
||||||
|
RSI realised 33,205 — it has no hard cap; honest 32k+ shape.
|
||||||
|
|
||||||
|
**H2O's trim, the named risk, confirmed:** the shape calibration plateaued at
|
||||||
|
32,082 realised tokens across builder lengths 46,953→51,097 (full history in
|
||||||
|
`arm-h2o/shape32k.stdout`) — the shim head/tail-trims at 32,000 exactly as on
|
||||||
|
10-04 (24,834 effective then, 32,082 now: the trim moved but did not go away).
|
||||||
|
The gate itself scores OK at every depth (the evidence sits mid-state and
|
||||||
|
survives the trim), so the failure mode is silent: **accuracy at 32k is
|
||||||
|
measured on a truncated state and the response says nothing about it.**
|
||||||
|
|
||||||
|
## Long-input gate (1k/8k/16k/30k + positive/null, tokens-to-model logged)
|
||||||
|
|
||||||
|
| arm | verdict | evidence |
|
||||||
|
|---|---|---|
|
||||||
|
| INC | clean | reused; no trim observed on seat path |
|
||||||
|
| H2O | **FAIL — silent truncation** | calibration plateau at 32,082; `usage.input_tokens` reports the post-trim count, not the sent count |
|
||||||
|
| CLEF bf16 | clean (window=32,768 honoured) | d30000 OK; no plateau below the window |
|
||||||
|
| CLEF int8 | clean | as bf16 |
|
||||||
|
| RSI | clean | 32,758 realised, no plateau; d30000 OK |
|
||||||
|
|
||||||
|
## Decision-rule application (fixed at 2e0539f)
|
||||||
|
|
||||||
|
**H2O-lightning-4b — SHORT-ROUTE CANDIDATE.** Fits the budget at every shape
|
||||||
|
incl. 16q32k (15,464 ≤ 16,847); battery CI lower bound +0.0023 > 0; Wyrd
|
||||||
|
−0.0167 ≥ −0.02; 16q32k warm 1.31 s ≤ 2.94 s (passes latency by the letter,
|
||||||
|
and it is genuinely the fastest at the worst shape because the shim's
|
||||||
|
single-count protocol answers over a *truncated* state). Fails only the gate
|
||||||
|
clause: silent truncation at 32k. Recorded, not acted on. Note for whoever
|
||||||
|
eventually evaluates it for the seat: the trim is inside the vendor's shipped
|
||||||
|
serving code — a supersede needs the vendor to fix the shim, not us to patch
|
||||||
|
around it.
|
||||||
|
|
||||||
|
**CLEF bf16 — LATERAL, memory-out.** Battery CI straddles zero; Wyrd fails;
|
||||||
|
rest 18,854 MiB does not fit the budget at all; 16q32k warm 2,962 ms fails the
|
||||||
|
latency clause by 22 ms (0.7%) at peak 32,130 MiB (1.9x the budget).
|
||||||
|
**CLEF int8 — LATERAL.** Fits short shapes (rest 12,084) but the battery CI
|
||||||
|
straddles zero, so it is not even a short-route candidate; 16q32k fails both
|
||||||
|
memory (23,080) and latency (3,284).
|
||||||
|
**RSI-Jev — LOSS.** Public −23 items (outside ±3), battery Δ −0.0149 with CI
|
||||||
|
[−0.0303, +0.0006], Wyrd −0.0548, negative control softest of the run. Its
|
||||||
|
serving numbers are respectable (clean gate, latency pass at 16q32k) — it is
|
||||||
|
the decision quality that loses.
|
||||||
|
|
||||||
|
## Housekeeping
|
||||||
|
|
||||||
|
- Repo pins: RSI-Jev serving code = main @ fa70834 (pinned commit recorded per
|
||||||
|
prereg; the 7f93b192 sha is the **HF model revision**, not a repo commit —
|
||||||
|
it is not an ancestor of main and was resolved as a snapshot).
|
||||||
|
- Licence flags as prereg'd: Clef Apache-2.0 fine; RSI has no LICENSE file at
|
||||||
|
source and lists a research-only training source — flagged once, does not
|
||||||
|
block a bench, deployment rights are the operator's call.
|
||||||
|
- Teardown: all bench servers stopped, no listeners on 1809x, GPU 3 guard read
|
||||||
|
2 MiB, no firewall rules were needed (loopback-bound ports only).
|
||||||
|
- Weights KEPT (operator ruling stands): model-runtime now 116 GB total
|
||||||
|
(adds h2o 9 GB blob-set, clef 18 GB, RSI 5.5 GB), plus materialized
|
||||||
|
`pkg-h2o` 9.0 GB and `pkg-clef` 18 GB serving dirs and the RSI-Jev clone
|
||||||
|
72 MB. `/tank` 63% used, 3.2 TB free.
|
||||||
|
- Artifacts: `fv-ml1:/tank/bench/arm-{h2o,clef-bf16,clef-int8,rsi,inc}/`
|
||||||
|
(jevbench ×3, sets.json battery, gate, shape32k, crit16; h2o additionally
|
||||||
|
`shape32k-forced.json` recording the trimmed-shape cells and
|
||||||
|
`shape32k.stdout` the plateau history).
|
||||||
|
|
||||||
|
## What was NOT run
|
||||||
|
|
||||||
|
Nothing from the prereg matrix was skipped. One methodology note: INC's
|
||||||
|
battery/gate rows are the 10-04 r1 rows (reused cells, harness unchanged and
|
||||||
|
bit-exact reproduced this morning at 202/231), and the paired bootstrap seeds
|
||||||
|
differ between the morning doc (42) and this one (1012) — both are seeded and
|
||||||
|
re-runnable; the h2o CI lower bound of +0.0023 is close enough to zero that
|
||||||
|
the choice of seed is worth knowing about, and it survived re-runs under
|
||||||
|
seed 42 unchanged ([+0.0023, +0.0257] on both).
|
||||||
Reference in New Issue
Block a user