esh-pfi-infrastructure/docs/pfi/jev-candidates-bench-2026-10-04.md

Jev candidates bench — 2026-10-04 (fv-ml1 GPU 3, cap 14.4 GiB @ 32,768 ctx)

Two candidates against the live baseline, per brokkr's brief: h2oai/h2o-lightning-4b (rev 542e9eff, Apache-2.0) and Cloudflare/clef-flash (rev 17f0b0ad, Apache-2.0, bf16 and bnb-int8). Baseline: Intern-Decision-4B native, re-measured fresh. Style follows jev-candidates-bench-2026-09-30.md (rest/peak MiB from a 0.2 s poller, CUDA context outside the vLLM pool; medians of 3 run-medians).

Decision

Neither candidate replaces Intern-Decision-4B at the fleet shape.

• H2O-lightning-4b is the strongest candidate we have measured: vendor figure reproduced exactly (205/231 ×3), best calibration (Brier 0.153 vs 0.178), best negative-control discipline (follows the moved definition 0.931, clings to the gold name only 0.028). But it cannot fit the budget as shipped and is 2.2× slower on the 21-criteria shape (184 ms vs 85 ms). On like-for-like nvidia-smi accounting the miss is small: ~17 MiB over at rest, ~130 MiB at peak, against the live seat's own 15,220-of-15,437 MiB headroom (the earlier "~1.2 GiB" line conflated the torch pool fraction — which excludes the CUDA context — with nvidia-smi totals that include it; brokkr correction 2026-10-05). Both the pool fraction and --max-num-seqs are tunable, so a leaner build closes the fit; but at parity on accuracy, 2.2× slower crit21, ~24.8k effective context through the seam, and a floored yes/no distribution, it stays a lateral move. If the cap is ever revisited, re-derive the gap from the nvidia-smi totals above, not from the 1.2 GiB figure.
• Clef-flash is a net loss on every axis that matters here: −11 JevBench pts (191 bf16 / 192 int8), hard tier 72–73/111 vs baseline 83, slower at short shapes (bf16 164 vs 85 ms; int8 297 ms), and its 32k context is not real as shipped — the release path silently slices every input to exactly 16,384 tokens (input_tokens pinned at 16384 on every cell) unless you deviate from the shipped max_length. int8 additionally needs a 126-tensor dtype alignment (bnb leaves layernorm/head fp32, vendor head forwards bf16) that is not in the vendor path, and it peaks 20.7 GiB on a single 32k request (activations; use_cache=False), 43% past the cap.

Headline (jevbench public /231; battery = our sets, labelled-evidence rows)

system	jevbench	hard /111	Brier	battery pooled /398 (Δ vs intern, 95% CI)	rest / peak MiB	crit21 ms	long16 ms
Intern-Decision-4B native (re-measure)	202 ×3, bit-exact vs 09-30 (0 rows flipped over 4+3 repeats)	83	0.178	362 (0.9095) baseline	9,752 / 9,752† — fits	85	214
H2O native (vendor vLLM 0.30.0 + shipped shim)	205 ×3 (exact vendor fig.)	86	0.153	367 (0.9221; +0.0126, CI −0.015..+0.044)	15,454 / 15,566 — over cap at rest	184	240
H2O floor-off calibration	205 — identical to floored	—	0.154	—	—	—	—
Clef bf16 (vendor loader, uncapped ref)	191	72	0.235	366 (0.9196; +0.010, CI −0.021..+0.042)	19,696 rest	164 (flat; use_cache=False)	390
Clef int8 (bnb, dtype-aligned)	192	73	0.235	365 (0.9171; +0.0075, CI −0.023..+0.038)	12,064 rest / 13,216 suite peak — fits; 20,718 gate peak	297	446

† Intern's 9,752 MiB is the capped-suite figure only — this bench's battery never pushed the baseline through the 32k long-input gate (it is already the live seat and was gate-proven at that shape on 09-30). H2O/Clef peaks include the 32k gate calls, so the peak column is not shape-matched for the baseline row. For like-for-like reference: the live seat's own 32k peak on GPU 1 is 15,220 MiB of a 15,437 MiB budget (n=3, persistent-memory intern-decision entry).

Row-level: 33/398 battery rows flip intern→H2O, 29/398 intern→clef-bf16, 30/398 intern→clef-int8; every battery delta straddles zero on a paired group bootstrap (group = set), i.e. at 4B class our battery cannot separate them. The separations are all in jevbench/hard, VRAM, and latency.

Battery by set (labelled-evidence acc, single)

set	intern	h2o	clef-bf16	clef-int8
authored144	0.917	0.958	0.924	0.910
perturbations108	0.926	0.917	0.954	0.963
cicada-w1	0.935	0.968	0.968	0.968
cicada-w2	0.710	0.806	0.806	0.806
wyrd	0.940	0.893	0.893	0.893

Same 09-30 pattern: candidate gains sit on the NLI-family public sets and partly reverse on Wyrd. Rotations/repeat/reversed/shuffled/multifield: all four systems 0 failed rows; H2O 0.937 rotations / 0.893 multifield; intern 0.897 / 0.917; clef-bf16 0.920 / 0.821; clef-int8 0.917 / 0.833.

Name-vs-definition cell (new; 144 rows, descriptions reversed)

Variant caveat (brokkr, 2026-10-05): this cell reverses descriptions under the same option names, which with odd option counts leaves the middle option's pair unmoved — an option can count as both name-followed and def-followed, so the two columns can sum above 1. It is a reasonable stand-in, not the loaded-name variant of arXiv 2609.26758; read the columns as overlaps, not a partition.

system	name-follow	def-follow
intern	0.375	0.847
h2o	0.306	0.958
clef-bf16	0.375	0.840
clef-int8	0.368	0.854

Negative control (gold label moved to another option's definition): same-top 0.097 / 0.028 / 0.111 / 0.090 respectively — no system is cue-clinging except mildly at baseline level; H2O is the cleanest. All four are definition-followers, not name-followers — true for the description-reversal manipulation run; the loaded-name variant is untested here.

Gates (fit / long-input, per amendment)

• H2O fit gate (vendor path, --max-num-seqs 16 required — Mamba blocks=181 < default seqs): pool accounting PASS at util 0.1506 (rest 14,356 / peak 15,540 shim-measured), served a 32,082-token request at 1,987 ms. nvidia-smi total is 15.4–15.6 GiB once the ~1.1 GiB CUDA context is counted — that is the overage.
• H2O long-input gate (clean rerun): PASS. Every depth cell ok, null control correct (supported=0). Max input_tokens reached: 24,834 — the head+tail seam truncates ~7k tokens off the nominal 32k state; head+tail keeps the tail (recent events), which is the wrong end for our trace-replay workload. Latency at depth: min/med/max 369/962/1,448 ms.
• Clef shipped-default gate (max_length=16384): PASS with pinning — every cell input_tokens exactly 16384, i.e. silent server-side slice, no error.
• Clef raised gate (max_length=32768), bf16 and int8: PASS clean. input_tokens tracks request size (22.9k–24.9k), no cliff at any depth cell, null control correct. Max state tokens reached: 24,898. Latency min/med/max 829/1,959/23,828 ms (bf16), 934/2,220/24,211 ms (int8) — the ~24 s max is the first cell after load (cold cache), median is the honest number. int8 32k call peaks 20,718 MiB.
• Latency at 32k-class inputs is where Clef's use_cache=False single-forward design shows: flat per-call cost, ~2 s for a full 24k-token forward.

Scoring ruling (below-floor) — empirically closed

Floor-off calibration cell: 205/231, identical to the floored runs, identical per-row. Below-floor outputs flow through as ordinary scored answers. MISS is the correct default for below-floor; no abstain handling needed in the harness.

Provenance / gotchas worth keeping

• Both snapshots sha256-verified against lfs.sha256 (6/6, 16/16 after materialisation).
• hf download without HF_HOME silently writes ~/.cache — the /tank snapshot then looks like 11K of configs (symlinks missing) while blobs are present. cp -rL materialises for bind mounts.
• vLLM on H2O needs --max-num-seqs 16 (Mamba cache blocks=181).
• Clef int8 dtype alignment: cast 126 non-bnb float tensors fp32→bf16 at load. Under the vendor's own API, but not shipped — treat as integration friction, not a fallback.
• First H2O long-input gate ran inside a container-collision window; archived as gate-h2o-collision-window, numbers above are from the clean rerun.
• Artifacts: fv-ml1:/tank/bench-jev-2026-10-04/out/ (vram.csv, sets.json, shape.json, jevbench/, gate.json per dir), harness code/ alongside; jevbench v1.2.16 @ 5e95f23cbb7.