esh-pfi-infrastructure/docs/pfi/jev-candidates-bench-2026-10-04.md

Jev candidates bench — 2026-10-04 (fv-ml1 GPU 3, cap 14.4 GiB @ 32,768 ctx)

Two candidates against the live baseline, per brokkr's brief: h2oai/h2o-lightning-4b (rev 542e9eff, Apache-2.0) and Cloudflare/clef-flash (rev 17f0b0ad, Apache-2.0, bf16 and bnb-int8). Baseline: Intern-Decision-4B native, re-measured fresh. Style follows jev-candidates-bench-2026-09-30.md (rest/peak MiB from a 0.2 s poller, CUDA context outside the vLLM pool; medians of 3 run-medians).

Decision

Neither candidate replaces Intern-Decision-4B at the fleet shape.

• H2O-lightning-4b is the strongest candidate we have measured: vendor figure reproduced exactly (205/231 ×3), best calibration (Brier 0.153 vs 0.178), best negative-control discipline (follows the moved definition 0.931, clings to the gold name only 0.028). But it cannot fit the 14.4 GiB budget as shipped (rest 15,454 MiB before any request, peak 15,566 at the 32k gate), and it is 2.2× slower on the 21-criteria shape (184 ms vs 85 ms). If the cap is ever raised ~1.2 GiB or a leaner engine build trims the Mamba cache, it becomes a real upgrade; at the cap, it is out.
• Clef-flash is a net loss on every axis that matters here: −11 JevBench pts (191 bf16 / 192 int8), hard tier 72–73/111 vs baseline 83, slower at short shapes (bf16 164 vs 85 ms; int8 297 ms), and its 32k context is not real as shipped — the release path silently slices every input to exactly 16,384 tokens (input_tokens pinned at 16384 on every cell) unless you deviate from the shipped max_length. int8 additionally needs a 126-tensor dtype alignment (bnb leaves layernorm/head fp32, vendor head forwards bf16) that is not in the vendor path, and it peaks 20.7 GiB on a single 32k request (activations; use_cache=False), 43% past the cap.

Headline (jevbench public /231; battery = our sets, labelled-evidence rows)

system	jevbench	hard /111	Brier	battery pooled /398 (Δ vs intern, 95% CI)	rest / peak MiB	crit21 ms	long16 ms
Intern-Decision-4B native (re-measure)	202 ×3, bit-exact vs 09-30 (0 rows flipped over 4+3 repeats)	83	0.178	362 (0.9095) baseline	9,752 / 9,752 — fits	85	214
H2O native (vendor vLLM 0.30.0 + shipped shim)	205 ×3 (exact vendor fig.)	86	0.153	367 (0.9221; +0.0126, CI −0.015..+0.044)	15,454 / 15,566 — over cap at rest	184	240
H2O floor-off calibration	205 — identical to floored	—	0.154	—	—	—	—
Clef bf16 (vendor loader, uncapped ref)	191	72	0.235	366 (0.9196; +0.010, CI −0.021..+0.042)	19,696 rest	164 (flat; use_cache=False)	390
Clef int8 (bnb, dtype-aligned)	192	73	0.235	365 (0.9171; +0.0075, CI −0.023..+0.038)	12,064 rest / 13,216 suite peak — fits; 20,718 gate peak	297	446

Row-level: 33/398 battery rows flip intern→H2O, 29/398 intern→clef-bf16, 30/398 intern→clef-int8; every battery delta straddles zero on a paired group bootstrap (group = set), i.e. at 4B class our battery cannot separate them. The separations are all in jevbench/hard, VRAM, and latency.

Battery by set (labelled-evidence acc, single)

set	intern	h2o	clef-bf16	clef-int8
authored144	0.917	0.958	0.924	0.910
perturbations108	0.926	0.917	0.954	0.963
cicada-w1	0.935	0.968	0.968	0.968
cicada-w2	0.710	0.806	0.806	0.806
wyrd	0.940	0.893	0.893	0.893

Same 09-30 pattern: candidate gains sit on the NLI-family public sets and partly reverse on Wyrd. Rotations/repeat/reversed/shuffled/multifield: all four systems 0 failed rows; H2O 0.937 rotations / 0.893 multifield; intern 0.897 / 0.917; clef-bf16 0.920 / 0.821; clef-int8 0.917 / 0.833.

Name-vs-definition cell (new; 144 rows, descriptions reversed)

system	name-follow	def-follow
intern	0.375	0.847
h2o	0.306	0.958
clef-bf16	0.375	0.840
clef-int8	0.368	0.854

Negative control (gold label moved to another option's definition): same-top 0.097 / 0.028 / 0.111 / 0.090 respectively — no system is cue-clinging except mildly at baseline level; H2O is the cleanest. All four are definition-followers, not name-followers.

Gates (fit / long-input, per amendment)

• H2O fit gate (vendor path, --max-num-seqs 16 required — Mamba blocks=181 < default seqs): pool accounting PASS at util 0.1506 (rest 14,356 / peak 15,540 shim-measured), served a 32,082-token request at 1,987 ms. nvidia-smi total is 15.4–15.6 GiB once the ~1.1 GiB CUDA context is counted — that is the overage.
• H2O long-input gate (clean rerun): PASS. Every depth cell ok, null control correct (supported=0). Max input_tokens reached: 24,834 — the head+tail seam truncates ~7k tokens off the nominal 32k state; head+tail keeps the tail (recent events), which is the wrong end for our trace-replay workload. Latency at depth: min/med/max 369/962/1,448 ms.
• Clef shipped-default gate (max_length=16384): PASS with pinning — every cell input_tokens exactly 16384, i.e. silent server-side slice, no error.
• Clef raised gate (max_length=32768), bf16 and int8: PASS clean. input_tokens tracks request size (22.9k–24.9k), no cliff at any depth cell, null control correct. Max state tokens reached: 24,898. Latency min/med/max 829/1,959/23,828 ms (bf16), 934/2,220/24,211 ms (int8) — the ~24 s max is the first cell after load (cold cache), median is the honest number. int8 32k call peaks 20,718 MiB.
• Latency at 32k-class inputs is where Clef's use_cache=False single-forward design shows: flat per-call cost, ~2 s for a full 24k-token forward.

Scoring ruling (below-floor) — empirically closed

Floor-off calibration cell: 205/231, identical to the floored runs, identical per-row. Below-floor outputs flow through as ordinary scored answers. MISS is the correct default for below-floor; no abstain handling needed in the harness.

Provenance / gotchas worth keeping

• Both snapshots sha256-verified against lfs.sha256 (6/6, 16/16 after materialisation).
• hf download without HF_HOME silently writes ~/.cache — the /tank snapshot then looks like 11K of configs (symlinks missing) while blobs are present. cp -rL materialises for bind mounts.
• vLLM on H2O needs --max-num-seqs 16 (Mamba cache blocks=181).
• Clef int8 dtype alignment: cast 126 non-bnb float tensors fp32→bf16 at load. Under the vendor's own API, but not shipped — treat as integration friction, not a fallback.
• First H2O long-input gate ran inside a container-collision window; archived as gate-h2o-collision-window, numbers above are from the clean rerun.
• Artifacts: fv-ml1:/tank/bench-jev-2026-10-04/out/ (vram.csv, sets.json, shape.json, jevbench/, gate.json per dir), harness code/ alongside; jevbench v1.2.16 @ 5e95f23cbb7.