diff --git a/docs/pfi/jev-candidates-bench-2026-10-04.md b/docs/pfi/jev-candidates-bench-2026-10-04.md new file mode 100644 index 0000000..5a22fd7 --- /dev/null +++ b/docs/pfi/jev-candidates-bench-2026-10-04.md @@ -0,0 +1,114 @@ +# Jev candidates bench — 2026-10-04 (fv-ml1 GPU 3, cap 14.4 GiB @ 32,768 ctx) + +Two candidates against the live baseline, per brokkr's brief: `h2oai/h2o-lightning-4b` +(rev 542e9eff, Apache-2.0) and `Cloudflare/clef-flash` (rev 17f0b0ad, Apache-2.0, bf16 +and bnb-int8). Baseline: Intern-Decision-4B native, re-measured fresh. Style follows +`jev-candidates-bench-2026-09-30.md` (rest/peak MiB from a 0.2 s poller, CUDA context +outside the vLLM pool; medians of 3 run-medians). + +## Decision + +**Neither candidate replaces Intern-Decision-4B at the fleet shape.** + +- **H2O-lightning-4b** is the strongest candidate we have measured: vendor figure + reproduced exactly (205/231 ×3), best calibration (Brier 0.153 vs 0.178), best + negative-control discipline (follows the moved definition 0.931, clings to the gold + name only 0.028). But it **cannot fit the 14.4 GiB budget as shipped** (rest 15,454 + MiB before any request, peak 15,566 at the 32k gate), and it is **2.2× slower on the + 21-criteria shape** (184 ms vs 85 ms). If the cap is ever raised ~1.2 GiB or a leaner + engine build trims the Mamba cache, it becomes a real upgrade; at the cap, it is out. +- **Clef-flash** is a net loss on every axis that matters here: −11 JevBench pts (191 + bf16 / 192 int8), hard tier 72–73/111 vs baseline 83, slower at short shapes + (bf16 164 vs 85 ms; int8 297 ms), and its 32k context is not real as shipped — the + release path **silently slices every input to exactly 16,384 tokens** (input_tokens + pinned at 16384 on every cell) unless you deviate from the shipped `max_length`. + int8 additionally needs a 126-tensor dtype alignment (bnb leaves layernorm/head + fp32, vendor head forwards bf16) that is not in the vendor path, and it peaks + 20.7 GiB on a single 32k request (activations; `use_cache=False`), 43% past the cap. + +## Headline (jevbench public /231; battery = our sets, labelled-evidence rows) + +| system | jevbench | hard /111 | Brier | battery pooled /398 (Δ vs intern, 95% CI) | rest / peak MiB | crit21 ms | long16 ms | +|---|---|---|---|---|---|---|---| +| **Intern-Decision-4B native** (re-measure) | **202 ×3**, bit-exact vs 09-30 (0 rows flipped over 4+3 repeats) | 83 | 0.178 | 362 (0.9095) baseline | **9,752 / 9,752** — fits | **85** | **214** | +| H2O native (vendor vLLM 0.30.0 + shipped shim) | **205 ×3** (exact vendor fig.) | 86 | 0.153 | 367 (0.9221; **+0.0126**, CI −0.015..+0.044) | 15,454 / 15,566 — **over cap at rest** | 184 | 240 | +| H2O floor-off calibration | 205 — *identical to floored* | — | 0.154 | — | — | — | — | +| Clef bf16 (vendor loader, uncapped ref) | 191 | 72 | 0.235 | 366 (0.9196; +0.010, CI −0.021..+0.042) | 19,696 rest | 164 (flat; `use_cache=False`) | 390 | +| Clef int8 (bnb, dtype-aligned) | 192 | 73 | 0.235 | 365 (0.9171; +0.0075, CI −0.023..+0.038) | **12,064 rest / 13,216 suite peak** — fits; **20,718 gate peak** | 297 | 446 | + +Row-level: 33/398 battery rows flip intern→H2O, 29/398 intern→clef-bf16, 30/398 +intern→clef-int8; every battery delta straddles zero on a paired group bootstrap +(group = set), i.e. at 4B class our battery cannot separate them. The separations are +all in jevbench/hard, VRAM, and latency. + +## Battery by set (labelled-evidence acc, single) + +| set | intern | h2o | clef-bf16 | clef-int8 | +|---|---|---|---|---| +| authored144 | 0.917 | 0.958 | 0.924 | 0.910 | +| perturbations108 | 0.926 | 0.917 | 0.954 | 0.963 | +| cicada-w1 | 0.935 | 0.968 | 0.968 | 0.968 | +| cicada-w2 | 0.710 | 0.806 | 0.806 | 0.806 | +| wyrd | 0.940 | 0.893 | 0.893 | 0.893 | + +Same 09-30 pattern: candidate gains sit on the NLI-family public sets and partly +reverse on Wyrd. Rotations/repeat/reversed/shuffled/multifield: all four systems 0 +failed rows; H2O 0.937 rotations / 0.893 multifield; intern 0.897 / 0.917; clef-bf16 +0.920 / 0.821; clef-int8 0.917 / 0.833. + +## Name-vs-definition cell (new; 144 rows, descriptions reversed) + +| system | name-follow | def-follow | +|---|---|---| +| intern | 0.375 | 0.847 | +| h2o | 0.306 | **0.958** | +| clef-bf16 | 0.375 | 0.840 | +| clef-int8 | 0.368 | 0.854 | + +Negative control (gold label moved to another option's definition): same-top 0.097 / +0.028 / 0.111 / 0.090 respectively — no system is cue-clinging except mildly at +baseline level; H2O is the cleanest. All four are definition-followers, not +name-followers. + +## Gates (fit / long-input, per amendment) + +- **H2O fit gate (vendor path, `--max-num-seqs 16` required — Mamba blocks=181 < + default seqs):** pool accounting PASS at util 0.1506 (rest 14,356 / peak 15,540 + shim-measured), served a 32,082-token request at 1,987 ms. nvidia-smi total is + 15.4–15.6 GiB once the ~1.1 GiB CUDA context is counted — that is the overage. +- **H2O long-input gate (clean rerun):** PASS. Every depth cell ok, null control + correct (supported=0). Max input_tokens reached: **24,834** — the head+tail seam + truncates ~7k tokens off the nominal 32k state; head+tail keeps the tail (recent + events), which is the wrong end for our trace-replay workload. Latency at depth: + min/med/max 369/962/1,448 ms. +- **Clef shipped-default gate (max_length=16384):** PASS with pinning — every cell + input_tokens **exactly 16384**, i.e. silent server-side slice, no error. +- **Clef raised gate (max_length=32768), bf16 and int8:** PASS clean. input_tokens + tracks request size (22.9k–24.9k), no cliff at any depth cell, null control correct. + Max state tokens reached: **24,898**. Latency min/med/max 829/1,959/23,828 ms + (bf16), 934/2,220/24,211 ms (int8) — the ~24 s max is the first cell after load + (cold cache), median is the honest number. int8 32k call **peaks 20,718 MiB**. +- Latency at 32k-class inputs is where Clef's `use_cache=False` single-forward design + shows: flat per-call cost, ~2 s for a full 24k-token forward. + +## Scoring ruling (below-floor) — empirically closed + +Floor-off calibration cell: **205/231, identical to the floored runs**, identical +per-row. Below-floor outputs flow through as ordinary scored answers. **MISS is the +correct default for below-floor; no abstain handling needed in the harness.** + +## Provenance / gotchas worth keeping + +- Both snapshots sha256-verified against lfs.sha256 (6/6, 16/16 after materialisation). +- `hf download` without `HF_HOME` silently writes `~/.cache` — the /tank snapshot then + looks like 11K of configs (symlinks missing) while blobs are present. `cp -rL` + materialises for bind mounts. +- vLLM on H2O needs `--max-num-seqs 16` (Mamba cache blocks=181). +- Clef int8 dtype alignment: cast 126 non-bnb float tensors fp32→bf16 at load. Under + the vendor's own API, but not shipped — treat as integration friction, not a + fallback. +- First H2O long-input gate ran inside a container-collision window; archived as + `gate-h2o-collision-window`, numbers above are from the clean rerun. +- Artifacts: `fv-ml1:/tank/bench-jev-2026-10-04/out/` (vram.csv, sets.json, + shape.json, jevbench/, gate.json per dir), harness `code/` alongside; + jevbench v1.2.16 @ 5e95f23cbb7. diff --git a/services/semif-serve/bench-jev-2026-09-30/code/bench_sets.py b/services/semif-serve/bench-jev-2026-09-30/code/bench_sets.py index 7b41c4a..735d3d8 100644 --- a/services/semif-serve/bench-jev-2026-09-30/code/bench_sets.py +++ b/services/semif-serve/bench-jev-2026-09-30/code/bench_sets.py @@ -76,7 +76,9 @@ def load_items() -> list[dict]: def post(url: str, body: dict, headers: dict, timeout: float = 300) -> tuple[int, dict, float]: data = json.dumps(body).encode() - req = urllib.request.Request(url, data=data, headers={"Content-Type": "application/json", **headers}) + import os as _os + _auth = {"Authorization": "Bearer " + _os.environ["JB_AUTH"]} if _os.environ.get("JB_AUTH") else {} + req = urllib.request.Request(url, data=data, headers={"Content-Type": "application/json", **_auth, **headers}) t = time.perf_counter() try: with urllib.request.urlopen(req, timeout=timeout) as resp: @@ -222,6 +224,17 @@ def rotated_descriptions(options: list[dict]) -> list[dict]: return [{**o, "description": d} for o, d in zip(options, descs)] +def contradicted_names(options: list[dict]) -> list[dict]: + """2026-10-04 NEW row (arXiv 2609.26758): the option ids ARE semantically loaded names + (supported / insufficient / contradicted). Bind each name to a CONTRADICTING rubric by + assigning descriptions in reverse order (ids keep their position). For the 3-option + authored144 shape this swaps supported<->contradicted descriptions and leaves insufficient + self-bound. Name-following = the id whose NAME matches the gold rubric; definition-following + = the id now carrying the gold DESCRIPTION.""" + descs = [o["description"] for o in options][::-1] + return [{**o, "description": d} for o, d in zip(options, descs)] + + def gold_desc_id(item: dict) -> str | None: """After rotated_descriptions, position j carries the description that was at j+1, so the gold description (position g) now sits at g-1.""" @@ -232,6 +245,16 @@ def gold_desc_id(item: dict) -> str | None: return ids[(g - 1) % len(ids)] +def gold_desc_id_reversed(item: dict) -> str | None: + """After contradicted_names (descriptions reversed), position j carries the description that + was at n-1-j, so the gold description (position g) now sits at n-1-g.""" + if not item["gold"]: + return None + ids = [o["id"] for o in item["options"]] + g = ids.index(item["gold"][0]) + return ids[len(ids) - 1 - g] + + def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--backend", choices=("semif", "systemone"), required=True) @@ -252,6 +275,7 @@ def main() -> int: "reversed": [(i, list(reversed(i["options"])), False) for i in a144], "shuffled": [(i, shuffled(i["options"], i["id"]), False) for i in a144], "negative": [(i, rotated_descriptions(i["options"]), False) for i in a144], + "namedef": [(i, contradicted_names(i["options"]), False) for i in a144], } report = {"label": args.label, "backend": args.backend, "url": args.url, "started_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "conditions": {}} @@ -283,7 +307,8 @@ def main() -> int: "gold": item["gold"], "forbid": item["forbid"], "option_ids": [o["id"] for o in options], # negative control: the id that now carries the gold option's description - "gold_desc_id": gold_desc_id(item) if cond == "negative" else None, + "gold_desc_id": (gold_desc_id(item) if cond == "negative" else + gold_desc_id_reversed(item) if cond == "namedef" else None), **r}) if fails >= 5 and fails == len(rows): print(f"[{args.label}] {cond}: first {fails} rows all failed: {r}", file=sys.stderr)