docs(pfi): amend 10-04 bench doc per brokkr review

- Fit gap restated on nvidia-smi accounting (~17 MiB rest / ~130 MiB peak
  over, vs the live seat's 15,220/15,437 headroom); the 1.2 GiB figure
  conflated the torch pool fraction with nvidia-smi totals
- Footnote: baseline peak row is suite-only, not shape-matched to the
  candidate gate peaks
- namedef cell labeled as the description-reversal stand-in, not the
  loaded-name variant of arXiv 2609.26758; columns are overlaps
This commit is contained in:
vh
2026-10-04 23:12:41 -07:00
parent 523f086939
commit 867f33d672
+26 -6
View File
@@ -13,10 +13,16 @@ outside the vLLM pool; medians of 3 run-medians).
- **H2O-lightning-4b** is the strongest candidate we have measured: vendor figure - **H2O-lightning-4b** is the strongest candidate we have measured: vendor figure
reproduced exactly (205/231 ×3), best calibration (Brier 0.153 vs 0.178), best reproduced exactly (205/231 ×3), best calibration (Brier 0.153 vs 0.178), best
negative-control discipline (follows the moved definition 0.931, clings to the gold negative-control discipline (follows the moved definition 0.931, clings to the gold
name only 0.028). But it **cannot fit the 14.4 GiB budget as shipped** (rest 15,454 name only 0.028). But it **cannot fit the budget as shipped** and is **2.2× slower on
MiB before any request, peak 15,566 at the 32k gate), and it is **2.2× slower on the the 21-criteria shape** (184 ms vs 85 ms). On like-for-like nvidia-smi accounting the
21-criteria shape** (184 ms vs 85 ms). If the cap is ever raised ~1.2 GiB or a leaner miss is small: ~17 MiB over at rest, ~130 MiB at peak, against the live seat's own
engine build trims the Mamba cache, it becomes a real upgrade; at the cap, it is out. 15,220-of-15,437 MiB headroom (the earlier "~1.2 GiB" line conflated the torch pool
fraction — which excludes the CUDA context — with nvidia-smi totals that include it;
brokkr correction 2026-10-05). Both the pool fraction and `--max-num-seqs` are
tunable, so a leaner build closes the fit; but at parity on accuracy, 2.2× slower
crit21, ~24.8k effective context through the seam, and a floored yes/no distribution,
it stays a lateral move. If the cap is ever revisited, re-derive the gap from the
nvidia-smi totals above, not from the 1.2 GiB figure.
- **Clef-flash** is a net loss on every axis that matters here: −11 JevBench pts (191 - **Clef-flash** is a net loss on every axis that matters here: −11 JevBench pts (191
bf16 / 192 int8), hard tier 72–73/111 vs baseline 83, slower at short shapes bf16 / 192 int8), hard tier 72–73/111 vs baseline 83, slower at short shapes
(bf16 164 vs 85 ms; int8 297 ms), and its 32k context is not real as shipped — the (bf16 164 vs 85 ms; int8 297 ms), and its 32k context is not real as shipped — the
@@ -30,12 +36,19 @@ outside the vLLM pool; medians of 3 run-medians).
| system | jevbench | hard /111 | Brier | battery pooled /398 (Δ vs intern, 95% CI) | rest / peak MiB | crit21 ms | long16 ms | | system | jevbench | hard /111 | Brier | battery pooled /398 (Δ vs intern, 95% CI) | rest / peak MiB | crit21 ms | long16 ms |
|---|---|---|---|---|---|---|---| |---|---|---|---|---|---|---|---|
| **Intern-Decision-4B native** (re-measure) | **202 ×3**, bit-exact vs 09-30 (0 rows flipped over 4+3 repeats) | 83 | 0.178 | 362 (0.9095) baseline | **9,752 / 9,752** — fits | **85** | **214** | | **Intern-Decision-4B native** (re-measure) | **202 ×3**, bit-exact vs 09-30 (0 rows flipped over 4+3 repeats) | 83 | 0.178 | 362 (0.9095) baseline | **9,752 / 9,752**† — fits | **85** | **214** |
| H2O native (vendor vLLM 0.30.0 + shipped shim) | **205 ×3** (exact vendor fig.) | 86 | 0.153 | 367 (0.9221; **+0.0126**, CI −0.015..+0.044) | 15,454 / 15,566 — **over cap at rest** | 184 | 240 | | H2O native (vendor vLLM 0.30.0 + shipped shim) | **205 ×3** (exact vendor fig.) | 86 | 0.153 | 367 (0.9221; **+0.0126**, CI −0.015..+0.044) | 15,454 / 15,566 — **over cap at rest** | 184 | 240 |
| H2O floor-off calibration | 205 — *identical to floored* | — | 0.154 | — | — | — | — | | H2O floor-off calibration | 205 — *identical to floored* | — | 0.154 | — | — | — | — |
| Clef bf16 (vendor loader, uncapped ref) | 191 | 72 | 0.235 | 366 (0.9196; +0.010, CI −0.021..+0.042) | 19,696 rest | 164 (flat; `use_cache=False`) | 390 | | Clef bf16 (vendor loader, uncapped ref) | 191 | 72 | 0.235 | 366 (0.9196; +0.010, CI −0.021..+0.042) | 19,696 rest | 164 (flat; `use_cache=False`) | 390 |
| Clef int8 (bnb, dtype-aligned) | 192 | 73 | 0.235 | 365 (0.9171; +0.0075, CI −0.023..+0.038) | **12,064 rest / 13,216 suite peak** — fits; **20,718 gate peak** | 297 | 446 | | Clef int8 (bnb, dtype-aligned) | 192 | 73 | 0.235 | 365 (0.9171; +0.0075, CI −0.023..+0.038) | **12,064 rest / 13,216 suite peak** — fits; **20,718 gate peak** | 297 | 446 |
† Intern's 9,752 MiB is the capped-suite figure only — this bench's battery never
pushed the baseline through the 32k long-input gate (it is already the live seat and
was gate-proven at that shape on 09-30). H2O/Clef peaks include the 32k gate calls, so
the peak column is not shape-matched for the baseline row. For like-for-like reference:
the live seat's own 32k peak on GPU 1 is 15,220 MiB of a 15,437 MiB budget (n=3,
persistent-memory intern-decision entry).
Row-level: 33/398 battery rows flip intern→H2O, 29/398 intern→clef-bf16, 30/398 Row-level: 33/398 battery rows flip intern→H2O, 29/398 intern→clef-bf16, 30/398
intern→clef-int8; every battery delta straddles zero on a paired group bootstrap intern→clef-int8; every battery delta straddles zero on a paired group bootstrap
(group = set), i.e. at 4B class our battery cannot separate them. The separations are (group = set), i.e. at 4B class our battery cannot separate them. The separations are
@@ -58,6 +71,12 @@ failed rows; H2O 0.937 rotations / 0.893 multifield; intern 0.897 / 0.917; clef-
## Name-vs-definition cell (new; 144 rows, descriptions reversed) ## Name-vs-definition cell (new; 144 rows, descriptions reversed)
**Variant caveat (brokkr, 2026-10-05):** this cell reverses *descriptions* under the
same option names, which with odd option counts leaves the middle option's pair
unmoved — an option can count as both name-followed and def-followed, so the two
columns can sum above 1. It is a reasonable stand-in, **not** the loaded-name variant
of arXiv 2609.26758; read the columns as overlaps, not a partition.
| system | name-follow | def-follow | | system | name-follow | def-follow |
|---|---|---| |---|---|---|
| intern | 0.375 | 0.847 | | intern | 0.375 | 0.847 |
@@ -68,7 +87,8 @@ failed rows; H2O 0.937 rotations / 0.893 multifield; intern 0.897 / 0.917; clef-
Negative control (gold label moved to another option's definition): same-top 0.097 / Negative control (gold label moved to another option's definition): same-top 0.097 /
0.028 / 0.111 / 0.090 respectively — no system is cue-clinging except mildly at 0.028 / 0.111 / 0.090 respectively — no system is cue-clinging except mildly at
baseline level; H2O is the cleanest. All four are definition-followers, not baseline level; H2O is the cleanest. All four are definition-followers, not
name-followers. name-followers — true for the description-reversal manipulation run; the loaded-name
variant is untested here.
## Gates (fit / long-input, per amendment) ## Gates (fit / long-input, per amendment)