From a9bae04ba6ffab2aeb18bb64f1eb324d454b5501 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 12 Oct 2026 08:08:39 -0700 Subject: [PATCH] =?UTF-8?q?docs(pfi):=20D3=20vs=20Intern-Decision-4B=20sea?= =?UTF-8?q?t=20bench=20=E2=80=94=20final=20stashed=20findings=20(halted=20?= =?UTF-8?q?by=20operator=20call)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured: bf16 ladder across six sizes, calibrated 32k shape cells with O(q*state) wire accounting, bs=1 memory/latency curve, Engine-vs-vendor fidelity rows, Stage 4 control quant trio (battery monotone in weight error), flash calibrated-NVFP4 with survive rules applied by the letter, and the Amendment 2 GPU 1 budget (16,851 MiB) re-read against every fit clause. Halted before the INC bench-copy arms and mini/27B/small-size NVFP4 arms; stated plainly. Weights kept pending operator ruling (80 GB). --- .../d3-vs-intern-decision-bench-2026-10-12.md | 192 ++++++++++++++++++ 1 file changed, 192 insertions(+) create mode 100644 docs/pfi/d3-vs-intern-decision-bench-2026-10-12.md diff --git a/docs/pfi/d3-vs-intern-decision-bench-2026-10-12.md b/docs/pfi/d3-vs-intern-decision-bench-2026-10-12.md new file mode 100644 index 0000000..c56274f --- /dev/null +++ b/docs/pfi/d3-vs-intern-decision-bench-2026-10-12.md @@ -0,0 +1,192 @@ +# D3 vs Intern-Decision-4B — decision-model seat bench (2026-10-12) + +Ordered by Prime; preregistration `brokkr-smithy` +`docs/surveys/vllm-sr-decision-landscape-2026-10-11/d3-bench-prereg.md` +(c93d7b0, amended b87a7ab), plus Amendment 1 (shape calibration, brokkr) and +Amendment 2 (operator: memory budget = what fits on the seat's GPU, e3dbfac). +Run by infra-hermes on fv-ml1 GPU 3. **HALTED by operator call at 15:07Z** — +"fails on a number of levels… stash the findings and call it." This doc is the +stash: everything below was measured; what was *not* run is stated plainly. + +Harness: jevbench v1.2.16 (public 231 = easy/original/hard), the S1 battery +(authored144 + perturbations108 + cicada W1/W2 + Wyrd, 1,748 paired scorable +rows), and the 10-04 shape-runner lineage with a per-model calibrated 32k +builder. All artifacts: `fv-ml1:/tank/bench/arm-*` and `/tank/bench-jev-2026-10-04` +(the latter untouched, lkraven-owned). + +## Headline + +The incumbent (intern-decision-serve 0.1.3, pinned Intern-Decision-4B) is not +displaced. Every d3 arm that clears the incumbent's accuracy needs either a +GPU budget the seat cannot hold at the worst shape, or fails the latency +contract, or both. The one arm inside the seat's own memory budget at the +worst shape (d3-edge at bs=1) loses 8 battery points and is 28x slower than +INC at 32k-class state. The quantization warrant closes: NVFP4 does not buy +the 32k shape, because the 32k problem is O(q x state) activation and protocol +accounting, not weight bytes. + +## 1. bf16 ladder (Engine primary, `vllm-sr` dev CLI, GPU 3) + +Revisions pinned per arm (edge e51ab425…, lite c91d55d6…, nano ccd7bc1a…, +mini 40391eb6…, flash 06d33616…, d3-27 7f257ad7…). jevbench acc = public 231, +median of 3 repeats (all repeats bit-exact per arm). Battery = labelled-evidence +acc, single condition. Peak VRAM = steady-state serve MiB. p50 = harness per-call +median ms. + +| arm | params | jevbench | battery single | peak MiB | p50 ms | +|---|---|---|---|---|---| +| **INC (baseline)** | 4B | **202** | .910 | 8,822 (32k shape) | ~1.9–2.0s at 32k | +| d3-edge | 0.59B | 158 | .749 | 2,398 | 15 | +| d3-lite | 1.4B | 168 | .852 | 2,676 | 28 | +| d3-nano | 3.8B | 175 | .907 | 5,304 | 29 | +| d3-mini | 7.3B | 188 | .945 | 10,540 | 36 | +| d3-flash | 8.39B | 190 | .947 | 17,600 | 37 | +| d3-27 (vendor path) | 26.09B | 206 | .970 | 50,300 | — | + +INC reproduction: 202/231 x3, bit-exact against the 10-04 record (`out/intern-native`). +d3-27 is the only arm that beats INC on the public suite, and it is a path +divergence: the Engine's placement estimator (4B/param + 2 GiB) refuses a 26B +model on any single card we have; the vendor `d3_server.py` runs it at 50.3 GiB. +Gate (jevbench >= INC - 3): PASS at 27B only. + +Argmax cross-check (vendor vs Engine, same snapshot, 231 items): edge 230, +lite 229, nano 229, mini 231, flash 231 — mismatches are borderline near-ties +(max prob gap 2.7e-2). Engine fidelity: argmax identical on all non-borderline +items; probabilities NOT exact (median gap 4.2e-4). The card's "matches exactly" +claim is false as written; argmax-equivalence holds. + +## 2. 32k shape cells (Amendment 1: per-model calibrated to ±2% of 32,768 realised tokens) + +Builder length iterated per model in its own tokenizer; every arm converged to +32,743 realised tokens for the 1q state. 16q cells carry 16 x 32,743 + framing = +**523,900 wire tokens** because the d3 wire protocol counts state once PER +QUESTION — O(q x state) accounting, measured, not predicted. INC's equivalent +is a single forward over one state. cold = first request after ready; launch to +ready wall time is reported separately below and is NOT folded into cold. + +| arm | crit21_short cold ms | 1q32k cold ms | 16q32k cold ms | 16q32k peak MiB | +|---|---|---|---|---| +| INC | (2-call 16+5, see note) | 1,970 (32,715 tok) | **rejected: >32,768 cap at 33,324; maxfit 16q = 32,708 tok, 1,960 ms** | 8,822 | +| d3-edge | 1,242.6 | 895.4 | 20,790 | 28,404 | +| d3-lite | 89.1 | 1,139.5 | 18,668 | 28,938 | +| d3-nano | 103.3 | 1,442.1 | 23,568 | 33,486 | +| d3-mini | 157.1 | 3,565.8 | **deadline-bound** (503 at 30.002s, all retries) | 71,404 | +| d3-flash | 242.3 | 4,469.5 | **deadline-bound** | 80,396 | +| d3-27 vendor | 642.7 | 8,731.0 | 147,724 (366,172 tok, builder cap) | 96,262 | + +Notes. (a) INC crit21 on 10-04 is a client-side 16+5 split summed across two +`/v1/systemone` calls (`bench_shape.py --max-questions 16`); `/v1/systemone` +enforces 1..16 questions, so a 21-question crit is structurally out of contract. +(b) The Engine's 30 s serving deadline is a configuration of the seat, not a +bug; the wall sits between 3.8B (nano completes at 23.6s) and 7.3B (mini). +(c) Activation peaks include 32k state at bs=8 (Engine BATCH_SIZE is hard-coded +8, `family.py:65`); every bs=8 arm exceeds even Amendment 2's budget at 16q32k. +(d) Latency clause (16q32k <= 1.5x INC): INC's honest 32k-class cell is 1.96s, +so the clause bar is ~2.9s. NO d3 arm meets it; closest is edge-b1 at 13.0s (6.6x). + +## 3. bs=1 curve (vendor `--batch-size 1`, the only memory knob) + +| arm | 16q32k peak MiB (bs=8 -> bs=1) | 16q32k cold ms (bs=8 -> bs=1) | +|---|---|---| +| d3-edge | 28,404 -> 4,778 | 20,790 -> 13,019 | +| d3-mini | 71,404 -> 18,514 | deadline -> 53,770 | + +The knob buys memory, never time: latency is protocol-intrinsic O(q x state). +No setting puts a mini-class model inside the seat at the worst shape (mini-b1 +still misses Amendment 2's budget by 1.6 GiB *and* fails latency at 18x INC). + +## 4. Stage 4 — control quant trio and the battery sensitivity curve + +d3-lite backbone, RTN (round-to-nearest, no calibration), vendor path, +`--verify none` (the manifest hashes the main shard; modified packages can't +self-verify). Paired deltas vs the path-matched bf16 vendor baseline, 1,748 +rows, 95% bootstrap CI, Wyrd split included: + +| arm | weight rel-L2 | jevbench | battery | paired delta [CI] | Wyrd delta [CI] | argmax flips | +|---|---|---|---|---|---|---| +| lite bf16 vendor (control) | 0 | 168 | .852 | — | — | — | +| lite NVFP4 RTN | 9.4% | 164 | .869 | +0.0034 [-0.0086, +0.0154] | -0.019 [-0.055, +0.017] | 117 | +| lite INT3 RTN | 19.9% | 158 | .739 | -0.095 [-0.118, -0.072] | -0.131 [-0.176, -0.088] | 476 | + +The battery gate is monotone in weight error and functional: it cannot see 9.4% +(NVFP4 RTN lands inside noise), it clearly sees 19.9% (INT3 collapses). This is +the calibration warrant's premise confirmed: a quant that survives must move +weights *less* than 9.4% AND keep paired deltas inside the band — i.e. the +headroom to demonstrate is measured, and it is small. + +## 5. Stage 5 — flash calibrated NVFP4 (the one arm completed before the halt) + +ModelOpt `NVFP4_DEFAULT_CFG`, calibrated on the shared 512-row corpus (sha256 +e7aa7b18…a31e, seeded `jev-nvfp4-calib-2026-10-12`, no scored item). Served +in-process over a `/v1/systemone` shim (`stage5_serve.py`, prepare/run/respond +mirroring the vendor contract, 1,405 quantizers): ModelOpt's fake-quant lives +in the quantizer forward, so no package rewrite; accuracy is a true fake-quant +read, memory is analytic + the byte table below. + +jevbench 194 x3 (bf16 Engine control 190). Survive rules applied by the letter, +paired vs `arm-d3-flash`, n=1,748: + +| rule | result | verdict | +|---|---|---| +| (a) battery pooled delta, CI lower bound >= -0.02 | -0.0011 [-0.0086, +0.0057] | PASS | +| (b) public suite within ±3 of bf16 median | +4 (194 vs 190) | **FAIL by the letter** (upward; 10 flips: 6 up, 2 down, 2 both-wrong) | +| (c) Wyrd within -0.02 | +0.0071 [-0.0119, +0.0262] | PASS | +| (d) argmax flips <= 4 of 231, both directions | 10 | **FAIL by the letter** | + +Byte table (bf16 snapshot index, total 15.63 GiB on disk): quantized linears +(attn 3,980 + mlp 9,216 MiB) at 4.5 b/w = 0.28125x -> 3,711 MiB; kept wide +(embed+head 1,940, vision 870, other 2) = 2,812 MiB. **Projected export 6,524 +MiB (~6.37 GiB)** — single-digit, as expected. ModelOpt `export_hf_checkpoint` +exists for a measured-export ground truth; not run before the halt. +Latency of the fake-quant arm is dequant-matmul timing and is NOT reported as a +speedup. **Analytic 32k verdict unchanged: quantization does not touch the +O(q x state) activation cost; the 16q32k cell fails identically at any weight width.** + +## 6. Amendment 2 — the seat GPU budget (measured read-only, GPU 1) + +Prime: "our budget is now whatever fits in the current gpu where it's running." + +- GPU 1 total: **97,887 MiB.** +- Non-seat processes: `vllm-meromero-r` 43,994 MiB + `vllm-gen-small` 34,994 MiB = 78,988 MiB. +- Seat (intern-decision) resting: 8,812 MiB. +- Safety margin: 2,048 MiB (brokkr's suggested figure; no seat headroom policy exists). +- **Budget: 16,851 MiB (~16.5 GiB).** + +Re-read of every fit clause against 16,851 MiB: +- Short shape: fits INC-class (8,822 measured) and every arm through mini + (10,540); NVFP4-flash analytic ~6,524 fits; flash bf16 (17,600) misses by + 749 MiB; 27B (50,300) does not. +- 16q32k: only edge-b1 (4,778) and INC-class fit. Every bs=8 arm is 27.6–96.3 + GiB. The latency clause independently kills mini/flash/INC at that shape. +- Net: the wider budget upgrades mini from "memory-out" to "memory-in, + latency-out" (53.8s at the only knob setting that fits), and leaves the + conclusion unchanged. + +## 7. What was NOT run because of the halt + +- INC bench-copy arms (positive control `inc-bf16-copy` and fake-quant + `inc-fakequant`, 249 language tensors RTN-NVFP4, median rel-L2 9.4%, vision + kept bf16): the 10-04 `intern_server.py` shim and both packages were built + and the arm was mid-boot when the halt landed. The INC fake-quant row is + therefore absent — note the production engine cannot load NVFP4 at all + (`intern_decision_serve/engine.py` pins the snapshot's `inference.py` by hash + and validates dtype against {bfloat16, float16, float32} with an explicit + raise; no compressed-tensors path exists in the image). +- NVFP4 arms for mini, 27B (fake-quant), nano, lite (calibrated), edge. +- ModelOpt measured export (ground-truth on-disk size). +- Paired battery deltas + Wyrd splits for the bf16 ladder arms (no quantized + sibling to pair against; would need battery re-runs). + +## 8. Teardown & disk + +Engine stack stopped, all bench servers on GPU 3 stopped, `:18080` iptables rule +removed, guard read confirms GPU 3 at 2 MiB. Production GPU-0/1 seats untouched +(one ~51 s exposure window during cross-checks earlier in the run, zero +degradation observed — all RestartCount=0, no Xid). + +Weights KEPT per instruction (Prime rules on deletion): +`/tank/bench/models/model-runtime` **80 GB** (all six d3 snapshots + INC copy +staged), plus bench working set: `inc-bf16-copy` 8.5 GB, `inc-fakequant` 8.5 GB, +`quant-flash-nvfp4` 16 GB, `quant-lite-nvfp4` 1.7 GB, `quant-lite-int3` 1.4 GB. +`/tank/bench` total 180 GB; `/tank` at 62% with 3.3 TB free, so this is +comfort, not pressure.