docs(pfi): D3 vs Intern-Decision-4B seat bench — final stashed findings (halted by operator call)

Measured: bf16 ladder across six sizes, calibrated 32k shape cells with
O(q*state) wire accounting, bs=1 memory/latency curve, Engine-vs-vendor
fidelity rows, Stage 4 control quant trio (battery monotone in weight error),
flash calibrated-NVFP4 with survive rules applied by the letter, and the
Amendment 2 GPU 1 budget (16,851 MiB) re-read against every fit clause.
Halted before the INC bench-copy arms and mini/27B/small-size NVFP4 arms;
stated plainly. Weights kept pending operator ruling (80 GB).

docs/pfi/d3-vs-intern-decision-bench-2026-10-12.md

@@ -0,0 +1,192 @@
+# D3 vs Intern-Decision-4B — decision-model seat bench (2026-10-12)
+
+Ordered by Prime; preregistration `brokkr-smithy`
+`docs/surveys/vllm-sr-decision-landscape-2026-10-11/d3-bench-prereg.md`
+(c93d7b0, amended b87a7ab), plus Amendment 1 (shape calibration, brokkr) and
+Amendment 2 (operator: memory budget = what fits on the seat's GPU, e3dbfac).
+Run by infra-hermes on fv-ml1 GPU 3. **HALTED by operator call at 15:07Z** —
+"fails on a number of levels… stash the findings and call it." This doc is the
+stash: everything below was measured; what was *not* run is stated plainly.
+
+Harness: jevbench v1.2.16 (public 231 = easy/original/hard), the S1 battery
+(authored144 + perturbations108 + cicada W1/W2 + Wyrd, 1,748 paired scorable
+rows), and the 10-04 shape-runner lineage with a per-model calibrated 32k
+builder. All artifacts: `fv-ml1:/tank/bench/arm-*` and `/tank/bench-jev-2026-10-04`
+(the latter untouched, lkraven-owned).
+
+## Headline
+
+The incumbent (intern-decision-serve 0.1.3, pinned Intern-Decision-4B) is not
+displaced. Every d3 arm that clears the incumbent's accuracy needs either a
+GPU budget the seat cannot hold at the worst shape, or fails the latency
+contract, or both. The one arm inside the seat's own memory budget at the
+worst shape (d3-edge at bs=1) loses 8 battery points and is 28x slower than
+INC at 32k-class state. The quantization warrant closes: NVFP4 does not buy
+the 32k shape, because the 32k problem is O(q x state) activation and protocol
+accounting, not weight bytes.
+
+## 1. bf16 ladder (Engine primary, `vllm-sr` dev CLI, GPU 3)
+
+Revisions pinned per arm (edge e51ab425…, lite c91d55d6…, nano ccd7bc1a…,
+mini 40391eb6…, flash 06d33616…, d3-27 7f257ad7…). jevbench acc = public 231,
+median of 3 repeats (all repeats bit-exact per arm). Battery = labelled-evidence
+acc, single condition. Peak VRAM = steady-state serve MiB. p50 = harness per-call
+median ms.
+
+| arm | params | jevbench | battery single | peak MiB | p50 ms |
+|---|---|---|---|---|---|
+| **INC (baseline)** | 4B | **202** | .910 | 8,822 (32k shape) | ~1.9–2.0s at 32k |
+| d3-edge | 0.59B | 158 | .749 | 2,398 | 15 |
+| d3-lite | 1.4B | 168 | .852 | 2,676 | 28 |
+| d3-nano | 3.8B | 175 | .907 | 5,304 | 29 |
+| d3-mini | 7.3B | 188 | .945 | 10,540 | 36 |
+| d3-flash | 8.39B | 190 | .947 | 17,600 | 37 |
+| d3-27 (vendor path) | 26.09B | 206 | .970 | 50,300 | — |
+
+INC reproduction: 202/231 x3, bit-exact against the 10-04 record (`out/intern-native`).
+d3-27 is the only arm that beats INC on the public suite, and it is a path
+divergence: the Engine's placement estimator (4B/param + 2 GiB) refuses a 26B
+model on any single card we have; the vendor `d3_server.py` runs it at 50.3 GiB.
+Gate (jevbench >= INC - 3): PASS at 27B only.
+
+Argmax cross-check (vendor vs Engine, same snapshot, 231 items): edge 230,
+lite 229, nano 229, mini 231, flash 231 — mismatches are borderline near-ties
+(max prob gap 2.7e-2). Engine fidelity: argmax identical on all non-borderline
+items; probabilities NOT exact (median gap 4.2e-4). The card's "matches exactly"
+claim is false as written; argmax-equivalence holds.
+
+## 2. 32k shape cells (Amendment 1: per-model calibrated to ±2% of 32,768 realised tokens)
+
+Builder length iterated per model in its own tokenizer; every arm converged to
+32,743 realised tokens for the 1q state. 16q cells carry 16 x 32,743 + framing =
+**523,900 wire tokens** because the d3 wire protocol counts state once PER
+QUESTION — O(q x state) accounting, measured, not predicted. INC's equivalent
+is a single forward over one state. cold = first request after ready; launch to
+ready wall time is reported separately below and is NOT folded into cold.
+
+| arm | crit21_short cold ms | 1q32k cold ms | 16q32k cold ms | 16q32k peak MiB |
+|---|---|---|---|---|
+| INC | (2-call 16+5, see note) | 1,970 (32,715 tok) | **rejected: >32,768 cap at 33,324; maxfit 16q = 32,708 tok, 1,960 ms** | 8,822 |
+| d3-edge | 1,242.6 | 895.4 | 20,790 | 28,404 |
+| d3-lite | 89.1 | 1,139.5 | 18,668 | 28,938 |
+| d3-nano | 103.3 | 1,442.1 | 23,568 | 33,486 |
+| d3-mini | 157.1 | 3,565.8 | **deadline-bound** (503 at 30.002s, all retries) | 71,404 |
+| d3-flash | 242.3 | 4,469.5 | **deadline-bound** | 80,396 |
+| d3-27 vendor | 642.7 | 8,731.0 | 147,724 (366,172 tok, builder cap) | 96,262 |
+
+Notes. (a) INC crit21 on 10-04 is a client-side 16+5 split summed across two
+`/v1/systemone` calls (`bench_shape.py --max-questions 16`); `/v1/systemone`
+enforces 1..16 questions, so a 21-question crit is structurally out of contract.
+(b) The Engine's 30 s serving deadline is a configuration of the seat, not a
+bug; the wall sits between 3.8B (nano completes at 23.6s) and 7.3B (mini).
+(c) Activation peaks include 32k state at bs=8 (Engine BATCH_SIZE is hard-coded
+8, `family.py:65`); every bs=8 arm exceeds even Amendment 2's budget at 16q32k.
+(d) Latency clause (16q32k <= 1.5x INC): INC's honest 32k-class cell is 1.96s,
+so the clause bar is ~2.9s. NO d3 arm meets it; closest is edge-b1 at 13.0s (6.6x).
+
+## 3. bs=1 curve (vendor `--batch-size 1`, the only memory knob)
+
+| arm | 16q32k peak MiB (bs=8 -> bs=1) | 16q32k cold ms (bs=8 -> bs=1) |
+|---|---|---|
+| d3-edge | 28,404 -> 4,778 | 20,790 -> 13,019 |
+| d3-mini | 71,404 -> 18,514 | deadline -> 53,770 |
+
+The knob buys memory, never time: latency is protocol-intrinsic O(q x state).
+No setting puts a mini-class model inside the seat at the worst shape (mini-b1
+still misses Amendment 2's budget by 1.6 GiB *and* fails latency at 18x INC).
+
+## 4. Stage 4 — control quant trio and the battery sensitivity curve
+
+d3-lite backbone, RTN (round-to-nearest, no calibration), vendor path,
+`--verify none` (the manifest hashes the main shard; modified packages can't
+self-verify). Paired deltas vs the path-matched bf16 vendor baseline, 1,748
+rows, 95% bootstrap CI, Wyrd split included:
+
+| arm | weight rel-L2 | jevbench | battery | paired delta [CI] | Wyrd delta [CI] | argmax flips |
+|---|---|---|---|---|---|---|
+| lite bf16 vendor (control) | 0 | 168 | .852 | — | — | — |
+| lite NVFP4 RTN | 9.4% | 164 | .869 | +0.0034 [-0.0086, +0.0154] | -0.019 [-0.055, +0.017] | 117 |
+| lite INT3 RTN | 19.9% | 158 | .739 | -0.095 [-0.118, -0.072] | -0.131 [-0.176, -0.088] | 476 |
+
+The battery gate is monotone in weight error and functional: it cannot see 9.4%
+(NVFP4 RTN lands inside noise), it clearly sees 19.9% (INT3 collapses). This is
+the calibration warrant's premise confirmed: a quant that survives must move
+weights *less* than 9.4% AND keep paired deltas inside the band — i.e. the
+headroom to demonstrate is measured, and it is small.
+
+## 5. Stage 5 — flash calibrated NVFP4 (the one arm completed before the halt)
+
+ModelOpt `NVFP4_DEFAULT_CFG`, calibrated on the shared 512-row corpus (sha256
+e7aa7b18…a31e, seeded `jev-nvfp4-calib-2026-10-12`, no scored item). Served
+in-process over a `/v1/systemone` shim (`stage5_serve.py`, prepare/run/respond
+mirroring the vendor contract, 1,405 quantizers): ModelOpt's fake-quant lives
+in the quantizer forward, so no package rewrite; accuracy is a true fake-quant
+read, memory is analytic + the byte table below.
+
+jevbench 194 x3 (bf16 Engine control 190). Survive rules applied by the letter,
+paired vs `arm-d3-flash`, n=1,748:
+
+| rule | result | verdict |
+|---|---|---|
+| (a) battery pooled delta, CI lower bound >= -0.02 | -0.0011 [-0.0086, +0.0057] | PASS |
+| (b) public suite within ±3 of bf16 median | +4 (194 vs 190) | **FAIL by the letter** (upward; 10 flips: 6 up, 2 down, 2 both-wrong) |
+| (c) Wyrd within -0.02 | +0.0071 [-0.0119, +0.0262] | PASS |
+| (d) argmax flips <= 4 of 231, both directions | 10 | **FAIL by the letter** |
+
+Byte table (bf16 snapshot index, total 15.63 GiB on disk): quantized linears
+(attn 3,980 + mlp 9,216 MiB) at 4.5 b/w = 0.28125x -> 3,711 MiB; kept wide
+(embed+head 1,940, vision 870, other 2) = 2,812 MiB. **Projected export 6,524
+MiB (~6.37 GiB)** — single-digit, as expected. ModelOpt `export_hf_checkpoint`
+exists for a measured-export ground truth; not run before the halt.
+Latency of the fake-quant arm is dequant-matmul timing and is NOT reported as a
+speedup. **Analytic 32k verdict unchanged: quantization does not touch the
+O(q x state) activation cost; the 16q32k cell fails identically at any weight width.**
+
+## 6. Amendment 2 — the seat GPU budget (measured read-only, GPU 1)
+
+Prime: "our budget is now whatever fits in the current gpu where it's running."
+
+- GPU 1 total: **97,887 MiB.**
+- Non-seat processes: `vllm-meromero-r` 43,994 MiB + `vllm-gen-small` 34,994 MiB = 78,988 MiB.
+- Seat (intern-decision) resting: 8,812 MiB.
+- Safety margin: 2,048 MiB (brokkr's suggested figure; no seat headroom policy exists).
+- **Budget: 16,851 MiB (~16.5 GiB).**
+
+Re-read of every fit clause against 16,851 MiB:
+- Short shape: fits INC-class (8,822 measured) and every arm through mini
+  (10,540); NVFP4-flash analytic ~6,524 fits; flash bf16 (17,600) misses by
+  749 MiB; 27B (50,300) does not.
+- 16q32k: only edge-b1 (4,778) and INC-class fit. Every bs=8 arm is 27.6–96.3
+  GiB. The latency clause independently kills mini/flash/INC at that shape.
+- Net: the wider budget upgrades mini from "memory-out" to "memory-in,
+  latency-out" (53.8s at the only knob setting that fits), and leaves the
+  conclusion unchanged.
+
+## 7. What was NOT run because of the halt
+
+- INC bench-copy arms (positive control `inc-bf16-copy` and fake-quant
+  `inc-fakequant`, 249 language tensors RTN-NVFP4, median rel-L2 9.4%, vision
+  kept bf16): the 10-04 `intern_server.py` shim and both packages were built
+  and the arm was mid-boot when the halt landed. The INC fake-quant row is
+  therefore absent — note the production engine cannot load NVFP4 at all
+  (`intern_decision_serve/engine.py` pins the snapshot's `inference.py` by hash
+  and validates dtype against {bfloat16, float16, float32} with an explicit
+  raise; no compressed-tensors path exists in the image).
+- NVFP4 arms for mini, 27B (fake-quant), nano, lite (calibrated), edge.
+- ModelOpt measured export (ground-truth on-disk size).
+- Paired battery deltas + Wyrd splits for the bf16 ladder arms (no quantized
+  sibling to pair against; would need battery re-runs).
+
+## 8. Teardown & disk
+
+Engine stack stopped, all bench servers on GPU 3 stopped, `:18080` iptables rule
+removed, guard read confirms GPU 3 at 2 MiB. Production GPU-0/1 seats untouched
+(one ~51 s exposure window during cross-checks earlier in the run, zero
+degradation observed — all RestartCount=0, no Xid).
+
+Weights KEPT per instruction (Prime rules on deletion):
+`/tank/bench/models/model-runtime` **80 GB** (all six d3 snapshots + INC copy
+staged), plus bench working set: `inc-bf16-copy` 8.5 GB, `inc-fakequant` 8.5 GB,
+`quant-flash-nvfp4` 16 GB, `quant-lite-nvfp4` 1.7 GB, `quant-lite-int3` 1.4 GB.
+`/tank/bench` total 180 GB; `/tank` at 62% with 3.3 TB free, so this is
+comfort, not pressure.