diff --git a/docs/pfi/model-quantization-playbook.md b/docs/pfi/model-quantization-playbook.md index e123d70..7b16943 100644 --- a/docs/pfi/model-quantization-playbook.md +++ b/docs/pfi/model-quantization-playbook.md @@ -541,6 +541,36 @@ thinking split, long-context needle, streaming), `serve_probe.sh`. --- +### 5.1 ⭐⭐ Acceptance is not throughput — always run the DEPTH control + +**Measured 2026-08-22**, same instrument (vLLM's own `spec_decode` counters, delta over a fixed +workload, temp 0), same target, same engine: + +| config | accepted tok/forward | throughput | +|---|---|---| +| MTP k=3 | 2.753 | 114.9 tok/s | +| MTP k=7 | **3.041** ⬆ | **74.0 tok/s** ⬇ | + +**Raising `num_speculative_tokens` improved acceptance and destroyed throughput.** Reporting +acceptance alone would have recommended a 36% regression. + +**Why:** a single-module MTP head (`mtp_num_hidden_layers: 1`, one `mtp.layers.0`) has no depth +of its own — vLLM runs it **autoregressively**, so k draft tokens cost **k sequential forward +passes**. Past a shallow depth the drafting cost exceeds what the extra accepted tokens save. +Check `mtp_num_hidden_layers` before assuming depth is cheap. + +**The rule: when comparing two speculative methods, match k, or you are measuring depth rather +than method.** A parallel-drafting drafter (DFlash2 and kin, which propose a whole block in one +pass) at k=7 versus an autoregressive MTP at k=3 is not a method comparison — the depth control +is what separates them. In our case the control showed most of the apparent acceptance win was +depth, while the *throughput* win was real and came from parallel drafting, not better drafts: +our MTP was **better at position 0** (79.6% vs 75.4%) and still lost overall. + +**Corollary — report both, always.** Acceptance rate, mean accepted length, and end-to-end +tok/s. Any one of the three alone can point the wrong way. + +--- + ## 6. Hardware and co-residency - **ana-ml2 = Blackwell (sm_120)**, 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths