From ab3a0ca5bc3f53de7b3b523331f808e8e1e63343 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 22 Aug 2026 00:52:06 -0700 Subject: [PATCH] docs(quant-playbook): acceptance is not throughput -- always run the depth control Measured 2026-08-22 on one target with one instrument: raising MTP num_speculative_tokens from 3 to 7 improved accepted length from 2.753 to 3.041 per forward pass while throughput fell from 114.9 to 74.0 tok/s. Reporting acceptance alone would have recommended a 36% regression. The cause is architectural rather than model-specific. A single-module MTP head has no depth of its own, so vLLM runs it autoregressively and k draft tokens cost k sequential forward passes. Past a shallow depth the drafting cost exceeds what the extra accepted tokens save. Records the comparison rule that follows: match k when comparing two speculative methods, or the measurement is of depth rather than method. A parallel-drafting drafter at k=7 against an autoregressive MTP at k=3 is not a method comparison. In the case that produced this, the depth control showed most of the apparent acceptance advantage was depth, while the throughput advantage was real and came from parallel drafting -- our MTP was better at position 0 and still lost overall. Only the measured, model-agnostic result is recorded here. The DFlash2-specific findings, the hypotheses that remain unproven, and the wrong turns taken along the way live in persistent-memory.d/2026-08-22-dflash2-spec-decode.md with explicit epistemic labels, deliberately kept out of the playbook. --- docs/pfi/model-quantization-playbook.md | 30 +++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/docs/pfi/model-quantization-playbook.md b/docs/pfi/model-quantization-playbook.md index e123d70..7b16943 100644 --- a/docs/pfi/model-quantization-playbook.md +++ b/docs/pfi/model-quantization-playbook.md @@ -541,6 +541,36 @@ thinking split, long-context needle, streaming), `serve_probe.sh`. --- +### 5.1 ⭐⭐ Acceptance is not throughput — always run the DEPTH control + +**Measured 2026-08-22**, same instrument (vLLM's own `spec_decode` counters, delta over a fixed +workload, temp 0), same target, same engine: + +| config | accepted tok/forward | throughput | +|---|---|---| +| MTP k=3 | 2.753 | 114.9 tok/s | +| MTP k=7 | **3.041** ⬆ | **74.0 tok/s** ⬇ | + +**Raising `num_speculative_tokens` improved acceptance and destroyed throughput.** Reporting +acceptance alone would have recommended a 36% regression. + +**Why:** a single-module MTP head (`mtp_num_hidden_layers: 1`, one `mtp.layers.0`) has no depth +of its own — vLLM runs it **autoregressively**, so k draft tokens cost **k sequential forward +passes**. Past a shallow depth the drafting cost exceeds what the extra accepted tokens save. +Check `mtp_num_hidden_layers` before assuming depth is cheap. + +**The rule: when comparing two speculative methods, match k, or you are measuring depth rather +than method.** A parallel-drafting drafter (DFlash2 and kin, which propose a whole block in one +pass) at k=7 versus an autoregressive MTP at k=3 is not a method comparison — the depth control +is what separates them. In our case the control showed most of the apparent acceptance win was +depth, while the *throughput* win was real and came from parallel drafting, not better drafts: +our MTP was **better at position 0** (79.6% vs 75.4%) and still lost overall. + +**Corollary — report both, always.** Acceptance rate, mean accepted length, and end-to-end +tok/s. Any one of the three alone can point the wrong way. + +--- + ## 6. Hardware and co-residency - **ana-ml2 = Blackwell (sm_120)**, 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths