memory: record prefill result (roughly doubled) on the gen-seat mixed requant

2026-08-15 02:34:04 -07:00
parent 4a5c3fcccf
commit 930197a56a
2 changed files with 8 additions and 1 deletions
@@ -89,6 +89,9 @@ names — 0 overlap, MLP union = layers 0-63 — before any GPU time was spent.
 | metric | W4A16 | mixed | delta |
 |---|---|---|---|
 | decode tok/s | 80.12 | **94.53** | **+18.0%** |
+| prefill tok/s (~6.7k prompt) | 3,206 | **6,334** | **+98%** |
+| prefill tok/s (~27k prompt) | 2,862 | **5,085** | **+78%** |
+| TTFT on a ~27k doc | 9.43 s | **5.31 s** | −44% |
 | MTP acceptance | 47.8% | 47.7% | unchanged |
 | perplexity (6 passages) | 6.941 | 7.059 | +1.7% worse |
 | abliteration compliance | 4/4 | 4/4 | preserved |
@@ -107,6 +110,10 @@ routing. Commit `74f596b`.
  uninitialised → 0% acceptance). `post_quant.py` now **re-injects it after the
  graft and re-verifies**. That check *fired on this run* — it was not
  hypothetical.
+- **Prefix caching silently fakes prefill numbers too.** The prefill harness originally used a
+  *seeded* nonce, so run 2 regenerated run 1's prompts verbatim and read **~41k tok/s of
+  cache-hit** instead of ~5k of real prefill. Same class of error as the decode bench. Use
+  `SystemRandom`; never seed a cache-busting nonce.
 - **vLLM's `prompt_logprobs` are garbage while speculative decoding is on** —
  ~uniform over the vocab (median rank ~10⁵, logprob ≈ log(1/vocab); " Paris"
  after "The capital of France is" ranked 69698). Perplexity must be measured on