fix(quant): stop baking the calibration truncation cap into the shipped tokenizer

load_calib tokenizes with tok(..., truncation=True, max_length=seqlen). For a
fast tokenizer that mutates the Rust backend's truncation state in place, and
the subsequent tok.save_pretrained() persisted it, so every mixed-NVFP4 build
shipped a tokenizer.json carrying

  "truncation": {"direction": "Right", "max_length": 2048, ...}

against a source whose value is null. Every prompt was clamped at the
calibration length, permanently.

It hid because older transformers does not enforce the text-vs-ids count
check. On a newer one the seat dies at startup with a message that names
images and never mentions tokenizers:

  ValueError: Mismatch in `image` token count between text and `input_ids`.
  Got ids=[2047] and text=[16384].

The cap also silently limited image resolution well before it killed
anything -- at 2048 the largest servable image is about 1448x1448, since
(edge/patch)^2 / merge^2 image tokens have to fit under it.

Fix saves a pristine tokenizer re-read from the source rather than the
mutated calibration object, and then asserts truncation is null so the
defect fails the build instead of shipping again.

Playbook gains section 3.14 with the symptom, the cause, the audit one-liner
and a table of which builds were affected, plus a fourth mandatory post-step.
The transferable lesson is called out: this is the third case of an artifact
carrying config authored against an older transformers that a newer one
begins enforcing, so an image bump is a config-compatibility event rather
than just a version change.
This commit is contained in:
2026-08-22 00:31:59 -07:00
parent ad21302474
commit 0755ba7d00
2 changed files with 108 additions and 2 deletions
@@ -30,7 +30,7 @@ MTP MUST stay in `ignore` -- otherwise vLLM loads the grafted bf16 MTP head as
quantized and it comes up uninitialised (0% acceptance). That bug cost two prior
rounds; do not remove `re:^mtp.*`.
"""
import argparse, json, sys
import argparse, json, os, sys
# --- group_0: FP8 W8A8 -------------------------------------------------------
G0_TARGETS = [
@@ -163,7 +163,32 @@ def main():
print(f"saving -> {a.out}", flush=True)
model.save_pretrained(a.out, save_compressed=True)
tok.save_pretrained(a.out)
# ⚠ DO NOT `tok.save_pretrained(a.out)` — that ships a CRIPPLED tokenizer.
# `load_calib` calls tok(..., truncation=True, max_length=seqlen), which
# MUTATES the fast tokenizer's Rust backend truncation state in place.
# save_pretrained then bakes {"truncation": {"max_length": <seqlen>}} into
# tokenizer.json, so every prompt is silently clamped at the CALIBRATION
# length forever. Latent on older transformers (it does not enforce the
# check) and fatal on newer ones: a vision model dies at startup because the
# dummy profiling image expands to more image tokens than the cap allows
# ("Mismatch in `image` token count ... Got ids=[<cap-1>]").
# Save a pristine tokenizer re-read from the SOURCE instead.
from transformers import AutoTokenizer as _AutoTokenizer
_AutoTokenizer.from_pretrained(a.model, trust_remote_code=True).save_pretrained(a.out)
# Fail loudly rather than shipping the defect again.
_tj = os.path.join(a.out, "tokenizer.json")
if os.path.exists(_tj):
with open(_tj) as _f:
_trunc = json.load(_f).get("truncation")
if _trunc:
raise SystemExit(
f"FAILED CHECK: saved tokenizer.json carries truncation={_trunc}. "
"It must be null — see quant playbook §3.14."
)
print("tokenizer saved pristine (truncation=null) — verified", flush=True)
print("DONE (post-steps still required: graft MTP, preprocessor_config, verify ignore)",
flush=True)
return 0