fix(quant): stop baking the calibration truncation cap into the shipped tokenizer
load_calib tokenizes with tok(..., truncation=True, max_length=seqlen). For a
fast tokenizer that mutates the Rust backend's truncation state in place, and
the subsequent tok.save_pretrained() persisted it, so every mixed-NVFP4 build
shipped a tokenizer.json carrying
"truncation": {"direction": "Right", "max_length": 2048, ...}
against a source whose value is null. Every prompt was clamped at the
calibration length, permanently.
It hid because older transformers does not enforce the text-vs-ids count
check. On a newer one the seat dies at startup with a message that names
images and never mentions tokenizers:
ValueError: Mismatch in `image` token count between text and `input_ids`.
Got ids=[2047] and text=[16384].
The cap also silently limited image resolution well before it killed
anything -- at 2048 the largest servable image is about 1448x1448, since
(edge/patch)^2 / merge^2 image tokens have to fit under it.
Fix saves a pristine tokenizer re-read from the source rather than the
mutated calibration object, and then asserts truncation is null so the
defect fails the build instead of shipping again.
Playbook gains section 3.14 with the symptom, the cause, the audit one-liner
and a table of which builds were affected, plus a fourth mandatory post-step.
The transferable lesson is called out: this is the third case of an artifact
carrying config authored against an older transformers that a newer one
begins enforcing, so an image bump is a config-compatibility event rather
than just a version change.
This commit is contained in:
@@ -392,6 +392,84 @@ fallback, the incumbent-vs-candidate A/Bs (47.2% acceptance, PPL 6.910, and the
|
||||
2026-08-20 Heretic-300 build) are apples-to-apples. This is unrealised upside, not a
|
||||
correction to past numbers.
|
||||
|
||||
### 3.14 ⭐⭐ Calibration BAKES a truncation cap into the shipped tokenizer
|
||||
|
||||
**Symptom (on a newer transformers, at startup, on a vision model):**
|
||||
|
||||
```
|
||||
ValueError: Mismatch in `image` token count between text and `input_ids`.
|
||||
Got ids=[2047] and text=[16384]. Likely due to `truncation='max_length'`.
|
||||
```
|
||||
|
||||
The engine never serves a request. The number in `ids=[…]` is your **calibration seqlen minus
|
||||
one**, which is the tell.
|
||||
|
||||
**Cause — an in-place mutation you never wrote.** Calibration tokenizes like this:
|
||||
|
||||
```python
|
||||
tok(b["text"], truncation=True, max_length=seqlen, add_special_tokens=False)
|
||||
```
|
||||
|
||||
For a **fast** tokenizer that call does not just return ids — it **mutates the Rust backend's
|
||||
truncation state in place**. A later `tok.save_pretrained(out)` then persists it:
|
||||
|
||||
```json
|
||||
"truncation": {"direction": "Right", "max_length": 2048, "strategy": "LongestFirst", "stride": 0}
|
||||
```
|
||||
|
||||
The source model has `"truncation": null`. **You shipped a tokenizer that clamps every prompt at
|
||||
the calibration length, permanently.**
|
||||
|
||||
**Why it hid for months.** Older transformers does not enforce the text-vs-ids count check, so
|
||||
the cap sits latent — the model serves, gates pass, vision works, nothing logs. It only detonates
|
||||
when you bump the image, and then it presents as a *vision* bug at startup with no mention of
|
||||
tokenizers. It also caps the effective image resolution long before it kills the seat: at a 2048
|
||||
cap the largest servable image is ~1448×1448, because `(edge/patch)² / merge²` image tokens must
|
||||
fit under it.
|
||||
|
||||
**The fix — never save the calibration tokenizer.** Re-read a pristine one from the source:
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer as _AutoTokenizer
|
||||
_AutoTokenizer.from_pretrained(a.model, trust_remote_code=True).save_pretrained(a.out)
|
||||
```
|
||||
|
||||
then **assert** it, because this is exactly the class of defect that returns silently:
|
||||
|
||||
```python
|
||||
if json.load(open(f"{a.out}/tokenizer.json")).get("truncation"):
|
||||
raise SystemExit("FAILED CHECK: saved tokenizer carries a truncation cap")
|
||||
```
|
||||
|
||||
Both live in `quant_mixed_nvfp4.py` as of 2026-08-22.
|
||||
|
||||
**Audit any build predating that.** One line per model:
|
||||
|
||||
```bash
|
||||
python3 -c 'import json,sys;print(json.load(open(sys.argv[1]+"/tokenizer.json")).get("truncation"))' <model_dir>
|
||||
```
|
||||
|
||||
Measured 2026-08-22 — every mixed-NVFP4 build from this pipeline was affected, and the two live
|
||||
ones were corrected in place (backup `tokenizer.json.bak-truncation-20260822`; only the
|
||||
`truncation` field changed, vocab and `added_tokens` byte-identical):
|
||||
|
||||
| build | truncation as found |
|
||||
|---|---|
|
||||
| `qwen38-27b-orcarouter-nvfp4-mixed` (live `gen`) | **2048** → fixed |
|
||||
| `mog-sec-27b-nvfp4-mixed` (live `sec`) | **2048** → fixed |
|
||||
| `qwen38-27b-heresy-nvfp4-mixed` (retired) | 2048, left as-is |
|
||||
| `G4-MeroMero-v2-31B-NVFP4A16` (different pipeline) | `null` ✓ |
|
||||
| `mog-sec-27b-bf16` (source) | `null` ✓ |
|
||||
|
||||
**Editing it is safe on a running seat** — vLLM reads the tokenizer at startup and holds its own
|
||||
copy, so the fix lands on the next restart with no disruption.
|
||||
|
||||
**The general lesson, which is the transferable part:** this is the third defect in this playbook
|
||||
where *the artifact carries config authored against an older transformers and a newer one starts
|
||||
enforcing it* (see also the Gemma-4 heterogeneous `head_dim`). **Treat "we bumped the image" as a
|
||||
config-compatibility event, not just a version change** — and prefer saving artifacts re-read
|
||||
from the source over saving objects the pipeline has touched.
|
||||
|
||||
---
|
||||
|
||||
## 4. Pipeline shape
|
||||
@@ -416,6 +494,9 @@ Never optional, always in this order, and the last one **verifies rather than as
|
||||
1. Graft `model-mtp.safetensors` + register its tensors in the output index.
|
||||
2. Restore `preprocessor_config.json` / `processor_config.json` / `video_preprocessor_config.json`.
|
||||
3. **Re-inject `re:^mtp.*` into `quantization_config.ignore` and confirm it is there** (§3.3).
|
||||
4. **Confirm the saved `tokenizer.json` has `truncation: null`** (§3.14) — calibration mutates the
|
||||
fast tokenizer in place and `save_pretrained` bakes the cap in. Latent on an older
|
||||
transformers, fatal on a newer one.
|
||||
|
||||
Reference implementation: `services/gen-seat-mixed-quant/post_quant.py`.
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@ MTP MUST stay in `ignore` -- otherwise vLLM loads the grafted bf16 MTP head as
|
||||
quantized and it comes up uninitialised (0% acceptance). That bug cost two prior
|
||||
rounds; do not remove `re:^mtp.*`.
|
||||
"""
|
||||
import argparse, json, sys
|
||||
import argparse, json, os, sys
|
||||
|
||||
# --- group_0: FP8 W8A8 -------------------------------------------------------
|
||||
G0_TARGETS = [
|
||||
@@ -163,7 +163,32 @@ def main():
|
||||
|
||||
print(f"saving -> {a.out}", flush=True)
|
||||
model.save_pretrained(a.out, save_compressed=True)
|
||||
tok.save_pretrained(a.out)
|
||||
|
||||
# ⚠ DO NOT `tok.save_pretrained(a.out)` — that ships a CRIPPLED tokenizer.
|
||||
# `load_calib` calls tok(..., truncation=True, max_length=seqlen), which
|
||||
# MUTATES the fast tokenizer's Rust backend truncation state in place.
|
||||
# save_pretrained then bakes {"truncation": {"max_length": <seqlen>}} into
|
||||
# tokenizer.json, so every prompt is silently clamped at the CALIBRATION
|
||||
# length forever. Latent on older transformers (it does not enforce the
|
||||
# check) and fatal on newer ones: a vision model dies at startup because the
|
||||
# dummy profiling image expands to more image tokens than the cap allows
|
||||
# ("Mismatch in `image` token count ... Got ids=[<cap-1>]").
|
||||
# Save a pristine tokenizer re-read from the SOURCE instead.
|
||||
from transformers import AutoTokenizer as _AutoTokenizer
|
||||
_AutoTokenizer.from_pretrained(a.model, trust_remote_code=True).save_pretrained(a.out)
|
||||
|
||||
# Fail loudly rather than shipping the defect again.
|
||||
_tj = os.path.join(a.out, "tokenizer.json")
|
||||
if os.path.exists(_tj):
|
||||
with open(_tj) as _f:
|
||||
_trunc = json.load(_f).get("truncation")
|
||||
if _trunc:
|
||||
raise SystemExit(
|
||||
f"FAILED CHECK: saved tokenizer.json carries truncation={_trunc}. "
|
||||
"It must be null — see quant playbook §3.14."
|
||||
)
|
||||
print("tokenizer saved pristine (truncation=null) — verified", flush=True)
|
||||
|
||||
print("DONE (post-steps still required: graft MTP, preprocessor_config, verify ignore)",
|
||||
flush=True)
|
||||
return 0
|
||||
|
||||
Reference in New Issue
Block a user