snapshot: NVFP4+MTP fast-seat quant recipe + failure state (gibberish, unisolated)
Captures the full pipeline recipe (graft->quant->splice->config->serve) with every gotcha found this session, the 3 gibberish suspects, and the diagnostic ladder (validate native-config no-MTP coherence FIRST) for a fresh session to finish the chase. Also stages the NVFP4 scripts + 512-row calib. Recent decisions: NEO-CODE seat swap (R36), webhook ALLOWED_HOST_LIST fix. Lessons: validate-tracer-bullet-first, mtp-graft-dropped-at-load, gitea-204-red-herring.
This commit is contained in:
+37
-1
@@ -102,7 +102,37 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-07-13 — two active tasks (per /snapshot args): the LiteLLM gateway #355-residual investigation and wiring the runner-side persistent BuildKit cache. The big WT #355 validation saga + Ledger provisioning + the assistant-role adds all LANDED this session; the concluded TTS/Zonos/RP-seat/ComfyUI/T1 in-flight was rolled out of this section (it lives in git + auto-memory)._
|
||||
_As of 2026-07-14 — ONE active task: finish the NVFP4+MTP fast char-rp-reasoning seat (the "nvfp4 quant chase"). Everything else this session LANDED: the char-rp-reasoning Deckard→NEO-CODE swap (#355 resolved), the Worldtree deploy-speed PR, and the soong-lab webhook fix — see Recent decisions + git; the old #355/deploy-speed detail below is kept as history (both DONE)._
|
||||
|
||||
### ★ ACTIVE — NVFP4+MTP fast char-rp-reasoning seat (the quant chase)
|
||||
|
||||
**GOAL:** NVFP4-quantize NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) + graft the Qwen3.6 MTP head → a ~2-4× faster **vLLM** char-rp-reasoning seat replacing the ~59.5 tok/s GGUF NEO-CODE seat (buys reasoning-budget headroom for soong's 45s client timeout). NVFP4 is **Blackwell-only** → ana-ml2 GPU0 is the ONLY place it runs (irv-ml1 is Ampere).
|
||||
|
||||
**★ REFERENCE — MATCH IT:** `pantheon-27b-mtp-nvfp4` (`/tank/aimodels/pantheon-27b-mtp-nvfp4`) is a WORKING 27B NVFP4+MTP seat on our Blackwell, SAME `Qwen3_5` arch. Its `config.json` (architectures `Qwen3_5ForConditionalGeneration`, empty mtp fields, `mtp.*` tensors BF16 top-level) + the `vllm-aeon-rp` serve command (`/opt/docker/compose/qwen36-27b-aeon/compose.yaml`) are the gold standard. When my output diverges from pantheon, **pantheon is right** — diff against it.
|
||||
|
||||
**ARTIFACTS** (ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/`, infra-ops-owned):
|
||||
- `heretic2-mtp-bf16/` — grafted BF16 (Heretic2 1184 + 15 MTP = 1199 tensors) [GOOD]
|
||||
- `heretic2-mtp-nvfp4-prod/` — NVFP4 output (spliced + config-merged) [**SERVES GIBBERISH — rebuild it**]
|
||||
- `production_calib_512.jsonl` + `calib/` — brokkr/Dvalin's render-verified 512-row workload-matched calib (tool_call_xml 128 / reasoning 128 / longctx 32 / nsfw 64 / rp 160)
|
||||
- SCRIPTS (committed in eshpfi `services/heretic2-nvfp4-quant/`): `graft_mtp.py`, `quant_nvfp4.py`, `splice_mtp.py`, `serve_nvfp4.sh`, `run_quant.sh`, `render_verify.py`
|
||||
|
||||
**THE RECIPE (steps 1-3 WORK; 4-5 are the gibberish suspects):**
|
||||
1. **GRAFT** (CPU, seats up): `graft_mtp.py --heretic2 <Heretic2 snap> --base <Qwen/Qwen3.6-27B snap> --out heretic2-mtp-bf16` in a vLLM container. Symlinks Heretic2 shards + splices 15 base MTP tensors (`mtp.*` top-level, base shards 13+15). [WORKS]
|
||||
2. **QUANT** (GPU0 freed): `run_quant.sh chat` → vLLM container + `pip install llmcompressor tiktoken sentencepiece` → `quant_nvfp4.py`. `QuantizationModifier(targets=Linear, scheme=NVFP4, ignore=[lm_head,embed,visual,linear_attn,norms,mtp])`; chat calib via `apply_chat_template` (args parsed string→dict). Out = compressed-tensors `nvfp4-pack-quantized`, 256 layers packed, GDN/vision/lm-head/norms BF16. ~19 min. [MAIN LAYERS GOOD] ⚠️ `AutoModelForCausalLM` loads it as `Qwen3_5ForCausalLM` (strips multimodal) → config arch becomes ForCausalLM AND the top-level `mtp.*` keys are DROPPED at load (0 mtp in output; the `re:mtp.*` ignore is moot — mtp never enters the module tree).
|
||||
3. **SPLICE** (CPU): `splice_mtp.py <nvfp4_out> <graft>` — load the 26GB quantized shard + add 15 BF16 mtp tensors → rewrite single shard (1634 tensors). Matches pantheon. [WORKS — mtp present, BF16]
|
||||
4. **CONFIG** (⚠️ #1 SUSPECT): merged grafted config (arch `Qwen3_5ForConditionalGeneration` + multimodal structure) + the quant's `quantization_config` so vLLM resolves `Qwen3_5MTP`. NEEDED for MTP but is the prime gibberish suspect (I never served with the quant's native ForCausalLM config).
|
||||
5. **SERVE** (GPU0): `serve_nvfp4.sh` → vllm/vllm-openai:v0.24.0. GOTCHAS (all captured in the script): ENTRYPOINT is `["vllm","serve"]` → pass `<model> --flags`, do NOT add `serve`. `--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'` (needs ConditionalGeneration arch to resolve). `--language-model-only` (skip vision; else "Can't load image processor"). `--mamba-cache-dtype float32 --max-num-seqs 256` (else "max_num_seqs 1024 exceeds Mamba cache blocks 304"). `--quantization compressed-tensors --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice`. → LOADS + MTP detected (embedding/lm_head sharing), healthy on :8018. [SERVES but **GIBBERISH**]
|
||||
|
||||
**THE FAILURE:** raw `/v1/completions` returns `!!!!!!…` (degenerate, token-0). Broken weight-load. Spec-decode is lossless so NOT the MTP draft. 3 unisolated suspects: (1) the config-merge [step 4, most likely], (2) the splice [step 3] mangling the packed shard on re-save, (3) the quant itself.
|
||||
|
||||
**★ DIAGNOSTIC LADDER (do this — needs a GPU0 window):**
|
||||
1. **FIRST, validate the tracer bullet:** re-quant clean (`run_quant.sh`) → serve with the quant's NATIVE `Qwen3_5ForCausalLM` config, **NO config-merge, NO MTP, NO splice** → smoke `curl /v1/completions -d '{"prompt":"The old tavern","max_tokens":40}'`. **Coherent → quant is FINE** (config-merge/splice/MTP broke it, incrementally add each back to isolate). **Gibberish → the quant is broken** (calib/scheme rabbit hole).
|
||||
2. Diff my `config.json` + serve flags byte-for-byte vs pantheon's (pantheon WORKS).
|
||||
3. Only after coherence is confirmed, add MTP back (arch→ConditionalGeneration + splice + spec-decode) and re-smoke at each step.
|
||||
|
||||
**GPU WINDOW ritual:** graft is CPU (do seats-up first). Take down GPU0: `docker stop llama-charrp llama-charrp-reasoning vllm-aeon-gen` (GPU0→97G free); restore `docker start …` (all healthy ~230s). Heads-up wt-dev (character/thoughtful-character + gen route through these). The GGUF NEO-CODE seat (:8018) is the always-restorable validated fallback. ⚠️ LESSON from the 2026-07-14 window (see Tried and abandoned): smoke a PLAIN coherence check on the simplest config FIRST — do not build MTP scaffolding on an unverified quant.
|
||||
|
||||
### Landed this session (2026-07-14) — history
|
||||
|
||||
### ✅ #355-residual — RESOLVED 2026-07-14 via MODEL SWAP (Deckard→NEO-CODE); was ACTIVE 1
|
||||
|
||||
@@ -138,6 +168,9 @@ _As of 2026-07-13 — two active tasks (per /snapshot args): the LiteLLM gateway
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`.
|
||||
- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]].
|
||||
- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528.
|
||||
- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1.
|
||||
|
||||
- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot.
|
||||
@@ -381,6 +414,9 @@ _142 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state.
|
||||
- `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip.
|
||||
- `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny '<ip>'` line) AND an instrumented receiver.
|
||||
- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction.
|
||||
- `[2026-07-13]` `althing-cli reply <THREAD_id>` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to <peer>`. Bit me several times this session.
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,132 @@
|
||||
[
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "bifrost.soong-lab.set_name",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"required": ["name"],
|
||||
"properties": {
|
||||
"name": { "type": "string", "maxLength": 128, "minLength": 1 }
|
||||
}
|
||||
},
|
||||
"description": "Set the agent's name (its display name; export slugifies it)."
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "bifrost.soong-lab.set_role",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"required": ["role"],
|
||||
"properties": {
|
||||
"role": {
|
||||
"enum": ["assistant", "thoughtful-assistant", "character", "thoughtful-character"],
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"description": "Set the agent's model-role — one of: assistant (general), thoughtful-assistant (reasoning general), character (RP/writing-tuned), thoughtful-character (reasoning RP). Export ships it as the native agents.define `role`."
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "bifrost.soong-lab.set_ocean",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"A": { "type": "number", "maximum": 1, "minimum": -1 },
|
||||
"C": { "type": "number", "maximum": 1, "minimum": -1 },
|
||||
"E": { "type": "number", "maximum": 1, "minimum": -1 },
|
||||
"N": { "type": "number", "maximum": 1, "minimum": -1 },
|
||||
"O": { "type": "number", "maximum": 1, "minimum": -1 }
|
||||
},
|
||||
"minProperties": 1
|
||||
},
|
||||
"description": "Merge OCEAN dials (each in [-1,1]); re-derives the disposition."
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "bifrost.soong-lab.edit_prompt",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"required": ["system_prompt"],
|
||||
"properties": {
|
||||
"system_prompt": { "type": "string", "maxLength": 32768, "minLength": 1 }
|
||||
}
|
||||
},
|
||||
"description": "Replace the authored role/instructions block (the exported system prompt)."
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "bifrost.soong-lab.attach_tool",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"required": ["id", "name"],
|
||||
"properties": {
|
||||
"id": { "type": "string", "maxLength": 256, "minLength": 1 },
|
||||
"name": { "type": "string", "maxLength": 256, "minLength": 1 },
|
||||
"description": { "type": "string", "maxLength": 2048 }
|
||||
}
|
||||
},
|
||||
"description": "Attach (upsert by id) a tool the designed agent will bind via Bifrost."
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "bifrost.soong-lab.author_first_message",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"required": ["first_message"],
|
||||
"properties": {
|
||||
"first_message": { "type": "string", "maxLength": 32768, "minLength": 1 }
|
||||
}
|
||||
},
|
||||
"description": "Set the agent's in-character opening turn (tone + format by example)."
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "bifrost.soong-lab.edit_psych_profile",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"required": ["psych_profile"],
|
||||
"properties": {
|
||||
"psych_profile": { "type": "string", "maxLength": 8192, "minLength": 1 }
|
||||
}
|
||||
},
|
||||
"description": "Author the character's Psychological Profile & Experience — a ~150-300 word prose lens (the self-report producer reads it every turn) covering four dimensions: disposition/appraisal bent, attention/salience focus, values/yardstick, and formative history. Integrate them as one paragraph and CLOSE ON ATTENTION — what the character characteristically notices ('...notices who is unwell, what is left unsaid'). Guardrails: (1) NEVER name a per-event output emotion ('is anxious', 'feels hurt') — naming PRIMES it, so it fires regardless of the event; describe the bent and let emotion follow from the appraisal. (2) Magnitude lives in the OCEAN dials, not the prose (don't narrate 'comes apart'/'takes it hard'). (3) Appraisal-STYLE yes ('reads others charitably until she can't'), output-emotion no. (4) Salience is character-relative — never rewrite what happened."
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "bifrost.soong-lab.generate_portrait",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"required": ["description"],
|
||||
"properties": {
|
||||
"description": { "type": "string", "maxLength": 500, "minLength": 1 },
|
||||
"style_mode": { "enum": ["anime", "cartoon", "photoreal"], "type": "string" }
|
||||
}
|
||||
},
|
||||
"description": "Generate the agent's persona portrait NOW from a visual `description` you author — the persona's LOOK (e.g. 'a stern woman in her 40s, short dark hair, sharp jaw'). A persona is personality, not appearance, so YOU compose the appearance in conversation and pass it here; describe only the subject, never the art style. `style_mode` (anime | cartoon | photoreal) selects the curated render style — the style preset owns the look. Generation runs in the background; the portrait pane shows it when ready. Offer/generate a portrait once the look is settled."
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "bifrost.soong-lab.export",
|
||||
"parameters": { "type": "object", "properties": {} },
|
||||
"description": "Export the native payload + sidecar bundle (stub — lands in E5)."
|
||||
}
|
||||
}
|
||||
]
|
||||
File diff suppressed because one or more lines are too long
@@ -1,39 +1,30 @@
|
||||
#!/usr/bin/env python3
|
||||
"""NVFP4 quantize the MTP-grafted Heretic2 model (llm-compressor / compressed-tensors).
|
||||
|
||||
Runs AFTER graft_mtp.py. Uses the fleet-proven compressed-tensors NVFP4 path
|
||||
(the pantheon-27b-mtp-nvfp4 serving pattern already works on our Blackwell) with
|
||||
the ignore-list from robbatt's on-fleet deckard-nvfp4 recipe PLUS the MTP head:
|
||||
keep GDN/linear-attn, vision tower, lm-head, all norms, AND mtp.* in BF16;
|
||||
NVFP4 only the dense Linear layers. (R36 fast-seat spike, brokkr calib spec.)
|
||||
Runs AFTER graft_mtp.py. Fleet-proven compressed-tensors NVFP4 path (pantheon-27b-
|
||||
mtp-nvfp4 serves this on our Blackwell) with the ignore-list from robbatt's on-fleet
|
||||
deckard-nvfp4 recipe PLUS the MTP head: keep GDN/linear-attn, vision, lm-head, all
|
||||
norms, AND mtp.* in BF16; NVFP4 only the dense Linear layers. (R36 fast-seat spike.)
|
||||
|
||||
⚠️ NEEDS: (a) an env with llmcompressor + a CUDA torch (run inside a vLLM
|
||||
container: `pip install llmcompressor` on vllm/vllm-openai:v0.24.0), (b) a freed
|
||||
Blackwell GPU (~55 GB — both ana-ml2 GPUs are normally full; needs an off-peak
|
||||
window). API validated against llm-compressor at run time — treat the exact
|
||||
symbol names as first-draft until a dry import confirms them.
|
||||
API validated against llmcompressor 0.12.0: oneshot(model, dataset, recipe,
|
||||
num_calibration_samples, max_seq_length) — dataset is a pre-tokenized datasets.Dataset.
|
||||
|
||||
Two calib modes (brokkr's two artifacts):
|
||||
--calib-mode text : AEON-baseline control (neuralmagic/calibration LLM split)
|
||||
--calib-mode chat : production 512-row mix (JSONL rows {messages, tools});
|
||||
each row rendered via apply_chat_template(enable_thinking=True)
|
||||
so the forward-pass sees the qwen3_coder tool-call XML =
|
||||
the seat's native activations (the #355-preservation point).
|
||||
Two calib modes:
|
||||
--calib-mode text : AEON-baseline control (neuralmagic/calibration LLM split)
|
||||
--calib-mode chat : production 512-row mix (JSONL {messages, tools}); each row
|
||||
rendered via apply_chat_template(enable_thinking=True) so the
|
||||
forward-pass sees the qwen3_coder tool-call XML the seat emits.
|
||||
|
||||
Usage:
|
||||
python3 quant_nvfp4.py --model /tank/aimodels/heretic2-mtp-bf16 \
|
||||
--calib-mode text --calib neuralmagic/calibration --num-samples 160 \
|
||||
--out /tank/aimodels/heretic2-mtp-nvfp4-baseline
|
||||
python3 quant_nvfp4.py --model /tank/aimodels/heretic2-mtp-bf16 \
|
||||
--calib-mode chat --calib /path/to/production_calib_512.jsonl \
|
||||
--out /tank/aimodels/heretic2-mtp-nvfp4-prod
|
||||
Run in a vLLM container on the freed GPU0:
|
||||
docker run --gpus '"device=0"' --ipc host -v /tank/aimodels:/tank/aimodels \
|
||||
--entrypoint bash vllm/vllm-openai:v0.24.0 -c \
|
||||
"pip install -q llmcompressor tiktoken sentencepiece && python3 quant_nvfp4.py ..."
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
|
||||
# Ignore list = robbatt deckard-nvfp4 recipe + MTP head (brokkr: keep mtp.* BF16).
|
||||
# Keeps in BF16: lm-head, embeddings, vision tower, GDN/linear-attn, all norms, MTP.
|
||||
IGNORE = [
|
||||
"lm_head",
|
||||
"re:.*embed_tokens$",
|
||||
@@ -43,27 +34,31 @@ IGNORE = [
|
||||
"re:.*norm.*",
|
||||
"re:.*q_norm.*",
|
||||
"re:.*k_norm.*",
|
||||
"re:mtp.*", # MTP head stays BF16 for the qwen3_5_mtp spec-decode head
|
||||
"re:.*mtp.*", # MTP head stays BF16 for qwen3_5_mtp spec-decode (match anywhere:
|
||||
# module path is model.mtp.*, so an anchored re:mtp.* misses it)
|
||||
]
|
||||
|
||||
|
||||
def _tok_rows_to_dataset(tok_rows):
|
||||
from datasets import Dataset
|
||||
return Dataset.from_list(tok_rows)
|
||||
|
||||
|
||||
def load_calib_text(name, tokenizer, n, seqlen):
|
||||
from datasets import load_dataset
|
||||
ds = load_dataset(name, split="train").shuffle(seed=42).select(range(n))
|
||||
ds = load_dataset(name, split="train").shuffle(seed=42).select(range(min(n, 100000)))
|
||||
col = "text" if "text" in ds.column_names else ds.column_names[0]
|
||||
return [tokenizer(x[col], truncation=True, max_length=seqlen) for x in ds]
|
||||
rows = [tokenizer(x[col], truncation=True, max_length=seqlen) for x in ds.select(range(n))]
|
||||
return _tok_rows_to_dataset(rows)
|
||||
|
||||
|
||||
def load_calib_chat(path, tokenizer, seqlen):
|
||||
"""Render {messages, tools} JSONL rows through the chat template with thinking on.
|
||||
The assistant tool_calls (OpenAI form) serialize to qwen3_coder XML here, so the
|
||||
calibration forward-pass sees the EXACT tool-call token distribution the seat emits."""
|
||||
rows = [json.loads(l) for l in open(path) if l.strip()]
|
||||
def load_calib_chat(path, tokenizer, seqlen, n):
|
||||
rows = [json.loads(l) for l in open(path) if l.strip()][:n]
|
||||
out = []
|
||||
for r in rows:
|
||||
# Tool_call arguments arrive as OpenAI wire-form JSON *strings*; the Qwen3.6
|
||||
# template does .items() on them → needs a dict. Parse string→dict so the
|
||||
# forward-pass sees the qwen3_coder XML the seat emits. (render_verify finding.)
|
||||
# Tool_call arguments may be OpenAI wire-form JSON strings; the Qwen3.6 template
|
||||
# does .items() on them → needs a dict. Parse string→dict (render_verify finding;
|
||||
# belt-and-suspenders even though brokkr canonicalized the rows to dicts).
|
||||
for m in r["messages"]:
|
||||
for tc in (m.get("tool_calls") or []):
|
||||
a = tc.get("function", {}).get("arguments")
|
||||
@@ -74,14 +69,14 @@ def load_calib_chat(path, tokenizer, seqlen):
|
||||
tokenize=False, add_generation_prompt=False, enable_thinking=True,
|
||||
)
|
||||
out.append(tokenizer(text, truncation=True, max_length=seqlen))
|
||||
return out
|
||||
return _tok_rows_to_dataset(out)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--model", required=True, help="grafted BF16 model dir (graft_mtp.py output)")
|
||||
ap.add_argument("--model", required=True)
|
||||
ap.add_argument("--calib-mode", choices=["text", "chat"], required=True)
|
||||
ap.add_argument("--calib", required=True, help="HF dataset name (text) or JSONL path (chat)")
|
||||
ap.add_argument("--calib", required=True)
|
||||
ap.add_argument("--out", required=True)
|
||||
ap.add_argument("--num-samples", type=int, default=512)
|
||||
ap.add_argument("--seqlen", type=int, default=8192)
|
||||
@@ -91,34 +86,33 @@ def main() -> int:
|
||||
from llmcompressor import oneshot
|
||||
from llmcompressor.modifiers.quantization import QuantizationModifier
|
||||
|
||||
print(f"loading grafted model: {args.model}")
|
||||
print(f"loading grafted model: {args.model}", flush=True)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
args.model, torch_dtype="auto", device_map="auto", trust_remote_code=True,
|
||||
)
|
||||
tok = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)
|
||||
|
||||
print(f"building calibration ({args.calib_mode}, {args.num_samples} samples, seq {args.seqlen})")
|
||||
print(f"building calibration ({args.calib_mode}, up to {args.num_samples} @ seq {args.seqlen})", flush=True)
|
||||
if args.calib_mode == "text":
|
||||
calib = load_calib_text(args.calib, tok, args.num_samples, args.seqlen)
|
||||
else:
|
||||
calib = load_calib_chat(args.calib, tok, args.seqlen)
|
||||
print(f" {len(calib)} calibration rows")
|
||||
calib = load_calib_chat(args.calib, tok, args.seqlen, args.num_samples)
|
||||
print(f" {len(calib)} calibration rows", flush=True)
|
||||
|
||||
recipe = QuantizationModifier(targets="Linear", scheme="NVFP4", ignore=IGNORE)
|
||||
|
||||
print("running NVFP4 oneshot (Linear-only; GDN/vision/lm-head/norms/MTP kept BF16)")
|
||||
print("running NVFP4 oneshot (Linear-only; GDN/vision/lm-head/norms/MTP kept BF16)", flush=True)
|
||||
oneshot(
|
||||
model=model, dataset=calib, recipe=recipe,
|
||||
max_seq_length=args.seqlen, num_calibration_samples=len(calib),
|
||||
num_calibration_samples=len(calib), max_seq_length=args.seqlen,
|
||||
)
|
||||
|
||||
print(f"saving -> {args.out}")
|
||||
print(f"saving -> {args.out}", flush=True)
|
||||
model.save_pretrained(args.out, save_compressed=True)
|
||||
tok.save_pretrained(args.out)
|
||||
print("DONE. serve: vllm --quantization compressed-tensors "
|
||||
"--speculative-config '{\"method\":\"qwen3_5_mtp\",\"num_speculative_tokens\":3}' "
|
||||
"--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice")
|
||||
print("NEXT: ping brokkr -> P00 rig (soong 9-tool k5); acceptance = hold ~0.967/perfect attach_tool")
|
||||
"--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice", flush=True)
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
#!/bin/bash
|
||||
# Run the Heretic2 NVFP4 production quant on ana-ml2 GPU0 (freed for the window).
|
||||
# llmcompressor + tokenizer deps pip-installed into a vLLM container (torch/CUDA ready).
|
||||
set -euo pipefail
|
||||
WORK=/tank/aimodels/heretic2-nvfp4-work
|
||||
MODE="${1:-chat}" # chat = production 512-row mix; text = AEON baseline
|
||||
CALIB="${2:-$WORK/production_calib_512.jsonl}"
|
||||
OUT="${3:-$WORK/heretic2-mtp-nvfp4-prod}"
|
||||
|
||||
docker run --rm --gpus '"device=0"' --ipc host \
|
||||
-v /tank/aimodels:/tank/aimodels \
|
||||
--entrypoint bash vllm/vllm-openai:v0.24.0 -c "
|
||||
set -e
|
||||
pip install -q llmcompressor tiktoken sentencepiece 2>&1 | tail -1
|
||||
python3 $WORK/quant_nvfp4.py \
|
||||
--model $WORK/heretic2-mtp-bf16 \
|
||||
--calib-mode $MODE --calib $CALIB \
|
||||
--num-samples 512 --seqlen 8192 \
|
||||
--out $OUT
|
||||
"
|
||||
@@ -0,0 +1,20 @@
|
||||
#!/bin/bash
|
||||
# Serve the NVFP4+MTP Heretic2 seat on ana-ml2 GPU0 for P00 validation.
|
||||
# vLLM compressed-tensors NVFP4 + qwen3_5_mtp spec-decode (pantheon pattern).
|
||||
# Served as char-rp-reasoning on :8018 (the GGUF seat's port; that seat is stopped).
|
||||
set -euo pipefail
|
||||
WORK=/tank/aimodels/heretic2-nvfp4-work
|
||||
docker rm -f vllm-charrp-nvfp4-test 2>/dev/null || true
|
||||
|
||||
docker run -d --name vllm-charrp-nvfp4-test --gpus '"device=0"' --ipc host \
|
||||
-v /tank/aimodels:/tank/aimodels -p 8018:8000 \
|
||||
vllm/vllm-openai:v0.24.0 \
|
||||
"$WORK/heretic2-mtp-nvfp4-prod" \
|
||||
--quantization compressed-tensors \
|
||||
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
|
||||
--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \
|
||||
--language-model-only \
|
||||
--mamba-cache-dtype float32 \
|
||||
--served-model-name char-rp-reasoning \
|
||||
--max-model-len 40960 --max-num-seqs 256 --gpu-memory-utilization 0.45 --trust-remote-code
|
||||
echo "container started: $(docker ps --filter name=vllm-charrp-nvfp4-test --format '{{.Status}}')"
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Splice the BF16 MTP head tensors into the NVFP4-quantized output.
|
||||
|
||||
The Qwen3_5ForConditionalGeneration model class does NOT expose the MTP head as a
|
||||
loadable module (it's a spec-decode drafting head vLLM reads separately from the
|
||||
checkpoint), so AutoModelForCausalLM.from_pretrained DROPS the top-level `mtp.*`
|
||||
state-dict keys on load — the quant output ends up with 0 mtp tensors. The working
|
||||
reference (pantheon-27b-mtp-nvfp4) keeps them by SPLICING the BF16 mtp.* tensors
|
||||
into the safetensors after quantization. This does the same: rewrites the single
|
||||
output shard to include the 15 BF16 mtp.* tensors (from the graft's model-mtp
|
||||
shard), matching pantheon's structure. (R36 fast-seat spike.)
|
||||
|
||||
Usage: python3 splice_mtp.py <nvfp4_out_dir> <grafted_dir_with_model-mtp.safetensors>
|
||||
"""
|
||||
import sys
|
||||
|
||||
from safetensors import safe_open
|
||||
from safetensors.torch import save_file
|
||||
|
||||
out_dir, graft_dir = sys.argv[1], sys.argv[2]
|
||||
out_st = f"{out_dir}/model.safetensors"
|
||||
mtp_st = f"{graft_dir}/model-mtp.safetensors"
|
||||
|
||||
# load the full quantized shard (round-trips packed/scale tensors exactly)
|
||||
tensors = {}
|
||||
with safe_open(out_st, framework="pt") as f:
|
||||
for k in f.keys():
|
||||
tensors[k] = f.get_tensor(k)
|
||||
n_main = len(tensors)
|
||||
|
||||
# add the BF16 mtp.* tensors
|
||||
with safe_open(mtp_st, framework="pt") as f:
|
||||
mtp_keys = list(f.keys())
|
||||
for k in mtp_keys:
|
||||
tensors[k] = f.get_tensor(k)
|
||||
|
||||
assert not any("mtp" in k.lower() for k in list(tensors)[:n_main]), "output already had mtp?"
|
||||
print(f"main tensors: {n_main} | splicing {len(mtp_keys)} BF16 mtp tensors")
|
||||
print("mtp dtypes:", {str(tensors[k].dtype) for k in mtp_keys})
|
||||
|
||||
save_file(tensors, out_st, metadata={"format": "pt"})
|
||||
print(f"DONE — rewrote {out_st} with {len(tensors)} tensors ({n_main} quantized + {len(mtp_keys)} BF16 mtp)")
|
||||
Reference in New Issue
Block a user