vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

snapshot: NVFP4+MTP fast-seat quant recipe + failure state (gibberish, unisolated)

Browse Source 
Captures the full pipeline recipe (graft->quant->splice->config->serve) with every
gotcha found this session, the 3 gibberish suspects, and the diagnostic ladder
(validate native-config no-MTP coherence FIRST) for a fresh session to finish the
chase. Also stages the NVFP4 scripts + 512-row calib. Recent decisions: NEO-CODE
seat swap (R36), webhook ALLOWED_HOST_LIST fix. Lessons: validate-tracer-bullet-first,
mtp-graft-dropped-at-load, gitea-204-red-herring.
This commit is contained in:
vh
2026-07-14 11:31:57 -07:00
parent 462d528bef
commit b972bef10e
11 changed files with 804 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+37 -1
View File
@@ -102,7 +102,37 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-07-13 — two active tasks (per /snapshot args): the LiteLLM gateway #355-residual investigation and wiring the runner-side persistent BuildKit cache. The big WT #355 validation saga + Ledger provisioning + the assistant-role adds all LANDED this session; the concluded TTS/Zonos/RP-seat/ComfyUI/T1 in-flight was rolled out of this section (it lives in git + auto-memory)._
_As of 2026-07-14 — ONE active task: finish the NVFP4+MTP fast char-rp-reasoning seat (the "nvfp4 quant chase"). Everything else this session LANDED: the char-rp-reasoning Deckard→NEO-CODE swap (#355 resolved), the Worldtree deploy-speed PR, and the soong-lab webhook fix — see Recent decisions + git; the old #355/deploy-speed detail below is kept as history (both DONE)._
### ★ ACTIVE — NVFP4+MTP fast char-rp-reasoning seat (the quant chase)
**GOAL:** NVFP4-quantize NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) + graft the Qwen3.6 MTP head → a ~2-4× faster **vLLM** char-rp-reasoning seat replacing the ~59.5 tok/s GGUF NEO-CODE seat (buys reasoning-budget headroom for soong's 45s client timeout). NVFP4 is **Blackwell-only** → ana-ml2 GPU0 is the ONLY place it runs (irv-ml1 is Ampere).
**★ REFERENCE — MATCH IT:** `pantheon-27b-mtp-nvfp4` (`/tank/aimodels/pantheon-27b-mtp-nvfp4`) is a WORKING 27B NVFP4+MTP seat on our Blackwell, SAME `Qwen3_5` arch. Its `config.json` (architectures `Qwen3_5ForConditionalGeneration`, empty mtp fields, `mtp.*` tensors BF16 top-level) + the `vllm-aeon-rp` serve command (`/opt/docker/compose/qwen36-27b-aeon/compose.yaml`) are the gold standard. When my output diverges from pantheon, **pantheon is right** — diff against it.
**ARTIFACTS** (ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/`, infra-ops-owned):
- `heretic2-mtp-bf16/` — grafted BF16 (Heretic2 1184 + 15 MTP = 1199 tensors) [GOOD]
- `heretic2-mtp-nvfp4-prod/` — NVFP4 output (spliced + config-merged) [**SERVES GIBBERISH — rebuild it**]
- `production_calib_512.jsonl` + `calib/` — brokkr/Dvalin's render-verified 512-row workload-matched calib (tool_call_xml 128 / reasoning 128 / longctx 32 / nsfw 64 / rp 160)
- SCRIPTS (committed in eshpfi `services/heretic2-nvfp4-quant/`): `graft_mtp.py`, `quant_nvfp4.py`, `splice_mtp.py`, `serve_nvfp4.sh`, `run_quant.sh`, `render_verify.py`
**THE RECIPE (steps 1-3 WORK; 4-5 are the gibberish suspects):**
1. **GRAFT** (CPU, seats up): `graft_mtp.py --heretic2 <Heretic2 snap> --base <Qwen/Qwen3.6-27B snap> --out heretic2-mtp-bf16` in a vLLM container. Symlinks Heretic2 shards + splices 15 base MTP tensors (`mtp.*` top-level, base shards 13+15). [WORKS]
2. **QUANT** (GPU0 freed): `run_quant.sh chat` → vLLM container + `pip install llmcompressor tiktoken sentencepiece` → `quant_nvfp4.py`. `QuantizationModifier(targets=Linear, scheme=NVFP4, ignore=[lm_head,embed,visual,linear_attn,norms,mtp])`; chat calib via `apply_chat_template` (args parsed string→dict). Out = compressed-tensors `nvfp4-pack-quantized`, 256 layers packed, GDN/vision/lm-head/norms BF16. ~19 min. [MAIN LAYERS GOOD] ⚠️ `AutoModelForCausalLM` loads it as `Qwen3_5ForCausalLM` (strips multimodal) → config arch becomes ForCausalLM AND the top-level `mtp.*` keys are DROPPED at load (0 mtp in output; the `re:mtp.*` ignore is moot — mtp never enters the module tree).
3. **SPLICE** (CPU): `splice_mtp.py <nvfp4_out> <graft>` — load the 26GB quantized shard + add 15 BF16 mtp tensors → rewrite single shard (1634 tensors). Matches pantheon. [WORKS — mtp present, BF16]
4. **CONFIG** (⚠️ #1 SUSPECT): merged grafted config (arch `Qwen3_5ForConditionalGeneration` + multimodal structure) + the quant's `quantization_config` so vLLM resolves `Qwen3_5MTP`. NEEDED for MTP but is the prime gibberish suspect (I never served with the quant's native ForCausalLM config).
5. **SERVE** (GPU0): `serve_nvfp4.sh` → vllm/vllm-openai:v0.24.0. GOTCHAS (all captured in the script): ENTRYPOINT is `["vllm","serve"]` → pass `<model> --flags`, do NOT add `serve`. `--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'` (needs ConditionalGeneration arch to resolve). `--language-model-only` (skip vision; else "Can't load image processor"). `--mamba-cache-dtype float32 --max-num-seqs 256` (else "max_num_seqs 1024 exceeds Mamba cache blocks 304"). `--quantization compressed-tensors --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice`. → LOADS + MTP detected (embedding/lm_head sharing), healthy on :8018. [SERVES but **GIBBERISH**]
**THE FAILURE:** raw `/v1/completions` returns `!!!!!!…` (degenerate, token-0). Broken weight-load. Spec-decode is lossless so NOT the MTP draft. 3 unisolated suspects: (1) the config-merge [step 4, most likely], (2) the splice [step 3] mangling the packed shard on re-save, (3) the quant itself.
**★ DIAGNOSTIC LADDER (do this — needs a GPU0 window):**
1. **FIRST, validate the tracer bullet:** re-quant clean (`run_quant.sh`) → serve with the quant's NATIVE `Qwen3_5ForCausalLM` config, **NO config-merge, NO MTP, NO splice** → smoke `curl /v1/completions -d '{"prompt":"The old tavern","max_tokens":40}'`. **Coherent → quant is FINE** (config-merge/splice/MTP broke it, incrementally add each back to isolate). **Gibberish → the quant is broken** (calib/scheme rabbit hole).
2. Diff my `config.json` + serve flags byte-for-byte vs pantheon's (pantheon WORKS).
3. Only after coherence is confirmed, add MTP back (arch→ConditionalGeneration + splice + spec-decode) and re-smoke at each step.
**GPU WINDOW ritual:** graft is CPU (do seats-up first). Take down GPU0: `docker stop llama-charrp llama-charrp-reasoning vllm-aeon-gen` (GPU0→97G free); restore `docker start …` (all healthy ~230s). Heads-up wt-dev (character/thoughtful-character + gen route through these). The GGUF NEO-CODE seat (:8018) is the always-restorable validated fallback. ⚠️ LESSON from the 2026-07-14 window (see Tried and abandoned): smoke a PLAIN coherence check on the simplest config FIRST — do not build MTP scaffolding on an unverified quant.
### Landed this session (2026-07-14) — history
### ✅ #355-residual — RESOLVED 2026-07-14 via MODEL SWAP (Deckard→NEO-CODE); was ACTIVE 1
@@ -138,6 +168,9 @@ _As of 2026-07-13 — two active tasks (per /snapshot args): the LiteLLM gateway
## Recent decisions
- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`.
- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]].
- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528.
- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1.
- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot.
@@ -381,6 +414,9 @@ _142 older entries archived to archival-memory.md._
## Tried and abandoned
- `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state.
- `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip.
- `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny '<ip>'` line) AND an instrumented receiver.
- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction.
- `[2026-07-13]` `althing-cli reply <THREAD_id>` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to <peer>`. Bit me several times this session.