From 05a4f54a2a2bae2cc62fa089488a4ae2fb7d12d5 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 15 Jul 2026 11:04:13 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=202026-07-15=20f?= =?UTF-8?q?leet-ops=20session?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrote Current state / in-flight to reflect the session's landed work + parked items (in-flight compressed from the now-done NVFP4/#355/deploy-speed history). Added Recent decisions (homepage AI-tab revamp 569e1af; esh-docker-vm reboot + NFS fstab fix 21d9a07; HA config repo; char-rp-reasoning OOM rescue; arbo->gen switch with image-bench eviction parked post-bake; soong-lab library persistence; statusline overhaul) and Tried-and-abandoned lessons (nofail defeats After=remote-fs.target; D-state wedge needs a reboot; max-model-len doesn't free vLLM VRAM; statusline cost is per-session). Nothing archivable (all dated entries <30 days). --- persistent-memory.md | 141 ++++++++++++------------------------------- 1 file changed, 37 insertions(+), 104 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index 6e2c359..004633f 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,11 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-07-14_ +_Last updated: 2026-07-15_ + +> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its +> `Written:` stamp is under an hour old, read it (it carries the in-flight +> handoff from the previous session), then delete it. Older than an hour: +> stale — delete it unread. ## Repo purpose @@ -102,115 +107,36 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-07-14 — ONE active task: the NVFP4 fast char-rp-reasoning seat. GIBBERISH RESOLVED (quant-namespace bug — NVFP4 now serves coherent) and the speed premise DISPROVEN on the llm-compressor/compressed-tensors format (base NVFP4 ≈ GGUF at batch-1; MTP 0%-accept). PIVOTED (operator) to a **modelopt-format re-quant for working MTP** — scoped + de-risked, needs `quant_modelopt.py` + one more GPU0 window (see ★ section). Everything else this session LANDED: the char-rp-reasoning Deckard→NEO-CODE swap (#355 resolved), the Worldtree deploy-speed PR, and the soong-lab webhook fix — see Recent decisions + git; the old #355/deploy-speed detail below is kept as history (both DONE)._ +_As of 2026-07-15 — a long fleet-ops + infra session. Everything below either LANDED (committed/pushed) or is PARKED on a clean async watch. No blocking work in flight._ -### ✅✅ DONE — modelopt NVFP4 + MTP fast char-rp-reasoning seat WORKS (2026-07-14) +**Landed this session (2026-07-14 -> 15):** +- **Homepage revamp** — the flat ~22-service "AI Systems" group split into a dedicated **AI tab** with 6 role-based groups (Inference / Eval & Retrieval / Gateways & Chat / Speech (TTS) / Audio Tools / Image & Media) + an **AI - Dormant** group for the 7 stopped stacks. Relabeled 22 live compose files (`homepage.group=`) via surgical live `sed` + canonical backfill; GPU0 seats recreated in the operator's VRAM order (char-rp -> gen -> char-rp-reasoning). Committed `569e1af`, pushed. Homepage runs on esh-docker-vm `:5100` (config `configs/homepage/`). +- **esh-docker-vm rescued + NFS hardened** — uptime-kuma + paperless-web were wedged in the D-state/phantom class; **only a host REBOOT cleared it** (infra-ops sudo). **NFS fstab properly fixed** (`21d9a07`, pushed): added `x-systemd.before=docker.service,x-systemd.mount-timeout=30` to the 4 `10.0.50.50:` lines — the prior `_netdev,nofail`+`After=remote-fs.target` was silently defeated by `nofail`. Playbook `playbooks/fix-esh-nfs-boot-ordering.yaml` updated. Also: **mOrpheus killed** (irv-ml1 `docker compose down`; backend OOM-looping on the packed 3090) and a **1.36 GB HA coredump cleared**. +- **Home Assistant config repo** — `vh/home-assistant-config` (private gitea) for the HA Container at `10.0.50.46:8123` (= `homeassistant` on esh-docker-vm, macvlan, config named-volume). **git-in-place in `/config`** (root/infra-ops) + scoped deploy key; **local clone `~/development/home-assistant-config`**. UI-managed HA -> allowlist model (YAML + curated secret-free `.storage` subset; secrets/DB/coredump gitignored). +- **soong-lab library persistence** (corviduo-dev) — `SOONG_LAB_LIBRARY_DIR=/home/infra-ops/soong-lab-data/library` added to soong-lab.env (mirrors PORTRAIT_DIR), `soong-lab-studio.service` restarted. Unblocked the operator (Sindra import -> library save survives redeploys). +- **char-rp-reasoning OOM rescue + tune** — the NVFP4+MTP seat (GPU0) crash-looped on a solo restart (GPU0 too tight). Fixed: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` + `REASONING_GPU_MEM_UTIL 0.39->0.38` + `REASONING_MAX_MODEL_LEN 196608->153600` (150K). Now ~4.5 GB GPU0 headroom, healthy; brokkr's text battery passed. +- **Statusline overhaul** (`~/.claude/statusline-command.sh`, NOT this repo) — git state (branch/dirty/unpushed), 🔔/🔕 althing-monitor-armed, project tag, absolute tokens, **per-session cost** (`.cost.total_cost_usd`), threshold-colored ctx% + rate %s (green<60 / yellow60-90 / red>90). -**WORKING at ~77 tok/s** (vs GGUF NEO-CODE ~59.5, base NVFP4 ~53 → **~1.3× over GGUF**), MTP -draft-acceptance **32–40%**, mean acceptance length **2.19**. Same Heretic2/NEO-CODE model, NVFP4 -+ native MTP. **★ FULL RECIPE + all gotchas: `docs/runbooks/heretic2-nvfp4-mtp-seat.md`** (the -fire-drill-killer the operator demanded). Scripts: eshpfi `services/heretic2-nvfp4-quant/` -(`quant_modelopt.py`, `finalize_modelopt_mtp.py`, `serve_modelopt_mtp.sh`, -`sitecustomize-mtp-workaround.py`). Seat artifact: ana-ml2 -`/tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp`. +**PARKED (operator-directed holds):** +- **image-bench (qwen-image-bench, GPU1) eviction — PENDING post-bake.** Operator directed a full arbo -> gen switch (comfy-dev executed it, LIVE in prod: `DEFAULT_LLM_MODEL=gen` + `ARBO_VISION_MODEL=gen`); image-judge now has ZERO arbo tenancy. Sequencing = **short prod bake on gen (~few days) THEN evict** -> reclaim ~30 GB GPU1. Keep `vllm-qwen-image-bench` WARM as the instant-revert backstop; comfy-dev watches NSFW-anatomy scoring + tight-grounding (flip `ARBO_VISION_MODEL` back on regression). On a clean bake: evict + ping comfy-dev. **Fully tracked in auto-memory `project_arbo_gen_switch_imagebench_evict`.** +- **granite right-size — un-started GPU1 win.** granite (GPU1) is ~29 GB (util 0.34 -> ~16.5 GB oversized KV for an 8B summarizer). Right-sizing util 0.34 -> ~0.13 frees ~16 GB WHILE keeping the summarizer (fleet's dedicated summarizer/classifier; aliases granite-4.1-8b/summarizer/classifier). Independent of image-bench; operator was freeing GPU1 for a TBD "something else". -**The four landmines (each cost hours — full detail in the runbook):** (1) load as -`AutoModelForImageTextToText` not `AutoModelForCausalLM` (namespace → else `!!!!`); (2) MODELOPT -format not compressed-tensors (compressed-tensors MTP = 0% accept; base NVFP4 alone ≈ GGUF, MTP is -the whole win); (3) modelopt 0.45↔transformers 5.12.1 FusedMoE crash — guarded in `quant_modelopt.py`; -(4) **⭐ vLLM 0.24.0 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → -the BF16 mtp head gets quantized → shape crash; **no checkpoint config fixes it** (`is_layer_skipped` -uses exact membership not glob, and the drafter never sees the mtp excludes) → **fix = a mounted -`sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (report upstream as a vLLM bug). - -**Operator's memory was right:** MTP ran before on the MoE `gen` (qwopus-122B +12% single-stream, -archival 2026-07-01) — dropped for `gen` because MTP HURTS concurrency; it belongs on the -single-stream RP seats. **NOT-YET-DONE (productionization):** compose stack with the workaround baked -in, brokkr P00 (soong 9-tool k5 — same base as GGUF so R36 should carry, confirm NVFP4-vs-Q5 quality), -gateway `char-rp-reasoning` repoint, wt-dev heads-up, file the vLLM upstream bug. - ---- - -### (historical) RESOLVED (gibberish) → PIVOT: modelopt-format re-quant for working MTP - -**2026-07-14 WINDOW OUTCOME (this session).** Ran the diagnostic ladder in one clean ~40-min GPU0 window. -- **GIBBERISH ROOT CAUSE = quant NAMESPACE (found from config diffs + vLLM source, ZERO GPU time).** `quant_nvfp4.py` loaded via `AutoModelForCausalLM` → resolves qwen3_5 to the text-only `Qwen3_5ForCausalLM` → weight keys born flat `model.layers.*` (no vision). But vLLM 0.24 registers ONLY `Qwen3_5ForConditionalGeneration` (registry.py:566), whose `hf_to_vllm_mapper` (qwen3_vl.py:1692) remaps `model.language_model.*`→`language_model.model.*` and has NO rule for a bare `model.layers.` prefix → every transformer-layer weight fails to match → uninitialized → `!!!!`. The step-4 config-merge to ConditionalGeneration was a doomed patch over a wrong-namespace checkpoint. **FIX = load as `AutoModelForImageTextToText`** (resolves qwen3_5 → `Qwen3_5ForConditionalGeneration` → keys born `model.language_model.*` + `model.visual.*`, pantheon namespace). One-class swap; committed to `quant_nvfp4.py`. -- **VALIDATED COHERENT.** Re-quant → `heretic2-nvfp4-cg` (arch ConditionalGeneration, nvfp4-pack-quantized, 1952 keys: 1618 language_model + 333 visual + 0 bare model.layers, 447-entry ignore = pantheon-shaped). Greedy `/v1/completions` = fluent ("...tavern in Baldur's Gate...", "Paris."). Spliced 15 BF16 mtp → `heretic2-nvfp4-cg-mtp` (1967 tensors, mtp key-set identical to pantheon+AEON). -- **BUT THE SPEED PREMISE IS DISPROVEN on this (llm-compressor / compressed-tensors) format:** (1) base NVFP4 no-MTP ≈ **53 tok/s decode ≈ the GGUF seat's ~59.5 at batch-1** (NO single-stream win — this GDN-hybrid is bandwidth-bound at batch=1 with the BF16 linear_attn layers dominating; NVFP4's edge is concurrency, not batch-1; llama.cpp is highly optimized single-stream). (2) MTP spec-decode = **0% acceptance** — vLLM's Qwen3_5MTP drafter can't load the bf16 mtp weights off a compressed-tensors main model (`Parameter ... not found in params_dict`); pantheon is identical (its "working MTP" was never real acceleration). KNOWN limit: the bf16 MTP head only loads on the **MODELOPT** main-model format. -- **PIVOT (operator 2026-07-14): pursue a modelopt-format re-quant for working MTP** — the ONLY path to the 2-4×. - -**⏭ MODELOPT PLAN (scoped + de-risked this session, ready for the next window):** -- **REFERENCE = AEON** `/tank/aimodels/qwen36-27b-aeon-nvfp4` (served by `vllm-aeon-rp`, `/opt/docker/compose/qwen36-27b-aeon/compose.yaml`). Confirmed: arch `Qwen3_5ForConditionalGeneration`, `quant_method: modelopt` (config.json quantization_config + a separate `hf_quant_config.json`), 1967 tensors, **15 BF16 `mtp.*` tensors with byte-identical keys to our graft** (in `exclude_modules`), namespace `model.language_model.*`. Weights stored as `weight`+`weight_scale`+`input_scale` (modelopt) vs compressed-tensors packing — THAT format diff is why MTP loads on modelopt but not compressed-tensors (mtp tensors themselves are identical). -- **THE PIPELINE IS UNCHANGED except the quant step:** reuse the graft (`heretic2-mtp-bf16`) → **quant with `nvidia-modelopt` instead of llm-compressor** → reuse `splice_mtp.py` (bf16 mtp) → serve `--quantization modelopt` + `--speculative-config qwen3_5_mtp` (AEON's `vllm-aeon-rp` is the exact serve reference). -- **modelopt API de-risked:** `pip install nvidia-modelopt` → **v0.45.0 installs** in the vLLM container; `mtq.quantize(model, config, forward_loop)` + `NVFP4_DEFAULT_CFG` + `export_hf_checkpoint(model, export_dir=…)` all present (the AEON PTQ→HF-export path). Load as `AutoModelForImageTextToText`; exclude visual/linear_attn/mtp/lm_head/embed to match AEON's `exclude_modules`; 512-row chat calib as the `forward_loop`. -- **RESIDUAL RISKS:** (1) modelopt 0.45 warns "not tested with transformers 5.12.1" → try `nvidia-modelopt[hf]` or a transformers pin; (2) modelopt PTQ tracing the GDN-hybrid arch for calibration (AEON proves feasible); (3) building the exclude/quant config to match AEON exactly. (4) needs another ~30-min GPU0 window (PTQ calibration runs on GPU). -- **NEXT STEPS:** write `quant_modelopt.py` (adapt `quant_nvfp4.py`'s load+calib, swap the quant body to `mtq.quantize`+`export_hf_checkpoint`) → GPU0 window → serve modelopt+MTP → verify acceptance >0% → brokkr P00. -- **Seats restored + healthy; char-rp-reasoning STAYS on the validated GGUF NEO-CODE seat** (~59.5 tok/s, R36-passing) until a modelopt-MTP seat proves a real win. GGUF is backstopped by worldtree's 300s wall-clock. Artifacts kept on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/`: `heretic2-nvfp4-cg` (coherent no-MTP), `heretic2-nvfp4-cg-mtp` (spliced, MTP-inert on compressed-tensors), `heretic2-mtp-nvfp4-prod` (original gibberish, keep for diff). Fixed `quant_nvfp4.py` staged at `ana-ml2:~/quant_nvfp4_cg.py`. - -### ★ (historical, SUPERSEDED by the above) NVFP4+MTP compressed-tensors recipe — the quant chase - -**GOAL:** NVFP4-quantize NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) + graft the Qwen3.6 MTP head → a ~2-4× faster **vLLM** char-rp-reasoning seat replacing the ~59.5 tok/s GGUF NEO-CODE seat (buys reasoning-budget headroom for soong's 45s client timeout). NVFP4 is **Blackwell-only** → ana-ml2 GPU0 is the ONLY place it runs (irv-ml1 is Ampere). - -**★ REFERENCE — MATCH IT:** `pantheon-27b-mtp-nvfp4` (`/tank/aimodels/pantheon-27b-mtp-nvfp4`) is a WORKING 27B NVFP4+MTP seat on our Blackwell, SAME `Qwen3_5` arch. Its `config.json` (architectures `Qwen3_5ForConditionalGeneration`, empty mtp fields, `mtp.*` tensors BF16 top-level) + the `vllm-aeon-rp` serve command (`/opt/docker/compose/qwen36-27b-aeon/compose.yaml`) are the gold standard. When my output diverges from pantheon, **pantheon is right** — diff against it. - -**ARTIFACTS** (ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/`, infra-ops-owned): -- `heretic2-mtp-bf16/` — grafted BF16 (Heretic2 1184 + 15 MTP = 1199 tensors) [GOOD] -- `heretic2-mtp-nvfp4-prod/` — NVFP4 output (spliced + config-merged) [**SERVES GIBBERISH — rebuild it**] -- `production_calib_512.jsonl` + `calib/` — brokkr/Dvalin's render-verified 512-row workload-matched calib (tool_call_xml 128 / reasoning 128 / longctx 32 / nsfw 64 / rp 160) -- SCRIPTS (committed in eshpfi `services/heretic2-nvfp4-quant/`): `graft_mtp.py`, `quant_nvfp4.py`, `splice_mtp.py`, `serve_nvfp4.sh`, `run_quant.sh`, `render_verify.py` - -**THE RECIPE (steps 1-3 WORK; 4-5 are the gibberish suspects):** -1. **GRAFT** (CPU, seats up): `graft_mtp.py --heretic2 --base --out heretic2-mtp-bf16` in a vLLM container. Symlinks Heretic2 shards + splices 15 base MTP tensors (`mtp.*` top-level, base shards 13+15). [WORKS] -2. **QUANT** (GPU0 freed): `run_quant.sh chat` → vLLM container + `pip install llmcompressor tiktoken sentencepiece` → `quant_nvfp4.py`. `QuantizationModifier(targets=Linear, scheme=NVFP4, ignore=[lm_head,embed,visual,linear_attn,norms,mtp])`; chat calib via `apply_chat_template` (args parsed string→dict). Out = compressed-tensors `nvfp4-pack-quantized`, 256 layers packed, GDN/vision/lm-head/norms BF16. ~19 min. [MAIN LAYERS GOOD] ⚠️ `AutoModelForCausalLM` loads it as `Qwen3_5ForCausalLM` (strips multimodal) → config arch becomes ForCausalLM AND the top-level `mtp.*` keys are DROPPED at load (0 mtp in output; the `re:mtp.*` ignore is moot — mtp never enters the module tree). -3. **SPLICE** (CPU): `splice_mtp.py ` — load the 26GB quantized shard + add 15 BF16 mtp tensors → rewrite single shard (1634 tensors). Matches pantheon. [WORKS — mtp present, BF16] -4. **CONFIG** (⚠️ #1 SUSPECT): merged grafted config (arch `Qwen3_5ForConditionalGeneration` + multimodal structure) + the quant's `quantization_config` so vLLM resolves `Qwen3_5MTP`. NEEDED for MTP but is the prime gibberish suspect (I never served with the quant's native ForCausalLM config). -5. **SERVE** (GPU0): `serve_nvfp4.sh` → vllm/vllm-openai:v0.24.0. GOTCHAS (all captured in the script): ENTRYPOINT is `["vllm","serve"]` → pass ` --flags`, do NOT add `serve`. `--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'` (needs ConditionalGeneration arch to resolve). `--language-model-only` (skip vision; else "Can't load image processor"). `--mamba-cache-dtype float32 --max-num-seqs 256` (else "max_num_seqs 1024 exceeds Mamba cache blocks 304"). `--quantization compressed-tensors --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice`. → LOADS + MTP detected (embedding/lm_head sharing), healthy on :8018. [SERVES but **GIBBERISH**] - -**THE FAILURE:** raw `/v1/completions` returns `!!!!!!…` (degenerate, token-0). Broken weight-load. Spec-decode is lossless so NOT the MTP draft. 3 unisolated suspects: (1) the config-merge [step 4, most likely], (2) the splice [step 3] mangling the packed shard on re-save, (3) the quant itself. - -**★ DIAGNOSTIC LADDER (do this — needs a GPU0 window):** -1. **FIRST, validate the tracer bullet:** re-quant clean (`run_quant.sh`) → serve with the quant's NATIVE `Qwen3_5ForCausalLM` config, **NO config-merge, NO MTP, NO splice** → smoke `curl /v1/completions -d '{"prompt":"The old tavern","max_tokens":40}'`. **Coherent → quant is FINE** (config-merge/splice/MTP broke it, incrementally add each back to isolate). **Gibberish → the quant is broken** (calib/scheme rabbit hole). -2. Diff my `config.json` + serve flags byte-for-byte vs pantheon's (pantheon WORKS). -3. Only after coherence is confirmed, add MTP back (arch→ConditionalGeneration + splice + spec-decode) and re-smoke at each step. - -**GPU WINDOW ritual:** graft is CPU (do seats-up first). Take down GPU0: `docker stop llama-charrp llama-charrp-reasoning vllm-aeon-gen` (GPU0→97G free); restore `docker start …` (all healthy ~230s). Heads-up wt-dev (character/thoughtful-character + gen route through these). The GGUF NEO-CODE seat (:8018) is the always-restorable validated fallback. ⚠️ LESSON from the 2026-07-14 window (see Tried and abandoned): smoke a PLAIN coherence check on the simplest config FIRST — do not build MTP scaffolding on an unverified quant. - -### Landed this session (2026-07-14) — history - -### ✅ #355-residual — RESOLVED 2026-07-14 via MODEL SWAP (Deckard→NEO-CODE); was ACTIVE 1 - -**UPDATE 2026-07-14 — FULLY RESOLVED via MODEL SWAP.** The custom llama.cpp (PR #25544 multi-terminator) was necessary but **INSUFFICIENT alone** (Deckard still ran away ~5% of tool-turns). Root cause was **MODEL-LEVEL**: Deckard emitted Qwen's NATIVE qwen3_coder XML tool format **MALFORMED** → llama.cpp leaked the closing tags into the arg value (`{"name":"\n\n..."}`) → Bifrost `attach_tool` schema error → retry → reasoning-runaway. **Swapped the model to NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** (GGUF base_model `DavidAU/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking`; "NEO-CODE" = DavidAU's imatrix quant recipe, not a code finetune) which emits the same native format **CLEANLY** (`{"name":"Sindra"}`) on the same seat/parser. **R36 gate (brokkr 2026-07-14): NEO-CODE ships on ALL axes** — tool-calling **0.967** (attach_tool 1.00, **0 runaways**, beats gen 0.856 + Deckard 0.08/0.80), prose genre-artifact-fine (D2-slop repetition = dialogue-tags/anatomical genre convention, LESS clichéd than gen — not degradation), refusal uncensored-as-spec (creative-dark 38/40) + CSAM-gate-clean (0/100 operational-harm → serve-layer guardrails are the NSFW-RP consumer's job). **LIVE in production** (Q5, 256K ctx, card samplers temp 1.0 — temp 1.0 IMPROVED tools; DRY dropped); Deckard = rollback (`.env`/compose `.bak-pre-neocode-20260714`). **Custom llama.cpp KEPT** (now needed for qwen3_coder parsing — stock b8840 predates it, so stock is NOT a valid NEO-CODE rollback — + #25544 belt-and-suspenders). There was **NEVER a "wrong parser"** — the XML IS Qwen3.5/3.6-native (byte-identical to `Qwen/Qwen3.6-27B`'s template); an earlier "misconfigured OpenHands template" config-override reframe was WRONG and retracted. **OPEN:** seat is ~50 tok/s GGUF → soong's 45s client-timeout + the reasoning-budget-latency squeeze remain the constraint; the DFlash spec-decode recipe (z-lab Qwen3.6-27B drafter, ~4× on vLLM) applied to an NVFP4 Heretic2 vLLM seat is the future faster-seat path (NOT AEON-7-the-model — a non-RP-tuned base abliteration whose 4× speed is DGX-Spark-locked). Full settled record: auto-memory [[charrp-custom-llamacpp-pr25544]]. Historical mid-investigation saga kept below for context. - -**BOTTOM LINE (historical):** char-rp-reasoning ran away to `max_tokens=32768` (~22 min, ~13% of turns) because stock llama.cpp **b8840's reasoning-budget forcing can't match Qwen3.5's `` reasoning terminator** (it only knows ``) → on the tool-retry path forcing never fires. **FIXED** by a custom build (master `6eddde0` + unmerged upstream **PR #25544**, multi-terminator) now **LIVE** on the seat (`llamacpp-charrp:custom-latest`). **PARTIALLY VALIDATED**: the fix cut the runaway RATE hugely (soong smoke — most tool-turns now complete ~90 tokens, reasoning ~40 tokens, clean) BUT has a **RESIDUAL GAP** — task 5674 (2026-07-13 ~21:43) still ran away to 32768 (~9min; reasoned >45s before any content → the cap didn't fire on that turn). b61's 300s = client backstop but does NOT propagate to the seat → seat still burns to 32768 (confirmed). **⚠️ DO NOT retire the custom build as "0-gain" — CORRECTED: the seat is meant to be an RPG-GM (tools + prose + NSFW), so the tool-path #355 fix is the TOOL-PATH ENABLER for any reasoning-RP model here → KEEP it.** The REAL problem is the MODEL: **Deckard (RP-abliterated) is a poor tool-caller** (rambles/loops on tool-turns even with the fix — abliteration + heavy RP-tuning damages tool-calling/instruction-following). **MODEL-SWAP underway:** brokkr running a hunt for an NSFW-abliterated + **RELIABLE-tool-calling** + GM-prose **~24-27B** replacement (thread `01KXEQWEN2…`; ★ tool-calling is now a DISQUALIFYING A/B axis — the original A/B that picked Deckard scored prose/loops/refusals but NEVER tested tools). **PLAN: leave soong on char-rp-reasoning; swap the MODEL under it when brokkr delivers (no double-push).** Still **REMOVE-WHEN-#25544-MERGES** for the custom llama.cpp itself (retire custom→stock llama.cpp when the fix ships upstream; independent of the model swap). Records: auto-memory `charrp-custom-llamacpp-pr25544` + eshpfi `stacks/char-rp-gguf/llamacpp-custom/`; rollback = `.env` `LLAMA_REASONING_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda`. Deploy-speed side-quest (#359+#360, ~11m→~4m) + soong-lab CI/CD (green-gated push-to-deploy + red-relay) also landed this session. Full diagnostic saga below (kept for the retire-decision context). - -### 🔺 #355-residual ROOT CAUSE (historical saga): seat reasoning-loop to max_tokens -- **DEFINITIVE root cause (3 independent sources, 2026-07-13):** char-rp-reasoning (Deckard-PKD, seat ana-ml2:8018 `llama-charrp-reasoning`) intermittently enters a **non-terminating REASONING loop** and runs to the request's **`max_tokens=32768`** → ~22 min at 24.7 tok/s. ~13% of requests (4 of ~30 in the wedge window). It is NOT the gateway holding, NOT a slot leak, NOT the seat idle — the seat genuinely GENERATES 32768 tokens. My earlier "LiteLLM holds ~21 min while seat idles / slot-leak" read was **DISPROVEN** (the seat wasn't idle; there's no `max_parallel_requests`/`router_settings` in the config at all → no semaphore to leak). -- **Evidence:** (a) spend_logs: 4 requests each `completion_tokens = EXACTLY 32768`, status=success (hit the length cap cleanly); normal requests EOS at 730–2250. (b) seat llama.cpp log: `eval time = 1325064ms / 32768 tokens (24.73 tok/s)`, `truncated=0`. (c) WT↔gateway pcap: runaway stream is **100% `reasoning_content` deltas**, coherent, planning tool-call retries ("The tool call failed due to input schema errors… Let me make these calls one at a time"). NEVER exits ``. -- **MECHANISM — RESOLVED (Step-1 non-disruptive + Dvalin source read, 2026-07-13):** the budget **ARMS but its force-close NEVER FIRES** in the seat's build. Seat log aggregate: 158 `reasoning-budget: activated, budget=400` / 110 `deactivated (natural end)` / **0 forced closes** / 48 never-closed arms (= the runaways). Build = **llama.cpp 8840 (`9e5647aff`)**, image `ghcr.io/mostlygeek/llama-swap:cuda`. Dvalin read commit 9e5647aff: force-inject is **NOT gated on `--reasoning-budget-message`** (server-task.cpp:493-495 `if (!end_tag.empty())`, no message guard; reasoning-budget.cpp:99-108 fires unconditionally at remaining≤0) → so it's genuinely broken in b8840, NOT the missing flag. NOT a re-arm loop (9e5647aff's DONE is a plain break; re-arm landed later on master). Every request carries 8 tools; runaways = the tool-retry-FAILURE subset (model won't self-terminate ``, forcing doesn't save it). `reasoning_control` API is ABSENT in 8840 (`--help` has no flag; binary has no `/control` strings) → brokkr's watchdog needs the bump too. -- **FIX (agreed with brokkr, NO max_tokens ceiling per operator):** (1) IMMEDIATE relief = push worldtree's **b61 270s wall-clock** — bounds the user-facing hang regardless of the seat loop (so the seat bump is NOT urgent). (2) DIAGNOSE-THEN-FIX (Eitri refinement, may SAVE the bump) — first capture the detailed tuple `tokens/start/end/forced` (needs a **verbose / low-budget-matrix seat restart**; 8840's default log has only the simpler activated/deactivated line — no `forced=`/`start=…end=`, confirmed by grep): **`end=0` → the FORCED-injection end_tag isn't reaching the budget path (Deckard template / reasoning-format mismatch) → CHEAP CONFIG fix = explicit budget start/end tags, a restart NOT a bump**; `end>0,forced=0` → end-tag tokenization bug; `end>0 + no forcing across arms` → real state bug → build bump. (NB: 110 `deactivated (natural end)` prove the DETECTION end-tag works, so an `end=0` would be specifically the FORCED-inject tag, a narrow config miss.) Only if the tuple says end>0 → **DURABLE = stage + validate a post-#22740** llama.cpp build OFF the hot path (⚠️ #22717 garbled-forcing regression ~b9018 fixed by #22740 → don't land in the b8840→b9018 window; validate vs `tests/test-reasoning-budget.cpp` + that it loads Deckard Q5_K_M), then surface the seat-image swap to Vuong; `--reasoning-budget-message` additive WITH the bump. Getting the tuple = the disruptive step (bounce the prod RP seat to --verbose/low-budget; a separate test instance won't fit — GPU0 has only ~4G margin). **brokkr FINAL rec (thread wrapped 2026-07-13): go STRAIGHT to the staged bump — the tuple-diagnostic / tag-probe is a LONG-SHOT (the 110 natural closes prove the parser knows the delimiter → more likely a build/state bug than a tag-carry miss). **RESOLVED WITHOUT the bounce (brokkr msg16 deduction): the `deactivated (natural end)` log is the budget SAMPLER's namespace → the sampler matched its `reasoning_budget_end` token → end_tag NON-empty (110×); and per source `forced = tokenize(message+end_tag)` is built in the SAME block → forced also NON-empty. So the tuple is logically end>0/forced>0 → forced tokens EXIST, forcing just never FIRES = STATE-MACHINE bug → the post-#22740 BUILD BUMP. The `end=0` cheap-config branch is RULED OUT. NO diagnostic bounce needed (would only confirm) → no prod-seat-restart go required from Vuong for diagnosis.** XTC HELD. b61 = the immediate relief → NO rush. Deliverable owed to brokkr = the validated bump-target (loads Deckard Q5_K_M + passes tests/test-reasoning-budget.cpp AND — the real acceptance test — a **live low-budget check on the Qwen3.5/Deckard arch showing an ACTUAL forced-close, not just natural ends**, since a forced-close-that-never-fires is exactly what 8840 gets wrong); Dvalin standing by for a 2nd read, Eitri if arch specifics get hairy.** **VERIFICATION (2026-07-13, WebFetch ggml-org/llama.cpp — the build-bump has NO released target; the dwarf "bump to post-#22740" was WRONG, #22740 fixed a DIFFERENT regression #22717):** the real fixes are UNMERGED — #22684 (Qwen3.5/3.6 tool-call-in-`reasoning_content` = our TRIGGER) is **CLOSED-AS-NOT-PLANNED**; #24202 (Qwen tool-parsing) is **OPEN**; **#25544** (reasoning-budget sampler → MULTIPLE terminating sequences) is **OPEN, awaiting review** = OUR forcing fix. MECHANISM NAILED: 8840's budget sampler knows only a SINGLE end tag (``), but Qwen3.5's tool-path terminates reasoning with `` which the sampler can't match → forcing never fires on the tool path (explains 110 natural-`` closes working + 0 forced + tool-retry specificity). **RE-FRAMED PLAN: no bump target exists yet → b61's 270s wall-clock is the ACTUAL mitigation (not just interim); TRACK PR #25544 and bump only when it merges into a release + validate the forced-close empirically. Seat XTC = optional loop-frequency fallback, not a fix.** **BUILT 2026-07-13 (operator directed the custom build after we verified NO released fix exists): image `llamacpp-charrp:6eddde0-pr25544`/`:custom-latest` on ana-ml2 (reports b10001, merged HEAD `6ce128d0b` = master 6eddde0 + PR #25544, CUDA 12.8.1 sm_120, server target `/app/llama-server`), binary smoke-tested OK. Reproducible recipe + records: eshpfi `stacks/char-rp-gguf/llamacpp-custom/` (build.sh + README) + auto-memory `charrp-custom-llamacpp-pr25544`. **SWAPPED + LIVE 2026-07-13 (operator: swap now)** — reasoning seat recreated on `llamacpp-charrp:custom-latest` (via new `LLAMA_REASONING_IMAGE` compose var; Magidonia char-rp stays stock); Deckard loads clean on Blackwell, serves coherent output, reasoning bounds at 400 (non-tool forcing intact). Per-request `reasoning_budget` override IGNORED (CLI-400 pins it, expected). New build's log DROPPED the `reasoning-budget: activated` line (b10001 format change) → can't watch forcing via seat log; DEFINITIVE tool-path proof = spend_logs `completion_tokens=32768` runaways STOP (baseline 4 today pre-swap). Rollback = set `LLAMA_REASONING_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda` in `/opt/docker/compose/char-rp-gguf/.env` + `docker compose up -d --no-deps llama-charrp-reasoning` (8840 image d6c39f55 preserved). On-seat acceptance = Deckard loads + serves + a FORCED close fires at low budget (not just natural ends). #355 DEPLOY-SPEED task separately CONFIRMED landed: #359 merged→build-and-deploy 11m→~5m (~2×, wt-dev corroborated 5m/4m vs 11m); b61 pushed (v1.0.0b61 747d7a0) = 300s client backstop live on deploy.** (3) Optional interim = mild XTC on the seat (behavior-shaping, not enforcement) — holding unless the loop-frequency reduction is wanted pre-bump. Seat config: canonical `stacks/char-rp-gguf/compose.yaml` (service `llama-charrp-reasoning`), live `/opt/docker/compose/char-rp-gguf/` on ana-ml2. **Cheap `--reasoning-budget-message`-alone test is DEAD (source-confirmed won't fix it) — did NOT waste a restart on it.** -- **Operator direction:** find the SERVER-side fix (params/config), do NOT ceiling max_tokens (legit long outputs must survive). Routed to brokkr-smithy-dev (thread 01KXEEQ0TD6D…) → he accepted, pulled dvalin for the known-bug search, running the dwarf consult. I offered to run the live-seat with/without-tools split-test on his signal (own the seat; mildly disruptive → off-peak). -- **Wasted-GPU side issue + b61 re-measure:** LiteLLM logged these `success` at full 1255–1390s → on b60 it did NOT abort at WT's cancel (whatever WT did, no upstream abort → ~18min wasted GPU/runaway). worldtree-dev's **unpushed b61** adds a **270s provider wall-clock** that runs `_bounded_stream_close`→httpx FIN (force-closes WT's upstream side). So post-b61 the FIN-check flips to "WT closes at ~270s"; the only open question is whether **LiteLLM propagates that client-disconnect to the seat** (my gateway domain). **RE-MEASURE after b61 deploys** — don't assume b60 behavior. -- **Data on hand:** pcaps at `corviduo-dev:~infra-ops/py-spy-355/wedge_*/` are only ~99s windows triggered on the attach_tool failure (too short for the 300s cancel — a limitation, not usable for the FIN-check). LiteLLM config `/opt/docker/conf/litellm/config.yaml` → `/app/config.yaml`; spend_logs = `litellm-db` (`docker exec litellm-db psql -U litellm -d litellm`; camelCase cols double-quoted; request body in `proxy_server_request::jsonb->>'max_tokens'` top-level). - -### 🔺 ACTIVE 2 — Worldtree deploy-speed: DELIVERED as PR #359 (green-lit, awaiting merge) -- **Real bottleneck was NOT uv sync (memory's assumption was WRONG).** Pulled the buildx STEP log (gitea `actions_log` on ana-docker, `docker exec gitea` → `/data/gitea/actions_log/vh/Worldtree/*/{task}.log.zst`, zstd). Sub-steps of the ~7min build: `#24 RUN useradd … chown -R /app /data … = **251.2s**` ← the monster; `#25 export image+push = 123.7s`; `#13 uv sync --no-cache = **only 35.6s**`; `#26 cache-export mode=max = 18.4s`. The runner-side registry LAYER cache (`--cache-from/--cache-to type=registry:buildcache,mode=max`) is **already wired in deploy.yml** — my prior "runner cache missing" note was STALE. -- **Root cause:** `chown -R worldtree:worldtree /app` forces BuildKit to copy-up the entire root-owned `.venv` (torch + 125 pkgs) into a fresh layer → ~4min + a duplicate venv baked into the image (also slows export/push). -- **Fix (PR #359, branch `infra/build-cache` off origin/main@b60):** (1) drop `/app` from the chown (keep `/data`); (2) add `--mount=type=cache,target=/root/.cache/uv` + drop `--no-cache`. **VALIDATED against BOTH live containers** (demo+personal): zero files written under `/app` post-boot (PID1-start ref) — all runtime writes → `/data/state`, `/data/kb`, `/home/worldtree`; `/app` is read-only for uid 1000, `/app/config` chowned at first boot by the root entrypoint shim. So the chown was pure waste. Expected **~5min off** (~11→~6min). -- **Status:** worldtree-dev GREEN-LIT (confirmed `--no-cache` not deliberate; b61 is theirs, core/llm-only, zero Dockerfile overlap). PR open + mergeable; they review, merge→main triggers the real build that measures the actual speedup. Deploy trigger unaffected (Dockerfile change → builds). **Done from my side.** - -### Landed this session (2026-07-13) — context for the fresh session -- **WT #355 turn-lifecycle fix VALIDATED** (worldtree b60): wedged turns self-terminate cancelled/stalled at the 300s watchdog (turns 2064/2065 = 302s/360s) vs pre-b60 turn 2061's 16-min no-terminal silence. The residual (ACTIVE 1) is a separate LiteLLM matter. -- **Ledger tier-3 provisioned** (`ledger:miranda`) on personal `:8081`: key minted (user `ledger`, key_id `b38932f5`), GPG-delivered + verified + cleartext shredded; scope confirmed (user tier has `bifrost:invoke`+`agents.define`); allowlist `10.100.10.50:8770` activated by the b60 recreate. -- **`assistant`+`thoughtful-assistant` roles** added (bind `gen`/`gen-reasoning`, kind capability, grants `[foundational]`) on personal+demo; canonicalized by worldtree-dev (`d8bd497`). -- **3 zombie soong sessions retired** (2ad34ec6+7d23052f+0775c9b1 via `POST /admin/sessions//retire`); pre-b60 stale-active backlog HELD (worldtree's #356 rehydrate fix reclaims that class). - -### Still-open standing items (carried) -- **T1 SFT LoRA (AEON-27B E-RP)** — adapter banked but BLOCKED on hot-swap serving; DORMANT (operator pivoted to off-the-shelf). Merge fallback primed (~2-3h) if hot-swap abandoned. -- **Backups** — rotate the 5 disclosed rest-server creds (operator, offline); confirm cross-site restic mirror. -- **althing monitor dies on /clear** — re-arm `/althing:monitor` in the fresh session (open await-reply loops: brokkr on the #355-residual, worldtree-dev on the deploy-resilience follow-ons #356/#357/#358). +**Open loops for the fresh session:** +- **`stacks/heretic2-charrp-reasoning/` is UNTRACKED in git** (`?? stacks/heretic2-charrp-reasoning/`) — the char-rp-reasoning compose stack is deployed but never committed; commit it (canonical mirror of the live seat). +- **althing monitor** — re-arm `/althing:monitor` in the fresh session (the wake-listener dies on /clear). Open watch: comfy-dev on the arbo bake (no news = clean). +- **Carried standing items** (non-blocking): T1 SFT LoRA dormant (operator pivoted off-the-shelf); rotate the 5 rest-server backup creds (operator, offline); brokkr P00 handoff for the NVFP4 char-rp-reasoning seat (in live traffic ahead of it). ## Recent decisions +- `[2026-07-15]` **arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake.** Operator-directed full switch (comfy-dev executed, live in prod). Established: gen (`qwen3.6-35b-a3b-heretic`) is vision-enabled and was image-bench's predecessor as arbo's hero-judge; image-judge actually serves 4 roles (vision quality-scoring + identity-scoring + bbox grounding + an uncensored text tier), not just grounding. comfy-dev spot-check: gen faster on every task, grounding within ~3px, uncensoring preserved, and it FIXED a bug (image-judge's reasoning preamble broke json_object + stalled the router). Sequencing = short prod bake then evict (~30 GB GPU1 reclaim); revert = flip `ARBO_VISION_MODEL`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`. +- `[2026-07-15]` **esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service`** (the prior `After=remote-fs.target` drop-in was silently defeated by `nofail`). Reached only after a REBOOT (D-state phantom containers uptime-kuma + paperless-web that no `docker`/`ctr`/daemon-restart could clear). Committed `21d9a07` + playbook updated. See Tried and abandoned. +- `[2026-07-15]` **Homepage AI-tab revamp** — flat "AI Systems" group -> dedicated AI tab, 6 role-based groups + AI-Dormant; committed `569e1af`, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.) +- `[2026-07-15]` **Home Assistant config repo created** (`vh/home-assistant-config`, private). UI-managed HA -> allowlist model (YAML + curated secret-free `.storage` subset). git-in-place in `/config` on esh-docker-vm + scoped deploy key + local clone `~/development/home-assistant-config`. +- `[2026-07-15]` **char-rp-reasoning OOM rescue** — solo-restart on the packed GPU0 crash-looped; fixed via `expandable_segments:True` + util 0.39->0.38 + max-model-len 192K->150K. LESSON (Tried): `max-model-len` does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom now. +- `[2026-07-15]` **soong-lab `SOONG_LAB_LIBRARY_DIR` made persistent** (corviduo-dev) — was on the redeploy-wiped code default; set to `/home/infra-ops/soong-lab-data/library` (mirrors PORTRAIT_DIR), restarted. Closed a queued no-rush item; unblocked the operator. +- `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost (`.cost.total_cost_usd`) / threshold-colored ctx+rate (green<60 / yellow60-90 / red>90). + +- `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug. - `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section. - `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`. - `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]]. @@ -458,6 +384,13 @@ _142 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-07-15]` **`docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts** — `nofail` drops a mount out of remote-fs.target's blocking set, so the drop-in ordering is silently defeated (paperless still Exited(255) on reboot). Real fix = DIRECT mount->docker ordering via the fstab `x-systemd.before=docker.service` option (verify `systemctl show docker -p After` lists the mnt-*.mount units). esh-docker-vm. +- `[2026-07-15]` **The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT** — reconfirmed: `docker stop/rm -f`, `ctr -n moby task delete`, AND `systemctl restart docker` all fail to clear it; `docker exec` into a wedged container ALSO fails (`setns ... exit status 1`), so the in-place restart escape hatch is out. Worse, a daemon restart can HALF-KILL other healthy containers (knocked paperless's granian down + left it wedged). Process dead but dockerd won't reap -> phantom. NFS mounts are `_netdev,nofail` so the reboot is boot-safe. +- `[2026-07-15]` **vLLM `max-model-len` does NOT free GPU VRAM** — the KV cache POOL is sized by `gpu-memory-utilization`, not max-model-len. Lowering max-model-len only caps per-request context + drops max concurrency; the pool still fills the util budget. To actually free VRAM, lower `gpu-memory-utilization`. (Bit the char-rp-reasoning "drop KV to 150K" ask: the 150K applied but freed 0 VRAM until util dropped 0.39->0.38.) +- `[2026-07-15]` **Claude Code statusline `.cost.total_cost_usd` is per-SESSION** (Claude Code's own cache/model-aware session accounting), not a lifetime aggregate — the large value just reflects a long, multiple-times-summarized session. And the old statusline hardcoded Sonnet pricing ($3/$15) on an Opus session -> ~5x cost understatement. + +- `[2026-07-14]` **MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround.** All crashed the same way (`qwen3_5_mtp.py:256` `param_data.shape == loaded_weight.shape` AssertionError — bf16 mtp head loaded into a quantized drafter param): (1) mtp excludes in `config.json` (WRONG file — vLLM modelopt reads `hf_quant_config.json`); (2) specific-unfused mtp names in hf_quant_config; (3) wildcards `mtp*`/`mtp.layers.0*` (`is_layer_skipped` is EXACT-membership, NOT glob — wildcards match nothing); (4) exact fused+unfused names in both `mtp.`/`model.` prefixes. Instrumenting `is_layer_skipped` proved the drafter's exclude list holds ONLY the main model's `linear_attn` entries — the mtp excludes never reach the draft-model quant config. ONLY fix = a mounted `sitecustomize` force-skipping `mtp.*`. LESSON: don't chase checkpoint-config fixes for the mtp-drafter crash; go straight to the runtime patch. Also `nvidia-modelopt[hf]==0.43` (AEON's producer version) is a trap — it pins transformers back to 4.57 which can't load `qwen3_5` at all; use 0.45 + the FusedMoE guard in `quant_modelopt.py`. +- `[2026-07-14]` **AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof.** `vllm-aeon-rp`'s .env → `AEON_RP_MODEL=pantheon-27b-mtp-nvfp4`, `AEON_RP_QUANT=compressed-tensors` — it LOADED (mtp silently skipped, `exited 0`) but never accelerated. Same vLLM image (`:latest` = `sha256:4091d55` = 0.24.0) as the failed Heretic2 test, so the "AEON ran on an older vLLM" theory was wrong. Don't treat a seat that "ran" as MTP-validated without checking its `SpecDecoding` acceptance. - `[2026-07-14]` **NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept.** Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp weights off a compressed-tensors main model → `Parameter … not found in params_dict`, `Avg Draft acceptance rate: 0.0%`). Pantheon is identical — its "working NVFP4+MTP" was working *structure*, never real acceleration. Working native MTP needs the **modelopt** main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt. - `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state. - `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip.