The fast char-rp-reasoning seat works: ~77 tok/s (vs GGUF ~59.5, base NVFP4 ~53),
MTP draft-acceptance 32-40%, mean acceptance length 2.19. Same Heretic2/NEO-CODE
model, NVFP4 + native qwen3_5_mtp spec-decode.
Full end-to-end recipe + the four landmines in docs/runbooks/heretic2-nvfp4-mtp-seat.md:
(1) load as AutoModelForImageTextToText not AutoModelForCausalLM (namespace/gibberish);
(2) modelopt format not compressed-tensors (compressed-tensors MTP = 0% accept);
(3) modelopt 0.45 <-> transformers 5.12.1 FusedMoE crash (guarded in quant_modelopt.py);
(4) vLLM 0.24.0 does NOT propagate modelopt exclude_modules to the spec-decode draft
model -> BF16 mtp head gets quantized -> shape crash; no checkpoint config fixes it
(is_layer_skipped is exact-membership not glob) -> fix is a mounted sitecustomize that
force-skips mtp.* in is_layer_skipped (upstream vLLM bug to report).
Scripts: quant_modelopt.py (FusedMoE guard + single-shard export + multimodal load),
finalize_modelopt_mtp.py (splice bf16 mtp), serve_modelopt_mtp.sh, run_quant_modelopt.sh,
sitecustomize-mtp-workaround.py.
Root-caused the NVFP4 gibberish to a quant-namespace bug: quant_nvfp4.py loaded
via AutoModelForCausalLM -> text-only Qwen3_5ForCausalLM -> flat model.layers.* keys,
but vLLM 0.24 serves only Qwen3_5ForConditionalGeneration (whose weight mapper needs
model.language_model.*). Fixed by loading as AutoModelForImageTextToText; NVFP4 now
serves coherent (validated greedy on ana-ml2 GPU0).
Base NVFP4 (compressed-tensors) measured ~53 tok/s (~= GGUF at batch-1, no single-stream
win) and its MTP is 0% acceptance (vLLM's Qwen3_5MTP drafter loads the bf16 mtp head only
off a modelopt main-model checkpoint). Added quant_modelopt.py (nvidia-modelopt PTQ,
matches AEON's NVFP4 W4A4 g16 + lm_head/linear_attn/visual exclusions) as the path to
working native MTP; graft + splice + serve otherwise unchanged.
Captures the full pipeline recipe (graft->quant->splice->config->serve) with every
gotcha found this session, the 3 gibberish suspects, and the diagnostic ladder
(validate native-config no-MTP coherence FIRST) for a fresh session to finish the
chase. Also stages the NVFP4 scripts + 512-row calib. Recent decisions: NEO-CODE
seat swap (R36), webhook ALLOWED_HOST_LIST fix. Lessons: validate-tracer-bullet-first,
mtp-graft-dropped-at-load, gitea-204-red-herring.
The ufw fix (prior commit) was necessary but insufficient. The DECISIVE blocker
was gitea webhook.ALLOWED_HOST_LIST = 'external, 10.100.0.0/16' (NH3 only) —
corviduo-dev is 10.250.50.152 (Anaheim), so gitea refused to deliver ('deny
10.250.50.152') and never opened the TCP connection. Fixed to 'external,
10.0.0.0/8' (whole fleet, matches the ufw choice) + gitea restart.
Listener now logs every delivery (source-IP/hmac_ok/ref/action) — the old
log_message=pass silence hid the whole failure. Proven end-to-end: real gitea
delivery -> hmac_ok=True, ref=main, 202 deploying -> green deploy.
render-verify caught it: Dvalin's calib tool_calls carry OpenAI wire-form JSON
string arguments, but the Qwen3.6 chat template does .items() on arguments (needs
a dict) → jinja TypeError. Parse string->dict in render_verify + the quant's
load_calib_chat. Confirmed: renders the exact qwen3_coder XML the seat emits
(prefixed bifrost.soong-lab.*, v0.3.13 generate_portrait, <think>, <tool_response>).
Extracted from the deployed backend (bifrost/tools.py) via the venv with a
capturing mock register_tool — the LIVE schema, not a stale copy. OpenAI-function
form for brokkr/Dvalin's ~128-row tool-call-XML calib slice (R36 #355 anchor).
soong-dev found the studio serving a stale web/ (52015 vs 55025 bytes — missing the
01-Role section, favicon, thinking-status): the deploy rsynced backend/ but never web/,
so SOONG_LAB_WEB_DIR stayed pinned to the initial manual copy while the backend updated.
Deploy now rsyncs BOTH backend/->studio AND web/->SOONG_LAB_WEB_DIR (read from the env)
on every green run. Verified: served frontend now 55025 bytes, current.
Per operator call (no gitea write token on the Worldtree-team VM): a 2-min systemd
--user timer on nh3-dev polls corviduo's last-deploy.json and pings soong-dev via
althing on a NEW red deploy (green stays silent). Delivers soong-dev's red-run
visibility without a credential on corviduo. Tested (red detect+format DRY, green quiet).
comfy-dev's explicit-over-implicit call: arbo now sends train_id, so the
worker no longer derives the loras/trained/{train_id}/ namespace from
output_dir.parent (which coupled it to arbo's handoff layout). train_id is
optional + path-safe-validated; when present it wins, else the path
derivation remains as the fallback. Wired through TrainRequest ->
validate_request -> published_relative_path -> _publish_lora. 18 tests green.
On a train reaching succeeded, IN ADDITION to output/{name}.safetensors
(unchanged download source), COPY it into ComfyUI's loras search path at
/storetank/arbo/models/loras/trained/{train_id}/{name}.safetensors and
return published_lora_name (the ComfyUI-relative LoraLoader string) in the
terminal GET /train/{id} payload (arbo Phase 2 auto-registration, §4.1/§7).
- Copy not move; a publish failure NEVER fails the train (keeps succeeded,
omits published_lora_name, logs the reason to the tailable run log).
- INV-T7-safe: a copy to a fixed computed path, no new free-form args.
- train_id derived from the handoff layout (output_dir.parent.name).
- Provisions loras/trained/ (arbotrain 2775, group-write per the Phase-1
lesson; world-readable/traversable for ComfyUI) via the deploy playbook.
- ComfyUI verified to resolve nested loras subfolders (no flat fallback).
- Pure path helper unit-tested; 16 tests green.
The first real arbo train 422'd: SDXL checkpoints live at
/storetank/arbo/models/checkpoints/ (the 2026-06-13 move to the 1.8TB
/storetank volume), which wasn't in ALLOWED_MODEL_ROOTS — the old roots
predated the move (/worktank/models is gone, /worktank/comfyui host path
is empty; ComfyUI mounts /storetank/arbo/models -> /basedir/models inside
its container). Allowlist /storetank/arbo/models (llmuser-readable,
world-readable tree), drop the two stale roots. Regression test added (15 green).
comfy-dev cross-check: Sindra v1/v2 used alpha=dim/2 (0.5 LoRA scaling),
which produced the validated likeness; the initial alpha=dim (1.0) was a
stronger, unvalidated default. Align the default to the proven value
(operator/per-request can still override). Tests updated (14 green).
Host service (runs as llmuser, owns /opt/fluxgym + GPU access) that runs
sd-scripts SDXL LoRA training on demand for arbo — the infra-ops half of the
in-arbo LoRA training Phase 1 ownership split (vh/arbo
docs/contracts/in-arbo-lora-training-phase1.contract.md §4.1/§2).
- Fixed-invocation only (INV-T7): bounded params -> one sd-scripts command
shape; every param range/allowlist/path-containment checked before spawn;
bad request = 422, never a silent downgrade. 14 unit tests green.
- Thin supervisor: never imports torch; subprocesses the fluxgym venv's
accelerate. 1-job-at-a-time (arbo lease is the serializer, 409 is backstop).
Durable job records + boot reconciliation (§4.3).
- API: POST /train, GET /train/{id}[/log], POST /train/{id}/cancel,
GET /gpu-status (per-device VRAM + tts_on_3090 co-OOM signal), GET /healthz.
- Wire-shape (§7 resolved with comfy-dev): shared /worktank/arbo/train handoff
(group arbotrain, setgid 2770); worker binds 0.0.0.0:8203, arbo reaches via
host.docker.internal:host-gateway (reachability proven on 172.20.0.1:8203);
device-aware TTS steering via /gpu-status.
Deployed to irv-ml1 via playbooks/deploy-lora-training-worker.yaml (elway,
idempotent); systemd unit active; /healthz + /gpu-status verified live.
vh