memory: snapshot — T1 SFT done (AEON-27B, adapter banked) + hot-swap BLOCKED (vLLM qwen3_5 LoRA no-op #47639), SGLang last-shot = smoke sft_adapter_zc; comfy-dev LoRA worker done (Phases 1/2/2.5)

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-07-06_
+_Last updated: 2026-07-07_
 
 ## Repo purpose
 
@@ -28,7 +28,7 @@ Sister repos (separate gitea repos, deployed by playbooks here):
 | `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) |
 | `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |
 | `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook |
-| `model-training-forge` (mtf-dev) | Fine-tuning recipe forge; **T1 = qwopus E-RP writing LoRA** (SFT→DPO, LitBench-RM reward) | training runs, not a deployed sidecar |
+| `model-training-forge` (mtf-dev) | Fine-tuning recipe forge; **T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06)** (SFT→DPO, LitBench-RM reward) | training runs, not a deployed sidecar |
 
 (`vh/volva` + Heid were re-architected from systemd daemons to Claude Code
 session orchestrators 2026-06-08; their nh3-dev `.service` units were removed —
@@ -102,80 +102,95 @@ no longer deployed sidecars here. See Recent decisions.)
 
 ## Current state / in-flight
 
-_As of 2026-07-06:_
+_As of 2026-07-07:_
 
-- **`gen` is now AEON Qwen3.6-27B (qwopus DISPLACED 2026-07-06).** Dual co-located NVFP4 serves on
-  ana-ml2 GPU0: `vllm-aeon-gen` :8015 (MTP off) + `vllm-aeon-rp` :8016 (native MTP), 256K, multimodal,
-  **vLLM 0.24.0 + LiteLLM v1.91.0**. Gateway gen/gen-reasoning/summarizer-large→:8015,
-  char-rp/char-rp-reasoning→:8016; retired qwen3.5-122-a10b[-reasoning]+qwen-large[-reasoning]; qwopus
-  container stopped (rollback). The reasoning-trace "bug" was the LiteLLM shared-config-mutation footgun,
-  FIXED durably via distinct `-thinking` served-names (→ distinct LiteLLM deployments). Worldtree personal
-  wired: character→char-rp, thoughtful-character→char-rp-reasoning. Full detail: auto-memory
-  `reference_aeon_27b_gen`; `stacks/qwen36-27b-aeon/` (commit e6ab51c). T1 (qwopus E-RP LoRA) is now a
-  SEPARATE parked effort — qwopus stays the LoRA base, but is no longer the live gen.
+- **T1 SFT COMPLETE (AEON-27B E-RP LoRA) — adapter banked, but BLOCKED on hot-swap serving.** SFT ran
+  clean on ana-ml2 GPU0 (all-TRL torch-GDN, batch1/accum8/GC-off, 1 epoch, ~9h @ 0.25 s/s steady, final
+  loss 2.445); adapter at `ana-ml2:~/model-training-forge/data/spike/sft_adapter` (124MB, r16 α32, targets
+  `self_attn.q/k/v/o_proj` + `linear_attn.in_proj_qkv/out_proj`, NO `in_proj_z`). DPO leg deferred (needs
+  mergekit resolved + on-policy `rejected` regen). **THE WALL: the adapter is INERT on the vLLM NVFP4
+  hot-load** — loads HTTP 200 but zero deltas reach inference (vLLM 0.24.0 qwen3_5 LoRA-APPLICATION silent
+  no-op; upstream #47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4
+  AND FP8 both inert) AND adapter-format-agnostic (separate-key `sft_adapter_zc` loads clean via vLLM's
+  allowlist but inert; the fused-key rekey rejected outright). The #47640 guard-patch overlay did NOT fix
+  it. **HARD REQUIREMENT (operator): hot-swappable or ABANDON — merge is NOT a solution now** (static;
+  reserved for a later RP-quant track only).
 
-- **T1 TRAIN — venue decision live; NEXT ACTION = the ana-ml2 SMOKE.** T1 (qwopus E-RP writing
-  LoRA) datasets + recipe + smoke set are all staged (mtf-dev, on `nh3-dev:data/export/qwen-3.5-122b-erp-lora/`
-  incl. `smoke/`); bf16 base is on ana-ml2 `/tank/aimodels/qwopus3.5-122b-a10b-bf16` (233G, PUBLIC
-  HF repo, zero-auth pull). On-prem RULED OUT except a full ana-ml2 shutdown → CPU-offload → ~1-day
-  fleet outage; cloud (Vast.ai 8×80GB, ~3-6h, ~$60-500, zero impact) is the clean alt. **Operator
-  chose smoke-first on ana-ml2** to measure real samples/sec before committing. ⚠️ the smoke ITSELF
-  needs both GPUs + the ~421G vLLM RAM freed (192G VRAM < 244G base → CPU offload) = a brief full
-  ana-ml2 serving pause; keeping any serving up drops it into the ~week NVMe regime. Full runbook +
-  both paths in `reference_t1_cloud_train_plan`. Post-train serve-path (swappable-LoRA-on-NVFP4
-  test) still queued: `reference_gen_qwopus_122b`.
+- **HOT-SWAP — LAST SHOT IN FLIGHT = SGLang (operator: no more formats after this).** SGLang 0.5.13
+  (`lmsysorg/sglang:latest`, on ana-ml2) is the only engine that ATTEMPTS to apply the LoRA (vs vLLM's
+  silent drop): bf16 serves the qwen3_5 arch fine + enters the apply path, but wants its OWN packed-LoRA
+  layout (base r16 + `get_stacked_multiply=3` qkv stacking + TP-sliced qkv B — a THIRD format). **Engine
+  mirror-image finding was PARTLY WRONG — see next.** **NEXT ACTION (mtf-dev msg 17, from reading the
+  0.5.13 source): smoke the EXISTING `sft_adapter_zc` on SGLang bf16 — do NOT build a new format.** SGLang's
+  `utils.py:139 params_mapping` wants SEPARATE q/k/v on disk and STACKS them itself into [r*3,hidden]=[48,5120];
+  the `sft_adapter_fused` I already tested was the WRONG direction (SGLang ×3'd the already-fused weight →
+  [144,5120] → the :773 assert). `sft_adapter_zc` (separate q/k/v/o + in_proj_qkv/in_proj_z, canonical) was
+  NEVER smoked on SGLang and should clear the qkv assert. REMAINING UNKNOWN = the GDN `in_proj_qkvz`
+  (get_stacked_multiply=4; the default utils.py `get_hidden_dim` RAISES NotImplementedError for it, so its
+  real dims live in the qwen3_5 MODEL CLASS override — not in the ref file I sent). Run: `--lora-target-modules
+  all --max-lora-rank 16 --lora-paths t1=/adapters/sft_adapter_zc`. ATTACHES → T1 hot-swaps; ASSERTS on
+  in_proj_qkvz → paste the assert + `cat` the qwen3_5 model-class `get_hidden_dim`/`get_stacked_multiply` to
+  mtf-dev (one targeted fix, or intractable-4-slice → cut to merge, NO more rounds — operator's LAST shot).
+  (SGLang GENERIC image can't LOAD our NVFP4 checkpoint — ModelOptModelLoader packing mismatch — so
+  NVFP4-on-SGLang later needs the dedicated dev image or a requant; the smoke is bf16.) Adapter variants
+  built: `sft_adapter` (separate), `sft_adapter_zc` (separate + zero `in_proj_z`, the SGLang + vLLM-canonical
+  one to smoke), `sft_adapter_fused` (fused — wrong for both engines).
 
-- **LitBench-RM TORN DOWN; comfyui RESTORED (2026-07-06).** Operator called litbench done →
-  `docker rm -f vllm-litbench-rm && docker start comfyui` on irv-ml1; comfyui booted clean (no
-  mmartial venv trap — venv intact, HTTP 200 :8188, A6000 reclaimed). Respin litbench on demand
-  per `reference_litbench_rm_irv_ml1` (weights staged, ~90s) — ⚠️ that re-displaces comfyui.
+- **MERGE FALLBACK — primed (~2-3h) if hot-swap is fully abandoned.** peft merge_and_unload (~20min, LOW
+  wall — separate keys merge directly, no rekey) → sanity greedy-diff (finally reveals if the ~10h training
+  is any good — we've NEVER seen the LoRA apply) → ModelOpt NVFP4 re-quant of the merged bf16 (~1-2h,
+  MEDIUM-HIGH wall = the GDN-hybrid requant + calibration + verify GDN modules quantize) → serve as plain
+  base (no `--enable-lora`). Static, NOT hot-swappable.
 
-- **character-rp role — SHIPPED; one QUEUED spot-check.** Proved per-request `extra_body`
-  (top_k/repetition_penalty) forwards through the `gen-reasoning` alias to the vLLM sampler (no
-  gateway cap needed); pre-staged the `character-rp` role in demo+personal bind-mount
-  `model_roles.yaml` (byte-verified live on b18; caught the cached-registry ordering so the deploy's
-  own restart activates it). worldtree-dev shipped **#344 (v1.0.0b19)** fixing the durable-agent
-  override-drop (role now applies the RP profile). QUEUED: ratatoskr-dev pings a turn-timestamp → I
-  spot-check the gateway spend_logs to confirm the RP sampler params (temp 0.75 / top_p 0.85 /
-  presence 1.5 / top_k 20 / rep 1.0) reach vLLM.
+- **comfy-dev in-arbo LoRA-training worker — DONE (all phases, this session).** Built/deployed the worker
+  (`services/lora-training-worker/`, irv-ml1 `0.0.0.0:8203`, runs as llmuser, fixed-invocation sd-scripts,
+  systemd, elway deploy playbook; commits 888ba6a→b617a8b). Phase 1 first real train SUCCEEDED (Sindra
+  fast/lean). Phase 2 publish-step (`published_lora_name` + optional `train_id` field). Phase 2.5 GC
+  contract (shape A: `GET /published-loras` + `POST /gc/published {train_ids}`) CONFIRMED — worker impl
+  DEFERRED to after the T1 hot-swap work. auto-memory `reference_lora_training_worker`.
 
-- **althing v2 (herald + receiver) = systemd-supervised on nh3-dev (the althing DEV box).**
-  Formalized this session: `althing-herald.service` (Restart=always, **Environment=PATH incl
-  ~/.cargo/bin** — the fix for the silent pane-dispatch outage) + `althing-receiver.service` (v2 →
-  pillar-3 `/owner/*` live). Stale forseti unit removed; both on v2.0.0b2, canonicalized by forseti.
-  `reference_nh3_dev_althing_herald`.
+- **Serving state:** AEON gen (:8015) + char-rp (:8016) NVFP4 on ana-ml2 GPU0, HEALTHY (repeatedly
+  stopped/restored for the hot-swap smokes via `docker start vllm-aeon-gen vllm-aeon-rp`, never compose up).
 
-- **glm-5.2 = z.ai 1M-in / 128K-out, NO gateway cap** (probed z.ai; recorded in the config comment
-  + `reference_litellm_gateway`; commit 624a07e). All GLM entries are pure z.ai passthrough — the
-  effective ceiling is z.ai's canonical, not a gateway limit.
+- **worldtree-dev vh/pypi read token** delivered (bifrost==1.1.0 pin, #349); dedicated read:package token
+  at `nh3-dev:~/.config/gitea-tokens/vh-pypi-read.txt`. The over-scoped write:package token the operator
+  pasted was revoked + verified dead.
 
-- **`/books` mounted (transient) on nh3-dev** (`10.0.50.50:/mnt/books` → NFSv4 ro,soft; re-mount via
-  `infra-ops@10.100.10.50` if it reboots).
+- **Backups — STILL OPEN:** rotate the 5 disclosed rest-server creds (operator, offline); confirm
+  esh-vm-db's resticprofile includes DB dumps. `docs/runbooks/backups.md`.
 
-- **Backups — recovered + hardened (2026-06-20), STILL OPEN:** rotate the 5 disclosed rest-server
-  creds (operator, offline); confirm esh-vm-db's resticprofile includes DB dumps. `docs/runbooks/backups.md`.
-
-- **Open follow-ups (low-priority):** clean phantom `qwen3.6-35b-a3b` off the gateway (lists in
-  /v1/models but 400s); docker-daemon default log-cap on ana-docker; drop the now-inert
-  `mood.decay_rate`/`mood.stale_hours` keys from the deployed `/opt/worldtree*/config` bind-mounts
-  (harmless-present per worldtree-dev; edit-only/no-restart at a no-deploy window).
-
-- **Standing / parked:** Mac Pro migration (hardware-gated); R17 v2 corpus HELD; disclosed-keys
-  rotation queue; clean legacy `news-digest`; R22 gateway-only full-access key at
-  `/home/lkraven/.r22-gateway-key` (mode 600, carries paid GLM, do NOT delete); granite-4.1-8b bf16
-  at `irv-ml1:/home/lkraven/granite-4.1-8b-bf16` (reap-on-request); Deckard staged on ana-ml2 as
-  T1's writing benchmark.
+- **Standing / parked:** phantom `qwen3.6-35b-a3b` gateway cleanup (400s in /v1/models); ana-docker
+  docker-daemon log-cap; inert `mood.decay_rate`/`stale_hours` keys in deployed WT config; Mac Pro
+  migration (hw-gated); R22 gateway-key at `/home/lkraven/.r22-gateway-key` (mode 600, paid GLM, do NOT
+  delete); Deckard staged on ana-ml2 as T1's writing benchmark; `/books` transient NFS mount on nh3-dev.
 
 - **Worldtree config-propagation (reference):** demo+personal bind-mount config from
-  `/opt/worldtree{,-personal}/config` (infra-ops-deployable, byte-identical from canonical); reload
-  via `docker restart <container>`, NEVER `compose up` (stale-`:latest` footgun). **The role registry
-  is loaded ONCE + CACHED at startup** (`ConversationService.role_registry`) → a bind-mount
-  `model_roles.yaml` change needs a container restart to take effect; pre-stage the bind-mount
-  BEFORE the activating deploy's restart (else it needs a second bounce). config REMOVALS are NOT
-  backward-compatible with the still-running image.
+  `/opt/worldtree{,-personal}/config` (infra-ops-deployable, byte-identical from canonical); reload via
+  `docker restart <container>`, NEVER `compose up` (stale-`:latest` footgun). Role registry loaded ONCE +
+  CACHED at startup → a bind-mount `model_roles.yaml` change needs a restart; pre-stage BEFORE the
+  activating deploy's restart. config REMOVALS are NOT backward-compatible with the running image.
 
 ## Recent decisions
 
+- `[2026-07-07]` **T1 (retargeted qwopus-122B→AEON-27B) SFT COMPLETE on ana-ml2** — all-TRL torch-GDN,
+  batch1/accum8/GC-off, 1 epoch ~9h @ 0.25 s/s, loss 2.445, adapter banked. Supersedes the 2026-07-05
+  cloud-vs-smoke venue decision: AEON-27B's 52G bf16 fits one Blackwell → ran on-prem in ~9h (no cloud,
+  no full-fleet outage). GC is NOT the throughput lever (batch1-no-GC == batch4+GC at 0.25; dvalin's 0.565
+  was warmup-noise from a 30-sample smoke); ~0.25 s/s is the torch-GDN floor (only FLA beats it, sm_120-
+  blocked #913). `reference_t1_cloud_train_plan`.
+- `[2026-07-07]` **T1 hot-swap on vLLM NVFP4 is DEAD — vLLM 0.24.0 qwen3_5 LoRA-application silent no-op**
+  (#47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4+FP8) AND
+  adapter-format-agnostic (separate `zc` loads-but-inert; fused rejected); #47640 guard-patch overlay didn't
+  fix. Root-caused by a 3-peer dwarf panel (eitri/regin/dvalin); their vLLM issue refs verified real via
+  GitHub API. Engine mirror-image: vLLM wants SEPARATE keys, SGLang wants FUSED.
+- `[2026-07-07]` **HARD REQUIREMENT restated: hot-swappable LoRA or ABANDON.** Merge is NOT a solution for
+  this requirement (static; reserved for a later RP-quant track). SGLang is the last hot-swap shot; if it
+  can't apply, park the adapter + file upstream repros. `reference_lora_training_worker` (adjacent).
+- `[2026-07-07]` **comfy-dev in-arbo LoRA-training worker DONE** — built/deployed/validated Phases 1+2
+  (first Sindra train succeeded, publish-step + `train_id`), Phase 2.5 GC contract (shape A) confirmed +
+  impl deferred. `services/lora-training-worker/`, commits 888ba6a→b617a8b. `reference_lora_training_worker`.
+- `[2026-07-06]` **infra-ops NOPASSWD sudo confirmed on irv-ml1 + ana-ml2** (the arbo memory's "NOT
+  infra-ops on irv-ml1" was stale) — enabled the worker deploy + every ana-ml2 hot-swap smoke.
 - `[2026-07-05]` **T1 training venue: CLOUD recommended; operator chose smoke-first on ana-ml2.**
   On-prem ruled out (ana-ml2 full — both 96G GPUs ~93G used): keep-serving = NVMe offload ~6-8 DAYS;
   full ana-ml2 shutdown = CPU offload ~1 DAY but a whole-fleet outage. Cloud Vast.ai 8×80GB (no
@@ -253,6 +268,25 @@ _118 older entries archived to archival-memory.md._
 
 ## Tried and abandoned
 
+- `[2026-07-07]` **vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639).** Adapter loads HTTP 200
+  but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate `zc`
+  adapter — correct per vLLM's `check_unexpected_modules` allowlist — loads clean but inert; the fused-key
+  rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of
+  `expand_packed_lora` — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no
+  version-bump helps. Merge bakes deltas in (bypasses this) but is static.
+- `[2026-07-07]` **SGLang generic image can't LOAD our NVFP4 AEON** — ModelOptModelLoader weight-shape/
+  packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated
+  `qwen36-27b-nvfp4` dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was
+  quant-loader-specific).
+- `[2026-07-07]` **SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for**
+  (invalid choice: 'in_proj_qkv'); `'all'` resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants
+  its OWN packed layout (base r16 + `get_stacked_multiply=3`, NOT a pre-fused rank-48 qkv → the [48]-vs-[144]
+  shape assert). A THIRD adapter format; version-exact source needed (`:latest`=0.5.13, NOT `main`).
+- `[2026-07-07]` **Engine invocation footguns cost several wasted serve-bounces this session** — `docker run
+  --rm` ate crash logs; duplicated `serve` (vLLM image entrypoint is already `["vllm","serve"]`);
+  `--max-lora-rank 48` invalid (choices 1/8/16/32/64… → use 64); parens in `echo` inside `ssh host -c "…"`
+  break the remote shell. LESSON: verify engine launch flags (`--help`, GPU-free) + never `--rm` a container
+  whose crash logs you need, BEFORE bouncing a production serve.
 - `[2026-07-04]` **LiteLLM (this gateway version) mutates the SHARED deployment config in-place on
   per-request sampler-param merge** → my deliberately-invalid `top_k=-5` forwarding-probe bled into a
   param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT