memory: snapshot — eRP dual-seat overhaul landed (MeroMero + Dark-Scarlett @ 256K); next = infra+access

persistent-memory.d/2026-08-12-erp-dual-seat-overhaul.md

@@ -0,0 +1,91 @@
+# eRP dual-seat overhaul — MeroMero-v2 + Dark-Scarlett, NVFP4A16 @ 256K on ana-ml2
+
+`[2026-08-12]` Replaced the two legacy char-rp seats with home-quantized NVFP4A16 vLLM
+seats. Operator-driven, end to end this session.
+
+## What landed
+
+| Seat (LiteLLM alias) | Model | Role | GPU | Context |
+|---|---|---|---|---|
+| `char-rp` (:8016) | **G4-MeroMero-v2-31B** (Gemma-4) | non-thinking PROSE, **multimodal (vision)** | GPU0 | 256K @ 2.07× (util 0.52) |
+| `char-rp-reasoning` (:8018) | **Dark-Scarlett-v1.0-27B** (Qwen3.6) | THINKING (default) | GPU1 | 256K @ 1.62× (util 0.44) |
+
+- Both **NVFP4A16 weight-only** (llm-compressor, `compressed-tensors`), `--kv-cache-dtype fp8`.
+- Replace: `char-rp-gguf` (Magidonia-24B GGUF/llama.cpp, :8016) + `heretic2-charrp-reasoning`
+  (DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP, :8018). Old stacks/containers **stopped +
+  retained** for rollback.
+- Compose-ified: `stacks/meromero-charrp` + `stacks/darkscarlett-charrp-reasoning` (ana-ml2
+  `/opt/docker/compose/`, mirrored to eshpfi, commit **`f08b6cb`**) → survive reboot.
+- Research that drove picks: `docs/pfi/erp-thinking-finetunes-2026.md` (from the `gecko-65` Booth).
+
+## Load-bearing lessons (the whole point of this file)
+
+1. **Load via the ConditionalGeneration WRAPPER class, never `AutoModelForCausalLM`.** For a
+   multimodal-capable base (Gemma-4, Qwen3.6), `AutoModelForCausalLM.from_pretrained` +
+   `save_pretrained` writes a FLAT text config (`Qwen3_5TextConfig`, `model.layers.*`) that
+   **both vLLM AND SGLang reject** (SGLang: "Qwen3_5ForCausalLM has no SGLang implementation";
+   vLLM wants `Qwen3_5ForConditionalGeneration`). Loading via `Qwen3_5ForConditionalGeneration` /
+   `Gemma4ForConditionalGeneration` keeps the wrapper config they accept. **This was the DS
+   blocker** — re-quant via the wrapper fixed it (`Dark-Scarlett-...-NVFP4A16-wrapper`).
+2. **NVFP4A16 is weight-only → DATA-FREE.** llm-compressor infers `DataFreePipeline`; calibration
+   data is unused (only matters for W4A4 activation quant). W4A16 chosen per NVIDIA's sm_120
+   long-context guidance (W4A4 KLD 2-4× worse past ~10k ctx).
+3. **Load on CPU (`device_map=None`)** so llm-compressor onloads one layer at a time. `device_map=
+   "auto"` packs the whole model onto the GPU and OOMs when the card isn't fully free.
+4. **Both models are KV-EFFICIENT — the "dense = KV-hungry" worry was WRONG.** MeroMero (Gemma-4)
+   uses **sliding-window attention** (most layers cache only a bounded window); DS (Qwen3.6) uses
+   **hybrid GatedDeltaNet linear-attention** (3:1 linear:full, linear layers carry no KV). Both
+   hit full native 256K easily. (MeroMero KV pool ~542K tokens at util 0.52.)
+5. **MeroMero vision reconstruction.** The finetune ships `processor_config.json` (image_processor
+   inline, `Gemma4ImageProcessor`) but NOT `preprocessor_config.json` — the old-format file vLLM's
+   feature-extractor loader wants. **Even google/gemma-4-31B-it (ungated!) ships only
+   processor_config.json.** FIX: extract the `image_processor` section → write
+   `preprocessor_config.json` verbatim, serve WITHOUT `--language-model-only`. Verified (model
+   correctly ID'd a red circle). Audio is config-declared but WEIGHTLESS (0 audio tensors).
+6. **GPU placement.** Match the KV-heavier model to the roomier GPU. GPU0 (gen neighbor, ~54GB
+   free) > GPU1 (utility cluster, ~45GB free). Swapped MeroMero→GPU0, DS→GPU1. Pins via compose
+   `deploy.resources.reservations.devices`.
+
+## Dead ends (tried + abandoned)
+
+- **DS via llm-compressor `AutoModelForCausalLM`** → flat config vLLM/SGLang reject. → wrapper class.
+- **DS via NVIDIA ModelOpt** → modelopt↔transformers **version deadlock**: current transformers
+  supports `qwen3_5` but crashes modelopt's sparse-moe plugin (`issubclass()` on a non-class);
+  modelopt 0.43.0 pulls an old transformers that can't load `qwen3_5` at all. Abandoned.
+- **DS via SGLang** → `Qwen3_5ForCausalLM has no SGLang implementation`. Abandoned, but it REVEALED
+  that both engines need the wrapper (→ the fix in lesson 1).
+- **`device_map="auto"` for the quant** → CUDA OOM in the weight observer. → `device_map=None`.
+
+## granite retired + gateway repoint
+
+- `vllm-granite` (granite-4.1-8b, fleet summarizer, GPU1) **`docker stop`ped** (reversible) to
+  reclaim ~13.6GB GPU1 for RP context.
+- LiteLLM (`ana-docker:/opt/docker/conf/litellm/config.yaml`, backed up
+  `.bak-pre-granite-down-*`): **`granite-4.1-8b` alias RETIRED** — commented out, now 404s cleanly
+  (the `*` wildcard→llama-swap was decommissioned 2026-06-20, so no fallthrough). **`summarizer` +
+  `classifier` REPOINTED to gen** (`hosted_vllm/qwen3.6-35b-a3b-heretic` @ :8015,
+  `enable_thinking:false`) — both verified. ⚠ This LiteLLM change is **server-only / not
+  version-controlled** (a follow-up).
+
+## MTP — deferred
+
+DS's MTP heads were dropped by the CausalLM loader; **deferred, not restored** (spec-decode is
+net-negative at RP temps: ~38-52% accept at temp 0.8-1.25, below vLLM's 0.5 cutoff). The
+splice-back path (`splice_mtp.py` in the heretic2 work dir) exists if ever wanted. MeroMero
+(Gemma-4) has no MTP by architecture.
+
+## On-disk / where things live
+
+- Quant pipelines: `ana-ml2:/tank/aimodels/meromero-v2-nvfp4-work/` +
+  `/tank/aimodels/darkscarlett-nvfp4-work/` (scripts, BF16 source, NVFP4 outputs).
+- Compose stacks: `ana-ml2:/opt/docker/compose/{meromero-charrp,darkscarlett-charrp-reasoning}/`.
+- Gateway aliases (unchanged, port-based): `char-rp`→:8016, `char-rp-reasoning`→:8018. (char-rp was
+  also fixed from the stale `magidonia-24b-v4.3` backend model name → `char-rp`.)
+
+## Open follow-ups
+
+1. LiteLLM granite/repoint change NOT version-controlled (server + backup only).
+2. eshpfi unpushed (many commits this session incl. `f08b6cb`, `7bd7375`, `398b58a`).
+3. MTP deferred (see above).
+4. DS thinks verbosely (~13:1 reasoning:content) — eval item; consumers need generous `max_tokens`.
+5. MeroMero full 256K needs util 0.55 (GPU0 ~1.8GB free, tight); ran at 0.52 for headroom (~4.6GB).

persistent-memory.md

@@ -109,9 +109,11 @@ no longer deployed sidecars here. See Recent decisions.)
 
 ## Current state / in-flight
 
-_As of 2026-08-12 — the secrets-broker + stonehenge-park work LANDED (detail in Recent decisions). **wgtunnel** moved OUT to its own repo (pre-seeded, verification done)._
+_As of 2026-08-12 — **eRP dual-seat overhaul LANDED** (both char-rp seats live at 256K, compose-ified); **wgtunnel DEPLOYED** by tunnel-dev + stack mirrored; secrets-broker + stonehenge-park landed. **NEXT PROJECT (operator, queued for the next session): infra setup with additional access.**_
 
-- **⟳ wgtunnel — OWN REPO (`vh/wgtunnel`, private); `/vor-plan` DONE, build underway by `tunnel-dev`.** WireGuard-over-wstunnel WSS/443 tunnel for captive-portal/DPI networks (plane, hotel, DPI firewall); design bundle in the repo's `docs/design/`. Host = **`boring.phasefinal.com`** (innocuous SNI, operator's call). Endpoint verified: `ana-wg` UDP/31337, traefik on ana-docker terminates TLS :443, edge `38.120.12.44`. **Phase-0 DNS DONE 2026-08-12** — `boring` CNAME → `ana-srv1` (DNS-only) published + verified (resolves 38.120.12.44 both authoritative NS + 1.1.1.1); wstunnel ACME can proceed once Unit 1 deploys. infra-ops role = infra + DNS; the build lives in `~/development/wgtunnel` (tunnel-dev's session), not here.
+- **✅ eRP DUAL-SEAT OVERHAUL — DONE + LIVE (2026-08-12).** Two home-quantized NVFP4A16 vLLM seats on ana-ml2 replace the legacy char-rp seats: **`char-rp` :8016 GPU0 = MeroMero-v2-31B** (Gemma-4, non-thinking PROSE, **multimodal/vision**) + **`char-rp-reasoning` :8018 GPU1 = Dark-Scarlett-v1.0-27B** (Qwen3.6, THINKING). Both full **256K** (MeroMero util 0.52 → GPU0 ~4.6 GB free, 2.07×; DS util 0.44, 1.62×). Compose-ified (`stacks/meromero-charrp` + `stacks/darkscarlett-charrp-reasoning`, commit `f08b6cb`) → reboot-durable. Old GGUF/heretic2 seats stopped-but-retained. Full arc + all lessons (wrapper-config fix, vision reconstruction, GPU swap, ModelOpt/SGLang dead-ends) → Recent decisions `[2026-08-12] eRP dual-seat overhaul`. ⚠ OPEN: the LiteLLM granite-retire + `summarizer`/`classifier`→gen repoint is **server-only, NOT version-controlled**; MTP deferred; DS thinks verbosely (needs generous `max_tokens`).
+
+- **⟳ wgtunnel — DEPLOYED + accepted (tunnel-dev).** WG-over-wstunnel WSS/443 for captive-portal/DPI networks. `wgtunnel-wstunnel-1` live on ana-docker (`erebe/wstunnel v10.6.2`, behind traefik, Host `boring.phasefinal.com`, Mode-A `anaprod` cert), `--restrict-to ana-wg:31337` (not an open relay). infra-ops did Phase-0 DNS (`boring` CNAME → `ana-srv1`, DNS-only) + mirrored the stack into eshpfi `stacks/wgtunnel/` (`398b58a`). Full project in `vh/wgtunnel` (tunnel-dev's repo/session).
 
 - **SECRETS-BROKER — LIVE.** Per-box credential store over Vaultwarden; the `secret` CLI (`put/get/list/rm/backfill`) on PATH at `~/.local/bin/secret`, 25 nh3-dev secrets backfilled + verified. Standing directive now global: **vault is the credential source of truth — store in it AND `secret get` creds FROM it.** Detail → Recent decisions `[2026-08-10→12]`.
 
@@ -133,11 +135,13 @@ _As of 2026-08-12 — the secrets-broker + stonehenge-park work LANDED (detail i
 
 **althing monitor** ARMED (handle `infra-ops`, task `b1uat4njl`; herald up). ⚠️ Re-arm ONLY after a real FIRE (`<task-notification> completed rc0`), NEVER after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, **NEVER chained with `&`/`&&`** (orphans it to init — bit me this session; reclaim with `althing-cli stop-monitor` then re-arm cleanly).
 
-**eshpfi push state:** `origin/main` well behind — this session added the dots.tts arc + the **secrets-broker arc** (`41359ea` CLI, `850a197` backfill, `a249073` rm+warn, `a1304b7` contract note) + dots-tts v3/extract (`a80f6e9`/`62672c9`) on top of prior unpushed work. **Push is the operator's call.** (Dotfiles + stonehenge-park are SEPARATE repos, both pushed this session.) `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's).
+**eshpfi push state:** `origin/main` well behind — this session added the **eRP dual-seat arc** (`7bd7375` research doc, `f08b6cb` the two char-rp compose stacks), the **wgtunnel mirror** (`398b58a`), memory updates (`a8c6d85`/`69597cb`) — on top of the earlier dots.tts + secrets-broker arcs (still unpushed). **Push is the operator's call.** ⚠ The LiteLLM **granite-retire + `summarizer`/`classifier`→gen** change lives **server-only** on ana-docker (`/opt/docker/conf/litellm/config.yaml` + timestamped `.bak-pre-granite-down-*`), NOT in this repo. `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's — now superseded by `darkscarlett-charrp-reasoning`).
 
 
 ## Recent decisions
 
+- `[2026-08-12]` **eRP dual-seat overhaul: MeroMero-v2 (`char-rp`) + Dark-Scarlett (`char-rp-reasoning`), both NVFP4A16 @ 256K on ana-ml2; granite retired.** Replaced the GGUF/heretic2 RP seats with two home-quantized vLLM seats. The DS blocker (an `AutoModelForCausalLM` save wrote a flat `Qwen3_5TextConfig` that **both vLLM AND SGLang reject**) was fixed by re-quanting via the `Qwen3_5ForConditionalGeneration` **wrapper class**; ModelOpt was a version deadlock, SGLang lacked the impl (but revealed the fix). MeroMero vision reconstructed by extracting `preprocessor_config.json` from `processor_config.json`. Both models KV-efficient (Gemma-4 sliding-window / Qwen3.6 hybrid linear-attn) → full 256K; GPU-swapped for headroom; compose-ified + committed `f08b6cb`. granite downed + LiteLLM `summarizer`/`classifier`→gen. Full arc, lessons, dead-ends → `persistent-memory.d/2026-08-12-erp-dual-seat-overhaul.md`
+
 - `[2026-08-12]` **infra-ops now holds an all-zones Cloudflare DNS-edit token (vaulted) + wgtunnel Phase-0 DNS landed.** Operator handed over a `Zone·DNS·Edit` (all zones) CF token → `secret put nh3-dev/.config/cloudflare/infra-ops-dns-token` (round-trip verified; /tmp drop shredded). Fleet DNS is now self-serve for infra-ops (⚠ HIGH blast radius — all zones). First use: created `boring.phasefinal.com` CNAME → `ana-srv1.phasefinal.com`, **DNS-only** (proxied:false), verified resolving to 38.120.12.44 on both authoritative NS (louis/wren) + 1.1.1.1 — NOT Cloudflare-proxied. Unblocks wgtunnel's wstunnel ACME cert. phasefinal.com zone id `f812ba74ed9a75cf21bbe7ce9188db50`. auto-memory `reference_infra_ops_cloudflare_dns_token`. (Earlier gap: the only prior vaulted CF token, jackdaw's, had `zone:read`+`worker:edit` but no `dns_records:edit`.)
 
 - `[2026-08-12]` **wgtunnel stood up as its own repo (`vh/wgtunnel`, private) after a live endpoint-verification pass.** Operator directed own-repo (mirrors stonehenge-park/tts-stack). Verified off the fleet before seeding: `ana-wg` WG server = **UDP/31337** (not 51820), subnet 10.30.10.0/24, MTU 1420, active roaming peer proves the public UDP DNAT works; traefik on ana-docker **terminates TLS :443** (ACME `anaprod` http-challenge, docker+file providers, CrowdSec bouncer) → confirms the clean design (wstunnel container on `traefik-net`, Host-routed, WS→UDP to `ana-wg:31337`); edge `38.120.12.44` direct-A, `tunnel.phasefinal.com` free (⚠ must be **direct**, NOT Cloudflare-proxied like vaultwarden). Repo pre-seeded (README/CLAUDE/persistent-memory/ROADMAP + `docs/verified-infrastructure.md` = ground truth) + pushed; commit `9584d38`, Vuong-attributed. vh gitea token pulled from the vault (`secret get`), not persisted to `.git/config`. **NEXT = `/vor-plan` or `/vor` (operator's call, interactive).** Deps to line up in the plan: DNS A-record, FortiGate :443 host-routing, a new ana-wg peer for the laptop, client tooling.