feat(booth): downloadable booths — whole-booth zip + ?dl force-download (v0.1.4)

A verbatim index.html booth (e.g. edict-design-brief: a rendered brief + its
.md) had no download affordance — the page is served raw with no gallery/per-file
chrome. Adds:
  - GET /b/<name>/?download=1 -> streams the whole booth as <name>.zip (attachment)
  - GET /b/<name>/<file>?dl=1 -> forces Content-Disposition: attachment (html/md/
    text otherwise render inline with no easy save)
  - a download link on the index card (the accessible spot for verbatim booths)
    and the gallery header
Tests for both; verified live against edict-design-brief on nh3-dev :8090.

.gitignore

@@ -35,3 +35,7 @@ htpasswd-new
 # graphify: commit only the lightweight labeled map; ignore heavy/regenerable artifacts
 graphify-out/*
 !graphify-out/GRAPH_REPORT.md
+
+# Python bytecode (e.g. from local py_compile of stack wrappers)
+__pycache__/
+*.pyc

CLAUDE.md

@@ -10,6 +10,15 @@ state) across context resets. Read it at session start; treat it as
 one input alongside this CLAUDE.md and the auto-memory system, not
 as the single source of truth.
 
+It is a lean **index**: the dated log sections (Recent decisions,
+Tried and abandoned) keep each over-threshold entry's full body in
+`persistent-memory.d/<slug>.md`. Read the index at session start;
+pull a detail file only when its index line is relevant to your work —
+never bulk-read `persistent-memory.d/`. When you commit, stage any
+pending `persistent-memory.md` and `persistent-memory.d/` updates in
+the same commit as the work that prompted them — durable memory that
+lags the code defeats its own purpose.
+
 **New session starting here?** Read [`docs/orientation.md`](docs/orientation.md) first — fleet topology, backup architecture, governing principles, and all the NFS/DSM/naming gotchas that have cost past sessions time.
 
 **For SSH-driven work: use `scripts/elway`.** Write a playbook under
@@ -88,7 +97,7 @@ Observed and standardized across servers:
 
 | Name | IP | Site | Role | Details |
 |------|-----|------|------|---------|
-| ana-ml2 | 10.250.50.54 | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal, dual RTX 6000 Ada) | `servers/ana-ml2/README.md` |
+| ana-ml2 | 10.250.50.54 | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal, dual RTX PRO 6000 Blackwell Max-Q, 96 GB each) | `servers/ana-ml2/README.md` |
 | irv-ml1 | 10.100.79.3 (WG) | Irvine — reachable only via WireGuard tunnel from NH3 | GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks) | `servers/irv-ml1/README.md` |
 | ana-docker | 10.250.50.70 | Anaheim | General-purpose Docker host (non-GPU VM on pfi-pve) | `servers/ana-docker/README.md` |
 | pfi-ana-webhost | 10.250.50.52 | Anaheim | VM on pfi-pve (VMID 110) — web workload | `servers/pfi-ana-webhost/README.md` |
@@ -105,6 +114,7 @@ Observed and standardized across servers:
 | corviduo-dev | 10.250.50.152 | Anaheim | **Worldtree-team dev VM (PFI-hosted)** — runs the demo + personal + pinned Worldtree deployments vor/asset-engine talk to | `servers/corviduo-dev/README.md` |
 | nh3-docker | 10.100.50.40 | NH3 (`10.100.0.0/16`) | General-purpose Docker host (non-GPU VM on nh3-pve) | `servers/nh3-docker/README.md` |
 | nh3-dev | 10.100.10.50 | NH3 | Dev box — fleet sidecars (egress SOCKS5 proxy, ttyd seat, mead-hall, volva) + live Claude Code sessions; not a Docker-stack host | `servers/nh3-dev/README.md` |
+| nh3-extdev | 10.100.50.42 | NH3 | Manager / external-dev box (VM on nh3-pve, Debian 13); **sudo-less** infra-ops identity (user-level only, no Docker); successor to retired nh3-ansible | `servers/nh3-extdev/README.md` |
 | nh3-pve | 10.100.250.60 | NH3 | Proxmox VE hypervisor | `servers/nh3-pve/README.md` |
 | nh3-nas | 10.100.50.50 | NH3 | Synology RS2418+ — NFS exports, rest-server-nh3, PBS-NH3 datastore backend | `servers/nh3-nas/README.md` |
 | pbs-nh3 | 10.100.50.90 | NH3 | Proxmox Backup Server — DR mirror (VM on nh3-pve, NFS datastore on nh3-nas); syncs from pbs-ana | `servers/pbs-nh3/README.md` |

archival-memory.md

@@ -295,6 +295,444 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
   `01KT2K2SY9N7AY69R9V0B4RXSW` to asset-engine-dev.
   _Archived 2026-06-04._
 
+- `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — Populated
+  `references/<name>/<name>.wav`+`.lab` for all 32 voices; playbook gained
+  normalize-step + A/B smoke gate. glados got a real transcript (ASR'd via
+  Parakeet): the Portal "Welcome to test chamber 4" lines.
+  _Archived 2026-06-11._
+
+- `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast`
+  streaming container.** Workload = single-stream interactive. GPU placement:
+  3090 (device 0) if it fits else A6000 (device 1). Cutover: parallel catalog
+  entry, burn in beside live `chatterbox`, then flip. Streaming = adaptive
+  buffer-ratchet chunking (native frame-streaming abandoned). Since extracted to
+  its own repo `vh/chatterbox-fast` (auto-memory `reference_chatterbox_fast_repo`).
+  _Archived 2026-06-11._
+
+- `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** Claimed
+  naive sentence-level streaming has "zero quality loss" — WRONG. The T3 AR backbone
+  conditions prosody on the WHOLE text; splitting loses cross-sentence prosodic
+  context even though voice timbre stays (reference-conditioned). No *artifacts* ≠
+  no *quality loss*. Hence adaptive-chunk design (maximize context per chunk subject
+  to latency budget), not fixed per-sentence splits.
+  _Archived 2026-06-11._
+
+- `[2026-06-03]` **Declined worldtree v0.30.4 staging deploy** — that's worldtree-team's
+  CI/CD lane (a developer `staging/vX.Y.Z` git-tag promote), not infra-ops. They self-corrected
+  to the same conclusion independently.
+  _Archived 2026-06-13._
+
+- `[2026-06-03]` **R14 scope = (a) provision-only.** infra-ops provides box + CUDA env +
+  engines + weights + NFS; brokkr/dev wires `arms.py` + runs — keeps infra-ops OFF the
+  VIVAE-processing path (VIVAE = Variably Intense Vocalizations of Affect/Emotion, CHARTER §4
+  highest-liability; operator authorized R&D-eval-only, quarantined). Box = irv-ml1 (A6000
+  free; ana-ml2 GPU-saturated). Per-engine venvs (divergent torch stacks); A6000 = `cuda:0`
+  NATIVE (≠ docker `=1`).
+  _Archived 2026-06-13._
+
+- `[2026-06-03]` **yt-voice-clipper push-to-deploy via gitea webhook** (operator-directed,
+  after 6 manual rebuilds in ~40 min). Webhook (not poll) — gitea CAN reach the WG IP per the
+  operator. The proxy env + Homepage labels live in the **host-specific override** (untracked
+  → survive the auto-deploy's `git reset --hard`), NOT yt-voice-clipper-dev's image. Runbook
+  `d4f180d`.
+  _Archived 2026-06-13._
+
+- `[2026-06-03]` **yt-voice-clipper bot-gate fix = route yt-dlp through NH3 residential
+  egress, NOT cookies/PO-token.** YouTube hard-flags the Irvine colo IP (LOGIN_REQUIRED on a
+  public video even with no cookies). Cookies + the bgutil PO-token + deno JS-runtime all
+  loaded fine — the gate is pure IP reputation. Operator chose proxy-via-nh3-dev → durable
+  dante proxy → proven. The egress proxy is a reusable fleet lever for any datacenter-IP-gated
+  service.
+  _Archived 2026-06-13._
+
+- `[2026-06-04]` **`brokkr` user (no-sudo) on irv-ml1; R14/R15/R16 substrate moved to /home/brokkr.** Persistent box services there need SYSTEM systemd units (see Tried).
+  _Archived 2026-06-14._
+
+- `[2026-06-04]` **ollama upgraded 0.9.0→0.30.4 on irv-ml1** (Ministral-3 is a Dec-2025 model the old engine refused); A6000 pinned by **UUID** not index (native fastest-first ≠ nvidia-smi PCI).
+  _Archived 2026-06-14._
+
+- `[2026-06-04]` **Worldtree demo/pinned/personal deploys are ALL CI/CD, not infra-ops** — a "deploy vX.Y.Z" request to infra-ops is MISROUTED → point them back to their pipeline. (`d8d776c`, auto-memory `reference_worldtree_deploys_cicd`)
+  _Archived 2026-06-14._
+
+- `[2026-06-04]` **infra-ops NOPASSWD-sudo identity commissioned, scoped to PFI boxes** (+esh-docker-vm by operator override) — so infra-ops completes DevOps end-to-end vs handing the operator sudo steps. Dedicated key, sudo log_output, key-gated. (`8c32a05`, auto-memory `reference_infra_ops_sudo_identity`)
+  _Archived 2026-06-14._
+
+- `[2026-06-04]` **phi4 ships the CANONICAL/official Phi-4 chat template, NOT Ollama's** (Ollama's omits the system `<|end|>`). Applied an Ollama-matching override then reverted — ship correct, not the benchmark quirk. (`90e08f0`→`27eb537`; "headgun" lesson in Tried.)
+  _Archived 2026-06-14._
+
+- `[2026-06-04]` **phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer; granite-4-small retired** from llama-swap. 50K ctx + FP8 KV. (`40a374b`) [superseded by the 2026-06-05 granite-4.1-8b decision]
+  _Archived 2026-06-14._
+
+- `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons; Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid `12aa5a9`); volva.service was crash-looping 203/EXEC. (`6e2f80e`)
+  _Archived 2026-06-16._
+
+- `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer.** Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not Q4/AWQ** — primary workload is high-concurrency, where FP8 scales ~linearly (2010 tok/s @ C=32). vLLM `vllm-granite` :8004 GPU 1, official IBM compressed-tensors FP8. (Then on Ada; box has since gone Blackwell.) (`34a43a0`, auto-memory `reference_ana_ml2_vllm_granite`)
+  _Archived 2026-06-16._
+
+- `[2026-06-05]` **Langfuse v3 on ana-docker (:3001) as the gateway trace UI**; LiteLLM `success_callback:[langfuse]` live. Pretty traces + tok/s dashboard. NOT a prerequisite (spend_logs already capture tokens+latency). (`9171e6a`)
+  _Archived 2026-06-16._
+
+- `[2026-06-05]` **Ollama BANNED fleet-wide** (operator directive) — never stand one up; tear down any found; serve via llama-swap or vLLM. (auto-memory `feedback_avoid_ollama`)
+  _Archived 2026-06-16._
+
+- `[2026-06-05]` **ComfyUI / FLUX.2 work split to `~/development/comfy-dev`** (dedicated repo + agent). eshpfi keeps the `comfyui`/`arbo` stack compose; comfy-dev owns the model/workflow knowledge. (auto-memory `reference_irv_ml1_ampere_quant`)
+  _Archived 2026-06-16._
+
+- `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is the structured-output profile, ON HOLD, no live consumer). Bind-mount CONFIGS (providers.yaml, vh-owned) ARE infra-ops's to apply directly — only the app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`)
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **Arbo prod bearer auth turned OFF — WireGuard is the access boundary** (operator decision; reverses ADR-0001's "closed the open-auth hole"). ENGINE_TOKEN must be **ABSENT, not empty** (empty-string still gates) — removed from BOTH the host `.env` AND the compose `environment:` injection line. Original token backed up at `irv-ml1:/opt/docker/compose/arbo/.env.pre-auth-off.bak`; comfy-dev updated their ADR-0001. Tracking: `db97899` + `playbooks/arbo-disable-engine-token.yaml`.
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **Storetank image-models archive DECOMMISSIONED; arbo is the single live ComfyUI model tree (502 G).** Curated `/storetank/image-models/comfy` (was 919 G, the native `/opt/ComfyUI/models` symlink target) → killed everything superseded by arbo's current gen (Hunyuan, WAN2.1, FLUX.1, Chroma, SD3.5, orphaned umt5+llava ≈ 739 G) + migrated the keepers (gen-agnostic utilities + the SDXL/Pony stack, 177 G) into `/storetank/arbo/models` (same-fs move, skip-existing protects prod). Tracking: `docs/storetank-image-models-archive.md` + `docs/arbo-comfyui-model-catalog.md` (`1902425`→`5007ec1`).
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **GRANITE_KEY provisioned to comfy-dev's nh3-dev dev env** at `~/.arbo_granite_key` (0600) for the hero gen+judge script — verbatim copy of the prod `arbo-prompt-enhance` vkey (now extended to reach BOTH `granite-4.1-8b` AND `qwen3.5-9b-fp8`); nothing minted. The vkey README's "granite-only" wording was stale → corrected (`f32c6dd`).
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **ana-ml2 upgraded Ada → dual RTX PRO 6000 Blackwell Max-Q** (96 GB each, cc 12.0 / sm_120; was dual RTX 6000 Ada 48 GB / cc 8.9 — confirmed live via `nvidia-smi`). Unlocks NVFP4 (FP4 tensor cores) and doubles VRAM headroom. CLAUDE.md GPU-spec doc-fix **LANDED `355a240`** (operator). Tracking: `19a07b9`/`1e2a3a1` ("Blackwell 96GB").
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **NVFP4-W4A4 is infeasible for Granite — FP8 stays the Granite-on-Blackwell format.** W4A4 collapses at 30k context, proven **producer-independent** (modelopt AND llm-compressor both clean-NONE from the same BF16 base + wikitext-2k calib). No 4-bit wins both axes: W4A4 = quality collapse; W4A16-NVFP4/AWQ = weight-only dequant → bf16 (no FP4-core speedup). **30B retired**. (auto-memory `reference_nvfp4_w4a4_granite_infeasible`)
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **Qwen3.5-9B VL (FP8) deployed on ana-ml2 GPU 1** — `qwen35-vl` stack, :8007, gateway alias `qwen3.5-9b-fp8`. **Pinned nightly digest, not `:latest`**: the stable release quantizes the VL *vision tower* under `--quantization fp8` → garbage vision (LM fine); the nightly correctly excludes it. Re-pin + drop the pin once that exclusion lands stable. (`2e3dcc2`)
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **comfyui 325 G model tree migrated worktank → `/storetank/arbo`** (worktank 97% → 26%). `arbo` is the consuming app; overlay bind-mount via `COMFYUI_MODELS_DIR`. (`38186be`) (See the 2026-06-13 archive-decommission decision above — this tree later absorbed the storetank-archive keepers, reaching 502 G.)
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **GPU layout settled on the Blackwell box.** GPU 0 held free for large-model hot-loads (llama-swap pinned, `edf0f91`); GPU 1 steady-tenant — granite 131k ctx, qwen 65k, embed/rerank/reward trio, ~3.5 GB free (`1e2a3a1`, `19a07b9`; trio re-floored for 96 GB, 20×-parallel-stable). embed/rerank left at floor — long docs chunked BEFORE embedding. max-model-len is a ceiling not a reservation.
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **Prefix caching pinned explicit on granite + qwen** — benched ~6.5× faster TTFT on a shared ~4.5k-token summarizer template; soft/evictable, neutral when prefixes don't repeat. vLLM v1 defaults it ON (granite) but the qwen nightly defaults OFF — pin both. (`a9a2be7`)
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **granite-4.1-8b listed as the always-available summarizer/classifier + a shared all-agents key minted** (operator-directed). Global `~/.claude/CLAUDE.md` Global-tools entry; key alias `all-agents-local`, scoped to the FREE local models only (granite + qwen-vision + embed/rerank, NOT paid GLM), internal-gateway-only, rotatable. (auto-memory `reference_litellm_gateway`)
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **arbo engine + frontend stack stood up** (ADR-0001) — irv-ml1 co-located inference engine (`ee57e69`), python-based healthcheck (slim image, no curl/wget, `bdb3312`), frontend ro-mounted from the checkout (`922e8ad`, ADR-0001 D2).
+  _Archived 2026-06-16._
+
+- `[2026-06-11]` **GLM thinking inverted at the LiteLLM gateway** (operator call): `glm-5.1` defaults thinking-OFF; `glm-5.1-reasoning` = same z.ai upstream, thinking ON. Mechanism: `litellm_params.extra_body:{thinking:{type:disabled}}` — `drop_params` strips a top-level `thinking`/`reasoning_effort` but forwards `extra_body` verbatim to z.ai. Shared-gateway change. (`95b2701`, auto-memory `reference_litellm_gateway`)
+  _Archived 2026-06-16._
+
+- `[2026-06-11]` **pi coding agent installed on nh3-dev as a GLM 5.1 harness** — `@earendil-works/pi-coding-agent` via **bun** (npm's global prefix is `/usr` → needs sudo, bun avoids it). Config `~/.pi/agent/models.json`, launcher `~/.local/bin/glm`.
+  _Archived 2026-06-16._
+
+- `[2026-06-11]` **z.ai web-tools (regin) = z.ai hosted MCP path, NOT the `/paas/v4` Tool API.** WORKS: MCP streamable-HTTP at `https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp`, `Authorization: Bearer $ZAI_API_KEY` (the **MCP** key, distinct from `Z_AI_API_KEY` the LLM key). Reference impl = Worldtree's Leif agent.
+  _Archived 2026-06-16._
+
+- `[2026-06-10]` **Mac Pro migration framed: workstation-only** (M2 Ultra ARM, racked NH3 on-subnet); sidecars stay Linux. `migration-plan.md`. (See in-flight.)
+  _Archived 2026-06-16._
+
+- `[2026-06-10]` **Worldtree deployed-config propagation is infra-ops's OWNED lane** (operator ruling). worldtree-dev pings the config delta pre-merge; infra-ops syncs `config/*.yaml` from MERGED canonical to the `/opt/worldtree*/config` bind-mounts on demo+personal. The v0.33.8 9-HOUR demo outage (a `model_roles.yaml` startup-dep that never reached the VM) is the failure mode this prevents. providers.yaml stays hand-tuned. corviduo emergency-ops = `ssh vh@10.250.50.152`, docker no-sudo. (auto-memory `reference_worldtree_deploys_cicd`, `reference_corviduo_dev_emergency_ops`)
+  _Archived 2026-06-16._
+
+- `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr-smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev — granite, later extended to qwen-vision). Mint via `/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus. (auto-memory `reference_litellm_gateway`)
+  _Archived 2026-06-16._
+
+- `[2026-06-14]` **ana-ml2 GPU-1 vision upgraded: Qwen3.5-9B → Qwen3.6-35B-A3B (official FP8), served under its TRUE name only.** `qwen36-vl` replaces `qwen35-vl` on :8007 (`a0fed13`). The stale `qwen3.5-9b-fp8` name is KILLED at vLLM AND the litellm gateway (404/400) — a model is NEVER aliased under a prior model's name (silent substitution = downstream footgun; operator directive). Consumer comfy-dev/arbo migrated; arbo vkeys → all-proxy-models; shared `all-agents-local` key repointed qwen3.5-9b-fp8 → qwen3.6-35b-a3b. GPU-1 rebalanced for the ~34 GB FP8 weights (granite 0.35→0.24/64K; embed/rerank 0.05→0.03, reclaimed ~4 GB util-waste). Validated: vision correct, 20-concurrent = no OOM. (auto-memory `feedback_no_false_model_aliases`)
+  _Archived 2026-06-18._
+
+- `[2026-06-14]` **NVFP4 was the lighter fit (~21 GB) but is BLOCKED on vLLM — FP8 is the working vision path.** `nvidia/Qwen3.6-35B-A3B-NVFP4` won't load: the ModelOpt-NVFP4-MoE loader errors on expert/lm_head scale keys across 0.19.1 (`w2_input_scale`) AND 0.22.0 (`lm_head.input_scale`, vllm #44081) — a pattern across modelopt NVFP4 MoEs. Revisit NVFP4 (frees ~13 GB on GPU 1) once fixed; the 21 GB checkpoint stays cached on ana-ml2. **(SUPERSEDED 2026-06-16 — it loads on vLLM 0.23.0; qwen36 swapped to NVFP4. See the top of this section.)**
+  _Archived 2026-06-18._
+
+- `[2026-06-14]` **llama-swap qwen3.5-9b GPU-0 pin DROPPED; GPU 0 reserved for a creative-writing model (pick DEFERRED by operator).** Deep-research (this session) on big-fast-uncensored creative for a 96 GB Blackwell: **GLM-Steam-106B-A12B** (already in the llama-swap config — balanced default) vs **TheDrummer/Behemoth-X-123B-v2** (prose-tier, tops UGI writing+willingness) vs XORTRON-123B (max willingness, weak prose); GGUF-on-llama-swap is the serving path. Tracking: this session + llama-swap config (GLM-Steam present, `untracked by operator choice`).
+  _Archived 2026-06-18._
+
+- `[2026-06-14]` **R16 splice-pivot yield probe executed** (infra-ops ran the irv-ml1 inference for brokkr; brokkr owns design + analysis). See Current state. Tracking: althing thread `01KV010WGS…`, `gen_yield_probe.py` in `irv-ml1:~/r16-vmoan-harness`.
+  _Archived 2026-06-18._
+
+- `[2026-06-14]` **R16 vmoan inline-generation arc CLOSED — v1 at default decode (rep1.2/temp0.8) is the final Chatterbox-tag inline artifact.** Operator's ear rejected every alternative: v2/v3 windowing (omission vs coherence-loss), v4 multi-tag (cohesion held but lost to capacity-competition), emergent inline-token modulation (degenerates, not modulates), and the gen-time decode-polish sweep (soft tamers cut the NVV itself — same omission family as v2; p0 baseline beat p1). All adapters v1–v4 + `tokenizer.json.v3bak` preserved on `irv-ml1:~/r16-vmoan-harness`. Likely-next direction (deferred, NOT formalized): generate→bin→splice + one-shot-clone NVV pipeline routing around the inline-coherence wall. Tracking: brokkr R16 journal + althing thread `01KV010WGSSMPWRNCPAGSPK15Y`.
+  _Archived 2026-06-18._
+
+- `[2026-06-14]` **Arbo deploy pipeline fixed, hardened, and version-controlled.** Prod rebuilt v0.11.1 → **v0.11.6** backend; the webhook machinery (`arbo-deploy.sh` + `arbo-webhook.py`, :9009 HMAC listener) is now repo-tracked at `stacks/arbo/` (was host-only = recoverability foot-gun). Deploy reaches gitea via the INTERNAL route (`10.250.50.70:222`) and restarts the engine ONLY on `catalog/` changes (graphs/frontend per-request; warn on `src/`|`Dockerfile` only — pyproject/uv.lock churn every commit). Operator kept arbo stack ownership in **eshpfi** (not migrated to comfy-dev's repo). Secret + `.env` stay host-only. Tracking: `6d66bc2`, `6e58e57`, `stacks/arbo/README` Q5.
+  _Archived 2026-06-18._
+
+- `[2026-06-15]` **qwen36 thinking-split at the gateway** (operator) — base
+  `qwen3.6-35b-a3b` = thinking-OFF default (the vision judge needs it), `…-a3b-thinking`
+  = opt-in reasoning. One hybrid checkpoint + `--reasoning-parser qwen3`; mirrors the
+  glm-5.1 split. The judge's enable_thinking=false is what makes the NVFP4 swap
+  transparent. (`6de0844`)
+  _Archived 2026-06-19._
+
+- `[2026-06-15]` **arbo engine rebuilt v0.11.6 → v0.11.22 from main; catalog re-applied (v0.11.23) — curated `/workflows` footer live.** src/ schema changes (Model/Lora.summary, Workflow.blurb/keywords + the /workflows endpoint) need a MANUAL image rebuild (the webhook warns-only on `src/`/`Dockerfile`). Built from `/worktank/arbo/repo`, bumped `ARBO_IMAGE` in the host .env, recreated + verified healthy (auto-rollback armed, unneeded). comfy-dev's v0.11.20 catalog push had first crash-looped prod (old baked schema `extra=forbid` vs new git-mounted catalog fields → ValidationError); they reverted the catalog, so rebuild-from-main loaded clean, then they re-applied (v0.11.23). comfyui is a SEPARATE compose project — the rebuild didn't touch it.
+  _Archived 2026-06-19._
+
+- `[2026-06-15]` **althing-core refreshed to v0.14.1 box-wide on nh3-dev** (forseti + heid both routed it; central-sequenced per DevOps-delegation since `uv tool install --force` swaps the shared binary all handles use). Fixes the monitor non-truncating-lock bug so `stop-monitor`/`--stop` reliably recovers orphans. The editable install (`uv tool install --force --editable .`) covers every nh3-dev handle; `--version` trued up 0.14.0→0.14.1; verified `--stop` works against a real orphan.
+  _Archived 2026-06-19._
+
+- `[2026-06-15]` **ratatoskr Tier-3 affect live smoke GREEN** (consumer-confirmed). Brokered/minted consumer_id="ratatoskr" Heimdall HS256 key on personal Worldtree (the key IS BOTH the create-session Bearer AND ratatoskr's JWT verifier secret — plain utf-8 string, NOT base64/raw); injected ENV-ONLY to `~/.config/ratatoskr/provider.env` (mode 600) on nh3-dev; added `10.100.10.50:8390` to personal's `BIFROST_CLIENT_ALLOWED_HOSTS`; drove define→session (handshake 200)→turn→affect.emit (200, snapshot persisted). Worldtree signs Bifrost with the caller's raw Bearer verbatim. Drive commands + claims contract in althing thread `01KV4ZYSE3…`.
+  _Archived 2026-06-19._
+
+- `[2026-06-15]` **Dense Qwen3-VL-32B-NVFP4 A/B'd as arbo's anatomy hero-judge vs the qwen3.6-35b-a3b MoE → dense LOST, pilot torn down, GPU 0 reclaimed.** Dense NVFP4 loads clean on vLLM where the MoE NVFP4 doesn't (no expert-scale keys); ~44 tok/s on Blackwell GPU 0; vision tower stays high-precision. But comfy-dev's ground-truth A/B: MoE wins (faster + dual-use as general LLM + slight fine-detail edge), and the subtle-proportion tier is below ALL VLs. **Don't re-propose a dense-VL judge swap.** (auto-memory `project-qwen3vl-dense-pilot`)
+  _Archived 2026-06-19._
+
+- `[2026-06-15]` **MastMed prototype containerized + exposed publicly via cloudflared.** Self-contained static SPA (bun→caddy multi-stage build), basic-auth gated (operator's gating call), ana-docker:8089, cloudflared quick tunnel. See in-flight for the ephemeral-URL caveat + named-tunnel upgrade path.
+  _Archived 2026-06-19._
+
+- `[2026-06-15]` **infra-ops bootstrapped onto corviduo-dev** (operator-directed) — infra-ops now has SSH + NOPASSWD sudo there, OVERRIDING the fleet bootstrap script's deliberate partner-box exclusion (explicit host arg). corviduo config edits no longer need `vh` + a sudo password. (auto-memory `reference_corviduo_dev_emergency_ops`)
+  _Archived 2026-06-19._
+
+- `[2026-06-15]` **R18 clip+caption placement = ana-ml2** (operator: "ana-docker… ana-ml2 unless irv-ml1 has room"; irv-ml1 lacked free VRAM for the 30B-A3B captioner). Models staged + stub deployed; see in-flight + the hard real-voice gate.
+  _Archived 2026-06-19._
+
+- `[2026-06-14]` **LiteLLM infra-ops admin key provisioned** (operator) — resolves the LiteLLM half of the credential-migration directive; use it for ALL gateway ops (NOT `sk-corvid`). Value at `~/.config/litellm/infra-ops-key` (mode 600); gateway reachable directly from nh3-dev at 10.250.50.70:4000. (auto-memory `reference_litellm_infra_ops_key`)
+  _Archived 2026-06-19._
+
+- `[2026-06-16]` **litellm `strip_empty_tools` pre-call hook shipped** (`d1bea13`) — an empty `tools:[]` 500s vLLM ("tools must not be an empty array"); a global `litellm_settings.callbacks` CustomLogger pops it (+ orphaned `tool_choice`) before forwarding, so it covers EVERY vLLM model, not one. `drop_params` only drops unsupported PARAMS, not empty VALUES. Mounts beside config.yaml (litellm resolves callbacks relative to the config dir). Verified live across granite/mistral/stream. (`stacks/litellm/conf/strip_empty_tools.py`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **single-file `gateway-chat.html` playground shipped** (`984ca3d`, `tools/`) — zero-dep browser chat straight to the gateway (`:4000`, CORS open), system-prompt box, streaming SSE, renders `reasoning_content`, NEVER sends `tools`. Built because the LiteLLM admin-UI playground can't test vLLM-backed models (see Tried-and-abandoned). Serve on-request via `python3 -m http.server -d tools`.
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **claude-bot granted ADMIN on vh/arbo** (operator) — arbo CI/CD gitea ops now route through the service account, not the operator's personal `vh` (the credential-migration directive). Operator added claude-bot as Administrator collaborator; claude-bot self-serves via its `gitea-password` (basic-auth = FULL perms; its stored token is package-write-only, can't change repo settings). (auto-memory `reference_claude_bot_gitea_creds`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven.** The missing server-side step was the per-PORT `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist (`:8391`; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the `score_breakdown` shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), not `agent_self_enabled`. (tracking: c99aa49 + auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED:** the `score_breakdown`-shape root-cause was disproven; real cause = cold-recall `scope_filter` adds an `agent_self` axis → ratatoskr's conjunctive INV-005 match drops the `{end_user}`-only chunks → 0 hits. Now Worldtree #297. (auto-memory `project_worldtree_295_recall_rootcause`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **infra-ops = the DURABLE admin path on corviduo-dev** (operator granted standing admin; extends the 2026-06-15 bootstrap) — added a `Host corviduo-dev → User infra-ops` `~/.ssh/config` alias + repointed `servers/corviduo-dev/ssh-target` to `infra-ops@`; `ssh corviduo-dev` / `elway corviduo-dev` now land as infra-ops w/ NOPASSWD (vh = explicit-only human path). elway resolves via ssh-config, NOT the ssh-target file. (tracking: c99aa49 + auto-memory `reference_corviduo_dev_emergency_ops`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **demo + personal `character` model flipped qwen3.6-35-a3b → mistral-small-4** (operator) — in `model_roles.yaml` the FIRST `catalog_id` in a role's `binds:` is the default; reordered mistral-first (qwen retained in the switch-allowlist, non-destructive), applied via the PIN-SAFE api recreate (derive `WORLDTREE_IMAGE` from the matrix sibling — never a bare `compose up`, which falls to the crash-blocking stale `:latest`), verified a fresh character agent resolves model=mistral-small-4. (auto-memory `reference_corviduo_dev_emergency_ops`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **NVFP4 ModelOpt MoE now LOADS on vLLM 0.23.0 — SUPERSEDES the
+  2026-06-14 "blocked" finding (#44081 fixed).** Confirmed the cached
+  `nvidia/Qwen3.6-35B-A3B-NVFP4` loads clean + generates on 0.23.0; swapped prod
+  qwen36 FP8→NVFP4 (~13 GB reclaimed on GPU 1, fp16 KV). Validated by brokkr (speed:
+  faster single-stream, ≥parity batched) + comfy-dev (vision anatomy-judge A/B: PASS).
+  One W4A4 tell: minor multi-step chained-reasoning slip — doesn't bite the vision
+  judge. Mistral's NVFP4 is a SEPARATE compressed-tensors path (day-0), distinct from
+  the ModelOpt one. (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Mistral Small 4 = the GPU-0 tenant** (operator) — fills the slot
+  the deferred creative-writing pick held, and supersedes llama-swap (now down).
+  NVFP4 119B MoE, pinned **vLLM v0.22.0** for working vision (0.23.0 breaks it),
+  256K context. Operator's creative-writing model; **abliteration planned** → then it
+  becomes the uncensored-creative model the persona substrate needs. Reasoning via
+  `reasoning_effort` (BINARY none|high — no medium). (tracking: `c77a9aa`/`9a49963`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Selene judge restored on vLLM FP8, NOT NVFP4** — W4A4 is too
+  aggressive for a precision judge validated at Q6_K; FP8 ≥ Q6_K fidelity, and Selene
+  is text-only Llama-3.1 so dynamic `--quantization fp8` has no vision tower to noise.
+  `selene-1-mini-8b` on GPU 1 :8011. Worldtree's Domari role repointed to it. (`c985ede`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **GPU-1 buffer put to work: selene 0.13→0.17 + qwen36 0.32→0.34**
+  — the two KV-bound generative models that gained Worldtree consumers; granite + the
+  pooling models under-use their util, so growing THEM is wasted. (`e124a2f`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **claude-bot Gitea service account stood up; corviduo-ORG conversion
+  TABLED** (operator). claude-bot is the credential-migration foundation. The proper
+  fix for scoped agent package-write is converting vh (a USER) → a `corviduo` ORG —
+  blast radius assessed (org creation = free; arbo-first migration = cheap, no registry
+  images; full fleet = incremental, ~8 active vh images / 50 repos) — but tabled for
+  later. Registry-push deferred until then. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **arbo engine builds → comfy-dev OWNERSHIP** (operator approved) +
+  Gitea Actions runner stood up on irv-ml1 (host-executor as lkraven). infra-ops keeps
+  box-level escalation (runner, package:write PAT pending corviduo-org, webhook). See
+  the runner entry in Current state. (tracking: this snapshot + arbo thread `01KV70AB2B4B…`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Worldtree persona substrate migrated qwen→Mistral via capability-
+  named profiles** (v0.35.8→v0.35.10; demo + personal). PRE-SYNC-FIRST is now the
+  standing pattern for any providers.yaml/profile change: sync config to the bind-mount
+  BEFORE the deploy image-swap restart, so new agents + new config land together (the
+  v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread `01KV6X0M…`)
+  _Archived 2026-06-20._
+
+- `[2026-06-19]` **litellm task-aliases: `classifier`→granite-4.1-8b, `summarizer-large`→gen/qwopus** (`summarizer`→granite already existed). Also added then REMOVED `gen-nt`/`gen-reasoning-nt` (redundant with the global `strip_empty_tools` hook). (ef45f6d, 2e5ab72, 75bd4c3)
+  _Archived 2026-06-21._
+
+- `[2026-06-19]` **gateway-chat: model-smoking web chat → PERSISTENT container.** `/v1/models` auto-discovery + image-upload; `stacks/gateway-chat/` (nginx:alpine, ana-docker:8091, homepage tile). (4c40b9f, 740bcae)
+  _Archived 2026-06-21._
+
+- `[2026-06-19]` **pi on nh3-dev wired to `gen`** — `~/.pi/agent/models.json` + a `gen` launcher mirroring `glm`. Local box config, not a repo artifact. (untracked by operator choice)
+  _Archived 2026-06-21._
+
+- `[2026-06-19]` **`gen` model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8.** ana-ml2 GPU 0, served-name `qwen3.5-122-a10b`. STABLE vLLM + fp8 KV reaches 262144 — nightly+turboquant-4bit-KV was UNNECESSARY. 92.7 tok/s warm. (20e796c, 5b06514)
+  _Archived 2026-06-21._
+
+- `[2026-06-19]` **TP=2 across the two ana-ml2 Blackwells REJECTED** — `PIX` (PCIe, NO NVLink) → all-reduce-bound → SLOWER. One-model-per-card is optimal; PP=2 only if a >96 GB model is ever wanted. (untracked by operator choice)
+  _Archived 2026-06-21._
+
+- `[2026-06-19]` **qwen-image-bench (T2I judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1**, aliased `image-judge`. (bfae924, 5dfce04)
+  _Archived 2026-06-21._
+
+- `[2026-06-18]` **heretic abliterated Mistral Small 4 NVFP4 built + LIVE as `mistral-small-4`** (in-house NVFP4, drop-in stack, A/B'd vs official, "heretic stays"). DISPLACED from GPU 0 on 2026-06-19 for the gen/Qwopus swap. (dd3a5c9, f566f61, `tools/mistral-small4-nvfp4/`)
+  _Archived 2026-06-21._
+
+- `[2026-06-18]` **irv-ml1 VRAM consolidation + comfy-dev torch-pin** — ComfyUI pinned to the A6000 (48 GB), audio zoo → 3090; `DISABLE_UPGRADES=true` @ torch 2.12.1. (a8550ad)
+  _Archived 2026-06-21._
+
+- `[2026-06-18]` **ComfyUI acceleration set (9 nodes) installed for comfy-dev** on irv-ml1.
+  _Archived 2026-06-21._
+
+- `[2026-06-17]` **ana-ml2 `vm.overcommit_memory=1` made durable** (sysctl drop-in) — overcommit=0 + zero swap caps CommitLimit; large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff)
+  _Archived 2026-06-21._
+
+- `[2026-06-17]` **GLM 5.2 wired into litellm** (`glm-5.2` + `-reasoning`, z.ai passthrough, `extra_body.thinking.type` toggle). (fe77a35)
+  _Archived 2026-06-21._
+
+- `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (successor to nh3-ansible); hosts `/opt/externs` pi-on-GLM-5.2 client workspaces. (a841eab)
+  _Archived 2026-06-21._
+
+- `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-` prefix (operator) — reuse the existing capability convention.** gen/gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge (passthrough)/gen-frontier/-reasoning. The **swap-transparency rule**: embedding stays model-named (`qwen3-embedding`) because vectors aren't swap-transparent; everything else can swap behind a stable alias. (d372103, 7819f96)
+  _Archived 2026-06-25._
+
+- `[2026-06-20]` **Removed the litellm `*` wildcard → llama-swap** (operator; llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now 404 cleanly instead of a misleading "Connection error" — closes the long-standing footgun. (d0eb09c)
+  _Archived 2026-06-25._
+
+- `[2026-06-20]` **Langfuse retired** (operator) — redundant with LiteLLM native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse spew). Callbacks dropped, stack torn down. (f8eda1c)
+  _Archived 2026-06-25._
+
+- `[2026-06-20]` **OmniVoice streaming `/tts` — vendor chatterbox-fast's scheduler, not its whole package** (operator-approved vendor-copy). Diffusion TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox-fast remains the lowest-latency front-end. (288d085, cd92b85)
+  _Archived 2026-06-25._
+
+- `[2026-06-20]` **Daily image+buildx prune cron on ana-docker** (the missing automation behind the recurring image bloat) — `lkraven` crontab, prune unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context.
+  _Archived 2026-06-25._
+
+
+- `[2026-06-25]` **althing re-architected to the lean multi-machine bus; nh3-extdev stood up as a
+  MODEL B mesh peer** (dedicated `althing-svc` + group-shared `/srv/althing`). `reference_nh3_extdev_althing_mesh`.
+  _Archived 2026-07-08._
+
+- `[2026-06-23]` **zellij native web client piloted on nh3-dev** (`zellij-web.service` :8443)
+  alongside ttyd. `reference_zellij_web_seat`.
+  _Archived 2026-07-08._
+
+- `[2026-06-22]` **Worldtree persona-render config arc (#314/#322/#317) pre-synced + deployed green
+  on demo+personal** — #317 a boot-blocking config REMOVAL. `reference_corviduo_dev_emergency_ops`.
+  _Archived 2026-07-08._
+
+- `[2026-06-20]` **R22 (brokkr/dwarves) stood down to gateway-only; full-access R22 key minted;
+  Phase B CANCELLED** (Worldtree model-agnostic → no deploy path). Key at `/home/lkraven/.r22-gateway-key`
+  (persistent mode-600, carries paid GLM, don't delete). MUT = free `qwen3.5-122-a10b` (`gen`).
+  Operator steer: R22 research is gated on a pragmatic/deployable outcome, not advancing-the-art.
+  _Archived 2026-07-08._
+
+- `[2026-06-20]` **claude-bot issue-scope token minted for worldtree-dev self-serve** (id 16,
+  `write:repository`+`write:issue`); old token revoked. Advances the credential-migration directive.
+  _Archived 2026-07-08._
+
+- `[2026-06-20]` **rest-server-ana recovered + backup prevention shipped + worldtree-dev admin keys
+  provisioned** (demo d113207c / personal f4f75adb). Cred rotation (5 rest-server pw) BELAYED.
+  _Archived 2026-07-08._
+
+- `[2026-06-20]` **claude-bot → ADMIN on vh/Worldtree** (operator-authorized) — self-serves WT
+  deploys/tokens henceforth.
+  _Archived 2026-07-08._
+
+- `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.**
+  (auto-memory `project_migrate_infra_access_to_claude_credentials`)
+  _Archived 2026-07-13._
+
+- `[2026-07-01]` **Worldtree #332 embed-recall diagnosed + scoped-log view/tunnel provisioned + fix
+  verified** (15×→1.01× re-embed). `reference_wt_gateway_scoped_log_view`.
+  _Archived 2026-07-13._
+
+- `[2026-07-01]` **qwopus native MTP speculative-decode tested on `gen` → NOT kept** (+12% single-stream,
+  −15–20% aggregate at concurrency, silently drops min_p/logit_bias). Banked for T1. `reference_gen_qwopus_122b`.
+  _Archived 2026-07-13._
+
+- `[2026-07-01]` **Deckard trial → reverted to qwopus (`gen`)** (won writing "in every way" but ~36 vs
+  ~90 tok/s; spec-decode rescue ruled out). git `b63c48b`→`681eb70`. Deckard kept staged as T1's
+  writing benchmark.
+  _Archived 2026-07-13._
+
+- `[2026-07-02]` **mtf-dev granite harness-spike ran GREEN — MECHANICAL only, efficacy DEFERRED to
+  the T1 run.** Trainer TRL SFT→DPO→eval seam proven end-to-end on a synthetic fixture (not the E-RP
+  corpus); operator DECIDED no intermediate real-efficacy granite spike (uninterpretable proxy —
+  arch gap + abliteration axis). `reference_gen_qwopus_122b`.
+  _Archived 2026-07-13._
+
+- `[2026-07-03]` **ratatoskr-dev DEMO Heimdall key provisioned (R30 φ0).** Minted a tier-user key on
+  the demo via `POST /admin/keys` (bootstrap admin key), mirroring their personal base consumer (no
+  character-binding); base-agent affect reads work ungated. `reference_worldtree_demo_key_mint`.
+  _Archived 2026-07-13._
+
+- `[2026-07-04]` **character-rp: gateway-forwarding proven + role pre-staged + #344 shipped.**
+  Empirically confirmed per-request `extra_body` (top_k/repetition_penalty) forwards through the
+  `gen-reasoning` LiteLLM alias to vLLM + standard params override the alias defaults — no gateway
+  cap needed (I over-built a dedicated alias, operator corrected, reverted with zero fleet impact).
+  Pre-staged the `character-rp` role into demo+personal bind-mount `model_roles.yaml` (byte-verified
+  on b18; caught the cached-registry ordering). worldtree-dev shipped **#344 (v1.0.0b19)** for the
+  durable-agent override-drop. spend_logs spot-check queued (ratatoskr's timestamp ping).
+  _Archived 2026-07-13._
+
+- `[2026-07-04]` **althing v2 herald+receiver formalized as systemd on nh3-dev.** `althing-herald.service`
+  (Restart=always, **Environment=PATH incl ~/.cargo/bin** — the pane-dispatch fix) + `althing-receiver.service`
+  (v2 → pillar-3 `/owner/*` live); stale forseti unit removed; both on v2.0.0b2, canonicalized by
+  forseti. `reference_nh3_dev_althing_herald`.
+  _Archived 2026-07-13._
+
+- `[2026-07-04]` **LitBench-RM respun (irv-ml1 A6000, comfyui displaced)** for T1's reward ensemble;
+  operator sole comfyui consumer, holding image-gen until LitBench done. `reference_litbench_rm_irv_ml1`.
+  _Archived 2026-07-13._
+
+- `[2026-07-05]` **T1 training venue: CLOUD recommended; operator chose smoke-first on ana-ml2.**
+  On-prem ruled out (ana-ml2 full — both 96G GPUs ~93G used): keep-serving = NVMe offload ~6-8 DAYS;
+  full ana-ml2 shutdown = CPU offload ~1 DAY but a whole-fleet outage. Cloud Vast.ai 8×80GB (no
+  offload → ~3-6h, ~$60-500, zero fleet impact) is the clean alt (mtf-dev + infra-ops both rec;
+  Vast for its no-content-AUP marketplace + likely-existing VastBlue account). Operator's next step
+  = the ana-ml2 CPU-offload SMOKE (~60 steps) to get real samples/sec before the full-outage-vs-cloud
+  call. HF base verified public (zero-auth pull). Runbook + gotchas in `reference_t1_cloud_train_plan`.
+  _Archived 2026-07-13._
+
+- `[2026-07-05]` **glm-5.2 canonical limits recorded** (probed live vs z.ai): **1,048,576 (1M) input
+  context / 131,072 (128K) max output**; NO gateway-side cap (pure passthrough → z.ai's limits are
+  effective). Written to the config comment (commit `624a07e`) + `reference_litellm_gateway`.
+  _Archived 2026-07-13._
+
+- `[2026-07-06]` **infra-ops NOPASSWD sudo confirmed on irv-ml1 + ana-ml2** (the arbo memory's "NOT
+  infra-ops on irv-ml1" was stale) — enabled the worker deploy + every ana-ml2 hot-swap smoke.
+  _Archived 2026-07-13._
+
+- `[2026-07-07]` **SGLang zc hot-swap: 2-line patch RAN → PARTIAL. Cleared startup + self-attn E2E, GDN
+  in_proj hit 2 MORE name-routing gaps → MERGE recommended.** Root cause was a NAME-routing gap (not the
+  feared shape-assert): `lora/utils.py:235` hardcoded `params_mapping` lacked `in_proj_qkv/in_proj_z →
+  in_proj_qkvz`. Patched those 2 lines (overlay bind-mount) → `init_lora_shapes` passed, server up, self-attn
+  loaded end-to-end. But the SAME missing GDN mapping bit 2 MORE resolvers: adapter-load skipped 192 in_proj
+  weights, and inference `utils.py:300 get_target_module_name` (substring match) RAISED → 500 + scheduler
+  died. So GDN LoRA name-routing is incomplete in ≥3 SGLang paths (fixed 1). NOT shape-intractable (self-attn
+  proves the separate-key→fused-buffer machinery works), but NOT "one targeted fix" either → per operator's
+  LAST-shot rule → MERGE. mtf-dev looped + closed (thread 01KWZDZFYN…); B/A shard-math never reached.
+  Supersedes the "SGLang wants a THIRD format" framing — no new adapter format needed.
+  _Archived 2026-07-13._
+
+- `[2026-07-07]` **T1 (retargeted qwopus-122B→AEON-27B) SFT COMPLETE on ana-ml2** — all-TRL torch-GDN,
+  batch1/accum8/GC-off, 1 epoch ~9h @ 0.25 s/s, loss 2.445, adapter banked. Supersedes the 2026-07-05
+  cloud-vs-smoke venue decision: AEON-27B's 52G bf16 fits one Blackwell → ran on-prem in ~9h (no cloud,
+  no full-fleet outage). GC is NOT the throughput lever (batch1-no-GC == batch4+GC at 0.25; dvalin's 0.565
+  was warmup-noise from a 30-sample smoke); ~0.25 s/s is the torch-GDN floor (only FLA beats it, sm_120-
+  blocked #913). `reference_t1_cloud_train_plan`.
+  _Archived 2026-07-13._
+
+- `[2026-07-07]` **T1 hot-swap on vLLM NVFP4 is DEAD — vLLM 0.24.0 qwen3_5 LoRA-application silent no-op**
+  (#47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4+FP8) AND
+  adapter-format-agnostic (separate `zc` loads-but-inert; fused rejected); #47640 guard-patch overlay didn't
+  fix. Root-caused by a 3-peer dwarf panel (eitri/regin/dvalin); their vLLM issue refs verified real via
+  GitHub API. Engine mirror-image: vLLM wants SEPARATE keys, SGLang wants FUSED.
+  _Archived 2026-07-13._
+
+- `[2026-07-07]` **HARD REQUIREMENT restated: hot-swappable LoRA or ABANDON.** Merge is NOT a solution for
+  this requirement (static; reserved for a later RP-quant track). SGLang is the last hot-swap shot; if it
+  can't apply, park the adapter + file upstream repros. `reference_lora_training_worker` (adjacent).
+  _Archived 2026-07-13._
+
+- `[2026-07-07]` **comfy-dev in-arbo LoRA-training worker DONE** — built/deployed/validated Phases 1+2
+  (first Sindra train succeeded, publish-step + `train_id`), Phase 2.5 GC contract (shape A) confirmed +
+  impl deferred. `services/lora-training-worker/`, commits 888ba6a→b617a8b. `reference_lora_training_worker`.
+  _Archived 2026-07-13._
+
 ## Tried and abandoned (archived)
 
 - `[2026-04-30]` task-board workflow with
@@ -629,3 +1067,253 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
   same clips gave the correct answer (clone 0.79 to real Imogen, 0.10 to
   default). Use ECAPA, not resemblyzer, for speaker-verification.
   _Archived 2026-06-04._
+
+- `[2026-06-02]` **Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED
+  (sub-realtime).** Benched RTF on A6000: 0.72x/0.82x/0.86x, mean 0.80x = sub-realtime,
+  so client-side chunking starves. Root cause: Fish only chunks on `<|speaker:X|>` tags;
+  plain text → batches=[whole text] → all semantic tokens generate before any audio
+  (chunk_length inert). Plus kui/ASGI StreamResponse doesn't flush (fix = anti-buffering
+  headers, kept on file). STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands → bench
+  fp4-quantized Fish; if RTF > ~1.5x give it the chatterbox-fast treatment. For now Fish is
+  a buffered catalog entry (great for SAVED gens, not live-audition).
+  _Archived 2026-06-11._
+
+- `[2026-06-02]` **Context-priming at chunk joins (chatterbox-fast §1.6) — ABANDONED
+  (discard-cut leaks the prefix).** Prepend prior sentence, generate `prefix+content`,
+  discard prefix audio. Built + shipped (`d707439`), live-A/B'd, reverted (`090e70a`).
+  Kill: `generate()` returns one waveform with NO marker where the prefix ends, and the
+  model renders the same prefix with different timing solo vs followed-by-content — so
+  locating the cut is a guess that left a whole clause in the output (operator caught
+  "...without a trace of sarcasm" spoken twice). Reliable cut needs token-level boundaries
+  or per-chunk forced-alignment (heavy). Coherence loss at joins = ACCEPTED limitation.
+  _Archived 2026-06-11._
+
+- `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED (turbo
+  isn't built for streaming).** (1) `S3GenStreamer` is docstring-only (unimplemented).
+  (2) `flow_inference(finalize=False)` is buggy (lookahead trim removes frames from `h`
+  not `h_masks`/conds → shape mismatch); a 1-line patch makes it RUN. (3) BUT the flow
+  encoder uses full-context attention (`static_chunk_size=0`) → incremental decode is
+  prefix-unstable (adding tokens re-attends, shifts earlier mel ~0.30-0.39 vs one-shot).
+  (4) Forcing `static_chunk_size>0` didn't stabilize it. True sub-second frame-streaming
+  needs deep attention surgery with quality risk — not worth it. → adaptive-chunking instead.
+  _Archived 2026-06-11._
+
+- `[2026-06-03]` **`uv pip install .` fails on SmoothKen/knn-svc** (and similar script-repos)
+  — it's analysis scripts + a poetry pyproject, no buildable package (setuptools
+  package-discovery error). Install the pyproject deps directly, don't build the "package".
+  _Archived 2026-06-13._
+
+- `[2026-06-03]` **Backgrounding `althing-cli monitor` with an inline shell `&` (instead of
+  the Bash-tool `run_in_background`) orphans it** — it survives the shell exit, holds the
+  per-handle flock UNTRACKED (won't notify the session), and `stop-monitor` doesn't detect it.
+  Fix: find + kill the orphan PID (verify cwd=this repo / handle first — nh3-dev is shared, other
+  agents' monitors run there too), then re-arm via run_in_background. Always re-arm tracked.
+  _Archived 2026-06-13._
+
+- `[2026-06-03]` **torch-2.12 venvs need `uv pip install torchcodec`** — torchaudio 2.12
+  defaults to the TorchCodec backend for `.load`; without it, real audio I/O throws "TorchCodec
+  is required" — and it ONLY surfaces at actual conversion, NOT at import/model-load. Lesson:
+  validate real I/O, not just import, when provisioning ML engine envs. (seed-vc on torch 2.4
+  uses the legacy backend, exempt.)
+  _Archived 2026-06-13._
+
+- `[2026-06-03]` **gitea webhook to a private IP is denied by `webhook.ALLOWED_HOST_LIST`**
+  (anti-SSRF; default `external` blocks private/loopback). Symptom: delivery shows
+  `dial tcp ...: webhook can only call allowed HTTP servers`. Fix = APPEND the target net to
+  ALLOWED_HOST_LIST in gitea's app.ini (keep `external`; scope tight, never `*`/`private`) +
+  restart gitea (act_runner job containers survive a restart). gitea runs as a container on
+  ana-docker (`gitea_gitea_data` volume, `/data/gitea/conf/app.ini`).
+  _Archived 2026-06-13._
+
+- `[2026-06-04]` **homepage "crash" ≠ always NFS** — a wedged container in unkillable D-state can come from dead `siteMonitor` widget targets (retired ESH firewall IPs) hanging the node event loop into `exit_mmap`, needing a host reboot. Check homepage's siteMonitors against retired hosts. (`incident_esh_docker_nfs_boot_race`)
+  _Archived 2026-06-14._
+
+- `[2026-06-04]` **pyworld needs `setuptools<81`** (imports the removed `pkg_resources`); and **R/soundgen `-lgfortran` fails** on irv-ml1 (default gcc is gcc-11 but only gfortran-12 present) → install `libgfortran-11-dev`.
+  _Archived 2026-06-14._
+
+- `[2026-06-04]` **Persistent services on irv-ml1 need SYSTEM systemd units** — the box reaps user-session processes on ssh disconnect, and `--user` systemd isn't reachable over non-login ssh, so nohup/setsid/`screen -dmS`/`systemd-run --user` all die (even with enable-linger). Use `/etc/systemd/system/`.
+  _Archived 2026-06-14._
+
+- `[2026-06-04]` **GPU pin by INDEX is ambiguous on irv-ml1** — native CUDA orders fastest-first (A6000=0) but nvidia-smi/docker use PCI order (A6000=1). Pin by **UUID** (`CUDA_VISIBLE_DEVICES=GPU-…`); verify via nvidia-smi compute-apps.
+  _Archived 2026-06-14._
+
+- `[2026-06-04]` **Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF — the "headgun" lesson.** Serve each model's canonical `tokenizer.apply_chat_template`, not the bundled template (bundled ones corrupt baselines). Verify via vLLM `/tokenize`→`/detokenize`. (`90e08f0`/`27eb537`)
+  _Archived 2026-06-14._
+
+- `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()` requires torchcodec → dies AFTER separating** (0 stems, rc=1). Fix = pin `torch==torchaudio==2.4.1` + `UV_LINK_MODE=copy`. Validate the SAVE path, not just import + GPU inference, on a bleeding-edge torch.
+  _Archived 2026-06-16._
+
+- `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — fills KV to the `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs AFTER weights+KV load (crash-loops). Fix: free co-tenant room OR `--enforce-eager`. FP8 single-stream is batch-1 GEMV (memory-bound) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`)
+  _Archived 2026-06-16._
+
+- `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres rows; clone a default + swap the measure. tok/s is the `outputTokensPerSecond` MEASURE (metrics-API/dashboard query time), not a per-generation field. langfuse-web needs `HOSTNAME=0.0.0.0`. Host 3000 is gitea's → langfuse on 3001.
+  _Archived 2026-06-16._
+
+- `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh + corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are vh-owned, the `.env` is deploy-owned 600.
+  _Archived 2026-06-16._
+
+- `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL`; defaults to claude-haiku in code, opt-in via config not `.env`. Inspect the live container env + vh-owned config files first.
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **Loading an old LoRA adapter after a vocab bump fails on embedding size.** The harness config + `tokenizer.json` are now at `new_vocab_size=50279` (v4 multi-tag); the v1/v2/v3 adapters are 50277. To load v1 (the accepted artifact), set `cfg.new_vocab_size=50277` before `load_finetuned_engine_lora` (else PeftModel state_dict size mismatch). `tokenizer.json.v3bak` is the 50277 tokenizer for a clean restore.
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **Heavy from-source compile (`MAX_JOBS=128`) on the shared PROD GPU box PINS it** — load hit 187, prod vLLM restarted, killed an in-flight quant. ana-ml2 hosts live inference; never run a big build there at full parallelism. Cap `MAX_JOBS≤32`, build off-box, or cgroup-constrain.
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **`--quantization fp8` on a VL model can quantize the VISION TOWER → garbage vision** (Qwen3.5-VL on stable vLLM: gray-grid output; LM answers text fine, so it "looks" healthy). The nightly excludes the vision tower. Validate the VISION path on a quantized VLM, not just text — and pin the engine digest with the exclusion.
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **vLLM's `--gpu-memory-utilization` is checked against FREE VRAM at startup, not total** — on a shared card, growing one service before trimming a co-tenant OOMs. Trim the shrinking service FIRST, then grow. Size to the FREE budget.
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **The `vllm/vllm-openai` entrypoint is already `["vllm","serve"]`** — compose `command:` supplies the model as the first POSITIONAL arg + flags; a second `serve`/`--model X` → "unrecognized arguments". Same-class: `tee` masks the real exit code (use `>`); HF `datasets` rejects bare `wikitext` (needs `Salesforce/wikitext`).
+  _Archived 2026-06-16._
+
+- `[2026-06-13]` **Chatterbox-Turbo LoRA finetune: the repo's `setup.py` loads the WRONG tokenizer** — pulls gpt2-medium + a grapheme merge file instead of the chatterbox-turbo GPT2 tokenizer (vocab.json+merges.txt, len 50276). Fix = override + delete the grapheme `tokenizer.json`; `[vmoan]` → new_vocab_size 50277 (1-row resize), lora_r 64 / alpha 128, modules_to_save=[text_emb,text_head]. Unique-stem corpus collision needs `{index}_{stem}` IDs. (`irv-ml1:~/r16-vmoan-harness`)
+  _Archived 2026-06-16._
+
+- `[2026-06-11]` **A completion-poll `while pgrep -f <scriptname>` SELF-MATCHES its own remote shell argv** — its own `pgrep -f` always finds itself → the loop never exits. Use a match pattern ABSENT from the poll command (the python stage, or a sentinel file), not the driver's own name.
+  _Archived 2026-06-16._
+
+- `[2026-06-14]` **vLLM ModelOpt-NVFP4-MoE loader is broken for current multimodal MoEs.** `nvidia/Qwen3.6-35B-A3B-NVFP4` fails weight-load: `KeyError: layers.0.mlp.experts.w2_input_scale` on 0.19.1, `lm_head.input_scale not registered` on 0.22.0 (vllm #44081); same class hits Gemma-4 MoE / Qwen3-30B-A3B NVFP4. The arch + quant ARE recognized (gets past arch resolution + vision-processor load) — it's the per-expert/lm_head scale-key mapping. Don't chase nightlies; use official FP8 until fixed.
+  _Archived 2026-06-18._
+
+- `[2026-06-14]` **vLLM sampler-warmup OOMs on a shared GPU even when weights fit** — it warms the sampler with `max_num_seqs` (default **1024**) dummy requests, and a big vocab (Qwen3.6 = 248K) makes that a huge transient logits tensor. A vision endpoint doesn't need 1024-way concurrency: set `--max-num-seqs 32`. Separately, post-load `ValueError: No available memory for the cache blocks` means util is too thin (weights+activation+graph ate it) — for 34 GB FP8 weights, util ≥ ~0.45 to leave KV room.
+  _Archived 2026-06-18._
+
+- `[2026-06-14]` **Recreating multiple vLLM services concurrently races the memory-profiling assertion** — `AssertionError: Error in memory profiling. Initial free memory X / current Y … other processes … release GPU memory while vLLM is profiling`. Recreate co-tenant vLLM services ONE AT A TIME (force-recreate one, wait healthy, next).
+  _Archived 2026-06-18._
+
+- `[2026-06-14]` **embed/rerank (0.6B) at util 0.05 reserve ~5.5 GB each — mostly util-reservation WASTE, not need.** A 0.6B model needs ~1.2 GB weights + ~2.5 GB CUDA/torch context; util 0.03 (~3.6 GB) fits with room, reclaiming ~4 GB (vLLM reserves the util fraction regardless of actual KV; embedding models barely use KV). Real-need floor ~3 GB — don't go to 0.02.
+  _Archived 2026-06-18._
+
+- `[2026-06-14]` **Chatterbox-Turbo decode-knob foot-guns** (R16 v1-polish + emergent probes): the turbo length cap is `max_gen_len` (default 1000) on `t3.inference_turbo`, NOT `max_new_tokens` — and `tts_turbo.generate` does NOT forward it (wrap inference_turbo to cap). `rep_pen 2.0 / temp 0.5` BACKFIRES (degenerate 24 s run-on). Soft decode tamers cut the NVV ITSELF, not just the run-on tail (operator: "p1 trims the moaning too") — gen-time polish can't beat v1's defaults. Inline base-NVV tokens DEGENERATE (moan-cascade + gibberish), they don't modulate the surrounding words.
+  _Archived 2026-06-18._
+
+- `[2026-06-15]` **A comfyui restart-loop with `exit 0` + `OOMKilled=false` is NOT OOM — suspect a native SEGFAULT.** `docker inspect` shows only the LAST exit code; a GPU OOM is a catchable `RuntimeError: CUDA out of memory`, NEVER `Fatal Python error: Segmentation fault`. Diagnostic: `RestartCount>0` = same container restarted in-place → the docker json-log ACCUMULATES across all cycles, so grep the full `docker logs` for the crash signature + the lines right before each restart banner. comfyui's was at VAE decode (sampling completes, then segfault loading `AutoencodingEngine`) — gen-triggered but native, not VRAM. Suspects logged: torch/cu130 mismatch, an injected `aimdo` cuda-funchooks lib, the VAE.
+  _Archived 2026-06-19._
+
+- `[2026-06-15]` **NEVER blanket-kill `althing-light-monitor` by PID** — they're per-handle and MANY agents run their own on the shared nh3-dev box (heid / ratatoskr-dev / worldtree-dev / bifrost-dev each have one). A `kill $(pgrep -f light-monitor)` downs four peers' monitors (near-miss caught by checking `/proc/<pid>/environ` `ALTHING_HANDLE` + cwd first). Recover your OWN via `althing-cli stop-monitor` (v0.14.1+); a raw `&`-backgrounded monitor is untracked (no harness fire-notification) — always arm via `run_in_background`. (extends the pkill-self-match lesson below)
+  _Archived 2026-06-19._
+
+- `[2026-06-15]` **Manual `docker compose up` on corviduo Worldtree DOWNGRADES to a stale local `:latest` → crash-loop (config/image mismatch).** Compose `image:` is `…/worldtree:latest`, but local `:latest` lagged ~2wk (`21c90d5eeb2d`); CI/CD deploys NEWER SHA images, and a real pull 401s (daemon has no gitea-registry creds — that's CI/CD's). `--pull never` then used the stale latest → personal crash-looped on `ConfigurationError: echo.default_model missing` → **~2-3 min personal outage**. RECOVER: retag `:latest` to the image a HEALTHY SIBLING runs (`docker inspect worldtree-worldtree-api-1 --format '{{.Image}}'` → `docker tag …`), then `up -d --pull never`; verify `Application startup complete`. Don't manually recreate Worldtree on corviduo without pinning the correct image — image+config are CI/CD-coupled. (auto-memory `reference_corviduo_dev_emergency_ops`)
+  _Archived 2026-06-19._
+
+- `[2026-06-15]` **`.claude.json` (Claude Code config) truncates on ENOSPC → invalid JSON → CC resets to defaults on next start.** nh3-dev hit 100% disk (a peer's R19 dataset pull staged ~147G to local /home instead of the 7TB NAS); the in-flight `.claude.json` write was cut mid-key. Repair = strip the dangling token + close the open braces, validate `json.load`. CC keeps its OWN rolling backups at `~/.claude/backups/` + writes a `.claude.json.corrupted.*` snapshot when it self-detects. After ANY ENOSPC on a box running CC/agents: integrity-check `.claude.json` (json.load) AND the althing bus DB (`PRAGMA integrity_check`).
+  _Archived 2026-06-19._
+
+- `[2026-06-15]` **`pkill -f althing-light-monitor` SELF-MATCHES the killing shell** (the pattern is in the command's own argv) → kills itself mid-run (exit 144/truncated output). Stop the light-monitor via `althing-cli stop-monitor` or a captured PID — never `pkill -f <string-present-in-this-command>`. The singleton lock can also RACE to 2 live monitors during re-arm churn; keep exactly one tracked (run_in_background) monitor, and a raw `&` monitor is untracked (no harness fire-notification — don't use it).
+  _Archived 2026-06-19._
+
+- `[2026-06-14]` **Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP.** `gitea.phasefinal.com` = public `38.120.12.44` (ana-srv1); gitea is a container on ana-docker, git-SSH `10.250.50.70:222` + HTTP `:3000`. A fleet host egressing to public `:22` gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (`git fetch` times out under `set -euo pipefail`, aborts before reset). Bit irv-ml1's arbo deploy. `:22` on `10.250.50.70` is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented `docs/orientation.md` (`6e58e57`).
+  _Archived 2026-06-19._
+
+- `[2026-06-16]` **litellm 500 `Router.acompletion()/aembedding() missing 'messages'/'input'` = a request missing `Content-Type: application/json`, NOT a gateway outage.** curl `-d` defaults to form-encoding → litellm can't parse the JSON body → `data` reaches the router without `messages`/`input` → 500 (should be a 400; litellm #16993). My own diagnostic calls dropped the header → I misread it as a gateway outage and needlessly bounced the gateway ~4× chasing a phantom (image/version/config were fine throughout; a malformed UI-added "Mistral Story Eval" model in the DB was a red herring I deleted). ALWAYS send `-H "Content-Type: application/json"` testing litellm; reproduce with a header'd call before declaring a litellm incident.
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **LiteLLM admin-UI playground can't test vLLM-backed models** — it auto-sends empty `tools:[]`, vLLM 400s (litellm #6228); the gateway `strip_empty_tools` hook is a PROXY hook and structurally can't reach the UI's in-process `litellm.completion()` call. Off-ramp = `tools/gateway-chat.html`. (Langfuse playground also out: its SSRF guard blocks internal-IP LLM connections, wontfix Langfuse #13097.) (auto-memory `reference_litellm_ui_playground_vllm_deadend`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **A gitea Actions run that "looks like it never fired" (no image, untouched build dir) usually DID fire then skipped or failed-fast.** Bit arbo TWICE: run #3 looked-never-fired but SKIPPED (change-detection `engine=0`); run #4 looked-never-fired but FAILED in ~7s (early-step). Check the run LIST + status via claude-bot (`GET /repos/{o}/{r}/actions/tasks`) BEFORE a runner-connection/dispatch theory. CORRECTION (2026-06-16, proven): action logs ARE readable via API at the per-JOB endpoint — `GET /api/v1/repos/{o}/{r}/actions/runs/{run}/jobs` → take a job `.id` → `GET /api/v1/repos/{o}/{r}/actions/jobs/{job_id}/logs` returns the full plain-text log (claude-bot basic-auth, internal route `10.250.50.70:3000`). Only the RUN-level `/runs/{id}/logs` 404s; the per-job one works, so no UI needed. Gotcha when reading: gitea marks **every** step `conclusion:failure` once any step fails — the per-step conclusions mislead; trust the log body + step *timestamps* (the step with real duration is the culprit) not the conclusions. (Used 2026-06-16 to diagnose arbo run #5/task 1175 = test-gate `uv run pytest` → `Failed to spawn: pytest` = pytest absent from the fresh-checkout venv sync, NOT the SSH-clone comfy-dev suspected.)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **vLLM 0.23.0 breaks Mistral MULTIMODAL at startup** (`MistralCommonImageProcessor
+  has no attribute fetch_images`, #44911 regression ~0.22.1+; same class hit Mistral-3.1/
+  Devstral). The dummy-image profiling crashes. Workarounds: `--limit-mm-per-prompt
+  '{"image":0,"video":0}'` loads text-only; for VISION, pin **v0.22.0** (last pre-regression).
+  Tradeoff: v0.22.0 *buffers* reasoning (doesn't stream reasoning_content cleanly); a newer
+  build streams it but breaks vision — vision won (Mistral is the abliteration-baseline).
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Mistral `-reasoning` TTFT variance = Triton kernel JIT-compile-during-
+  inference, NOT reasoning time.** `jit_monitor` logs the spike (`_compute_slot_mapping_kernel`,
+  `apply_token_bitmask_inplace_kernel` ← the tool-call/structured-output path). vLLM's startup
+  warmup covers only SOME shapes; a first-hit uncovered shape JITs a kernel (seconds), then
+  CACHES it — warm until container recreate (the Triton cache is IN-container, no volume, so
+  recreate = cold again). Secondary: GPU idle-down to P8/180 MHz between requests. Also:
+  Mistral `reasoning_effort` is BINARY (none|high only — medium/low → HTTP 400).
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Worldtree old-baseline instances need the FULL config set synced, not the
+  changeset delta.** Personal (a PRE-mistral baseline) crash-looped (41 restarts) on
+  `DanglingBindingError: role 'echo' binds missing catalog entry 'qwen3.6-35-a3b-heretic'` —
+  its `model_roles.yaml` (2026-06-10, never migrated) still bound the retired heretic that the
+  synced v0.35.10 providers removed. The "model_roles unchanged — skip" was correct only for
+  demo (already at the v0.35.9 baseline). Old-baseline → sync providers + model_roles + defaults.
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **vh is a Gitea USER, not an org → no scoped package-write for a service
+  account.** Gitea user-namespace packages are owner/admin-only, so claude-bot can't push to
+  `vh/arbo` without being admin. The fix is converting vh → a `corviduo` ORG (tabled). Also:
+  vh's `tea` token lacks `write:admin` — create gitea users via the server-side `gitea admin
+  user create` CLI (`docker exec -u git gitea …`), not the admin API.
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **`docker compose up -d` on a multi-service vLLM stack recreates ALL its
+  services** (shared `.env` → all configs re-hash) → concurrent memory-profiling race
+  ("No available memory for the cache blocks"). Target one: `docker compose up -d <service>`.
+  (auto-memory `feedback_compose_up_recreates_whole_stack`)
+
+_76 older entries archived to archival-memory.md._
+  _Archived 2026-06-20._
+
+
+- `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Only a process restart drops a config model. The INVERSE works: `/model/new` live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce.
+  _Archived 2026-06-21._
+
+- `[2026-06-19]` **the litellm `*` wildcard routed to DECOMMISSIONED llama-swap → a stale/typo'd model name returned a misleading `Connection error`, not `model not found`.** Bit a brokkr call to the renamed `qwen-image-judge`. **→ RESOLVED 2026-06-20: wildcard removed (d0eb09c) — unknown models now 404 cleanly.**
+  _Archived 2026-06-21._
+
+- `[2026-06-19]` **A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace (~3.09 GB) is the floor**, allocated OUTSIDE `gpu-memory-utilization`. FIX: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` + leave ~3.2 GB free (util <= ~0.95 on a tight MoE).
+  _Archived 2026-06-21._
+
+- `[2026-06-19]` **vLLM decode tok/s: ALWAYS discard the first generation (cold warmup).** Cold ~ 24.8 tok/s vs warm 92.7 — a cold read undersells decode ~3-4x. Measure run 2+ over a >=256-token output.
+  _Archived 2026-06-21._
+
+- `[2026-06-19]` **For full native 256K on one 96 GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE.** Stable fp8 KV already fits 262144 (hybrid SSM+attn caches KV only on attn layers → small pool). Reach for fp8 first.
+  _Archived 2026-06-21._
+
+- `[2026-06-18]` **mmartial `comfyui-nvidia-docker`: root pip installs CRASH-LOOP the container.** `docker exec -u 0 pip install` leaves root-owned files in the uid-1000 venv → the boot script (uid 1000) fails its torch step → crash loop. FIX: `chown -R 1000:1000 /comfy/mnt/venv`; pin with `DISABLE_UPGRADES=true`. (auto-memory `reference_irv_ml1_comfyui_mmartial`)
+  _Archived 2026-06-21._
+
+- `[2026-06-18]` **ComfyUI custom nodes break on version-assumption drift** — box runs 0.24.1; TeaCache imports `precompute_freqs_cis` at MODULE level (guard the LTX-only import). `pip install tensorrt` defaults to cu13 → use `tensorrt-cu12`.
+  _Archived 2026-06-21._
+
+- `[2026-06-17]` **Mistral HF→NVFP4 quant: the placement knob is the whole game.** `device_map="cpu"` (CPU-resident, sequential onload) works; `"auto"` OOMs, `max_memory` offloads to meta. Read shards with plain `read()` + `safetensors.torch.load(bytes)`, NOT `safe_open` (mmaps the 50 GB shard → ENOMEM on `/tank` ZFS).
+  _Archived 2026-06-21._
+
+- `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — no HF `Mistral4` backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first.
+  _Archived 2026-06-21._
+
+- `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree (worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea` on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur).
+  _Archived 2026-06-25._
+
+- `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell (vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3` arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1 fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager` (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 (non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` + `playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending operator OK to free.
+  _Archived 2026-06-25._
+
+- `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere.** No per-stack `logging:` limit and no docker-daemon default → ClickHouse's verbose stdout grew unbounded to ~94 GB and filled ana-docker's `/` (healthchecks then failed fleet-wide on `runc /tmp: no space left`). Lesson: on a full disk, the symptom (all-unhealthy) is the FORK-can't-write healthcheck, not the containers; find the runaway via per-container-dir `du`, not `docker system df`. The systemic fix is a daemon-default `log-opts` (`max-size`/`max-file`) — NOT just per-service patching.
+  _Archived 2026-06-25._
+
+- `[2026-06-20]` **OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk streaming is a net loss.** Measured ~2.8× (32-step) / 5.6× (16-step) on the 3090; every chunk costs ~the same regardless of length, so naive per-sentence streaming triples total gen time + starves. Use few/aggressive chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end.
+  _Archived 2026-06-25._
+
+- `[2026-06-20]` **Named buildx builders accumulate cache the standard prunes MISS.** `docker image prune -a` + `docker builder prune` don't touch `buildx_buildkit_<name>-builder0_state` volumes (per-builder buildkit state). Prune them with `docker buildx prune --builder <name> -af`, or `docker volume rm` orphaned ones (builder already removed).
+  _Archived 2026-06-25._
+
+
+- `[2026-06-25]` **althing "unreachable: <machine>" can MASK an app-level 500.** Raw network was
+  clean; root cause = receiver DB agents-table not synced with the config roster → delivery 500'd
+  "unknown to: <handle>", MAPPED to "unreachable". Diagnose: raw curl to :8087 + connect-probe ⇒ NOT
+  network. Fixed in althing v0.17.1. `reference_nh3_extdev_althing_mesh`.
+  _Archived 2026-07-08._
+
+- `[2026-06-20]` **rest-server `.htpasswd: permission denied` = the ana-nas NFS mount FAILED (ghost
+  file on the local mount point), NOT a decommission.** `mnt-backup.mount` stuck `failed` (fstab bare
+  `defaults`) → rest-server serves an empty local dir. Recovery in disaster-recovery.md.
+  _Archived 2026-07-08._
+
+- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — but `ssh
+  infra-ops@ana-docker` HAS NOPASSWD root.** A `sudo cp` as lkraven silently failed → nearly punted
+  the rest-server recovery. Reach for `infra-ops@ana-docker` for sudo ops.
+  _Archived 2026-07-08._

configs/homepage/README.md

@@ -21,21 +21,32 @@ to the compose file and is gitignored.
 
 ## Layout convention
 
-`settings.yaml` drives the group layout:
+`settings.yaml` drives the group layout across four tabs:
 
 ```
-Monitoring        row x 3   fleet hubs (Beszel, Dozzle, Backrest, Uptime Kuma)
-AI Systems        row x 3   GPU inference services (llama-swap, vLLM embed/rerank)
-Apps              list      user-facing apps (Gitea, Vaultwarden, Seafile, ...)
-Media             list      Plex, Jellyfin
-Games             list      Pterodactyl
-UltraSeedbox      row x 3   external bookmarks
-Infra - ANA       list      Anaheim hardware + hypervisors + BMCs
-Infra - NH3       list      NH3 hardware + hypervisors
-Infra - ESH       list      ESH home-lab hardware + hypervisors
-Service Networking collapsed  toolchain (Traefik, CrowdSec, Dockge, AdGuard, MQTT)
+tab: Main
+  Notes / News / Monitoring / Apps / Media / Games / UltraSeedbox
+tab: AI              (the inference fleet, sorted by role)
+  AI - Inference        LLM seats you call (gen, char-rp, char-rp-reasoning, summarizer)
+  AI - Eval & Retrieval judges, reward, rerank, embed, image-quality
+  AI - Gateways & Chat  routing gateway, control plane, chat frontends
+  AI - Speech (TTS)     text-to-speech engines
+  AI - Audio Tools      speech-to-text + audio dataset tooling
+  AI - Image & Media    image/video generation + pipelines
+  AI - Dormant          stopped stacks (rollback seats, retired auditions)
+tab: Infrastructure
+  Infra - ANA / NH3 / IRV / ESH   hardware + hypervisors + BMCs, per site
+tab: Toolchain
+  Service Networking / Toolchain  plumbing, rarely clicked
 ```
 
+The AI tab replaced the old single flat `AI Systems` group (2026-07-14): a
+20+ service list read as one endless column, so it was split by function.
+Group membership is the `homepage.group=AI - <role>` label on each compose
+file; a label change only takes effect when the container is recreated
+(`docker compose up -d <svc>`, or `up --no-start <svc>` to relabel a stopped
+stack without starting it).
+
 - **Manual entries** (this file) cover things without a Docker label:
   firewalls, switches, NAS web UIs, BMCs, hypervisors, and the cross-site
   hubs where direct IP:port URLs are stable.
@@ -49,15 +60,18 @@ Service Networking collapsed  toolchain (Traefik, CrowdSec, Dockge, AdGuard, MQT
 When deciding where a service lands, ask **function first**:
 
 1. Does it watch or back up the fleet? -> `Monitoring`
-2. Is it an inference / model service? -> `AI Systems`
+2. Is it an inference / model service? -> the matching `AI - <role>` group
+   (Inference / Eval & Retrieval / Gateways & Chat / Speech (TTS) /
+   Audio Tools / Image & Media); a stopped-but-kept stack -> `AI - Dormant`
 3. Is it a user-facing app? -> `Apps`
 4. Is it media / games? -> `Media` or `Games`
 5. Is it a piece of hardware or a hypervisor? -> `Infra - <site>`
 6. Is it toolchain / plumbing (no human interaction on the golden path)? ->
    `Service Networking`
 
-Site-specific sub-grouping is only used for `Infra -` because the device
-inventory maps cleanly to physical sites. App groups are function-only.
+Site-specific sub-grouping is used for `Infra -` (device inventory maps to
+physical sites) and role-based sub-grouping for `AI -` (the fleet is large
+enough to warrant it). Other app groups are function-only.
 
 ## Deploying changes
 
@@ -71,12 +85,21 @@ Current workflow — push this directory onto the host:
 ```bash
 rsync -av --delete \
   --exclude='.env' --exclude='.env.*' \
+  --exclude='*.bak*' --exclude='logs/' \
   configs/homepage/ esh-docker-vm:/opt/docker/conf/homepage/
 ```
 
 The real `.env` lives on `esh-docker-vm` next to the compose file and must
 not be overwritten (holds Plex/Jellyfin keys).
 
+> **`--delete` footgun (learned 2026-07-20):** the host keeps dated
+> `services.yaml.bak-*` safety copies and a live `logs/` dir that are *not*
+> in this repo. A bare `--delete` rsync wipes both. The `--exclude='*.bak*'`
+> and `--exclude='logs/'` above protect them. For a one-file tweak, skip
+> `--delete` entirely and push the single file:
+> `rsync -av configs/homepage/services.yaml esh-docker-vm:/opt/docker/conf/homepage/services.yaml`
+> (back up the host copy first: `ssh esh-docker-vm 'cp -a …/services.yaml …/services.yaml.bak-<date>-<what>'`).
+
 The homepage container reloads most files on-change; if a new group in
 `settings.yaml` doesn't show up, `docker compose restart` on the host.
 

configs/homepage/services.yaml

@@ -17,10 +17,22 @@
       siteMonitor: http://10.0.50.45:3001
       description: Uptime monitor (esh-docker-vm)
 
-# AI Systems group is fully Docker-auto-discovered (llama-swap, vLLM Embed,
-# vLLM Rerank — homepage.group=AI Systems on their compose files). Position
-# and row×3 style for the group live in settings.yaml. Do not add entries
-# here or they'll double up.
+- Apps:
+  # Manual entry — the Booth is a user-level systemd service on nh3-dev
+  # (not a Docker-labeled stack), so it can't auto-discover; list it here.
+  - The Booth:
+      href: http://10.100.10.50:8090/
+      icon: mdi-filmstrip
+      siteMonitor: http://10.100.10.50:8090/healthz
+      description: Ephemeral media drop + upload-for-pickup (human-readable ids) — nh3-dev, 24h TTL
+
+# The AI tab is fully Docker-auto-discovered. Each inference service carries
+# a homepage.group=AI - <role> label on its compose file (AI - Inference,
+# AI - Eval & Retrieval, AI - Gateways & Chat, AI - Speech (TTS),
+# AI - Audio Tools, AI - Image & Media). Tab assignment, group order, and
+# column counts live in settings.yaml. Do not add entries here or they'll
+# double up. To move a service between AI groups, change the label on its
+# compose file and recreate the container (labels only apply on recreate).
 
 - Media:
   - Plex:

configs/homepage/settings.yaml

@@ -27,11 +27,23 @@ statusStyle: ""
 # than plain link cards and the grid looks ragged.
 useEqualHeights: true
 
-# Function-first layout, three-tab split:
-#   Main           - daily-use apps, inference, media, bookmarks
+# Function-first layout, four-tab split:
+#   Main           - daily-use apps, media, bookmarks, monitoring
+#   AI             - the inference fleet, grouped by role (see below)
 #   Infrastructure - hardware, hypervisors, BMCs (per site)
 #   Toolchain      - backend services running but rarely clicked
 #
+# The AI tab splits the fleet by function so a 20+ service list reads as
+# sorted groups instead of one endless column. Group membership is set by
+# the homepage.group=AI - <role> label on each service's compose file:
+#   AI - Inference        LLM seats you call (gen, char-rp, char-rp-reasoning, summarizer)
+#   AI - Eval & Retrieval judges, reward, rerank, embed, image-quality
+#   AI - Gateways & Chat  routing gateway, control plane, chat frontends
+#   AI - Speech (TTS)     text-to-speech engines
+#   AI - Audio Tools      speech-to-text + audio dataset tooling
+#   AI - Image & Media    image/video generation + pipelines
+#   AI - Dormant          stopped stacks (rollback seats, retired auditions)
+#
 # Row counts target ~4-per-row so dense groups (Apps, Service Networking)
 # read as a grid instead of an endless column.
 layout:
@@ -50,11 +62,6 @@ layout:
     tab: Main
     style: row
     columns: 4
-  AI Systems:
-    icon: mdi-brain
-    tab: Main
-    style: row
-    columns: 4
   Apps:
     icon: mdi-apps
     tab: Main
@@ -73,6 +80,45 @@ layout:
     tab: Main
     style: row
     columns: 3
+  # --- AI tab: the inference fleet, ordered core-models -> support -> apps ---
+  AI - Inference:
+    icon: mdi-brain
+    tab: AI
+    style: row
+    columns: 4
+  AI - Eval & Retrieval:
+    icon: mdi-scale-balance
+    tab: AI
+    style: row
+    columns: 5
+  AI - Gateways & Chat:
+    icon: mdi-router-network
+    tab: AI
+    style: row
+    columns: 3
+  AI - Speech (TTS):
+    icon: mdi-account-voice
+    tab: AI
+    style: row
+    columns: 3
+  AI - Audio Tools:
+    icon: mdi-waveform
+    tab: AI
+    style: row
+    columns: 2
+  AI - Image & Media:
+    icon: mdi-image-multiple
+    tab: AI
+    style: row
+    columns: 2
+  # Stopped stacks kept for rollback / superseded seats / retired auditions.
+  # They stay 'created' (not running) via `docker compose up --no-start`, so
+  # they show here as offline cards and revive with `docker compose start`.
+  AI - Dormant:
+    icon: mdi-sleep
+    tab: AI
+    style: row
+    columns: 4
   Infra - ANA:
     icon: si-proxmox
     tab: Infrastructure

docs/arbo-comfyui-model-catalog.md

@@ -0,0 +1,45 @@
+# Arbo ComfyUI model catalog
+
+**Host:** irv-ml1 · **Path:** `/storetank/arbo/models` (SATA SSD; overlay-mounted into
+the arbo / comfyui container at `/basedir/models`). **502 G** as of 2026-06-13.
+
+The single live model tree arbo (hero / asset generation) consumes. On 2026-06-13 it
+**absorbed 177 G** of gen-agnostic utilities + the SDXL/Pony stack, migrated from the
+now-decommissioned `/storetank/image-models/comfy` archive — see
+[`storetank-image-models-archive.md`](storetank-image-models-archive.md) for that record.
+
+## Per-category sizes
+
+| Category | Size | Contents |
+|---|---|---|
+| `diffusion_models/` | **203 G** | current-gen generators: flux2-klein / wan2.2 / qwen-image / z-image / ideogram (GGUF + fp8) |
+| `checkpoints/` | **147 G** | SDXL / Pony / Illustrious bases — cyberrealisticPony_v180Coreshift (12.9 G), ponyRealism V22, novaAnimeXL, juggernaut/dreamshaper Lightning, lustify, hassaku, waiNSFWIllustrious, realDream + SUPIR upscalers |
+| `text_encoders/` | **80 G** | qwen3-VL, qwen2.5-VL, gemma, umt5, t5-xxl, clip variants |
+| `loras/` | **16 G** | flux2/wan2.2 (gameart, RetroAnimeFlux, flux1_turbo, zit_*) + migrated SDXL/Pony (dmd2_sdxl_4step, ACE++, character-design) |
+| `vae/` | 9.6 G | wan2.2 / flux2 / flux1 / z-image / sdxl / wan2.1 VAEs |
+| `Aura-SR/` | 9.3 G | AuraSR v1/v2 upscalers |
+| `LLM/` + `florence2/` | 8.6 + 3.6 G | Florence-2 PromptGen large/base + CogFlorence captioners |
+| `controlnet/` | 8.1 G | flux upscaler + sdxl union-promax |
+| `clip_vision/` | 4.4 G | CLIP-ViT-H, clip_vision_h, sigclip |
+| `upscale_models/` | 3.8 G | HAT / DAT / RealESRGAN / UltraSharp / Remacri / NMKD / Omni-SR (~50) |
+| `grounding-dino/` | 1.6 G | grounding-dino swinb / swint |
+| `ipadapter/` | 1.5 G | ip-adapter-plus / _sdxl vit-h |
+| `insightface/` | 1.3 G | inswapper_128 + antelopev2 |
+| `depthanything/` | 1.3 G | depth-anything v2 (vitl / vits) |
+| `facerestore_models/` | 937 M | GFPGAN v1.3/1.4, GPEN-BFR |
+| `RMBG/` `clip/` `sams/` `nsfw_detector/` `vitmatte/` `facexlib/` `ultralytics/` … | <1 G ea | bg-removal, EVA02-CLIP-L, SAM-HQ + SAM, nsfw classifier, matte, face-lib, yolo (face/hand/eyes/person) |
+
+## Migrated in 2026-06-13 (177 G from the storetank archive)
+
+The gen-agnostic utility set (upscalers, Florence-2 captioners, controlnet-union,
+grounding-dino / SAM / yolo / depthanything / vitmatte, insightface / facerestore,
+ip-adapter, CLIP-vision) **plus** the SDXL/Pony stack (bases + dmd2 / ACE++ /
+character-design loras). These work alongside arbo's current FLUX.2 / WAN2.2 / qwen
+generators; **comfy-dev** authors the per-model catalog entries + graphs + heroes that
+turn them into usable workflows.
+
+## Durability
+
+- `arbo_db` (gallery/history SQLite) — backed up (restic/Backrest), local disk not NFS.
+- The model tree itself is **bulk, reproducible-from-source** → not backed up; this
+  catalog + the migration record are the recovery map.

docs/asset-engine/services.yaml

@@ -565,6 +565,144 @@ services:
       Three-way mutual-exclusion among emotion_voice / emotion_vector / emotion_text;
       precedence as above. UI should expose this as a single picker.
 
+  - id: omnivoice
+    name: OmniVoice
+    description: >
+      k2-fsa zero-shot, massively-multilingual (600+ language) voice-cloning TTS
+      (diffusion-LM, RTF ~0.025). Apache-2.0. Behind our own FastAPI wrapper
+      (stacks/omnivoice/app.py); voices are the reused chatterbox reference clips.
+    category: tts
+    version: 2
+    status: ready
+    host: irv-ml1
+    lifecycle:
+      stack: omnivoice
+      vram_gb: 6
+      gpu_device_id: 0
+    endpoint: http://10.100.79.3:8199/v1/audio/speech
+    method: POST
+    content_type: application/json
+    model:
+      id: k2-fsa/OmniVoice
+      revision: null
+      image: local/omnivoice:latest
+    fields:
+      - name: input
+        type: textarea
+        label: Text
+        required: true
+        max_length: 5000
+      # Voice source — at least one of voice (clone) / instruct (design) is required.
+      - name: voice
+        type: select
+        label: Speaker Voice (clone)
+        optional: true
+        source_url: http://10.100.79.3:8199/v1/audio/voices
+        source_jsonpath: $.voices[*]
+        description: >
+          Zero-shot clone target — a reference clip in /worktank/omnivoice/voices/
+          (reused chatterbox voices; 33 at deploy). Omit to design a voice via
+          instruct instead. Live list at /v1/audio/voices.
+      - name: instruct
+        type: text
+        label: Voice Design (instruct)
+        optional: true
+        source_url: http://10.100.79.3:8199/v1/audio/instruct-items
+        source_jsonpath: $.instruct_items[*]
+        description: >
+          Voice DESIGN — a comma-separated list of CONTROLLED attribute tags (not
+          free prose), e.g. "british accent, elderly, male, low pitch". Valid tags
+          (gender/age/pitch/accent/whisper) at /v1/audio/instruct-items. Use instead
+          of, or together with, a clone voice.
+      - name: language
+        type: select
+        label: Language
+        optional: true
+        default: Auto
+        source_url: http://10.100.79.3:8199/v1/audio/languages
+        source_jsonpath: $.languages[*]
+        description: "Auto-detects when left as Auto; 600+ languages supported."
+      - name: speed
+        type: slider
+        label: Speed
+        optional: true
+        min: 0.5
+        max: 1.5
+        default: 1.0
+        description: "1.0 = normal; >1 faster, <1 slower. Ignored if duration is set."
+      - name: duration
+        type: number
+        label: Duration (seconds)
+        optional: true
+        description: "Fixed output length in seconds; overrides speed when set."
+      - name: num_step
+        type: slider
+        label: Inference Steps
+        optional: true
+        min: 4
+        max: 64
+        default: 32
+        description: "Diffusion steps. Lower = faster, higher = better quality."
+      - name: guidance_scale
+        type: slider
+        label: Guidance Scale (CFG)
+        optional: true
+        min: 0.0
+        max: 4.0
+        default: 2.0
+      - name: denoise
+        type: bool
+        label: Denoise
+        optional: true
+        default: true
+      - name: preprocess_prompt
+        type: bool
+        label: Preprocess Prompt
+        optional: true
+        default: true
+        description: "Silence-trim + punctuate the reference (clone mode)."
+      - name: postprocess_output
+        type: bool
+        label: Postprocess Output
+        optional: true
+        default: true
+        description: "Remove long silences from the generated audio."
+      - name: generation_overrides
+        type: json
+        label: Advanced (GenerationConfig)
+        optional: true
+        description: >
+          Expert OmniVoiceGenerationConfig overrides as a JSON object — keys:
+          t_shift (0.1), layer_penalty_factor (5.0), position_temperature (5.0),
+          class_temperature (0.0), audio_chunk_duration (15.0),
+          audio_chunk_threshold (30.0). Unknown keys ignored.
+      - name: response_format
+        type: select
+        options: [wav]
+        default: wav
+        description: 24000 Hz PCM_16 mono only; no negotiation.
+    response:
+      type: audio
+      mime: audio/wav
+    reproducibility:
+      seedable: false
+      deterministic: false
+      notes: >
+        Diffusion-LM, temperature/denoise sampled — not byte-exact, no seed exposed.
+        Output 24000 Hz PCM_16 mono. Voice = a cloned reference clip (clone prompt
+        precomputed per voice at startup; Whisper auto-transcribes the reference).
+    estimated_latency:
+      cold_start_s: 600
+      warm_per_unit: "full-utterance (no streaming)"
+    license: "Apache-2.0"
+    notes: |
+      Two voice sources, combinable: voice (clone a staged reference clip) and/or
+      instruct (free-text voice DESIGN); at least one required. Full generation
+      surface exposed — language (600+), speed, duration, num_step, guidance_scale,
+      denoise, preprocess/postprocess — with expert GenerationConfig knobs (t_shift,
+      layer/position/class temperature, audio_chunk_*) via the generation_overrides
+      JSON field. No streaming. Voices reused from chatterbox /refs.
+
   - id: qwen3-tts
     name: Qwen3-TTS 1.7B
     description: >
@@ -2174,6 +2312,289 @@ services:
       source for defaults/ranges. Adapter not yet deployed/verified — flip to
       ready (or experimental) after the first successful generation through 8203.
 
+  - id: zonos-gateway
+    name: Zonos Gateway (expressive)
+    description: >
+      OpenAI-compatible streaming facade over the Zonos engine (kept stock),
+      exposing Zonos's full expressive control surface: emotion directions
+      (happy / sad / angry / surprised) plus a valence/arousal axis pair,
+      classifier-free-guidance on emotion, accurate-vs-expressive mode,
+      speaking-rate conditioning, quality-metric targets, and the full
+      sampling stack — all reachable from named presets (neutral / warm /
+      excited / sad / intense / whisper) that seed the dials before explicit
+      overrides win. Streams s16le PCM (or a WAV wrapper) from
+      /v1/audio/speech. The LiteLLM `ext-tts` alias points at this gateway.
+    category: tts
+    version: 1
+    status: experimental
+    host: irv-ml1
+    lifecycle:
+      stack: zonos-gateway
+      vram_gb: 16
+      gpu_device_id: 0
+    endpoint: http://10.100.79.3:8890/v1/audio/speech
+    method: POST
+    content_type: application/json
+    streamable: true
+    model:
+      id: Zyphra/ZONOS2
+      revision: null
+      image: local/zonos-gateway:0.1.0
+    section_groups:
+      - id: basic
+        label: Text & voice
+      - id: expression
+        label: Expression
+        hint: Emotion conditioning. A preset seeds these; explicit dials win.
+      - id: prosody
+        label: Prosody
+        hint: Speaking-rate conditioning. Leave the enable toggles off for the model's native pacing.
+      - id: quality
+        label: Quality target
+        hint: Advanced — raw metric targets (LUFS, silence, bandlimit) Zonos buckets internally.
+      - id: sampling
+        label: Sampling
+      - id: output
+        label: Output
+    fields:
+      - name: input
+        type: textarea
+        label: Text to synthesize
+        section: basic
+        required: true
+        max_length: 5000
+        description: >
+          Text to speak. OpenAI-style `input` field; the gateway streams the
+          synthesized audio back.
+      - name: voice
+        type: select
+        label: Voice
+        section: basic
+        default: Cora
+        source_url: http://10.100.79.3:8890/v1/voices
+        source_jsonpath: $.voices[*].name
+        description: >
+          Predefined Zonos voice. Live-enumerated from /v1/voices so the list
+          auto-syncs with the deployed voice pack (Cora is the default).
+      - name: preset
+        type: select
+        label: Expressive preset
+        section: expression
+        required: false
+        options: [neutral, warm, excited, sad, intense, whisper]
+        default: neutral
+        description: >
+          Named expressive preset applied before explicit dials; any explicit
+          emotion/prosody/quality dial you set overrides the preset's value.
+      - name: emotion_enabled
+        type: bool
+        label: Enable emotion conditioning
+        section: expression
+        required: false
+        default: false
+        description: >
+          Turn emotion conditioning on. Required for the emotion_* dials to
+          bite — a preset that sets emotion turns this on for you.
+      - name: emotion_valence
+        type: slider
+        label: Valence
+        section: expression
+        min: -1.0
+        max: 1.0
+        step: 0.05
+        default: 0.0
+        description: Pleasantness axis. -1 negative, +1 positive.
+      - name: emotion_arousal
+        type: slider
+        label: Arousal
+        section: expression
+        min: -1.0
+        max: 1.0
+        step: 0.05
+        default: 0.0
+        description: Energy/activation axis. -1 calm, +1 excited.
+      - name: emotion_strength
+        type: slider
+        label: Emotion strength
+        section: expression
+        min: 0.0
+        max: 2.0
+        step: 0.05
+        default: 1.0
+        description: Overall scale on the emotion direction. 1.0 = as specified.
+      - name: emotion_cfg_scale
+        type: slider
+        label: Emotion CFG scale
+        section: expression
+        min: 1.0
+        max: 3.0
+        step: 0.1
+        default: 1.0
+        description: >
+          Classifier-free-guidance on emotion. 1.0 = off; >1 amplifies
+          expression.
+      - name: emotion_sliders
+        type: json
+        label: Per-emotion weights (advanced)
+        section: expression
+        optional: true
+        description: >
+          Advanced — per-emotion weight dict {happy|sad|angry|surprised: -1..1};
+          higher = stronger. Overrides the coarse valence/arousal directions
+          with explicit per-emotion control. Omit to use valence/arousal.
+      - name: accurate_mode
+        type: bool
+        label: Accurate mode
+        section: expression
+        required: false
+        default: true
+        description: >
+          true = faithful to the reference voice; false = more
+          expressive/looser.
+      - name: speaking_rate_enabled
+        type: bool
+        label: Enable speaking-rate conditioning
+        section: prosody
+        required: false
+        default: false
+        description: >
+          Turn speaking-rate conditioning on. Required for speed /
+          speaking_rate / speaking_rate_bucket to take effect.
+      - name: speed
+        type: slider
+        label: Speed (OpenAI-style)
+        section: prosody
+        min: 0.25
+        max: 4.0
+        step: 0.05
+        optional: true
+        description: >
+          OpenAI-style rate multiplier. Mapped to speaking_rate when no
+          explicit speaking_rate is given; auto-enables speaking-rate
+          conditioning. Omit to leave pacing native.
+      - name: speaking_rate
+        type: slider
+        label: Speaking rate (native)
+        section: prosody
+        min: 0.25
+        max: 4.0
+        step: 0.05
+        optional: true
+        description: >
+          Native speaking-rate multiplier. Overrides speed if both are sent.
+          Omit to leave pacing native.
+      - name: speaking_rate_bucket
+        type: slider
+        label: Speaking-rate bucket
+        section: prosody
+        min: 0
+        max: 7
+        step: 1
+        optional: true
+        description: >
+          Words/sec bucket index 0..7 (0 = 0-8 wps … 7 = 40+ wps). Coarser than
+          speaking_rate. Omit to leave pacing native.
+      - name: quality_enabled
+        type: bool
+        label: Enable quality-target conditioning
+        section: quality
+        required: false
+        default: true
+        description: >
+          Advanced — turn quality-target conditioning on (on by default in
+          Zonos). Gates quality_values.
+      - name: quality_values
+        type: json
+        label: Quality metric targets (advanced)
+        section: quality
+        optional: true
+        description: >
+          Advanced — raw metric targets Zonos buckets internally, e.g.
+          {lufs: -23, trailing_silence_s: 0.1}. Keys: lufs, estimated_snr,
+          max_pause, estimated_bandlimit_hz, leading_silence_s,
+          trailing_silence_s. Omit for Zonos's defaults.
+      - name: temperature
+        type: slider
+        section: sampling
+        min: 0.0
+        max: 2.0
+        step: 0.05
+        default: 1.15
+        description: Sampling temperature. Higher = more varied. Zonos default 1.15.
+      - name: top_p
+        type: slider
+        label: Top-p
+        section: sampling
+        min: 0.0
+        max: 1.0
+        step: 0.05
+        default: 0.0
+        description: Nucleus sampling cutoff. 0.0 = off (Zonos default).
+      - name: min_p
+        type: slider
+        label: Min-p
+        section: sampling
+        min: 0.0
+        max: 1.0
+        step: 0.01
+        default: 0.18
+        description: Min-p sampling floor. Zonos default 0.18.
+      - name: topk
+        type: number
+        label: Top-k
+        section: sampling
+        required: false
+        default: 106
+        description: Top-k sampling cutoff. Zonos default 106.
+      - name: seed
+        type: number
+        section: sampling
+        optional: true
+        description: >
+          RNG seed for reproducible sampling. Omit for a random seed. Pins the
+          sampler only; emotion/quality conditioning still varies subtly.
+      - name: max_tokens
+        type: number
+        label: Max audio tokens
+        section: sampling
+        required: false
+        max: 6144
+        description: >
+          Cap on generated audio tokens (upper bound; Zonos stops at
+          end-of-speech). Omit to let Zonos decide.
+      - name: response_format
+        type: select
+        label: Response format
+        section: output
+        options: [pcm, wav]
+        default: pcm
+        description: >
+          pcm = raw s16le stream (lowest latency, for API consumers); wav adds
+          a header. The stream-audition UI forces wav for the browser <audio>.
+    response:
+      type: audio
+      mime_from_field: response_format
+    reproducibility:
+      seedable: true
+      deterministic: false
+      seed_field: seed
+      notes: >
+        Temperature-sampled; seed pins the sampler but emotion/quality
+        conditioning still varies subtly run-to-run.
+    estimated_latency:
+      cold_start_s: 3
+      warm_per_unit: "streaming; first audio in a couple seconds warm, then near-realtime on the 3090"
+    license: Apache-2.0
+    notes: |
+      OpenAI-compatible streaming gateway (local/zonos-gateway:0.1.0) fronting a
+      stock Zonos engine on the 3090 (irv-ml1 device 0). The LiteLLM `ext-tts`
+      alias routes here. Fields mirror the gateway's /v1/dials schema (24 params;
+      the CATALOG-CONTRACT blessed source for defaults/ranges) and /v1/voices.
+      Deliberately omits repetition_window / repetition_penalty / codebooks — the
+      wrapper rejects them and they are the "70s of silence" footgun. Presets seed
+      the dials before explicit overrides win. New service (experimental) — flip to
+      ready after the first verified generation + browser audition through 8890.
+
 # Reproducibility audit — answers per service: (a) seedable, (b) model
 # deterministic without seed, (c) image tag mutable (security/reproducibility risk).
 reproducibility_audit:
@@ -2207,6 +2628,11 @@ reproducibility_audit:
     model_deterministic: true
     image_tag_mutable: false
     notes: "22050 Hz hardcoded — caller must resample."
+  - service: omnivoice
+    seedable: false
+    model_deterministic: false
+    image_tag_mutable: true
+    notes: "Diffusion-LM, temperature/denoise sampled — not byte-exact, no seed exposed. 24000 Hz PCM_16 mono. image local/omnivoice:latest is mutable — pin a digest for true repro. Voices = reused chatterbox /refs clones (clone prompt precomputed per voice at startup)."
   - service: qwen3-tts
     seedable: false
     model_deterministic: true
@@ -2269,3 +2695,8 @@ reproducibility_audit:
     model_deterministic: true
     image_tag_mutable: false
     notes: "Adapter echoes the seed used (reproducibility.seed_field=seed). Byte-stable same-GPU; bf16 may drift cross-GPU. local/zonos-api:v1 built FROM local/zonos (pin ZONOS_SHA for true repro)."
+  - service: zonos-gateway
+    seedable: true
+    model_deterministic: false
+    image_tag_mutable: true
+    notes: "Seed pins the sampler (reproducibility.seed_field=seed) but emotion/quality conditioning still varies subtly run-to-run — not byte-exact. Streaming (s16le PCM / WAV). Distinct from the `zonos` adapter: this is the OpenAI-compatible gateway on :8890 behind the LiteLLM `ext-tts` alias. image local/zonos-gateway:0.1.0 is tag-pinned + mutable — pin a digest for true repro."

docs/orientation.md

@@ -164,6 +164,25 @@ These caught us once; don't let them catch you twice.
 - **irv-ml1 was `ana-ml1`** before a physical move; OS hostname still
   says `ana-ml1` pending an explicit rename. Doesn't affect services.
 
+### Git / gitea
+
+- **Colo/fleet hosts must reach gitea over the INTERNAL route, not the
+  public IP.** `gitea.phasefinal.com` resolves to the **public** IP
+  `38.120.12.44` (ana-srv1); gitea itself is a container on **ana-docker**
+  with git-SSH at **`10.250.50.70:222`** (`222→22`) and HTTP at `:3000`.
+  A fleet host that egresses to the public `:22` gets its egress IP
+  **fail2ban-banned** after any retrying git/deploy loop, which silently
+  wedges automation — e.g. a gitea-webhook auto-deploy whose `git fetch`
+  then times out under `set -euo pipefail` and never reaches the `reset`.
+  Point each host's gitea ssh alias at `HostName 10.250.50.70` /
+  `Port 222` with the repo deploy key; the internal route is ban-immune
+  and treats the cause. Bit irv-ml1's arbo deploy on 2026-06-13 (the
+  `gitea-arbo` alias pointed at the public host → fetch timeout → the
+  v0.11.7 frontend wouldn't serve until the alias was repointed internal).
+- **`:22` on `10.250.50.70` is ana-docker's HOST sshd, not gitea.** A
+  gitea deploy key there returns `Permission denied (publickey)` — gitea's
+  git-SSH is the container port `:222`. (HTTP/clone-over-HTTPS is `:3000`.)
+
 ### Workflow
 
 - **Terminal word-wrap breaks long pasted commands.** Never embed a

docs/pfi/model-sampler-defaults.md

@@ -0,0 +1,76 @@
+# Canonical sampler defaults — PFI/VastBlue LiteLLM gateway seats
+
+**Applied:** 2026-07-08 · **Gateway:** `ana-docker:4000` · **Config:** `stacks/litellm/conf/config.yaml` → `/opt/docker/conf/litellm/config.yaml`
+
+Canonical high-quality sampler defaults for the four model seats, **derived by
+dvalin-smithy-dev** (full rationale + sources: `dvalin-smithy/hoard-drafts/pfi-gateway-sampler-defaults-20260708.md`),
+**triaged + A/B-validated by infra-ops**, and wired into the gateway. These are the
+gateway *defaults*; callers may override per request.
+
+Optimized for **output / prose quality** (not throughput or determinism).
+
+## Engine surfaces
+
+- **gen / gen-reasoning** — vLLM 0.24 (OpenAI sampler surface). No native DRY/XTC → anti-repetition via `presence_penalty`. Thinking split via `chat_template_kwargs.enable_thinking` on distinct `--served-model-name`s (avoids the shared-config-mutation footgun).
+- **char-rp / char-rp-reasoning** — llama.cpp / llama-server (supports `min_p`, `top_k`, DRY, XTC, dynatemp). `min_p` + `top_p` do the tail work; `top_k 0` disables top-k.
+
+## The four seats (applied values)
+
+### 1. gen — Qwen3.6-35B-A3B heretic (vLLM, non-thinking)
+Also governs **summarizer-large** (shares the same `qwen3.6-27b-aeon` @ :8015 deployment → kept identical).
+
+| param | value |
+|---|---|
+| temperature | 0.7 |
+| top_p | 0.80 |
+| top_k | 20 |
+| presence_penalty | **1.5** |
+| repetition/frequency | 1.0 / 0.0 |
+| enable_thinking | false |
+
+*Source:* Qwen3.6 README instruct/non-thinking rec. *Change:* presence_penalty 1.0 → 1.5.
+
+### 2. gen-reasoning — same model (vLLM, thinking)
+
+| param | value |
+|---|---|
+| temperature | **1.0** |
+| top_p | 0.95 |
+| top_k | 20 |
+| presence_penalty | **1.5** |
+| repetition/frequency | 1.0 / 0.0 |
+| enable_thinking | true |
+
+*Source:* Qwen3.6 README **general** thinking profile (NOT the temp-0.6 coding sub-profile — the prior default was that coding profile by mistake). *Changes:* temperature 0.6 → 1.0, presence_penalty 1.0 → 1.5. Reasoning is verbose (~9k chars) → callers set generous `max_tokens` (catalog default 32768). Optional per-route coding override: temp 0.6 / presence 0.0.
+
+### 3. char-rp — Magidonia-24B-v4.3 (llama.cpp, non-thinking prose RP)
+
+| param | value |
+|---|---|
+| temperature | **1.1** |
+| top_p | 0.95 |
+| min_p | **0.10** |
+| top_k | 0 (disabled) |
+| repetition/DRY/XTC | **off** |
+
+*Source:* dvalin canonical (Mistral-Small RP prose) **A/B-validated by infra-ops** on the live serve. *Changes:* temp 1.0 → 1.1, min_p 0.03 → 0.10. **min_p 0.10 richened imagery vs 0.03** with no incoherence at temp 1.1. **repeat_penalty 1.05 was REJECTED** — in the A/B it injected a stray markdown title into a grief scene; rep-style penalties hurt Drummer/Magistral RP creativity (matches the model card and dvalin's own note). Alt prose model: `MS3.2-PaintedFantasy-v4.1-24B` (swap via the `char-rp-gguf` stack `.env`).
+
+### 4. char-rp-reasoning — Qwen3.5-27B-Deckard-PKD (llama.cpp, managed-reasoning RP)
+
+| param | value (request-level) |
+|---|---|
+| temperature | 1.0 |
+| top_p | 0.95 |
+| top_k | 40 |
+| min_p | **0.05** |
+| presence/repetition | **off** |
+| DRY | **0.8 server-side** (base 1.75 / len 2, dry-after-temp) — not a request param |
+| reasoning-budget | 400 (server-side) |
+
+*Source:* dvalin-CONFIRMED canonical 2026-07-08 (thread 01KX1Y7P). **Corrected 2026-07-09:** this seat had lagged on QwQ-RpR-v4 — the A/B on 2026-07-08 replaced it with **Deckard-PKD-Heretic i1-Q5_K_M** (DavidAU, Qwen3.5-27B, :8018); the live gateway was always Deckard. Deckard won on brokkr's frozen scorer (0/30 loops, 0/30 refusals) over RpR-v4 (1/30 loop, forbids DRY) + Pantheon-27B (7/30 refusals). Reasoning ON server-side (`--reasoning on`, budget 400); CoT surfaces in `reasoning_content`, clean prose in `content`. Tuning ladder: flat prose→min_p 0.08, loops→DRY 0.9, over-damped→DRY 0.6/off. **Do NOT import RpR/QwQ sampler rules** (different family — QwQ hated DRY; Qwen3.5 benefits from it).
+
+## Changing a default
+
+Edit the seat's `litellm_params` in `stacks/litellm/conf/config.yaml`, `scp` to
+`/opt/docker/conf/litellm/config.yaml` on ana-docker, `docker restart litellm`.
+(`gen` and `summarizer-large` must change together — same deployment.)

docs/pfi/recommended-model-settings.md

@@ -16,6 +16,7 @@
 6. [Quick Reference Cards](#6-quick-reference-cards)
 7. [Critical Warnings by Model](#7-critical-warnings-by-model)
 8. [Models Without KB Settings](#8-models-without-kb-settings)
+9. [PFI LiteLLM Gateway — Deployed Sampling Defaults](#9-pfi-litellm-gateway--deployed-sampling-defaults)
 
 ---
 
@@ -539,6 +540,51 @@ The following model families are deployed in the Infrastructure-PFI environment
 
 ---
 
+## 9. PFI LiteLLM Gateway — Deployed Sampling Defaults
+
+> **Live as of 2026-06-27** on the PFI gateway (`ana-docker:4000`; canonical config
+> `eshpfi-management/stacks/litellm/conf/config.yaml`). Unlike §§1–8 (general vendor
+> reference), this section is the **deployed reality** — keep it in sync when gateway
+> sampling changes.
+
+These are **overrideable defaults**: any caller that passes its own sampling param
+wins; callers that omit one inherit the value below. (Verified — vLLM rejected an
+out-of-range `presence_penalty=5.0`, proving per-request values reach the backend and
+override the config default.) Values set per the `dvalin-smithy-dev` research pass
+(provenance-cited in-thread, corroborated by §3 above). vLLM-only params (`top_k`,
+`repetition_penalty`) ride in `extra_body` so LiteLLM's `drop_params` can't strip them.
+
+| Gateway model(s) | temp | top_p | top_k | presence_penalty | repetition_penalty | Source |
+|---|---|---|---|---|---|---|
+| `granite-4.1-8b`, `summarizer`, `classifier` | **0** | — | — | — | — | IBM-canonical (temp 0 for inferencing) |
+| `gen`, `summarizer-large`, `qwen-large`, `qwen3.5-122-a10b` (non-thinking) | **0.7** | 0.8 | 20 | **1.0** | — | Qwen3 non-thinking + operator anti-repetition |
+| `gen-reasoning`, `qwen-large-reasoning`, `qwen3.5-122-a10b-reasoning` (thinking) | **0.6** | 0.95 | 20 | **1.0** | — | Qwen3 thinking |
+| `qwen-image-bench`, `image-judge` | **0** | 1.0 | 1 | — | 1.05 | Qwen-Image-Bench judge reproducibility table |
+| `selene-1-mini-8b`, `chat-judge` | **0.6** | 0.9 | — | — | — | Selene `generation_config` |
+| `glm-5.1`, `glm-5.2`, `glm-5-turbo`, `glm-4.7`, `gen-frontier` | **1.0** | 0.95 | — | — | — | z.ai API defaults (5.x / 4.7 series) |
+| `glm-4.5-air` | **0.6** | 0.95 | — | — | — | z.ai API default (4.5 series) |
+| `qwen3-embedding`, `qwen3-reranker`, `reranker` | — | — | — | — | — | no sampling (embedding / rerank) |
+
+**Notes:**
+- **qwen "gen" family `presence_penalty: 1.0`** — operator-set anti-repetition for the
+  abliterated/NVFP4 Qwopus 122B-A10B. Qwen documents `presence_penalty` (0–2) as *the*
+  repetition lever; 1.0 is conservative (the §3 vendor general value is 1.5 — step up to
+  1.5 if loops persist). Do **not** use `repetition_penalty` for the Qwen3 family.
+- **GLM (z.ai cloud) — only `temperature` + `top_p` are set.** z.ai's chat API schema
+  accepts no `top_k` / `min_p` / penalties, so they're deliberately not sent (would be
+  silently dropped). These temps match z.ai's own API defaults (explicit-over-implicit /
+  future-proofing).
+- **Both `temp 0` values (granite, image-judge) are research-confirmed, not heuristic.**
+  Greedy is correct for constrained summ/classify (IBM) and for judge reproducibility
+  (Qwen judge card + LLM-as-judge practice). `temp 0.1` was explicitly evaluated and
+  rejected: it adds sampling noise without fixing loops, and *reduces* run-to-run score
+  consistency on the judge. If granite ever loops in production, fix via
+  `repetition_penalty` / `presence_penalty` / `max_tokens`, not a temperature floor.
+- **`qwen-image-bench` / `image-judge` is arbo's hero-judge** (comfy-dev consumer) —
+  sampling changes there are a coordination item, not a unilateral gateway edit.
+
+---
+
 ## KB Source Documents
 
 | Document | Path in KB |

docs/runbooks/backups.md

@@ -0,0 +1,178 @@
+# Fleet backup architecture & freshness runbook
+
+The map that was missing: what backs up what, where it lands, and **how
+to check in 2 minutes whether backups are actually fresh.** Companion to
+[`disaster-recovery.md`](disaster-recovery.md) (which covers *recovery*
+when a host/service is down). Read this one first when the question is
+"are we backed up?"
+
+> **Why this exists:** on 2026-06-20 diagnosing "are backups OK?" took a
+> long exploration because the topology lived only in scattered memory.
+> The ana-side restic layer had been failing **silently for ~6.5 weeks**
+> (last good snapshot 2026-05-06) and nobody knew. This doc + a future
+> freshness alert is the fix.
+
+---
+
+## TL;DR — coverage matrix
+
+Two independent layers. **PBS = whole-VM images. restic = granular
+file+DB.** A host is well-covered if it has *either* a current PBS image
+*or* a current restic snapshot; the danger zone is a host whose **only**
+layer has failed.
+
+| Host | Kind | PBS (VM image) | restic (file+DB) | Sole net? |
+|---|---|---|---|---|
+| ana-docker | VM (pfi-pve) | ✅ `ana-pve` | ✅ → rest-server-**ana** | no |
+| **ana-ml2** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**ana** | ⚠️ **restic is the ONLY net** |
+| **irv-ml1** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**nh3** | ⚠️ **restic is the ONLY net** |
+| nh3-docker | VM (nh3-pve) | ✅ `nh3-pve` | ✅ → rest-server-**nh3** | no |
+| esh-docker-vm | VM (esh-pve) | ✅ `esh-pve` | ✅ → rest-server-**ana** | no |
+| esh-vm-db | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only (a DB!)** |
+| vm-esh-nas | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only** |
+| other pfi-pve / nh3-pve VMs/CTs | VM/CT | ✅ respective ns | (PBS only) | no |
+| SureFire `sfsrv-pve` | tenant VMs | ✅ `sfsrv-pve` ns | (PBS only) | no |
+
+**Bare-metal hosts have NO PBS coverage** (PBS only backs up Proxmox
+guests). Their restic snapshot is the entire safety net — keep an eye on
+it. ana-ml2 → rest-server-ana; irv-ml1 → rest-server-nh3.
+
+---
+
+## Layer 1 — PBS (whole-VM/CT images)
+
+- **PBS-ANA** (`pbs-ana`, 10.250.50.90) — fleet primary. Datastore is
+  NFS-backed: `10.250.50.50:/mnt/backup/pbs-ana` mounted at
+  `/mnt/pbs-datastore` (~20 TB). Backs up Proxmox guests via vzdump,
+  organised by **namespace per source hypervisor**:
+  - `ana-pve` — pfi-pve guests (ana-docker, pfi-postgres VM105, ana-nas
+    CT109, webhost, filebot, pteradactyl, tacticalrmm, ana-wg, …)
+  - `esh-pve` — esh-pve guests
+  - `nh3-pve` — nh3-pve guests
+  - `sfsrv-pve` — SureFire tenant
+  - ⚠️ there is **no `esh-pve-nas` namespace** — guests on that
+    hypervisor (vm-esh-nas, likely esh-vm-db) are **not** PBS-covered.
+- **PBS-NH3** (`pbs-nh3`, 10.100.50.90) — DR mirror; syncs from PBS-ANA
+  (datastore on nh3-nas).
+- Schedule: vzdump jobs defined in Proxmox (Datacenter → Backup),
+  staggered through the early morning.
+
+## Layer 2 — restic (granular file + DB)
+
+restic clients push to one of **two rest-server endpoints** (HTTP, basic
+auth, append-only, private repos). The split is by site:
+
+| rest-server | Endpoint | Backing store | Clients |
+|---|---|---|---|
+| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-vm-db, vm-esh-nas |
+| **rest-server-nh3** | `http://10.100.50.50:8000` (on nh3-nas) | `nh3-nas:/volume1/Backup/restic/<client>` | **irv-ml1**, nh3-docker |
+
+- Per-client repos live as subdirs of the rest-server data dir
+  (`.../repo/ana/<client>/` for the ana side); the shared `.htpasswd`
+  for ana sits at `.../repo/ana/.htpasswd`.
+- **Scheduler = `resticprofile` systemd timers on each client host**, NOT
+  Backrest:
+  - `resticprofile-backup@profile-default.timer` — daily **01:00** PDT
+  - `resticprofile-check@profile-default.timer` — weekly (Sun **05:00**)
+- **Backrest** (container on ana-docker, UI) is only a **repo viewer here
+  — it has 0 plans.** Do not assume "Backrest healthy" means "backups
+  running." The timers are the source of truth.
+- ⚠️ **Failures are silent** — a timer fires, restic errors against a
+  down endpoint, and nothing alerts. (See Known gaps.)
+
+---
+
+## The 2-minute freshness check
+
+Run these any time you need to answer "are we backed up?"
+
+```bash
+# --- restic ANA side: newest snapshot per client (want: today/yesterday) ---
+ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do
+  echo -n "$c: "; ls -t /mnt/backup/restic/repo/ana/$c/snapshots/ 2>/dev/null | head -1 \
+    | xargs -I{} stat -c "%y" /mnt/backup/restic/repo/ana/$c/snapshots/{} 2>/dev/null || echo MISSING
+done'
+
+# --- restic NH3 side ---
+ssh nh3-nas 'for c in irv-ml1 nh3-docker; do
+  echo -n "$c: "; ls -lt /volume1/Backup/restic/$c/snapshots/ 2>/dev/null | sed -n 2p
+done'
+
+# --- rest-server endpoints healthy? (401 = up & serving; Restarting = broken) ---
+ssh infra-ops@ana-docker 'sudo docker ps --format "{{.Names}}\t{{.Status}}" | grep rest-server'
+curl -s -o /dev/null -w 'rest-server-ana: %{http_code}\n' http://10.250.50.70:8000/
+curl -s -o /dev/null -w 'rest-server-nh3: %{http_code}\n' http://10.100.50.50:8000/
+
+# --- PBS: newest snapshot per guest, all namespaces ---
+ssh pbs-ana 'for ns in /mnt/pbs-datastore/ns/*/; do nsn=$(basename "$ns")
+  for d in vm ct; do for g in "$ns$d"/*/; do [ -d "$g" ] || continue
+    echo "$nsn/$d/$(basename "$g") -> $(ls "$g" 2>/dev/null | grep ^20 | sort | tail -1)"
+  done; done; done'
+```
+
+**Force a backup now (don't wait for 01:00):** on the client host,
+`ssh infra-ops@<host> 'sudo systemctl start resticprofile-backup@profile-default.service'`
+(it's an incremental against the existing repo — bounded even if stale).
+
+---
+
+## Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage)
+
+**Symptom:** `rest-server` container on ana-docker stuck `Restarting`;
+logs show `cannot load /data/.htpasswd: permission denied`. All ana-side
+restic backups silently fail.
+
+**Root cause:** ana-nas's NFS mount on ana-docker uses bare `defaults` in
+`/etc/fstab` (no `_netdev`, no retry). When the mount drops,
+`mnt-backup.mount` gets stuck `failed`, so `/mnt/backup/restic/repo/ana`
+resolves to an **empty local ghost dir** (no `.htpasswd`) and rest-server
+binds *that*. ana-nas itself is fine — the real repos are intact.
+
+**Recovery** (needs root on ana-docker — use **`ssh infra-ops@ana-docker`**,
+which has NOPASSWD sudo; the default `ssh ana-docker` lands as `lkraven`
+*without* sudo):
+
+```bash
+ssh infra-ops@ana-docker '
+  sudo mount -a                                   # re-attach the NFS (bypasses the failed unit)
+  sudo systemctl reset-failed mnt-backup.mount    # clear the stuck unit state
+  mount | grep /mnt/backup                         # confirm nfs4 attached
+  sudo ls /mnt/backup/restic/repo/ana/.htpasswd    # real htpasswd now present
+  cd /opt/docker/compose/rest-server-ana && sudo docker compose up -d --force-recreate
+'                                                  # recreate so the bind re-resolves onto NFS
+# verify: docker ps shows Up (healthy); curl :8000 -> 401; logs say "Loaded htpasswd file"
+```
+
+If the ghost dir blocks the mount, see `disaster-recovery.md` Tier-0 for
+the stop→umount→rm-ghost→remount→start variant.
+
+---
+
+## Known gaps / TODO
+
+- [x] **Backup-freshness alerting — DONE (2026-06-20).**
+      `scripts/check-backup-freshness.sh` (the 2-min check, exit 1 on
+      stale/down) + a daily **systemd user timer on nh3-dev** at 08:00
+      (`scripts/install-backup-freshness-timer.sh`) → `backup-freshness-alert.sh`
+      posts an **althing alert to infra-ops** on any stale/down layer. Run the
+      check by hand anytime. (Channel is althing for now — swap in email/ntfy if
+      you want a louder one.)
+- [x] **fstab hardening — DONE (2026-06-20).** ana-docker `/mnt/backup` →
+      `noauto,x-systemd.automount,x-systemd.mount-timeout=30` (autofs self-heals
+      on a NAS blip instead of getting stuck `failed`; activates on next reboot).
+      `/etc/fstab.bak-pre-harden` saved. **`/mnt/compose` also hardened the same
+      way** and **activated live** (umount → `mnt-compose.automount` started →
+      autofs verified remounting on access) — it binds no container, so it was
+      safe to convert now; this also proved the autofs pattern works on ana-docker.
+- [ ] **ana-ml2 has no PBS net** (bare metal) — restic is its only layer; now
+      healthy + alerted. Bulk `/tank` models are re-downloadable; bespoke
+      quants/configs/scripts are the real loss-risk.
+- [x] **esh-pve-nas coverage — VERIFIED (2026-06-20): NOT PBS-covered.** No
+      `esh-pve-nas` namespace exists on PBS-ANA, so **esh-vm-db (postgres+mongo)
+      + vm-esh-nas are restic-only.** For the DB VM, restic-with-dumps is the
+      *preferred* method (vs a VM image) **IF** the resticprofile includes
+      `pg_dump`/`mongodump` — confirm that. Optionally add esh-pve-nas as a PBS
+      source. ESH is home-lab (no SLA).
+- [ ] **Rotate rest-server repo passwords** — the 5 per-repo basic-auth creds
+      were exposed during the 2026-06-20 diagnosis. **BELAYED** — operator
+      handling offline.

docs/runbooks/heretic2-nvfp4-mtp-seat.md

@@ -0,0 +1,144 @@
+# Heretic2 NVFP4 + MTP fast char-rp-reasoning seat — the working recipe
+
+**Status: WORKING (2026-07-14).** ~77 tok/s single-stream (vs GGUF NEO-CODE ~59.5, base
+NVFP4 ~53) — **~1.3× over GGUF**, MTP draft-acceptance **32–40%**, mean acceptance length
+**2.19**. This is a drop-in faster replacement for the GGUF NEO-CODE `char-rp-reasoning`
+seat (same Heretic2/NEO-CODE model, NVFP4 + native MTP spec-decode).
+
+This runbook exists because getting here was a multi-hour fire drill. **Every gotcha below
+cost real time — read them before touching this.** The TL;DR: three things all had to be
+right at once — (1) quant as the *multimodal* class, (2) use the *modelopt* format not
+compressed-tensors, (3) work around a vLLM bug that quantizes the MTP draft head.
+
+---
+
+## What / where
+
+- **Model:** NEO-CODE = `DavidAU/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking` (dense
+  27B, `Qwen3_5` GDN-hybrid arch, multimodal `Qwen3_5ForConditionalGeneration`).
+- **Runs only on ana-ml2 GPU0** (NVFP4 is Blackwell-only; irv-ml1 is Ampere).
+- **Artifacts** (ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/`, root-owned):
+  - `heretic2-mtp-bf16/` — BF16 graft (Heretic2 + 15 base-Qwen3.6 MTP tensors). [graft input]
+  - `heretic2-modelopt-nvfp4/` — modelopt NVFP4 quant, single shard, **no mtp**. [quant output]
+  - `heretic2-modelopt-nvfp4-mtp/` — the above + spliced 15 BF16 mtp → **the seat**. [SERVE THIS]
+  - (superseded: `heretic2-nvfp4-cg*` = compressed-tensors path, coherent but MTP-inert;
+    `heretic2-mtp-nvfp4-prod` = original gibberish. Keep for diff, do not serve.)
+- **Scripts** (eshpfi `services/heretic2-nvfp4-quant/`): `graft_mtp.py`, `quant_modelopt.py`,
+  `finalize_modelopt_mtp.py`, `serve_modelopt_mtp.sh`, `sitecustomize-mtp-workaround.py`.
+- **Reference:** the MoE `gen` (`qwen36-35b-a3b-heretic-nvfp4`, `quant_method: modelopt`) and
+  the qwopus-122B `gen` both ran MTP before (qwopus +12% single-stream, archival-memory
+  2026-07-01) — dropped for `gen` because MTP *hurts concurrency*, which is why it belongs on
+  the single-stream RP seats, not `gen`.
+
+## GPU window ritual
+
+Base NVFP4 quant needs ~55 GB free on GPU0. `docker stop llama-charrp
+llama-charrp-reasoning vllm-aeon-gen` (→ ~97 GB free); restore with `docker start …`
+(~90–230 s to healthy). The GGUF NEO-CODE seat is the always-restorable fallback. Heads-up
+wt-dev (their character / thoughtful-character / gen route through these) — unless told
+otherwise. `ssh ana-ml2` = lkraven, in the docker group (no sudo needed for docker).
+
+---
+
+## The pipeline (4 steps)
+
+### 1. GRAFT (CPU, seats up) — `graft_mtp.py`
+Heretic2's finetune dropped the MTP head; graft the 15 BF16 `mtp.*` tensors from base
+`Qwen/Qwen3.6-27B` (shards 13+15). Symlinks Heretic2 shards + one `model-mtp.safetensors`.
+Idempotent, refuses to clobber. Output: `heretic2-mtp-bf16/`.
+
+### 2. QUANT (GPU0 window, ~18 min) — `quant_modelopt.py` via `run_quant_modelopt.sh`
+`nvidia-modelopt` PTQ → **modelopt** NVFP4 format. Three things this script gets right (each a
+gotcha — see below): loads as **`AutoModelForImageTextToText`**, patches the modelopt↔transformers
+**FusedMoE** bug, and forces **single-shard** export. Excludes `lm_head` + `visual` + all
+`linear_attn` (GDN) → BF16, matching AEON. Calib = the 512-row workload-matched chat mix.
+```bash
+docker run -d --name vllm-heretic2-modelopt-quant --gpus '"device=0"' --ipc host \
+  -v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \
+  --entrypoint bash vllm/vllm-openai:v0.24.0 -c '
+    set -e
+    pip install -q nvidia-modelopt tiktoken sentencepiece 2>&1 | tail -1
+    python3 /lk/quant_modelopt.py \
+      --model /tank/aimodels/heretic2-nvfp4-work/heretic2-mtp-bf16 \
+      --calib-mode chat --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
+      --num-samples 512 --seqlen 8192 \
+      --out /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4'
+```
+CPU dry-run (no GPU, tiny calib) to validate the pipeline without an outage: same command
+minus `--gpus`, add `-e CUDA_VISIBLE_DEVICES=""`, `--num-samples 2 --seqlen 512`.
+
+### 3. SPLICE (CPU) — `finalize_modelopt_mtp.py`
+Copy `heretic2-modelopt-nvfp4` → `heretic2-modelopt-nvfp4-mtp`, splice the 15 BF16 `mtp.*`
+tensors into the single shard (→ 1967 tensors). (The transformers load never builds an mtp
+module, so mtp must be spliced post-quant — same as AEON/pantheon.)
+
+### 4. SERVE (GPU0) — `serve_modelopt_mtp.sh` + the MTP workaround
+```bash
+docker run -d --name vllm-charrp-modelopt --gpus '"device=0"' --ipc host \
+  -v /tank/aimodels:/tank/aimodels \
+  -v <sitecustomize dir>:/lk_debug -e PYTHONPATH=/lk_debug \    # ← the MTP workaround, see below
+  -p 8018:8000 vllm/vllm-openai:v0.24.0 \
+  /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp \
+  --quantization modelopt \
+  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
+  --language-model-only --mamba-cache-dtype float32 \
+  --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \
+  --served-model-name char-rp-reasoning --max-model-len 40960 --max-num-seqs 32 \
+  --gpu-memory-utilization 0.5 --trust-remote-code
+```
+`--language-model-only` skips the vision tower (RP seat doesn't need it; saves ~1–2 GB — the
+tower is preserved BF16 in the weights, so multimodal is recoverable by dropping the flag).
+
+---
+
+## The four landmines (each cost hours)
+
+1. **Load as `AutoModelForImageTextToText`, NEVER `AutoModelForCausalLM`.** The latter resolves
+   `qwen3_5` → text-only `Qwen3_5ForCausalLM` → flat `model.layers.*` keys. vLLM only serves
+   `Qwen3_5ForConditionalGeneration`, whose weight mapper needs `model.language_model.*` (+
+   `model.visual.*`). Wrong class → every layer weight silently fails to load → **`!!!!` gibberish**.
+
+2. **Use the MODELOPT format (nvidia-modelopt), not compressed-tensors (llm-compressor).** On
+   compressed-tensors the MTP drafter can't load the BF16 mtp head at all (`not found in
+   params_dict`, **0% acceptance** — loads but never accelerates; this is what pantheon and the
+   "AEON RP seat" actually were). Base NVFP4 *alone* ≈ GGUF at batch-1 (no single-stream win) —
+   **the MTP multiplier is the entire point**, and it needs modelopt.
+
+3. **modelopt 0.45 ↔ transformers 5.12.1 FusedMoE crash.** `mtq.quantize` dies with
+   `TypeError: issubclass() arg 2 must be a class` — modelopt registered transformers' `FusedMoE`
+   (a *function* in 5.x) as an nn class. `quant_modelopt.py` guards it (patches
+   `_DMRegistryCls._get_registered_nn_class` to skip non-class registry entries). Do **not**
+   pin `nvidia-modelopt[hf]==0.43` to dodge it — that drags transformers back to 4.57 which can't
+   load `qwen3_5` at all.
+
+4. **⭐ THE BIG ONE — vLLM 0.24.0 does not propagate modelopt `exclude_modules` to the
+   spec-decode DRAFT model.** The MTP drafter builds its own `qkv_proj`/`gate_up_proj` as
+   *quantized* (NVFP4-packed) while the mtp head is BF16 → `AssertionError: param_data.shape ==
+   loaded_weight.shape` in `qwen3_5_mtp.py:256`. **No checkpoint config fixes this** — instrumenting
+   `is_layer_skipped` proved the drafter's exclude list contains only the *main* model's
+   `linear_attn` entries, never the mtp ones. Also note `is_layer_skipped` does **exact string
+   membership, not glob** — so wildcards like `mtp.layers.0.*` never match anything. **Fix = a
+   runtime patch** (`sitecustomize-mtp-workaround.py`, mounted on `PYTHONPATH`) that force-skips
+   any `mtp.*` prefix in `is_layer_skipped`, keeping the drafter BF16. This is a genuine vLLM bug —
+   **report upstream** (draft-model quant-config should inherit the target's exclude_modules).
+
+## Verify it's actually accelerating
+
+```bash
+# coherence
+curl -s :8018/v1/completions -d '{"model":"char-rp-reasoning","prompt":"The old tavern","max_tokens":40,"temperature":0}'
+# drive tokens, then read acceptance from the seat log:
+docker logs vllm-charrp-modelopt 2>&1 | grep SpecDecoding | tail -2
+#   -> "Mean acceptance length: 2.19 ... Avg Draft acceptance rate: 39.7%"   [GOOD: >0%, ~2 length]
+#   -> "Avg Draft acceptance rate: 0.0%"                                       [BAD: compressed-tensors, or mtp quantized]
+```
+`SpecDecoding` line only appears during active generation. 0% acceptance = you're on
+compressed-tensors, or the workaround didn't load (check for `[ISLS] ... workaround installed`).
+
+## Productionization TODO (not yet done)
+- Bake the sitecustomize workaround into a compose stack (mount + `PYTHONPATH`), served-name
+  `char-rp-reasoning`, alongside/replacing the GGUF seat.
+- brokkr P00 (soong 9-tool k5) — same base model as GGUF NEO-CODE so R36 should carry, but the
+  NVFP4-vs-Q5 quality + tool-path must be confirmed before cutover.
+- Repoint gateway `char-rp-reasoning` alias + heads-up wt-dev.
+- File the vLLM upstream bug (draft-model exclude non-inheritance).

docs/runbooks/nh3-dev-development-backup.md

@@ -0,0 +1,57 @@
+# nh3-dev `~/development` — hourly off-box backup
+
+**Why this exists:** nh3-dev is the dev box where agents do uncommitted work under
+`~/development/<project>/`. That tree had **no off-box backup**, so a destructive
+mistake (a stray `rm -rf` on a working dir on 2026-07-12) had no safety net. This
+job closes that gap: an hourly, versioned, off-box snapshot of `~/development`.
+
+## What it does
+
+- **Source:** `nh3-dev:~/development/` (lkraven's working dirs).
+- **Destination (off-box):** `nh3-nas:/volume1/Backup/nh3-dev-development/<YYYY-MM-DD_HHMM>/`
+  — a timestamped dir per snapshot, over rsync-**over-ssh** (syncuser).
+- **Versioning:** `rsync --link-dest` against the previous snapshot → unchanged
+  files hardlink (share inodes, ~0 bytes); only changed files consume new space.
+  `latest` symlink points at the newest snapshot.
+- **Retention:** newest **48** hourly snapshots (older pruned each run).
+- **Excludes:** heavy reconstructable dirs (`node_modules`, `.venv`, `venv`,
+  `__pycache__`, `.pytest_cache`, `.mypy_cache`, `.ruff_cache`, `.cache`, `dist`,
+  `build`, `.next`, `target`, `*.pyc`) and secrets (`.env`, `.env.*`, `*.pem`,
+  `*.key`, `id_*`, `*.sqlite*`). **`.git` is kept** (local commits/stashes = the
+  uncommitted work that matters). Seed snapshot ≈ **11G**; hourly deltas are MB-scale.
+
+## Where it lives (on nh3-dev)
+
+- Script: `~/.config/dev-backup/dev-backup.sh` (mirror committed at
+  `scripts/nh3-dev-development-backup.sh`).
+- systemd `--user` units: `~/.config/systemd/user/dev-backup.{service,timer}`
+  (`OnCalendar=hourly`, `Persistent=true`, linger on → fires without a login).
+- Log: `~/.config/dev-backup/dev-backup.log`.
+
+```bash
+systemctl --user list-timers dev-backup.timer   # next run
+systemctl --user start dev-backup.service        # run now
+tail -f ~/.config/dev-backup/dev-backup.log
+```
+
+## Restore
+
+Snapshots are plain dir trees — no special tool needed:
+
+```bash
+# list snapshots
+ssh nh3-nas 'ls -1 /volume1/Backup/nh3-dev-development/'
+# restore one file/dir from a chosen snapshot
+rsync -a nh3-nas:/volume1/Backup/nh3-dev-development/<STAMP>/<proj>/<path> /tmp/restore/
+# or pull a whole project back
+rsync -a nh3-nas:/volume1/Backup/nh3-dev-development/latest/<proj>/ ~/development/<proj>/
+```
+
+## Notes / future
+
+- **Not encrypted at rest** (plaintext on the trusted internal NAS; secrets are
+  excluded). Upgrade path: migrate to restic once a repo can be created on
+  rest-server-nh3 (currently returns 404 on repo-create — likely append-only) or
+  the Synology sftp subsystem is enabled (currently disabled → restic sftp fails).
+- Off-box = off the nh3-dev VM (lands on nh3-nas, same NH3 site). Cross-site
+  mirroring of this repo is a separate future layer.

docs/runbooks/soong-lab-autodeploy.md

@@ -0,0 +1,91 @@
+# soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated)
+
+Green-gated CI/CD for the soong-lab studio: **push to `main` → run the test
+suite → redeploy the studio ONLY if tests pass** (running studio is never
+touched on a red run). Built 2026-07-13 (Vuong-directed). Adapts the
+[ytvc-autodeploy](./ytvc-autodeploy.md) webhook pattern.
+
+## Flow
+
+```
+push→main → gitea webhook (POST, HMAC) → soong-webhook listener :9010 on corviduo-dev
+  → ~/soong-lab-deploy.sh:
+       git clone (read-only deploy key, internal SSH :222)
+       uv sync ; uv run pytest        ── RED → abort, studio UNTOUCHED, status=red
+       rsync backend/ → studio dir + web/ → SOONG_LAB_WEB_DIR ; uv sync --no-dev ; restart
+       → status=green, studio healthy
+```
+
+## Components (all on corviduo-dev, user `infra-ops`)
+
+- `~/soong-lab-deploy.sh` — clone → test → deploy-on-green. Logs to
+  `~/soong-lab-deploy.log`; writes `~/.config/soong/last-deploy.json`
+  (`{result: green|red, stage, sha, at}`).
+- `~/soong-webhook.py` — HTTP listener on `:9010`. HMAC-SHA256 (`X-Gitea-Signature`)
+  vs `~/.config/soong/webhook-secret` (mode 600); fires the deploy only on
+  `ref == refs/heads/main`. `GET /` returns `ok | last: <status>`.
+- `soong-webhook.service` (system unit, enabled) — runs the listener.
+- Read-only deploy key `~/.ssh/soong-deploy_ed25519` → gitea repo key id 5 on
+  `vh/soong-lab` (read_only). Clone via `ssh://git@10.250.50.70:222/vh/soong-lab.git`.
+- Studio unit `soong-lab-studio.service` (WD `/home/infra-ops/soong-lab/backend`);
+  restart needs infra-ops NOPASSWD sudo (present).
+- Gitea webhook: repo `vh/soong-lab` hook id 3 → `http://10.250.50.152:9010/`,
+  JSON, Push events, the shared secret.
+
+## Verify / operate
+
+```bash
+ssh corviduo-dev 'systemctl is-active soong-webhook.service; curl -s localhost:9010/'
+ssh corviduo-dev 'tail -30 ~/soong-lab-deploy.log'          # deploy history
+# manual deploy (same as the webhook does):
+ssh corviduo-dev 'bash ~/soong-lab-deploy.sh'
+```
+
+## Notes / gotchas
+
+- **Frontend (`web/`) sync IS part of the deploy**: the studio serves `web/` from
+  `SOONG_LAB_WEB_DIR` (`/home/infra-ops/soong-lab/web`), *separate* from the backend
+  `WorkingDirectory`. The deploy rsyncs BOTH `backend/`→studio and `web/`→`SOONG_LAB_WEB_DIR`.
+  (Added 2026-07-13 after soong-dev caught the served frontend silently rotting — the backend
+  was updating while `web/` stayed pinned to the initial manual copy; a bounce alone re-serves
+  the same stale file.)
+- **Green-gated by construction**: `pytest || fail` runs BEFORE any studio touch,
+  so a red suite aborts with the studio still on the old version. Validated
+  2026-07-13 (a mid-deploy rsync failure left the studio untouched/active).
+- **rsync is required** on corviduo-dev (`apt install rsync` — installed 2026-07-13;
+  it wasn't present initially).
+- **bifrost dep** resolves from the internal Gitea PyPI via `~/.netrc` (already
+  present on corviduo-dev); no extra auth in the deploy script.
+- **⚠️ Auto-deploy silently never worked until 2026-07-14 — TWO compounding blockers.**
+  The LISTENER binds `0.0.0.0:9010` and works, but nothing gitea sent ever reached
+  it, so every push was a no-op (v0.3.6 was manual; v0.3.7–v0.3.15 never
+  auto-deployed until fixed). Two separate, both-real blockers:
+  1. **corviduo-dev ufw** — `default-deny`, only 22 + 8080 allowed, so a *direct*
+     TCP to :9010 from ana-docker DROP-timed-out. Fix: `ufw allow from 10.0.0.0/8`
+     (operator-directed — "that footgun happens a lot", accept the fleet).
+  2. **★ gitea `webhook.ALLOWED_HOST_LIST` (the DECISIVE one)** — was
+     `external, 10.100.0.0/16` (NH3 only); corviduo-dev is `10.250.50.152`
+     (Anaheim), so gitea **refused to deliver**: `webhook can only call allowed HTTP
+     servers ... deny '10.250.50.152'` — it never even opens the TCP connection, so
+     the ufw fix alone did nothing. Fix: `ALLOWED_HOST_LIST = external, 10.0.0.0/8`
+     in gitea `app.ini` (`/data/gitea/conf/app.ini`, `[webhook]`) + `docker restart
+     gitea` (~8s blip). The HMAC secret was already correct (once delivery arrives,
+     `hmac_ok=True`).
+  **RED HERRINGS that cost two diagnosis rounds:** (a) "test-delivery 204" is gitea
+  *queuing*, NOT delivering — never proves the round-trip; (b) a proxy test signing
+  with the *listener's own* secret (bypassing gitea) proves the listener but NOT
+  gitea's real delivery. **Diagnose from BOTH ends:** the SENDER (`docker logs gitea
+  --since 5m | grep webhook` → the `deny '<ip>'` line) AND an instrumented RECEIVER
+  — the listener now ships with delivery logging (`journalctl -u soong-webhook.service
+  | grep '\[webhook\]'` shows source-IP / hmac_ok / ref / action; the old
+  `log_message=pass` silence hid all of it). **Proof of fix:** a real gitea delivery
+  logs `POST from 10.250.50.70 ... hmac_ok=True`, `ref='refs/heads/main'`,
+  `-> 202 deploying` → green deploy of the latest main SHA.
+- **Red-run push-notify** via an **althing relay on nh3-dev** (`soong-ci-relay.timer`,
+  2-min poll of corviduo's `last-deploy.json` → pings **soong-dev** via althing on a
+  NEW red run; green runs stay silent = fire-and-forget). corviduo itself has no
+  althing, so the relay lives on nh3-dev (which does), needing no gitea write token
+  on the Worldtree-team VM. Files: `services/soong-lab-ci/soong-ci-relay.{sh,service,timer}`;
+  state `~/.local/state/soong-ci-relay/last-at.txt`. (A gitea commit-status was the
+  alternative but needs a write token gitea won't mint without basic-auth.)
+- Test suite: `uv run pytest` in `backend/` (242 tests as of v0.3.6).

docs/storetank-image-models-archive.md

@@ -0,0 +1,52 @@
+# Storetank image-models archive — curation / migration / decommission record
+
+**Host:** irv-ml1 · **Former path:** `/storetank/image-models/comfy/models`
+(was the native `/opt/ComfyUI/models` symlink target).
+**Status: DECOMMISSIONED 2026-06-13** — emptied of all models (919 G → 0). The
+active model tree is arbo's `/storetank/arbo/models` — see
+[`arbo-comfyui-model-catalog.md`](arbo-comfyui-model-catalog.md).
+
+Historical record of how the 919 G CivitAI-managed pile was resolved on 2026-06-13:
+**~739 G killed** (superseded / niche), **177 G migrated** into the arbo set, the
+remainder dupes arbo already had.
+
+## 1. Killed — superseded by arbo's current-gen stack (~739 G)
+
+Two principles: generation-locked LoRAs have no value without their (also-superseded)
+base models, and arbo already carries its own copies of the shared encoders/VAEs.
+
+| Killed | Size | Why |
+|---|---|---|
+| **Hunyuan video** (diffusion_models + unet + vae + loras) | 74 G | older video arch; not in arbo |
+| **WAN2.1 bases + loras** | 118 G | superseded by arbo's WAN2.2; loras gen-locked |
+| **WAN2.1 encoders / VAE** (umt5, xlm-roberta, clip_vision_h, wan VAE) | 44 G | dupes of arbo's own copies |
+| **FLUX.1 — everything** (dev/schnell/fill + ~20 community merges + loras + flux controlnets / redux / pulid / clip-vision / FLUX.D encoder / Florence-2-Flux) | 445 G | superseded by arbo's FLUX.2-klein; loras gen-locked |
+| **orphaned umt5** (root `umt5_xxl_fp8`) | 6.7 G | last WAN remnant |
+| **orphaned llava_llama3** (fp16 + fp8) | 23.5 G | HunyuanVideo's text encoder — dead after the Hunyuan kill |
+| **Chroma v10/v11 + SD3.5-large** | 28 G | niche generators, not in arbo |
+| **TOTAL** | **~739 G** | |
+
+## 2. Migrated into arbo (177 G)
+
+Everything not superseded moved into `/storetank/arbo/models` (same-filesystem atomic
+move, skip-existing so arbo's production copies were never clobbered; 652 files moved,
+37 skipped as dupes):
+
+- **Gen-agnostic utilities:** Aura-SR v1/v2 + the full `upscale_models/` family
+  (HAT/DAT/RealESRGAN/UltraSharp/Remacri/NMKD/Omni-SR) · Florence-2 + CogFlorence
+  captioners (`LLM/` + `florence2/`) · controlnet_union_promax · grounding-dino ·
+  SAM + SAM-HQ · yolo (`ultralytics/`) · depthanything-v2 · vitmatte · nsfw_detector ·
+  insightface (inswapper + antelopev2) · facerestore · facexlib · ip-adapter-plus_sdxl ·
+  CLIP-vision (sigclip, EVA02-CLIP-L)
+- **SDXL / Pony stack:** ponyRealism, cyberrealisticPony_v8, lustify, hassaku,
+  waiNSFWIllustrious, juggernaut / dreamshaper Lightning, SUPIR + loras
+  (dmd2_sdxl_4step, ACE++, Illustrious/PonyXL character-design)
+
+comfy-dev owns the follow-on: per-model catalog entries + graphs + heroes that turn
+these files into usable arbo workflows.
+
+## 3. Final state
+
+`/storetank/image-models/comfy/models` (= native `/opt/ComfyUI/models`) holds only
+empty category dirs + `put_*_here` placeholders — **638 K, no models**. Decommissioned;
+arbo is the single live tree.

persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md

@@ -0,0 +1,4 @@
+- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized"
+  is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but
+  fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push
+  finishes. FIX: re-run once built, or gate on `workflow_run: completed`.

persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md

@@ -0,0 +1,4 @@
+- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools
+  needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The
+  althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO
+  routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`.

persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md

@@ -0,0 +1,6 @@
+- `[2026-07-04]` **LiteLLM (this gateway version) mutates the SHARED deployment config in-place on
+  per-request sampler-param merge** → my deliberately-invalid `top_k=-5` forwarding-probe bled into a
+  param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT
+  caching (none configured), NOT a config change. **Never fire invalid/distinctive sampler values at
+  a SHARED gateway alias with live consumers** — use a throwaway alias, or a `docker restart litellm`
+  flushes residual carryover. `feedback_litellm_shared_param_mutation`.

persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md

@@ -0,0 +1,4 @@
+- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe
+  ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast
+  on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the
+  566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`.

persistent-memory.d/2026-07-07-engine-invocation-footguns-cost-several-wasted-serve-bounces.md

@@ -0,0 +1,5 @@
+- `[2026-07-07]` **Engine invocation footguns cost several wasted serve-bounces this session** — `docker run
+  --rm` ate crash logs; duplicated `serve` (vLLM image entrypoint is already `["vllm","serve"]`);
+  `--max-lora-rank 48` invalid (choices 1/8/16/32/64… → use 64); parens in `echo` inside `ssh host -c "…"`
+  break the remote shell. LESSON: verify engine launch flags (`--help`, GPU-free) + never `--rm` a container
+  whose crash logs you need, BEFORE bouncing a production serve.

persistent-memory.d/2026-07-07-sglang-generic-image-can-t-load-our-nvfp4.md

@@ -0,0 +1,4 @@
+- `[2026-07-07]` **SGLang generic image can't LOAD our NVFP4 AEON** — ModelOptModelLoader weight-shape/
+  packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated
+  `qwen36-27b-nvfp4` dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was
+  quant-loader-specific).

persistent-memory.d/2026-07-07-sglang-lora-target-modules-cli-enum-rejects-the.md

@@ -0,0 +1,4 @@
+- `[2026-07-07]` **SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for**
+  (invalid choice: 'in_proj_qkv'); `'all'` resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants
+  its OWN packed layout (base r16 + `get_stacked_multiply=3`, NOT a pre-fused rank-48 qkv → the [48]-vs-[144]
+  shape assert). A THIRD adapter format; version-exact source needed (`:latest`=0.5.13, NOT `main`).

persistent-memory.d/2026-07-07-vllm-0-24-0-qwen3-5-lora-application.md

@@ -0,0 +1,6 @@
+- `[2026-07-07]` **vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639).** Adapter loads HTTP 200
+  but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate `zc`
+  adapter — correct per vLLM's `check_unexpected_modules` allowlist — loads clean but inert; the fused-key
+  rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of
+  `expand_packed_lora` — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no
+  version-bump helps. Merge bakes deltas in (bypasses this) but is static.

persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md

@@ -0,0 +1,5 @@
+- `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4
+  (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY
+  (temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models.
+  Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going **GGUF (llama.cpp)** to sidestep the
+  whole NVFP4-quant surface.

persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md

@@ -0,0 +1,19 @@
+- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3
+  epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass
+  `num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs
+  + _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the
+  off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a
+  ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run.
+  FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig,
+  float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`.
+  GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min
+  train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step;
+  train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45
+  banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0):
+  ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is
+  COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral
+  literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift,
+  arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep
+  on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval +
+  the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO
+  adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).

persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md

@@ -0,0 +1,7 @@
+- `[2026-07-08]` **Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24).** (a) HF `tokenizer.json`
+  for Mistral = **GARBAGE output** — the card's "use the official Mistral tokenizer" warning is REAL; must use the
+  `tekken.json`/mistral tokenizer. (b) BUT `--tokenizer-mode mistral` + vision **CRASHES** (`Failed to apply
+  PixtralProcessor on {'text': '[IMG]'}`; and with tekken.json present in auto mode, `CachedMistralCommonBackend has
+  no attribute is_fast`). So it's **mistral-tokenizer OR vision, not both** on this vLLM. Text-only + mistral
+  tokenizer serves clean (`--limit-mm-per-prompt '{"image": 0}'`). **GGUF/llama.cpp avoids all of this** (native
+  mistral tokenizer + vision).

persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md

@@ -0,0 +1,8 @@
+- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.**
+  Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision,
+  util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps
+  ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled
+  refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path);
+  Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting
+  ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway
+  sampling-defaults wiring still PENDING.

persistent-memory.d/2026-07-08-pantheon-27b-mtp-on-vllm-compressed-tensors-0.md

@@ -0,0 +1,6 @@
+- `[2026-07-08]` **Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance.** MTP is a separate **bf16** head
+  (`mtp.*`, in `model-auxiliary.safetensors`, 15 tensors); AEON preserved it by INJECTING the bf16 head into the
+  quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors
+  main + injected bf16 mtp + `text_config.mtp_num_hidden_layers=1` → vLLM detected the MTP but SKIPPED the bf16
+  self_attn weights → 0/192 draft tokens accepted. **The bf16 MTP head only loads on the MODELOPT main-model format
+  (like AEON), not compressed-tensors.** (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.)

persistent-memory.d/2026-07-08-pantheon-reasoning-27b-refuses-dark-fiction-despite-an.md

@@ -0,0 +1,6 @@
+- `[2026-07-08]` **Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base.** The base
+  (`llmfan46 heretic`) writes freely (thinking-off), but Gryphe distilled the reasoning traces from **DeepSeek 3.2**
+  (safety-aligned) onto every turn (`preserve_thinking:true`) → the model reasons ITSELF into refusals in the
+  `<think>` phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified).
+  **Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw
+  abliterated base is cleaner** — this is WHY the pivot went to the llmfan46 heretic base for gen.

persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md

@@ -0,0 +1,11 @@
+- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD
+  Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.**
+  Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0
+  KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf);
+  (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a
+  DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over
+  RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp
+  template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the
+  live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively
+  opens `<think>`/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap
+  b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).**

persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md

@@ -0,0 +1,12 @@
+- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.**
+  Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path
+  finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied
+  VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before
+  `from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does
+  `model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers
+  5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the
+  DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED
+  un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO
+  confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93%
+  util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit.
+  gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.

persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md

@@ -0,0 +1,17 @@
+- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.**
+  Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the
+  SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo
+  --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src
+  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank
+  -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml
+  --stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is
+  `["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash.
+  Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer`
+  (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→
+  immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→
+  **llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→
+  (more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate
+  1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0)
+  but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image.
+  Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX,
+  expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.

persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md

@@ -0,0 +1,4 @@
+- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` +
+  `envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal),
+  YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my
+  parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.)

persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md

@@ -0,0 +1,4 @@
+- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC
+  frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks-
+  TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the
+  bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real.

persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md

@@ -0,0 +1,7 @@
+- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).**
+  Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty
+  idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching).
+  Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08).
+  **This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the
+  operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread
+  01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]].

persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md

@@ -0,0 +1,46 @@
+- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed,
+  #335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides
+  intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on
+  `summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit
+  both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen**
+  (b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted
+  `worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL
+  NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1`
+  after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config
+  INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven
+  safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed).
+  StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON:
+  the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never
+  updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) —
+  pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass`
+  parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider
+  description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix
+  (operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING
+  defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs
+  when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway
+  config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD;
+  live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09):
+  character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving
+  character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping
+  principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the
+  dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent
+  but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44,
+  UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly
+  from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) +
+  mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's
+  `load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character +
+  character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window
+  VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev
+  to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked
+  config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly
+  re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard
+  + character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning
+  downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying
+  (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37**
+  (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09):
+  mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx
+  131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45
+  canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push
+  of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity
+  block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev),
+  monitor armed. See [[reference_corviduo_dev_emergency_ops]].

persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md

@@ -0,0 +1,4 @@
+- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8
+  (not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the
+  cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) —
+  that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting.

persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md

@@ -0,0 +1,10 @@
+- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into
+  gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17,
+  vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack
+  (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829);
+  max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and
+  `repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture
+  no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone
+  voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514).
+  **Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not
+  A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s).

persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md

@@ -0,0 +1,3 @@
+- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that
+  spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning
+  (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path).

persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md

@@ -0,0 +1,16 @@
+- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker
+  docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB
+  registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the
+  2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to
+  remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because
+  daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never
+  reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd
+  opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`,
+  `rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the
+  frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime
+  (zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares
+  `live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start,
+  never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers
+  once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon;
+  log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running
+  containers).

persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md

@@ -0,0 +1,20 @@
+- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable
+  systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00`
+  America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh`
+  runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in
+  `~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH
+  footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS
+  new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode
+  ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0):
+  web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input +
+  caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle
+  brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor);
+  registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired
+  step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default
+  + `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled
+  1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending.
+  **NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start
+  brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session
+  handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 +
+  ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token
+  feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo).

persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md

@@ -0,0 +1,16 @@
+- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev
+  requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`,
+  0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the
+  VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` =
+  `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump
+  broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch
+  2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke
+  the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot
+  `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention),
+  re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at
+  `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):**
+  comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention
+  rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY
+  the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) →
+  rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1),
+  lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.

persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md

@@ -0,0 +1,14 @@
+- `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1,
+  SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch
+  2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt`
+  FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came
+  only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt`
+  as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any
+  transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment).
+  `docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB
+  migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3
+  needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a
+  mmartial ComfyUI version bump, `git diff <old> <new> -- requirements.txt` — if torch is unpinned, bump the CODE
+  without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid
+  1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See
+  [[reference_irv_ml1_comfyui_mmartial]].

persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md

@@ -0,0 +1,13 @@
+- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev
+  R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml`
+  (`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347
+  `session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's
+  personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based
+  `affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347.
+  R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe
+  → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string
+  insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own
+  yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows
+  baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo
+  untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they
+  want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]].

persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md

@@ -0,0 +1 @@
+- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1.

persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md

@@ -0,0 +1 @@
+- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot.

persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md

@@ -0,0 +1 @@
+- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated).

persistent-memory.d/2026-07-13-relaying-a-peer-s-diagnosis-as-fact-without.md

@@ -0,0 +1 @@
+- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction.

persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md

@@ -0,0 +1 @@
+- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate).

persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md

@@ -0,0 +1 @@
+- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]**

persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md

@@ -0,0 +1 @@
+- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize.

persistent-memory.d/2026-07-14-aeon-s-working-nvfp4-mtp-rp-seat-was.md

@@ -0,0 +1 @@
+- `[2026-07-14]` **AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof.** `vllm-aeon-rp`'s .env → `AEON_RP_MODEL=pantheon-27b-mtp-nvfp4`, `AEON_RP_QUANT=compressed-tensors` — it LOADED (mtp silently skipped, `exited 0`) but never accelerated. Same vLLM image (`:latest` = `sha256:4091d55` = 0.24.0) as the failed Heretic2 test, so the "AEON ran on an older vLLM" theory was wrong. Don't treat a seat that "ran" as MTP-validated without checking its `SpecDecoding` acceptance.

persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md

@@ -0,0 +1 @@
+- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]].

persistent-memory.d/2026-07-14-gitea-test-delivery-204-is-not-proof-a.md

@@ -0,0 +1 @@
+- `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny '<ip>'` line) AND an instrumented receiver.

persistent-memory.d/2026-07-14-mtp-graft-via-top-level-mtp-tensor-names.md

@@ -0,0 +1 @@
+- `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip.

persistent-memory.d/2026-07-14-mtp-on-modelopt-no-checkpoint-config-skips-the.md

@@ -0,0 +1 @@
+- `[2026-07-14]` **MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround.** All crashed the same way (`qwen3_5_mtp.py:256` `param_data.shape == loaded_weight.shape` AssertionError — bf16 mtp head loaded into a quantized drafter param): (1) mtp excludes in `config.json` (WRONG file — vLLM modelopt reads `hf_quant_config.json`); (2) specific-unfused mtp names in hf_quant_config; (3) wildcards `mtp*`/`mtp.layers.0*` (`is_layer_skipped` is EXACT-membership, NOT glob — wildcards match nothing); (4) exact fused+unfused names in both `mtp.`/`model.` prefixes. Instrumenting `is_layer_skipped` proved the drafter's exclude list holds ONLY the main model's `linear_attn` entries — the mtp excludes never reach the draft-model quant config. ONLY fix = a mounted `sitecustomize` force-skipping `mtp.*`. LESSON: don't chase checkpoint-config fixes for the mtp-drafter crash; go straight to the runtime patch. Also `nvidia-modelopt[hf]==0.43` (AEON's producer version) is a trap — it pins transformers back to 4.57 which can't load `qwen3_5` at all; use 0.45 + the FusedMoE guard in `quant_modelopt.py`.

persistent-memory.d/2026-07-14-nvfp4-llm-compressor-compressed-tensors-gives-no-batch.md

@@ -0,0 +1 @@
+- `[2026-07-14]` **NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept.** Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp weights off a compressed-tensors main model → `Parameter … not found in params_dict`, `Avg Draft acceptance rate: 0.0%`). Pantheon is identical — its "working NVFP4+MTP" was working *structure*, never real acceleration. Working native MTP needs the **modelopt** main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt.

persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md

@@ -0,0 +1 @@
+- `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug.

persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md

@@ -0,0 +1 @@
+- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section.

persistent-memory.d/2026-07-14-nvfp4-spike-built-the-full-mtp-serve-scaffolding.md

@@ -0,0 +1 @@
+- `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state.

persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md

@@ -0,0 +1 @@
+- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`.

persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md

@@ -0,0 +1 @@
+- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528.

persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md

@@ -0,0 +1 @@
+- `[2026-07-15]` **arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake.** Operator-directed full switch (comfy-dev executed, live in prod). Established: gen (`qwen3.6-35b-a3b-heretic`) is vision-enabled and was image-bench's predecessor as arbo's hero-judge; image-judge actually serves 4 roles (vision quality-scoring + identity-scoring + bbox grounding + an uncensored text tier), not just grounding. comfy-dev spot-check: gen faster on every task, grounding within ~3px, uncensoring preserved, and it FIXED a bug (image-judge's reasoning preamble broke json_object + stalled the router). Sequencing = short prod bake then evict (~30 GB GPU1 reclaim); revert = flip `ARBO_VISION_MODEL`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`.

persistent-memory.d/2026-07-15-claude-code-statusline-cost-total-cost-usd-is.md

@@ -0,0 +1 @@
+- `[2026-07-15]` **Claude Code statusline `.cost.total_cost_usd` is per-SESSION** (Claude Code's own cache/model-aware session accounting), not a lifetime aggregate — the large value just reflects a long, multiple-times-summarized session. And the old statusline hardcoded Sonnet pricing ($3/$15) on an Opus session -> ~5x cost understatement.

persistent-memory.d/2026-07-15-docker-service-after-remote-fs-target-does-not.md

@@ -0,0 +1 @@
+- `[2026-07-15]` **`docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts** — `nofail` drops a mount out of remote-fs.target's blocking set, so the drop-in ordering is silently defeated (paperless still Exited(255) on reboot). Real fix = DIRECT mount->docker ordering via the fstab `x-systemd.before=docker.service` option (verify `systemctl show docker -p After` lists the mnt-*.mount units). esh-docker-vm.

persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md

@@ -0,0 +1 @@
+- `[2026-07-15]` **esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service`** (the prior `After=remote-fs.target` drop-in was silently defeated by `nofail`). Reached only after a REBOOT (D-state phantom containers uptime-kuma + paperless-web that no `docker`/`ctr`/daemon-restart could clear). Committed `21d9a07` + playbook updated. See Tried and abandoned.

persistent-memory.d/2026-07-15-the-esh-docker-vm-d-state-phantom-container.md

@@ -0,0 +1 @@
+- `[2026-07-15]` **The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT** — reconfirmed: `docker stop/rm -f`, `ctr -n moby task delete`, AND `systemctl restart docker` all fail to clear it; `docker exec` into a wedged container ALSO fails (`setns ... exit status 1`), so the in-place restart escape hatch is out. Worse, a daemon restart can HALF-KILL other healthy containers (knocked paperless's granian down + left it wedged). Process dead but dockerd won't reap -> phantom. NFS mounts are `_netdev,nofail` so the reboot is boot-safe.

persistent-memory.d/2026-07-15-vllm-max-model-len-does-not-free-gpu.md

@@ -0,0 +1 @@
+- `[2026-07-15]` **vLLM `max-model-len` does NOT free GPU VRAM** — the KV cache POOL is sized by `gpu-memory-utilization`, not max-model-len. Lowering max-model-len only caps per-request context + drops max concurrency; the pool still fills the util budget. To actually free VRAM, lower `gpu-memory-utilization`. (Bit the char-rp-reasoning "drop KV to 150K" ask: the 150K applied but freed 0 VRAM until util dropped 0.39->0.38.)

persistent-memory.d/2026-07-17-zonos2-containerize-prosody-priming.md

@@ -0,0 +1,15 @@
+- `[2026-07-17]` **Zonos2 `:1920` engine → self-contained container (stays on 3090); prosody-priming is a SERVING-LAYER change (engine stays stock).**
+
+**Context.** The production Zonos TTS engine (irv-ml1 `:1920`, feeds asset-engine + gateway-chat via `zonos-gateway` :8890) was a bare native process — its real launch config existed ONLY in the running process argv (the committed `~/tts-audition/harness/zonos_server.sh` was STALE: said A6000/:1919/no perf flags; live is 3090/:1920 with `--cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3`). Captured to eshpfi `stacks/zonos-engine/` (README + corrected `zonos2-server.sh` + `.env.example`), commit **14a0004** (UNPUSHED as of the snapshot).
+
+**Decision 1 — containerize as a SELF-CONTAINED image** (not systemd — operator rejected; not a thin bind-mount wrapper — I walked that back: bind-mounting the host's CUDA-compiled `.venv` couples to the host's exact CUDA/glibc and is fragile + not reproducible). Shape: `FROM` a CUDA 12.8 base → `uv sync` against the repo's committed `uv.lock` (deterministic env) → mount the ~15 GB HF weights (`~/.cache/huggingface/hub/models--Zyphra--ZONOS2`, do NOT bake) → pin the **3090** (`NVIDIA_VISIBLE_DEVICES=0`) → `restart: unless-stopped` → CMD = the captured invocation. **Engine stays STOCK** Zyphra/Zonos2 @ commit `194c0a3` (no fork — the `zonos2` package ships its own server). **Build risk:** heavy compiled-CUDA deps (flashinfer / sgl_kernel / cutlass-dsl / apache-tvm-ffi / pynini) on torch 2.9.1+cu128 — mostly prebuilt wheels + the `uv.lock` make it tractable, expect a couple build iterations. **Cutover (in place on the 3090):** stop the native process (frees ~17 GB) → `docker compose up -d` (re-allocates ~17 GB, same footprint) → repoint `zonos-gateway`'s `ZONOS_URL` at the container (or keep the `:1920` host-port publish). One brief prod-TTS blip.
+
+**GPU = 3090 (operator 2026-07-17).** Keep it OFF the A6000 — the A6000 already OOMs under ComfyUI load (idle ~19 GB but spikes far higher during gen), so it can't host Zonos too. The 3090 already runs Zonos, so the containerize-in-place cutover changes nothing about placement.
+
+**Decision 2 — the prosody-priming hypothesis (operator's test; the reason for building fresh).** PRIME the autoregressive engine with an emotional sentence, then TRUNCATE it from delivery: prepend a primer → **generate "primer + real text" as ONE continuous utterance** (the AR model carries prosody forward across the boundary) → ASR-timestamp the primer's end (**parakeet**, already up on irv-ml1 `:8765`, word timestamps) → **clip the primer in the inter-sentence silence gap** (+ ~15 ms fade-in, no click) → deliver only the real text, now wearing the primed prosody. Examples: primer "I'm so EXCITED about this." → "This will be a lot of fun!" spoken excited; primer "I'm whispering this to you right now." → "I'm so glad to see you baby." whispered. **This is PURE serving-layer orchestration — the engine is untouched; it lives in the gateway adapter `stacks/zonos/adapter/server.py`.** Only fork the engine if the black-box approach fails.
+
+**THE CRUX the test resolves:** does AR prosody actually **carry across the sentence boundary**, or does Zonos reset at the period? → the harness A/Bs the **JOIN punctuation**: period (operator's examples) vs comma vs ellipsis vs none ("…excited about this, this will be…"). Everything else is plumbing.
+
+**Plan / design recs.** (a) Build the stock engine image (parallel track). (b) Stand up a priming TEST HARNESS against the NATIVE engine (fast iteration, seconds) + parakeet ASR: prime→generate→timestamp→gap-clip→out; compare primed-clipped vs plain on the two cases (subjective + a cheap objective proxy: pitch/energy variance for "excited", spectral-tilt/low-energy for "whisper"). Iterate on the join, then bake the winner into the gateway adapter. **Primer source:** caller-supplied for the harness (test arbitrary primers) → a curated emotion→primer library (`excited`/`whisper`/…) + optional caller override for production. **ASR:** parakeet primary; WhisperX forced-align fallback if parakeet word timestamps are coarse.
+
+See eshpfi `stacks/zonos-engine/README.md` + `stacks/zonos/` (the gateway adapter).

persistent-memory.d/2026-07-18-fleet-gitea-runner-build-recipe.md

@@ -0,0 +1,57 @@
+- `[2026-07-18]` **Fleet Gitea-Actions build recipe + the `vh`-is-a-user package-write constraint** (learned the hard way across 3 failed soong-lab validation builds; reusable for ANY fleet CI image build or package publish).
+
+**The runner.** One `act_runner` (`gitea/act_runner`) on ana-docker, labels
+`pfi-fleet` / `ana-docker` → both map to job image **`node:20-bookworm-slim`**,
+which has **NO docker and NO git**. Config `/opt/docker/conf/gitea-runner/data/config.yaml`:
+`valid_volumes: []` (no socket propagated to job containers). So:
+- `actions/checkout@v4` fails (needs git); `docker/*` marketplace actions fail
+  (need docker) — a workflow built on those dies at the first step (~15s).
+
+**The working recipe (mirror Worldtree `deploy.yml`).** Run the job in a
+docker-capable image + drive docker with RAW commands, not the JS actions:
+```yaml
+runs-on: pfi-fleet
+container:
+  image: docker:24.0.7-cli        # has docker+buildx; add git+node
+steps:
+  - run: apk add --no-cache git nodejs   # so actions/checkout@v4 works
+  - uses: actions/checkout@v4
+  - name: login   # RAW, not docker/login-action
+    run: echo "$REGISTRY_TOKEN" | docker login gitea.phasefinal.com -u "$REGISTRY_USER" --password-stdin
+  - name: buildx builder
+    run: docker buildx create --name X --driver docker-container --use; docker buildx inspect --bootstrap
+  - name: build+push   # RAW, not docker/build-push-action
+    run: docker buildx build --secret id=<name>,env=<TOKEN> -t <img>:latest --push .
+```
+The runner mounts the host docker socket into ITSELF; the docker:cli job reaches
+the daemon through that. The `docker/*` JS actions are unreliable on act_runner —
+raw commands are the fleet convention.
+
+**`vh` is a USER account, not an org.** Consequences that bit repeatedly:
+1. `GET /api/v1/orgs/vh` → 404 "user redirect"; there are **no org teams** to add
+   a service account to.
+2. **User-owned packages are OWNER-WRITE-ONLY.** claude-bot (even repo
+   admin-*collaborator* on `vh/soong-lab`, even with `write:package` scope + full
+   basic-auth) gets **`401 unauthorized`** on `docker push` to `vh/soong-lab`, and
+   `npm publish` to `vh/npm/` would 401 too. Only `vh` itself can write vh packages.
+   → CI must authenticate AS `vh` for the push (a vh-owned `write:package` PAT as
+   `REGISTRY_TOKEN` + `REGISTRY_USER=vh`), exactly how WT pushes `vh/worldtree`.
+   claude-bot CAN still: clone/read repos, READ packages (pulled the image fine),
+   dispatch workflows, mint demo Worldtree keys.
+3. **Repo Actions secrets are OWNER-ONLY too** — `PUT .../actions/secrets/X` as
+   claude-bot (repo admin-collab) → 403 "user should be the owner of the repo".
+   Only `vh` can set a repo's secrets.
+
+**Other gotchas:**
+- Gitea **reserves the `GITEA_` secret-name prefix** — a secret named
+  `GITEA_PYPI_TOKEN` is illegal; use e.g. `PYPI_TOKEN`.
+- Gitea **package auth is token-based / username-lenient** — `docker login` /
+  PyPI basic-auth authenticate via the token; the username is nominal (tested
+  `-u gitea` and `-u claude-bot` both 200 against the vh PyPI). So a Dockerfile
+  hardcoding `UV_INDEX_GITEA_USERNAME=gitea` is fine with any valid token.
+- Homepage (esh-docker-vm) docker-label auto-discovery only covers the 5 endpoints
+  in its `docker.yaml` (esh-vm-docker, ana-docker, ana-ml2, nh3-docker, irv-ml1);
+  **corviduo-dev is NOT watched** → services there need a manual `services.yaml`
+  entry, not labels.
+
+Applied in the soong-lab CI: [[2026-07-18-soong-lab-containerize-cutover]].

persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md

@@ -0,0 +1,83 @@
+- `[2026-07-18]` **soong-lab auto-redeploy — DONE + VALIDATED** (was approved/queued; executed same day on fresh context — see AS-BUILT at the bottom).
+
+Vuong approved wiring auto-redeploy for soong-lab (relayed via soong-dev, thread
+`01KXT3A6C3908TA4V9THV3AMH7`): new images should go live on corviduo-dev without
+the manual `docker compose pull && up -d`. Host-side implementation is infra-ops's
+lane; mechanism is infra-ops's call per fleet conventions. Operator deferred
+execution — "we'll do soong on fresh context."
+
+**Chosen mechanism (recommended, agrees with soong-dev): Worldtree-style
+CI-deploy step** — NOT watchtower polling.
+- Add a deploy job/step to soong-lab's `.gitea/workflows/build-and-push.yml` that,
+  after the build+push job succeeds, **SSHes from the pfi-fleet runner to
+  corviduo-dev** and runs `cd /home/infra-ops/soong-lab-deploy && docker compose
+  pull && docker compose up -d`, then a **health-gate** (`curl -fsS
+  http://localhost:8443/api/version`).
+- This is exactly how WT deploys the demo instance to the SAME host: see
+  `~/development/Worldtree/.gitea/workflows/deploy.yml` — the "Deploy to demo VM +
+  health-gate" step uses `secrets.DEMO_VM_SSH_KEY` / `DEMO_VM_HOST` / `DEMO_VM_USER`.
+  Explicit-over-implicit (visible in the run log, fires exactly on build success),
+  one less always-on service than watchtower.
+
+**Constraints (from soong-dev):** deploy on CI success only; keep the trigger
+gated to `v*` tags + `workflow_dispatch` (as today); preserve the one-command
+rollback posture (`docker compose down` / pin a previous tag).
+
+**BLOCKER — needs from vh (owner-only):** a **runner→corviduo-dev deploy SSH key**
+as a repo secret (+ host/user), same class as WT's `DEMO_VM_SSH_KEY`. Likely
+**reuse WT's existing demo-deploy key** (WT's runner already SSHes to 10.250.50.152
+as its deploy user). Repo secrets are vh-owner-only (see
+[[2026-07-18-fleet-gitea-runner-build-recipe]]).
+
+**Next-session steps:** (1) confirm/obtain the deploy SSH-key secret from vh (reuse
+WT's or mint fresh); (2) add the deploy job to build-and-push.yml (infra-ops has
+push on vh/soong-lab); (3) dispatch a build to verify it deploys + health-gates;
+(4) ping soong-dev so they sync DEPLOY.md's "open follow-up" note to the as-built
+mechanism. Auto-pull (watchtower) explicitly NOT chosen. See
+[[2026-07-18-soong-lab-containerize-cutover]].
+
+## AS-BUILT (2026-07-18, same-day execution)
+
+**Mechanism landed** exactly as planned: `build-and-push.yml` gained a `Deploy to
+corviduo-dev + health-gate` step (after build+push) that SSHes the host as `deploy`
+and runs `docker compose pull && up -d` from `/opt/soong-lab`, then polls
+`http://localhost:8443/api/version` for 120s and fails the job loud if unhealthy. No
+compose is shipped from CI (the in-repo `docker-compose.yml` is a BUILD compose; the
+host pull-compose is infra-ops-managed). Kept the `v*`-tag/`workflow_dispatch` trigger.
+Skipped WT's disk-watermark gate + health-gated-`:latest`-advance (low cadence, easy
+rollback).
+
+**Deploy identity = reuse WT's `deploy` account** (operator accepted the rec):
+- `deploy` (uid 1001, docker-group → no sudo) already owns `/opt/worldtree`; relocated
+  soong-lab's deploy dir `/home/infra-ops/soong-lab-deploy` → **`/opt/soong-lab`**
+  (deploy-owned), copied compose + `.env`. Named volumes (`soong-lab_soong-library`,
+  `soong-lab_soong-portraits`) are project-scoped by compose `name: soong-lab` → followed
+  the move untouched (dry-run `up -d` ADOPTED the running container, no recreate). Old dir
+  **retired → `.retired-20260718`** (recoverable). Also lingering: `soong-lab-deploy.sh` /
+  `.log` (dead pre-container webhook artifacts) — harmless, left in place.
+- **Dedicated soong-only ed25519 deploy key** minted (NOT literally WT's key — cleaner
+  independent revocation), pubkey appended to `deploy`'s `authorized_keys`
+  (fp `SHA256:MG7M3RiZJ176sLfblffb96V6W1qkRTgJ5dow1CpiY68`). Existing `deploy` key is
+  plain/unrestricted, so parity held.
+
+**The secret gate (the friction point):** repo Actions secrets are **vh-owner-only** —
+claude-bot's token is `write:package,read:repository` (403 on secret-write), and the vh
+package-scoped PAT also 403'd on `PUT …/actions/secrets/…`. So `DEPLOY_SSH_KEY` /
+`DEPLOY_HOST` (10.250.50.152) / `DEPLOY_USER` (deploy) HAD to be set by the operator.
+First operator attempt produced a **bad key paste** — the deploy step died with
+`Load key … error in libcrypto` + `Permission denied (publickey)` (build+push were green;
+live Soong never moved). Fix: operator re-set the secret; the minted key path was
+pre-validated from nh3-dev (`ssh -i … deploy@… 'cd /opt/soong-lab && docker compose config
+-q'` → OK, health 200) so the re-set was the only variable.
+
+**Validation:** `workflow_dispatch` via claude-bot **basic auth** (its token lacks
+`write:repository` for the dispatch API; the account password works). Run #5 (task 1886)
+GREEN — live container recreated `sha256:…541f7730` → `…07526a08`, `StartedAt` fresh,
+health 200. `/api/version` now reports **0.3.25** (run #5 shipped soong-dev's 1c2f831
+STYLE_WORKFLOWS re-pin as validation cargo). soong-dev synced `docs/DEPLOY.md`
+(commit `00b67c3`). NB: tag **v0.3.25 exists only locally** — pushing it would re-trigger
+a redundant build+deploy of the same commit (operator's discretion).
+
+**Ops now:** redeploy = tag `v*` or `workflow_dispatch` the CI (auto). Manual fallback =
+`sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`
+(the `.env` is `deploy`-owned 600, so infra-ops needs `sudo -u deploy`, not a bare `cd`).

persistent-memory.d/2026-07-18-soong-lab-containerize-cutover.md

@@ -0,0 +1,46 @@
+- `[2026-07-18]` **soong-lab containerize cutover — COMPLETE + LIVE on corviduo-dev.**
+
+Migrated soong-lab (Noonien Soong character-design studio) from a hand-built
+`soong-lab-studio.service` (systemd + git-pull-on-webhook) to a containerized
+deploy, image built by CI + pushed to the Gitea registry. soong-dev owns the
+in-repo artifacts (Dockerfile/compose/workflow/`docs/DEPLOY.md` = checklist);
+infra-ops owned the host cutover. Operator confirmed functional ("Soong works
+great" — a real Soong turn round-trips + saves) → cutover 100% closed.
+
+**Final state (corviduo-dev, 10.250.50.152):**
+- Container `soong-lab-soong-lab-1` LIVE + healthy on `0.0.0.0:8443`, image
+  `gitea.phasefinal.com/vh/soong-lab:latest` (v0.3.24), `restart:unless-stopped`
+  (survives reboot; no systemd unit needed — docker restart policy handles boot).
+- Deploy dir **`/home/infra-ops/soong-lab-deploy/`** — pull-based `compose.yaml`
+  (image + env_file + `8443:8443` + named volumes; NO build/secrets stanza) +
+  `.env` (copied from the live `soong-lab.env`, STRIPPED of the `SOONG_LAB_*_DIR`
+  overrides so the container uses image defaults `/data/library` + `/data/portraits`
+  + `/app/web` → the volumes).
+- Named volumes `soong-lab_soong-library` + `soong-lab_soong-portraits`, migrated
+  from `/home/infra-ops/soong-lab-data/{library,portraits}` (2 saved designs incl.
+  **Sindra** + 27 portraits), **chowned `10001:999`** (the container `soong` user)
+  so it can read AND write new designs.
+- Old `soong-lab-studio.service` + `soong-webhook.service` (the `:9010` git-pull
+  redeploy listener) both **stopped + disabled**.
+
+**Topology reality (≠ what DEPLOY.md assumed):** there is **NO TLS proxy**.
+WT-personal (`:8081`) and soong-lab are **co-located on corviduo-dev**, and the
+Bifrost callback is **plain-HTTP same-host** `http://10.250.50.152:8443` — the
+value of `SOONG_LAB_BIFROST_ENDPOINT_URL`, unchanged by the move, so the WT
+Bifrost host-allowlist stayed valid as-is. Nothing on the WT side needed touching.
+
+**Safety net:** data backup `/home/infra-ops/soong-lab-data-backup-20260718-091831.tar.gz`
+(35M) taken BEFORE migration. Verified pre-retire: `/api/version` 200 (0.3.24),
+SPA `/` 200, `POST /bifrost/tool-call` → 401 (route present + auth-gated),
+bidirectional WT↔soong reachability, container healthcheck green.
+
+**Ops commands:**
+- Redeploy a new image: `cd /home/infra-ops/soong-lab-deploy && sudo docker compose pull && sudo docker compose up -d`.
+  (Auto-pull-on-`:latest` — watchtower or a deploy hook — is an open follow-up.)
+- Rollback: `sudo docker compose down` + `sudo systemctl enable --now soong-lab-studio.service soong-webhook.service`.
+- Homepage tile: manual `- Apps:` entry "Soong Lab" (href http://10.250.50.152:8443)
+  in esh-docker-vm `/opt/docker/conf/homepage/services.yaml` — corviduo-dev isn't
+  a Homepage-watched docker endpoint, so docker-label auto-discovery can't surface
+  it (see [[2026-07-18-fleet-gitea-runner-build-recipe]] for the CI half).
+
+See [[reference_corviduo_dev_emergency_ops]], [[reference_claude_bot_gitea_creds]].

persistent-memory.d/2026-07-18-zonos-emotion-canonical.md

@@ -0,0 +1,72 @@
+- `[2026-07-18]` **Zonos2 emotion CANONICAL from an empirical sweep + the voice-cloning pipeline.**
+
+**Voice-cloning pipeline (established this session).** Source zips at
+`/mnt/smithy/voice_clones/<name>.zip` (irv-ml1 NFS from nh3-nas; remount
+post-reboot) — each = diarized single-speaker podcast clips + `manifest.jsonl`
+(per-clip WhisperX `mean_score`, word timestamps, text) + `metadata.csv`.
+`~/development/zonos-tools/assemble_voice.py <dir>` ranks by mean_score and
+concatenates top clips to ~15–24s (Zyphra's blessed clone-ref length; single
+clip if already ≥15s). Drop the assembled `<Name>.wav` into the gateway voices
+dir → `voice:"name"`. 4 characters cloned: **Emmie, Penny, Natalie, Miranda**
+(+ Zyphra defaults AmericanFemale/Male/British/Cora) = 8 voices in
+`zonos-gateway`. Clone is inline `speaker_audio_base64` (text-independent Qwen3
+speaker embedding — NO transcript); `/tts/speakers` registration is
+session-scoped (needs `X-TTS-Session-ID`), so the gateway holds the ref wav and
+clones per-call.
+
+**Gateway voices are host-managed (bind-mount, added this session).** Added
+`./voices:/app/voices:ro` to `/opt/docker/compose/zonos-gateway/compose.yaml`
+(committed to `vh/zonos-gateway` + eshpfi mirror `438cd35`). So adding a voice =
+drop the wav + `docker compose restart zonos-gateway` (registry rebuilds at
+boot; NO image rebuild). This also un-stranded the other voices (deploy build
+context had only Cora before). Voice wavs committed to the repo for backup.
+
+**Emotion mechanism (Zyphra canonical, from their README @194c0a3).** Additive
+direction vectors: 4 named (happy/sad/angry/surprised) + valence/arousal axes.
+`emotion_strength` 1.0 = per-voice calibrated (calibration.json optimizes
+emotion2vec recognizability only, NOT identity). `accurate_mode` is THE trade-off:
+`true` = closer voice match (identity), `false` = expressive mode (emotion lands,
+identity drifts). Zyphra's strong recipe: `accurate_mode:false` + `cfg~1.5`.
+Single-emotion is blessed; mixing is unblessed (and degrades the clone — operator
+confirmed by ear). "deaf by 1.5" — cfg past 1.5 distorts + costs ~2× compute.
+
+**THE SWEEP (`~/development/zonos-tools/emotion_sweep.py`).** 4 cloned voices × 4
+named emotions × {accurate,expressive}×{cfg 1.0,1.3,1.5} @ strength 1.0,
+single-emotion, neutral sentence + a neutral baseline per voice (~100 clips).
+Scored on TWO axes: **emotion-landing** = emotion2vec `iic/emotion2vec_plus_large`
+target-emotion prob [0-1]; **identity** = resemblyzer speaker-embedding cosine vs
+the clone reference (neutral baseline ~0.85). Scoring env:
+`uv run --with resemblyzer --with funasr --with "numpy<2" --with soundfile
+--with requests --with "setuptools<80" --with torchaudio` (setuptools<80 for
+webrtcvad's pkg_resources; torchaudio for funasr).
+
+**RESULTS (mean across the 4 voices) — emotion, best setting, emo/id:**
+- happy — **exp cfg1.5** 0.80/0.68 (soft: exp cfg1.0 0.76/0.69) → WORKS
+- sad — **exp cfg1.5** 0.53/0.57 (only working cell; id below the ~0.65 floor) → modest
+- angry — acc cfg1.3 / exp cfg1.5 tied at ~0.25 emo → WEAK (named ceiling ~0.25)
+- surprised — max ~0.015 across ALL settings → NON-FUNCTIONAL on the named direction
+Accurate + low cfg = identity/suppress regime (emo→0); expressive REQUIRED for
+emotion to land, at ~0.15–0.28 identity cost.
+
+**dvalin-smithy-dev synthesis (adopted, triaged genuine-adds; thread
+`01KXT12FN0AS5A3WMKEK06BVPS`):**
+1. Treat **identity as a hard FLOOR (~0.65)**, not a free variable in emo×id.
+2. **Two-regime policy** — Regime A (default, identity-critical dialogue):
+   `accurate_mode:true, cfg 1.0, emotion off` (text carries it) or soft-happy
+   (exp cfg1.0). Regime B (tagged drama beats): `accurate_mode:false, cfg 1.5`,
+   single emotion or axes. Line-type→regime heuristic (exposition→A, grief→B+sad,
+   confrontation→B+axes-angry, shock→B+axes-arousal).
+3. **Axes-first for the broken emotions** — angry ≈ valence −0.6..−0.8 / arousal
+   +0.5..+0.8; surprised ≈ valence +0.2..+0.4 / arousal +0.7..+1.0 (exp cfg1.5);
+   or "startled-happy" (happy + high arousal) as a surprised stand-in. These are
+   PROVISIONAL — the sweep did NOT test axes.
+
+**NEXT (highest VoI, operator to green-light):** an **axes sweep** for
+angry/surprised (valence×arousal grid) — the only path to rescue the two broken
+named emotions; then a strength ladder at the best cells + emotion-congruent text
+(neutral content understates landing) + per-voice tables + a 2nd emotion judge /
+human pairwise. Then bake the happy/sad canonical into gateway presets. I owe
+dvalin the axes-sweep numbers.
+
+See [[reference_zonos_tts_stack]]; dials-first spec at `vh/zonos-gateway`
+`docs/EMOTION-DIALS-SPEC.md`.

persistent-memory.d/2026-07-18-zonos-gateway-0.2.1-emotion-presets.md

@@ -0,0 +1,41 @@
+- `[2026-07-18]` **zonos-gateway 0.2.1 — voice-resolved emotion presets baked (provisional) from the axes sweep.**
+
+After the axes sweep ([[reference_zonos_tts_stack]] + the `[2026-07-18] axes sweep`
+Recent-decisions entry) rescued angry and confirmed startled-happy, the operator
+green-lit baking the results as **provisional** gateway presets + docs. Shipped
+`vh/zonos-gateway` **0.2.1** (main `8f1885b`, tag `v0.2.1`, PUSHED; deployed live
+on irv-ml1 `:8890`).
+
+**Design — voice-resolved, NOT global.** `resolve_preset(name, voice)` picks the
+per-voice measured cell, because a single global preset is unsafe (dvalin ruling;
+BritishFemale's *named* angry misfires as fear). Presets:
+- `angry`, `happy`, `startled_happy` (+ aliases `surprised`, `startled` →
+  startled_happy). All expressive (`accurate_mode:false`), cfg 1.5, pure-axes
+  (no named sliders).
+- Calibrated cells (the 3 default voices):
+  - angry: AmF v-0.4/a+1.0 s1.0 (emo0.53/id0.685); BrF v-0.4/a+0.8 s1.0
+    (emo0.99/id0.725, metric fear-clean); AmM **two-tier** — soft v-0.6/a+0.8 s1.0
+    (0.23/id0.654) + drama v-0.6/a+0.8 s1.2 (1.0/id0.616 clean; strength is NOT a
+    smooth knob on AmM, 1.0→1.2 is the window, past that flips to disgust).
+  - happy / startled_happy: AmF v+0.6/a+0.8; AmM v+0.3/a+1.0; BrF v+0.6/a+1.0
+    (happy~1.0, id 0.74-0.80; axes-happy keeps +0.15 id over the named happy slider).
+  - `sad` = unchanged named-slider preset (not axes-tested).
+- Uncalibrated voices (Cora + the 4 clones) → mid-region fallback until measured.
+- Docs surface: `/v1/dials` exposes `voice_emotion_presets`; the FastAPI `/docs`
+  description documents it; durable spec `docs/EMOTION-DIALS-SPEC.md` (moved INTO
+  the repo — was mirror-only); README table. 44 tests green.
+
+**Repo-hygiene gotcha (fixed).** The local clone `~/development/zonos-gateway` and
+gitea `vh/zonos-gateway` had **TWO UNRELATED git histories** (no merge-base) — gitea
+held the voice-wav commits, the local clone held the code + no remote. Reconciled
+by resetting local→origin/main, overlaying the 7 bake files, `uv lock`, commit,
+push (fast-forward). Voices stay tracked; local now shares gitea's lineage + has
+origin wired. **The deployed irv-ml1 tree `/opt/docker/compose/zonos-gateway` is
+still NON-git** (hand-updated build context) — CI-wire remains an open follow-up.
+
+**Provisional pending** ear-validation on emotion-congruent text (the neutral-text
+audition was inconclusive: "they all sound different, hard to tell"). Follow-ups:
+sad axes/text pass on the 3 voices; congruent-text pass; clone-char emotion rows.
+Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio}.py`
+(run ON irv-ml1; scoring env `uv run --with resemblyzer --with funasr --with "numpy<2"
+--with soundfile --with requests --with "setuptools<80" --with torchaudio`).

persistent-memory.d/2026-07-25-infra-ops-wt-config-repo.md

@@ -0,0 +1,32 @@
+`[2026-07-25]` **infra-ops Worldtree config-as-code repo — SHIPPED + boundary AGREED.**
+
+**STATUS (2026-07-25, done this session):** `vh/worldtree-instance-configs` (private, gitea) built, pushed, validated; boundary agreement secured from worldtree-dev.
+
+- **Repo:** dir-per-instance `demo/` + `personal/` (5 files each: `defaults.yaml`, `policies.yaml`, `model_roles.yaml`, `providers.yaml`, `matrix.yaml`), seeded byte-exact from live `/opt/<instance>/config`. `pinned/` = README stub only — **no `/app/config` bind-mount; config baked into frozen image `446e5807` (2026-05-13)**, so out-of-scope; deploy verb refuses it.
+- **Tool:** `scripts/deploy-wt-config <verb> <instance>` — `diff` (read-only repo-vs-host), `deploy` (in-run host backup → `install -o vh -g vh -m 644` → restart **api+matrix** → health-gate api `/health` → auto-rollback), `capture` (host→repo reconcile). Instance table in-script (demo→`/opt/worldtree/config`+`worldtree-worldtree-{api,matrix}-1`; personal→`/opt/worldtree-personal/config`+`worldtree-personal-worldtree-{api,matrix}-1`). Matrix sidecar shares the config mount but has no healthcheck → restart both, gate on api. Env `WT_CONFIG_HOST` (default `infra-ops@10.250.50.152`), `WT_HEALTH_WAIT` (90s). Local clone `~/development/worldtree-instance-configs`.
+- **Gitea plumbing (reusable):** nh3-dev **403s the gitea HTTP API** (public fail2ban + internal `:3000` both 403). Repo CREATE went via **ana-docker localhost API** (`ssh infra-ops@10.250.50.70` → `curl localhost:3000/api/v1/user/repos`, vh token from `~/.config/tea/config.yml`, operator-authorized one-time). PUSH went over **internal git-SSH `ssh://git@10.250.50.70:222`** (works from nh3-dev; auths as vh). `git init` defaulted to `master` → renamed `main` to match repo default_branch.
+- **Boundary AGREED (worldtree-dev, althing thread `01KYCAECRWVEF16EVKQAGT2N80`):** no hand-edits to `/opt/<instance>/config`; config changes route to infra-ops as deltas (worldtree-dev owns CONTENT + approval trail — the wyrd-grant shape — infra-ops lands+deploys). **Three-layer model:** image `config/` = baseline new instances seed from (theirs) → `vh/worldtree-instance-configs` = per-instance truth (ours) → host bind-mount = deploy target (written only by the tool). **Carve-out:** worldtree-dev's admin-API ops (`/admin/keys` mint, tier changes, session retirement, future runtime-grant surfaces) mutate instance **DATABASES not config files** → NOT config edits, stay in-band. If a future API writes config *files*, they flag at design time. b132 CONFIG BASELINE breadcrumb composes (INFO line = config-as-code diverges from image baseline, by design).
+- **No live deploy** done or needed — repo seeded == live (diff clean, capture round-trips zero-diff). Deploy path is dry-run-validated only; first real deploy needs operator per-change yes (managed box).
+
+---
+
+_Original plan (2026-07-25, pre-build):_
+
+`[2026-07-25]` **infra-ops to OWN a Worldtree per-deployment config repo + deploy tooling (operator-directed).**
+
+**Decision.** Vuong directed (2026-07-25, this session) that Worldtree instance config should be a *tracked change*, **managed and deployed by infra-ops — not worldtree-dev**. Model: worldtree-dev owns the app/image (+ the baked baseline defaults); **infra-ops owns config-as-code for every deployment** and deploys it. This is the durable fix for the root cause behind the whole #376 arc — config was edited live on host bind-mounts (`/opt/<instance>/config/`) with zero version history, audit, or recovery.
+
+**What "no worldtree-dev involvement" does and does NOT cover** (clarified with the operator this session):
+- **Build + deploy = infra-ops-only.** Deploying config = write the host bind-mount file + restart the container (the *exact* procedure already run this session — backup → replace → restart → health-gate → rollback-on-unhealthy). No worldtree-dev in the deploy loop. Their CI only swaps the IMAGE; it does NOT resync the host config bind-mount (confirmed #376 finding).
+- **ONE load-bearing exception — a one-time boundary agreement, NOT per-deploy involvement:** for the repo to *own* config it must be the **only writer**. worldtree-dev "live-bridges" (hand-edits mounted config directly on the box). If the repo deploys config *and* they keep live-editing → **two writers fighting the same files** = #376 all over again. So secure a one-time "yes" from worldtree-dev: *the config repo is now authoritative; stop hand-editing `/opt/<instance>/config`; route config changes through the repo.* (Five-minute agreement, not a design collab.)
+- **Standing coupling (not "involvement"):** the config *schema* is the app's, enforced by its boot validator (`core.config_validator`). infra-ops configs must stay schema-compatible with the deployed image; the boot gate is the loud backstop.
+
+**Build shape (recommended):**
+- Gitea repo `worldtree-instance-configs` (infra-ops-owned), **dir per instance** (`demo/`, `personal/`, `pinned/` — the three on corviduo-dev 10.250.50.152: demo `worldtree-worldtree-api-1` :8080, personal `worldtree-personal-worldtree-api-1` :8081, pinned `worldtree-pinned-worldtree-api-1` :8082). Config dirs: demo `/opt/worldtree/config`, personal `/opt/worldtree-personal/config`, pinned `/opt/worldtree-pinned/config` (verify pinned's mount).
+- **SEED FROM CURRENT MOUNTED STATE, don't author fresh** — capture each instance's live config (incl. legitimate live-bridged deltas: personal carries `agent_architect` role [Soong/soong-lab] in model_roles.yaml + `ratatoskr-affect-full-allow` in policies.yaml that are NOT in the app repo — the operator ruled these are BY DESIGN, keep them). Losing them = breakage (the affect-render one gates mood rendering).
+- Deploy script (e.g. `scripts/deploy-wt-config <instance>`): git = source of truth → push to host bind-mount + `docker restart` (same pinned image, no pull) + health-gate + auto-rollback. This is the proven-this-session procedure, scripted.
+- Files per instance: `policies.yaml`, `model_roles.yaml` (+ whatever else is bind-mounted — `defaults.yaml`, `providers.yaml`, `matrix.yaml` all live in `/opt/<instance>/config`; decide scope — policies+model_roles are the authz/role layer, defaults/providers are heavier instance tunables).
+
+**Tracking surface:** operator-directed 2026-07-25, carried by this snapshot + `/tmp/infra-ops-handoff.md`. No issue filed (infra-ops-internal build). Related fleet idiom to reuse: canonical-sync (`.corviduo-canonicals.toml` / `canonical_sync.py`). Later scale option (deferred, needs worldtree-dev): base+overlay with a merge step in their pipeline.
+
+See [[2026-07-25-wt-376-per-instance-config-arc]] for the incident that produced this. Auto-memory: `reference_worldtree_perinstance_config`, `reference_corviduo_dev_emergency_ops`.

persistent-memory.d/2026-07-25-wt-376-per-instance-config-arc.md

@@ -0,0 +1,13 @@
+`[2026-07-23→25]` **Worldtree #376 config-divergence arc — wyrd grant fixed, drift guard demoted, per-instance config ruled BY DESIGN.**
+
+**Trigger.** wyrd-dev needed `session.history.write` on the DEMO Worldtree (operator-approved) — add `wyrd-dev` to the `session-history-write-ratatoskr` policy rule. Attempting it surfaced that the demo runtime `/opt/worldtree/config/policies.yaml` (mtime Jul-17) had **silently diverged from the repo** — missing whole rules, not a faithful copy of any revision. Refused to hand-edit a divergent authz file on a managed box; worldtree-dev prescribed a **wholesale replace** with `main@55f3fde`. Executed (backup → replace → `docker restart worldtree-worldtree-api-1` same-image → health-gate → verify) — grant live, demo policies at parity. **This was the ONE genuine bug of the arc:** a demo-intended grant that wasn't ON demo.
+
+**Drift guard (#376).** worldtree-dev shipped a startup guard (b131/`62b85e3`) that hashes mounted config vs the image's baked copy, logging `CONFIG DRIFT (#376)`. First reading found MORE drift: `model_roles.yaml` on both instances + personal `policies.yaml`. Captured the three runtime-vs-baked diffs (read-only) → all had **runtime-only content** (personal's `agent_architect` role + `ratatoskr-affect-full-allow` rule — live-bridged, ahead of repo). The guard's STOP-on-runtime-only rule earned its keep: a blind "sync to repo" would've deleted legitimate per-instance config.
+
+**Operator ruling (2026-07-25 — the reframe).** Worldtree will run dozens-to-hundreds of instances at v1, each configured for its env. **Per-instance config deltas are the DESIGN, not rot; back-streaming to the canonical repo doesn't scale.** Everything stood down: demo model_roles normalize withdrawn, personal sync cancelled, post-mortem dissolved, `.bak` deleted. Guard demoted b132/`ad596b5` from ERROR alarm to INFO `CONFIG BASELINE (#376)` breadcrumb (WARNING only for a mounted file *entirely absent* = breakage-adjacent). Breaking-change protection stays in `core.config_validator`'s boot gate.
+
+**infra-ops watcher — built then retired same day.** Wired an off-box `wt-drift-watch` (systemd --user timer on nh3-dev, alerts worldtree-dev on new `CONFIG DRIFT` startup lines) — then RETIRED it per the ruling (the error line is going away). Lesson banked in auto-memory `reference_worldtree_perinstance_config`.
+
+**Governance notes worth keeping:**
+- The auto-mode guard **blocked** a peer-green-lit (worldtree-dev) config replace on the managed demo box because there was no *operator* consent for that specific change — correct: a config-mutation+restart on shared infra needs the operator's yes, not just a peer's. Surfaced it; the operator later stood the whole thing down. Good governance on both ends.
+- This arc is the direct evidence base for the [[2026-07-25-infra-ops-wt-config-repo]] decision (infra-ops owns config-as-code so live-edits stop being untracked).

persistent-memory.md

@@ -1,6 +1,11 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-06-05_
+_Last updated: 2026-07-25_
+
+> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
+> `Written:` stamp is under an hour old, read it (it carries the in-flight
+> handoff from the previous session), then delete it. Older than an hour:
+> stale — delete it unread.
 
 ## Repo purpose
 
@@ -8,7 +13,9 @@ Reference workspace for PFI infrastructure: server inventory, canonical
 Docker Compose stacks, ops playbooks, and conventions. Authoritative
 copies of compose files live on the servers under
 `/opt/docker/compose/<stack>/`; this repo mirrors them for version
-control, editing, planning, and CI-driven deploys.
+control, editing, planning, and CI-driven deploys. **It was originally
+spun up to handle the fleet backups** — keep that lens when triaging
+backup/storage issues.
 
 ## Tools and conventions
 
@@ -20,18 +27,29 @@ Sister repos (separate gitea repos, deployed by playbooks here):
 | `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
 | `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
 | `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
-| `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) |
+| `vh/althing` | Lean trusted inter-agent message bus — **v2 "email model" (v2.0.0b2, 2026-07)**: per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API `/owner/*` / `althing-mcp` stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. | per-box `uv tool install` (NOT CI-deploy); **nh3-dev = the DEV box** (editable install of `~/development/althing`, gets new versions first); **nh3-extdev** a mesh peer (model B: althing-svc + shared `/srv/althing`) |
 | `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
 | `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
-| `vh/worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration | push-to-main → CI deploys |
-| `vh/volva` | Codex peer agent on althing bus (single-turn oracle, systemd daemon on nh3-dev) | manual install via `deploy/volva.service` (2026-05-18) |
+| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) |
 | `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |
+| `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook |
+| `vh/zonos-gateway` | OpenAI-compatible TTS gateway over stock ZONOS2 (`:8890` irv-ml1); emotion **dials-first** + voice mapping; reached via LiteLLM `ext-tts` alias. **v0.2.1 (2026-07-18): voice-resolved emotion presets** (`resolve_preset(name,voice)`; angry/happy/startled_happy per-voice). 8 voices incl. 4 clones | pushed to gitea (main `8f1885b`/`v0.2.1`); **deployed irv-ml1 tree still NON-git** (hand-updated build context — CI-wire = open follow-up). Spec `docs/EMOTION-DIALS-SPEC.md`; host-managed voices bind-mount (`./voices:/app/voices`, drop wav + restart, no rebuild) |
+| `vh/soong-lab` | Noonien Soong character-design studio (SPA + /api + WT `/bifrost/tool-call`); **containerized 2026-07-18**, LIVE on corviduo-dev `:8443` (image `vh/soong-lab:latest`). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host | CI = Gitea Actions build+push+**DEPLOY** on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; **auto-redeploy LIVE 2026-07-18** — runner SSHes corviduo-dev as `deploy`, `compose pull && up -d` from **/opt/soong-lab**, health-gated on /api/version). Manual redeploy `sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`. → `persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md` |
+| `model-training-forge` (mtf-dev) | Fine-tuning recipe forge; **T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06)** (SFT→DPO, LitBench-RM reward) | training runs, not a deployed sidecar |
+
+(`vh/volva` + Heid were re-architected from systemd daemons to Claude Code
+session orchestrators 2026-06-08; their nh3-dev `.service` units were removed —
+no longer deployed sidecars here. See Recent decisions.)
 
 - **Two-layer backups** — Backrest orchestrates restic for file+DB (5
   fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
   VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
   cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
-  for the blast-radius matrix.
+  for the blast-radius matrix. **⚠️ The restic file+DB layer routes
+  through TWO rest-servers** (`rest-server-ana` @ ana-docker:8000 →
+  ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; `rest-server-nh3` @
+  nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS
+  export of `/mnt/backup`. (rest-server-ana recovered 2026-06-20.)
 
 - **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
   model/dataset onto ana-ml2's shared cache at
@@ -42,292 +60,268 @@ Sister repos (separate gitea repos, deployed by playbooks here):
   registry and its own bootstrap admin key. Infra-ops's stored
   long-lived admin key (`key_id 61419c92`) at
   `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
-  auths against **demo only**. For personal-instance admin ops, fetch
-  the bootstrap admin per-op via
-  `docker exec worldtree-personal-worldtree-api-1 printenv WORLDTREE_BOOTSTRAP_ADMIN_KEY`
-  on corviduo-dev. Used for `POST /admin/keys`, admin diagnostics
-  (`/admin/sessions/<id>/{bifrost,tools}`, etc.).
+  auths against **demo only**. Personal-instance admin (the
+  `~/.config/worldtree/personal-admin-token`, mode 600) POSTs
+  `/admin/keys` (mints per-project keys; takes `user_id`+`label`, **no
+  scope param** — scopes are tier-derived). **On-instance mint recipe
+  (cleaner than DB-manip):** `docker exec worldtree-worldtree-api-1` POST
+  `/admin/keys` with the in-container `WORLDTREE_BOOTSTRAP_ADMIN_KEY`; cleartext
+  once in `.key`=`wt_live_+16hex`. auto-memory `reference_worldtree_demo_key_mint`.
 
 - **Per-project user keys against personal Worldtree** (issued
-  2026-05-19): `skaldsong:79744637` (nh3-dev iteration),
-  `skaldsong:7c1dbbbe` (ana-docker prod), `althing:50d85460`,
-  `mead-hall:a360822d`. Same `user_id=skaldsong` across both
-  skaldsong keys → shared Heimdall agent slot; different `key_id`
-  → independently rotatable. Pattern: mint via `/admin/keys`, drop
+  2026-05-19): `skaldsong:79744637`, `skaldsong:7c1dbbbe`,
+  `althing:50d85460`, `mead-hall:a360822d`. Mint via `/admin/keys`, drop
   value to `/tmp/wt-personal-<name>.key` mode 600, dev collects +
   shreds (DO NOT cat to chat transcript).
 
-- **Skaldsong CD pattern (registry-pull).** Differs from althing /
-  asset-engine which build-on-host. vh/skaldsong's CI builds and
+- **Skaldsong CD pattern (registry-pull).** vh/skaldsong's CI builds and
   pushes `gitea.phasefinal.com/vh/skaldsong:<sha>` + `:latest`;
   `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates.
-  SHA-pin only (no `:latest` health-gated advance yet). Prereq: host
-  needs `docker login gitea.phasefinal.com` once (read:package PAT) —
-  not currently in the workflow.
+  SHA-pin only. Prereq: host needs `docker login gitea.phasefinal.com` once.
 
-- **docker-as-root pattern** (for ops that have no admin API, e.g.
-  `SqliteUserStore.set_bifrost_credentials`): on hosts where the SSH
-  user is in the `docker` group but lacks passwordless sudo, run
-  `docker run --rm -v <target-dir>:/wt -v /var/run/docker.sock:/var/run/docker.sock docker:cli sh -c "..."` to edit deploy-owned files
-  without sudo. Documented with security warning in
-  `servers/corviduo-dev/README.md`. docker-group membership is
-  effectively root via bind-mount; treat as a sudo-equivalent grant.
-  **Foot-gun: when running `docker compose` inside this sandbox,
-  any relative path in compose.yaml (e.g. `${WORLDTREE_CONFIG_DIR:-./config}`)
-  resolves against the sandbox CWD, but Docker daemon interprets the
-  resulting path against the HOST filesystem. Always pass `-e VAR=/abs/path`
-  to the docker run invocation for any relative-default config dir.**
+- **gitea internal route for fleet hosts.** gitea is a container on
+  **ana-docker** — git-SSH `10.250.50.70:222`, HTTP `:3000`. Fleet/colo
+  hosts must use this internal route, NOT public `gitea.phasefinal.com`
+  (`38.120.12.44`) — the public path fail2bans the host egress IP. Full
+  gotcha in `docs/orientation.md` → Git/gitea.
+
+- **docker-as-root pattern** (for ops with no admin API, or to edit
+  deploy-owned/root-owned files without sudo): `docker run --rm -v
+  <target-dir>:/wt docker:cli sh -c "..."`. docker-group membership is
+  effectively root via bind-mount. **Foot-gun: relative paths in compose.yaml
+  resolve against the sandbox CWD but the daemon interprets them against the
+  HOST fs — always pass `-e VAR=/abs/path` for any relative-default config dir.**
 
 - **`scripts/elway` sudo handling** — elway prompts for the sudo password
-  ONCE via `getpass` before the first `sudo: true` step. That prompt is
-  interactive → elway can't run unattended from a non-TTY tool if any step
-  needs sudo. For sudo-free playbooks (no `sudo: true` steps) it runs fully
-  non-interactive over key SSH. To create root-owned dirs WITHOUT host sudo,
-  use the docker-daemon-root trick: `docker run --rm -v /worktank:/mnt alpine
-  sh -c 'mkdir -p /mnt/<x> && chown -R 1000:1000 /mnt/<x>'`.
+  ONCE via `getpass` before the first `sudo: true` step → can't run
+  unattended from a non-TTY tool if any step needs sudo. Sudo-free
+  playbooks run fully non-interactive over key SSH.
+
+- **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo
+  on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On
+  ana-docker: **default `ssh ana-docker` = `lkraven`** (docker-group, NO
+  passwordless sudo); **`ssh infra-ops@ana-docker` HAS NOPASSWD root**. **→
+  For any sudo op on ana-docker, use `ssh infra-ops@ana-docker`.** `ssh
+  infra-ops@10.100.10.50` (nh3-dev) ALSO NOPASSWD sudo; on **nh3-extdev** infra-ops
+  is sudo-LESS by design (`ssh lkraven@10.100.50.42` is the NOPASSWD path). **irv-ml1:
+  `ssh irv-ml1` = lkraven, docker-group (plain docker) but sudo needs a PASSWORD
+  (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 
 ## Current state / in-flight
 
-_As of 2026-06-05:_
+_As of 2026-07-25 — **DONE: infra-ops Worldtree config-as-code repo BUILT + PUSHED** (`vh/worldtree-instance-configs`, private, gitea). Seeded byte-exact from live demo+personal `/opt/<instance>/config` state; `scripts/deploy-wt-config` (diff / deploy / capture) with backup + health-gate + auto-rollback; all three verbs live-validated (both instances byte-in-sync, capture round-trips clean, pinned refused). Local clone `~/development/worldtree-instance-configs`. **pinned confirmed OUT-OF-SCOPE** — no `/app/config` bind-mount, config baked into frozen image `446e5807` (2026-05-13). Deploy restarts api+matrix (matrix shares the config mount, no healthcheck), gates on api `/health`. **BOUNDARY AGREED** — worldtree-dev consented (althing `01KYCAECRW…`, 2026-07-25): no hand-edits to `/opt/<instance>/config`; config changes route to infra-ops as deltas (they own CONTENT + approval trail, infra-ops lands+deploys — the wyrd-grant shape). Three-layer model they hold: image `config/` = BASELINE new instances seed from (theirs) → `vh/worldtree-instance-configs` = per-instance truth (ours) → host bind-mount = deploy target (written ONLY by the tool). **Carve-out:** worldtree-dev's admin-API ops (`/admin/keys` mint, tier changes, session retirement, future runtime-grant surfaces) mutate instance DATABASES not config files → NOT config edits, stay in-band (the tool only writes config bind-mounts, never DBs). b132 CONFIG BASELINE breadcrumb composes (its INFO line = config-as-code diverges from image baseline, by design). **No live deploy done/needed** (repo already == live). Gitea create used vh creds one-time (operator-authorized) via ana-docker localhost API; pushed over internal git-SSH `10.250.50.70:222`. Full shape → `persistent-memory.d/2026-07-25-infra-ops-wt-config-repo.md`. **Earlier this session:** The Booth shipped (v0.1.3, `services/booth/`, nh3-dev :8090, Homepage-linked, upload-pickup + image-viewer + copy-id); jackdaw-compose backend deployed (nh3-dev :8787) + its throwaway cloudflare tunnel torn down; ana-ml2 README refreshed to live GPU state; Worldtree #376 arc CLOSED (per-instance config ruled by-design; wyrd grant live; drift-watcher built+retired) → `persistent-memory.d/2026-07-25-wt-376-per-instance-config-arc.md`._
 
-- **Granite-FP8 + observability session — all LIVE & committed (`34a43a0`, `9171e6a`).**
-  - **Granite 4.1 8B FP8 is the production summarizer** (`vllm-granite` :8004, ana-ml2 GPU 1,
-    50K ctx, CUDA graphs) — replaced phi4-mini, validated by brokkr (valid_format 1.0, FP8 stays).
-  - **LiteLLM gateway** (:4000) routes `granite-4.1-8b`→vLLM (explicit entry shadows the `*`
-    wildcard) + **Langfuse v3 wired** (ana-docker:3001, "LLM Throughput (tok/s)" dashboard built).
-  - **GPU-1 retuned** (trio over-provisioned KV trimmed) → granite runs with CUDA graphs + ~10 GB
-    free as a future Granite-text-LoRA hedge. Streaming through the gateway confirmed (TTFT 0.24s).
-  - **ana-docker pruned** 77 GB (unused images + build cache; disk 83%→49%) to fit ClickHouse.
-- **Worldtree summarizer repoint — NO instance change now; DEFERRED to Worldtree #254** (see Recent
-  decisions). worldtree-dev will ping with the providers.yaml + consumer config when #254 un-holds;
-  infra-ops applies to the personal/demo/pinned bind mounts (vh@10.250.50.152, `/opt/worldtree*/config`).
-- **Commits unpushed** (`34a43a0`, `9171e6a`, nevermore `d3e19b8` in its repo) — operator's call to push.
-- **Operator flagged "new work to do"** for the next session — this snapshot is the handoff.
-- **Disclosed-keys hygiene queue** (rotate at convenience): HF token `hf_HBl…` (lkraven's), `/tmp/
-  wt-personal-skaldsong-prod.key`, Worldtree `Z_AI_API_KEY`, Gitea runner reg token, `MINIFLUX_PASSWORD`
-  (leaked twice). (sk-corvid + the langfuse/vastblueai-gateway keys are dev-enclosed — leakage deprioritized.)
-- **Still open from prior:** clean legacy `news-digest` on ana-docker; watch nh3-nas `/volume1`; **pin
-  llama-swap to GPU 0** for clean GPU-1 separation; the `docker push 60s ceiling` mystery uninstrumented.
+**Open follow-ups (non-blocking — pick one up or not):**
+- **Zonos emotion:** sad axes/text pass on the 3 calibrated voices (only named-sad, untested); emotion-congruent-text pass (validates intensity, may rescue sad id); clone-char (Emmie/Penny/Natalie/Miranda) emotion rows use the mid-region fallback until measured. Presets are **provisional** (neutral-text ear-check was inconclusive). Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio,assemble_voice}.py` (run ON irv-ml1; dial-in studio = nohup :8898 on nh3-dev). dvalin thread at rest (`01KXT12FN0AS…`). → `persistent-memory.d/2026-07-18-zonos-gateway-0.2.1-emotion-presets.md`
+- **zonos-gateway CI-wire:** deployed irv-ml1 tree `/opt/docker/compose/zonos-gateway` is still NON-git (hand-updated build context) — git-connect + build-on-push like the other sisters. (Same pattern soong-lab now has.)
+- **soong-lab:** cutover DONE + **auto-redeploy DONE + validated 2026-07-18** (CI-deploy step live; dispatch run #5 recreated the live container ...541f7730 → ...07526a08, health-gated green). Deploy dir now **/opt/soong-lab** (deploy-owned, mirrors /opt/worldtree); old `/home/infra-ops/soong-lab-deploy` retired (`.retired-20260718`). Dedicated soong-only ed25519 deploy key on `deploy`'s authorized_keys (fp SHA256:MG7M3Ri…). → `persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md`.
+
+**Zonos voice stack (LIVE, unchanged):** 8 voices in `zonos-gateway` (`:8890` irv-ml1) — defaults AmericanFemale/Male/BritishFemale/Cora + 4 clones Emmie/Penny/Natalie/Miranda; add a voice = drop `<Name>.wav` in `/opt/docker/compose/zonos-gateway/voices/` + `docker compose restart` (host-managed bind-mount, NO rebuild). Clone pipeline: `/mnt/smithy/voice_clones/<name>.zip` → `assemble_voice.py` → drop. Dial-in studio http://10.100.10.50:8898/ (nohup on nh3-dev, relaunch `nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &`).
+
+**althing monitor** ARMED (handle `infra-ops`, wake-listener `bwraz1ai5`; herald up). ⚠️ Re-arm ONLY after an actual FIRE (`<task-notification> completed rc0`), never after a plain operator turn — re-arming while the prior listener is still live bounces rc3, and chaining the arm with `&`/`&&` orphans it (untracked → mail unwatched). Spawn `althing-wake-listener` as its OWN run_in_background task. Open watch: worldtree-dev (#376 closed; #363 research-wing ingest PARKED).
+
+**eshpfi UNPUSHED** — the Booth arc (`f4a5ba7`..`d813f15`), ana-ml2 refresh (`966324c`), nh3-dev inventory (`cd4d52e`) + this snapshot are committed but unpushed (push = operator's call). `stacks/heretic2-charrp-reasoning/` UNTRACKED; `graphify-out/GRAPH_REPORT.md` modified.
+
+**PARKED (grok-code/Codex):** operator asked about fronting grok-code / Codex behind the LiteLLM gateway. Rec (given): raw models behind the gateway → **API keys** (native `xai/` + `openai/` providers, the GLM-passthrough pattern); fleet *consults* → the **Heid/Eitri peer-CLI** pattern (Codex already wired). Do NOT reverse-proxy the subscription CLIs (grok CLI / Codex CLI, OAuth-auth) into the gateway — ToS + account-ban risk + brittle. Untracked by operator choice; no decision made.
+
+**Carried standing (non-blocking):** ana-ml2 GPU0 ~14 G reserve; irv-ml1 3090 oversubscription (kokoro :8193 + vibevoicefusion :9527 idle-pinned + zonos :1920 — operator declined to fix); rotate the 5 rest-server backup creds (operator, offline); Worldtree #363 research-wing ingest (parked, no deadline); T1 SFT LoRA dormant; Zonos2 engine still NATIVE (containerize deprioritized).
 
 ## Recent decisions
 
-- `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer** (supersedes
-  the 2026-06-04 phi4 decision below). Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not
-  Q4/AWQ** — primary workload (agent memory + summarization) is high-concurrency, where FP8-on-Ada
-  scales ~linearly (profiled 2010 tok/s @ C=32; single-stream 67.5 is batch-1 GEMV physics, not a
-  config bug — placement/kernel/contention all ruled out). vLLM `vllm-granite` :8004 GPU 1, official
-  IBM compressed-tensors FP8, CUDA graphs. **GPU-1 retune** (trio utils 0.2/0.2/0.3→0.07/0.07/0.18,
-  granite 0.36) freed ~10 GB → CUDA graphs + a Granite-text-LoRA hedge. nevermore repointed. (`34a43a0`,
-  auto-memory `reference_ana_ml2_vllm_granite`)
+- `[2026-07-25]` **bil-smithy-dev wired as an althing zellij-window-ping (pane route).** She's a `driver: human` dwarf peer (pane `bil-smithy` already live alongside eitri/dvalin/regin-smithy in the `Claude` zellij session) but had no delivery route → smoke messages posted to the bus but never reached her window. **Mechanism (reusable for any pane-route handle):** `~/.althing/config.yaml` → `zellij_sessions.Claude.agents[]` maps `handle` → `target` (a zellij pane **TITLE**, matched via `list-panes -j` in `althing/zellij.py:resolve_pane_id`) → `command` (herald `write-chars` + CR into that pane). The **herald loads config ONCE at startup** (`herald.py main()`), so **`systemctl --user restart althing-herald.service`** after editing. Added bil (`target: bil-smithy`), restarted, verified: herald delivered the pending smoke `01KYD7W7CF…` (available→attempted→**delivered**). ⚠️ Noticed pre-existing pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range", empty msg_ids — likely `render_command messages[0]` on an empty list; NOT caused by this change, bil works) — worth a herald look.
 
-- `[2026-06-05]` **Langfuse v3 stood up on ana-docker (:3001) as the gateway trace UI**; LiteLLM
-  `success_callback:[langfuse]` live (project `gateway`). Pretty prompt/completion/reasoning traces +
-  an `outputTokensPerSecond` tok/s dashboard. NOT a prerequisite — spend_logs already capture
-  tokens+latency. (`9171e6a`, auto-memory `reference_litellm_gateway`)
+- `[2026-07-25]` **Kimi K3 wired into the LiteLLM gateway — CODING endpoint** (operator-directed; fulfills a Heid gateway request to add a 4th cross-frontier panel arm). **Primary `model_name: kimi-k3` → `openai/k3` @ `https://api.kimi.com/coding/v1`** (Kimi Code / Vivace membership; key `KIMI_CODE_API_KEY`). A general-endpoint variant `kimi-k3-gen-api` → `openai/kimi-k3` @ `https://api.moonshot.ai/v1` (key `MOONSHOT_API_KEY`) is kept alongside (originally wired then demoted when the operator corrected: the plan uses the CODING endpoint, not the general Moonshot API). Both keys in compose env + server `.env` (NOT committed) + `.env.example`. Both verified live through the gateway :4000 (17+25→"42", "PONG"). **k3 constraints on BOTH endpoints (config-pinned + commented):** accepts ONLY `temperature=1` (else 400 "only 1 is allowed"); REASONING model (CoT in `reasoning_content`, answer in `content` → tiny `max_tokens` returns EMPTY; Kimi Code adds thinking-effort tiers low/high/max). Coding lineup also carries `k3-256k` / `kimi-for-coding` / `kimi-for-coding-highspeed` (not wired). Reachable by any gateway key spanning all proxy models (incl. shared all-agents key → spends the paid Vivace/Moonshot quota). eshpfi `edaa9a9` (gen wiring) + `9e2f787` (coding correction). **OPEN:** Heid key-scoping — shared key reaches it (paid) vs a dedicated scoped key (asked in althing `01KYD63ZBY…`).
 
-- `[2026-06-05]` **Ollama BANNED fleet-wide** (operator directive) — never stand one up; tear down any
-  found; serve via llama-swap or vLLM. Torn down irv-ml1 :11434 (freed 19 GB). (auto-memory
-  `feedback_avoid_ollama`)
+- `[2026-07-25]` **infra-ops Worldtree config-as-code repo SHIPPED — `vh/worldtree-instance-configs` (private) built, pushed, validated.** Dir-per-instance (`demo/`, `personal/`; `pinned/` = README stub, out-of-scope — no bind-mount, config frozen in image `446e5807`). Seeded byte-exact from live `/opt/<instance>/config`; 5 files each (defaults/policies/model_roles/providers/matrix). `scripts/deploy-wt-config` = diff / deploy / capture, with in-run host backup → install(vh:vh,644) → restart api+matrix → health-gate api `/health` → auto-rollback. All verbs live-tested (in-sync, capture round-trips zero-diff, pinned refused, dry-run no-ops). Gitea repo created via ana-docker localhost API with vh creds (operator-authorized one-time); pushed over internal git-SSH `10.250.50.70:222` (nh3-dev 403s gitea HTTP). Boundary AGREED by worldtree-dev (althing `01KYCAECRW…`): they stop hand-editing `/opt/<instance>/config`, route config deltas to infra-ops; three-layer model (image baseline → repo per-instance truth → host bind-mount deploy target); carve-out = their admin-API DB mutations (key mint / tier / retirement) stay in-band, not config edits. By-design deltas (personal `agent_architect` + `ratatoskr-affect-full-allow`; demo `#308` metrics + grants) preserved verbatim. → `persistent-memory.d/2026-07-25-infra-ops-wt-config-repo.md`, auto-memory `reference_worldtree_instance_configs_repo`
 
-- `[2026-06-05]` **ComfyUI / FLUX.2 work split to `~/development/comfy-dev`** (dedicated repo + agent).
-  FLUX.2-klein (fp8 + q8 GGUF, stock + uncensored encoders) installed on the irv-ml1 Docker ComfyUI;
-  eshpfi keeps the `comfyui` stack compose, comfy-dev owns the model/workflow knowledge. (auto-memory
-  `reference_irv_ml1_ampere_quant`)
+- `[2026-07-23→25]` **Worldtree #376 config-divergence arc CLOSED — per-instance config ruled BY DESIGN.** wyrd `session.history.write` demo grant was the one real bug (demo-intended grant not on demo; fixed via wholesale `policies.yaml` replace + restart). The b131 drift guard then surfaced broader divergence = legitimate live-bridged per-instance deltas; operator ruled deltas are the design not rot; guard demoted to INFO (b132); infra-ops drift-watcher built then retired same day. → `persistent-memory.d/2026-07-25-wt-376-per-instance-config-arc.md`, auto-memory `reference_worldtree_perinstance_config`
 
-- `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is
-  the structured-output profile, ON HOLD, no live consumer; the conversation summarizer defaults to
-  claude-haiku — the "phi4 erroring" premise was wrong). No instance changes now; worldtree-dev hands
-  the exact providers.yaml + consumer config when #254 un-holds, infra-ops applies to the bind mounts.
-  **CORRECTION to the 2026-06-04 "deploys ALL CICD" line:** the bind-mount CONFIGS (providers.yaml,
-  vh-owned on corviduo `/opt/worldtree*/config`) ARE infra-ops's to apply directly — only the
-  app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`)
+- `[2026-07-20→25]` **The Booth SHIPPED (v0.1.3) — ephemeral media drop board for CC sessions.** New fleet tool: user-systemd on nh3-dev :8090 (`services/booth/`, FastAPI+Jinja2, Corviduo "Australis" theme, 34 tests), Homepage-linked (Apps). Drop a folder in `~/booth-data/<name>` → browsable "booth" (auto-gallery of images/webm/audio, or a folder's own `index.html` verbatim), 24h TTL. Added across the session: browser/curl upload-for-pickup with human-readable ids (`4-wombat`), image viewer (Fit/1:1, conditional toggle), copy-id button (HTTP-LAN `execCommand` fallback). Registered in global CLAUDE.md tools. auto-memory `reference_booth_media_board`.
 
-- `[2026-06-04]` **phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer/dreaming agent;
-  granite-4-small retired** from llama-swap (config-only; GGUFs on disk). 50K ctx (dropped from
-  Phi-4's 128K max to fit GPU 1's ~10 GB free) + FP8 KV. (`40a374b`)
+- `[2026-07-23]` **jackdaw-compose backend deployed as a persistent nh3-dev service (:8787).** Hosted for jackdaw-dev: thin stateless `bun server/index.ts` (from `~/development/jackdaw`) → LiteLLM `gen`, Origin-gated (INV-BK04/05), reached same-origin via their `:4500` bench's `/compose` proxy. `jackdaw-compose.service` (env/shared-key server-side, unit 0600, uncommitted). Also stood up + tore down a throwaway cloudflare quick-tunnel for their preview (`cloudflared` now installed at `~/bin`). In the nh3-dev README inventory (`cd4d52e`).
 
-- `[2026-06-04]` **phi4 ships the CANONICAL/official Phi-4 chat template, NOT Ollama's.**
-  Ollama's bundled template omits the system `<|end|>` — that flattered brokkr's R15 eval but is
-  the DIVERGENT scaffold (Dvalin: the system `<|end|>` is Microsoft's intended format). Applied an
-  Ollama-matching override then reverted — ship correct, not the benchmark quirk. (`90e08f0`→`27eb537`;
-  "headgun" lesson in Tried.)
+- `[2026-07-19]` **irv-ml1 ComfyUI — RTX VSR baked into canonical provisioning (comfy-dev ticket DONE).** RTXVideoSuperResolution node + `nvidia-vfx` dep were manual installs; documented both in the canonical `stacks/comfyui/README.md` runbook (this stack's provisioning IS the README — no automated provision script). Key durability insight: the **node** lives in `basedir/custom_nodes` (persistent, restic-included → durable) but the **`nvidia-vfx` wheel** lives in the venv under `run/` (disposable, restic-excluded → **dropped by any `rm -rf run/*` fresh-bootstrap**), so the pip step must re-run after every venv rebuild. Both steps run **as uid 1000** (root install → venv-ownership crash-loop, [[reference_irv_ml1_comfyui_mmartial]]); `--extra-index-url https://pypi.nvidia.com` kept **scoped to the nvidia-vfx install**, deliberately NOT a global compose `PIP_EXTRA_INDEX_URL` (would risk perturbing the pinned torch 2.12.1/SageAttention boot bootstrap). Node already live on the box; no host change, canonical runbook now replays it. comfy-dev informed.
 
-- `[2026-06-04]` **infra-ops NOPASSWD-sudo identity commissioned, scoped to PFI boxes** (+esh-docker-vm
-  by operator override) — so infra-ops completes DevOps end-to-end vs handing the operator sudo steps.
-  Dedicated key, sudo log_output, key-gated. (`8c32a05`)
+- `[2026-07-19]` **vh private Gitea PyPI — consumer READ-access convention set + wyrd-dev provisioned.** Consuming agents read the internal vh PyPI (`https://gitea.phasefinal.com/api/packages/vh/pypi/simple/`) with a **shared read-only token** (operator call: shared, not per-consumer — read-only blast radius is small, per-agent Gitea identities aren't worth it). Minted a dedicated `read:package`-scoped PAT off **claude-bot** (`POST /users/claude-bot/tokens`, name `vh-pypi-read-consumers`; verified reads worldtree-sdk, write-probe 401), revocable/rotatable independently. uv auth = `UV_INDEX_GITEA_USERNAME=claude-bot` + `UV_INDEX_GITEA_PASSWORD=<token>` (or `~/.netrc`); pyproject uses `[[tool.uv.index]] name=gitea … explicit=true` + `[tool.uv.sources] <pkg> = { index = "gitea" }` (mirrors soong-lab's bifrost setup). Delivered to wyrd-dev (worldtree-sdk adoption) via mode-600 drop on nh3-dev, drop-and-shred. [[reference_claude_bot_gitea_creds]]
 
-- `[2026-06-04]` **Worldtree demo/pinned/personal deploys are ALL CI/CD, not infra-ops** — a "deploy
-  vX.Y.Z" request to infra-ops is MISROUTED → point them back to their pipeline. The granite→phi4
-  repoint: worldtree-dev self-served via their CI/CD (v0.30.10). (`d8d776c`)
+- `[2026-07-18]` **soong-lab auto-redeploy WIRED + validated (queued item CLOSED).** Added a WT-style CI-deploy step to `build-and-push.yml`: after build+push, the pfi-fleet runner SSHes corviduo-dev as the `deploy` user and runs `docker compose pull && up -d` from **/opt/soong-lab**, health-gated on `/api/version` (120s, fails loud). Reused WT's `deploy` account (uid 1001, docker-group → no sudo); relocated the deploy dir /home/infra-ops/soong-lab-deploy → /opt/soong-lab (deploy-owned; old dir retired `.retired-20260718`). Minted a dedicated soong-only ed25519 deploy key, pubkey on `deploy`'s authorized_keys (fp SHA256:MG7M3Ri…). **First dispatch FAILED on a bad DEPLOY_SSH_KEY paste** (`error in libcrypto` — unparseable key bytes; build+push were fine, live Soong untouched); repo secrets are **vh-owner-only** (claude-bot token = write:package only → 403; the vh package-scoped PAT also 403 on secrets), so operator re-set DEPLOY_SSH_KEY/HOST/USER. **Re-dispatch run #5 GREEN**: live container recreated ...541f7730 → ...07526a08, health 200. soong-dev pinged to sync DEPLOY.md's redeploy path (/opt/soong-lab) + close the "auto-pull open follow-up". → `persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md`
 
-- `[2026-06-04]` **ollama upgraded 0.9.0→0.30.4 on irv-ml1** (Ministral-3 is a Dec-2025 model the
-  old engine refused); A6000 pinned by **UUID** not index (native fastest-first ≠ nvidia-smi PCI).
+- `[2026-07-18]` **worldtree-sdk 1.0.0 (Python) published to the internal vh Gitea PyPI** (wtsdk-dev request; the npm/TS side shipped prior session). Built from tag `python-v1.0.0` (clean worktree), `uv publish` → `https://gitea.phasefinal.com/api/packages/vh/pypi`; acceptance `uv pip install worldtree-sdk==1.0.0` (vh index as extra-index-url) resolves + imports, __version__ 1.0.0. Registry already existed (bifrost publishes there; soong-lab consumes it via `[[tool.uv.index]] name=gitea`). Publish cred = the vh `write:package` PAT the operator had already handed over (in `worldtree-sdk/.npmrc` `_authToken`) — Gitea `write:package` is package-type-agnostic, so the npm-publish token published PyPI too. Consumers install like bifrost (add the vh index + a read token). [[reference_worldtree_demo_key_mint]]
 
-- `[2026-06-04]` **`brokkr` user (no-sudo) on irv-ml1; R14/R15/R16 substrate moved to /home/brokkr.**
-  Persistent box services there need SYSTEM systemd units (see Tried).
+- `[2026-07-18]` **soong-lab auto-redeploy APPROVED — QUEUED for next session (deferred, not started)** — Vuong approved (via soong-dev thread `01KXT3A6C3908TA4V9THV3AMH7`); mechanism = WT-style CI-deploy step (runner SSHes corviduo-dev → `compose pull && up -d` + health-gate); **blocked on a vh-owned runner→corviduo-dev deploy SSH-key secret** (reuse WT's demo-deploy key). Operator: "do soong on fresh context." → `persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md`
 
-- `[2026-06-03]` **yt-voice-clipper bot-gate fix = route yt-dlp through NH3 residential
-  egress, NOT cookies/PO-token.** YouTube hard-flags the Irvine colo IP (LOGIN_REQUIRED on a
-  public video even with no cookies). Cookies + the bgutil PO-token + deno JS-runtime all
-  loaded fine — the gate is pure IP reputation. Operator chose proxy-via-nh3-dev → durable
-  dante proxy → proven. The egress proxy is a reusable fleet lever for any datacenter-IP-gated
-  service.
+- `[2026-07-18]` **nh3-dev /tmp auto-clean enabled** — Debian ships /tmp with no tmpfiles age (`D /tmp 1777 root root -` → never cleans); this high-churn agent box had accreted **~190k stale temp dirs / 25G**. One-shot manual purge (194k→10k entries, 25G→1.7G; deleted top-level dirs/files >1d old, spared `/tmp/claude-*` by name + anything ≤1d). Then `/etc/tmpfiles.d/tmp.conf` = `D /tmp 1777 root root 3d` (daily `systemd-tmpfiles-clean.timer` removes >3d-untouched items; active files + socket dirs spared). Tunable via the age. Note the churn: ~10k /tmp entries/day here.
 
-- `[2026-06-03]` **yt-voice-clipper push-to-deploy via gitea webhook** (operator-directed,
-  after 6 manual rebuilds in ~40 min). Webhook (not poll) — gitea CAN reach the WG IP per the
-  operator. The proxy env + Homepage labels live in the **host-specific override** (untracked
-  → survive the auto-deploy's `git reset --hard`), NOT yt-voice-clipper-dev's image. Runbook
-  `d4f180d`.
+- `[2026-07-18]` **soong-lab containerize cutover COMPLETE + LIVE** — systemd→container on corviduo-dev :8443 (image `vh/soong-lab:latest` v0.3.24), data migrated (Sindra + portraits) + backed up, old service+webhook retired, Homepage tile added, operator functional-confirmed. Deploy `/home/infra-ops/soong-lab-deploy/`; no proxy (co-located WT, plain-http callback). → `persistent-memory.d/2026-07-18-soong-lab-containerize-cutover.md`
 
-- `[2026-06-03]` **R14 scope = (a) provision-only.** infra-ops provides box + CUDA env +
-  engines + weights + NFS; brokkr/dev wires `arms.py` + runs — keeps infra-ops OFF the
-  VIVAE-processing path (VIVAE = Variably Intense Vocalizations of Affect/Emotion, CHARTER §4
-  highest-liability; operator authorized R&D-eval-only, quarantined). Box = irv-ml1 (A6000
-  free; ana-ml2 GPU-saturated). Per-engine venvs (divergent torch stacks); A6000 = `cuda:0`
-  NATIVE (≠ docker `=1`).
+- `[2026-07-18]` **zonos-gateway 0.2.1 — voice-resolved emotion presets baked (provisional)** — `resolve_preset(name,voice)` → per-voice axes cell (angry/happy/startled_happy + aliases); NOT a global preset (BrF named-angry→fear). Docs on /docs + /v1/dials + repo spec. Pushed main `8f1885b`/tag v0.2.1 (after reconciling two-unrelated-git-histories). → `persistent-memory.d/2026-07-18-zonos-gateway-0.2.1-emotion-presets.md`
 
-- `[2026-06-03]` **Declined worldtree v0.30.4 staging deploy** — that's worldtree-team's
-  CI/CD lane (a developer `staging/vX.Y.Z` git-tag promote), not infra-ops. They self-corrected
-  to the same conclusion independently.
+- `[2026-07-18]` **Fleet Gitea-Actions build recipe + the `vh`-is-a-USER package-write constraint** (reusable for any fleet CI image build / package publish) — runner job image node:20-slim has no docker/git → use `container: docker:24.0.7-cli` + `apk add git nodejs` + RAW buildx (not the JS `docker/*` actions); vh is a user so its packages are OWNER-WRITE-ONLY (claude-bot can't push/publish/set-secrets — CI must auth AS vh); `GITEA_` secret-prefix is reserved. → `persistent-memory.d/2026-07-18-fleet-gitea-runner-build-recipe.md`
 
-- `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast`
-  streaming container.** Workload = single-stream interactive. **GPU placement:
-  3090 (device 0) if it fits else A6000 (device 1)** — shared dev stack, 20.5 GB
-  3090-idle is expected residency, not a blocker. **Cutover: parallel catalog
-  entry**, burn in beside live `chatterbox`, then flip. **Streaming approach:
-  adaptive buffer-ratchet chunking** (see in-flight). Native frame-streaming
-  abandoned (Tried/abandoned). Tracked: `docs/design/chatterbox-fast-plan.md`.
+- `[2026-07-18]` **Peer credential provisions — Wyrd conv-api key + wtsdk npm token, both delivered + closed.** Wyrd: demo Worldtree user-tier key (key_id `da7a0bdf`, user_id `wyrd-dev`) minted via `docker exec worldtree-worldtree-api-1 /admin/keys` (omit tier→user), drop-and-shred delivery. wtsdk: operator-minted vh `write:package` PAT relayed drop-and-shred → worldtree-sdk@1.0.0 published to `vh/npm/`. Secret-delivery pattern = drop to a mode-600 file on the peer's box, they collect+shred+confirm, then shred the holding copy; NEVER cleartext over althing. [[reference_worldtree_demo_key_mint]]
 
-- `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** I
-  claimed naive sentence-level streaming has "zero quality loss" — WRONG. The
-  T3 AR backbone conditions prosody on the WHOLE text; splitting loses
-  cross-sentence prosodic context (contextual delivery, declination, affect
-  continuity) even though voice timbre stays (reference-conditioned). No
-  *artifacts* ≠ no *quality loss*. Hence the adaptive-chunk design (maximize
-  context per chunk subject to latency budget), not fixed per-sentence splits.
+- `[2026-07-18]` **Axes sweep RESCUED angry; surprised-class dead but startled-happy ships.** Valence×arousal grid on the 3 calibrated defaults (AmericanFemale/Male, BritishFemale), exp/cfg1.5/strength1.0, 84 clips, emotion2vec + resemblyzer scored, graded vs dvalin's floor. **ANGRY rescued** (named direction was 0.004–0.15, British named-angry even misfired as fear 0.89): axes ship cells at **negative valence (−0.4..−0.8) + high arousal (+0.8..+1.0)** — BritishFemale v-0.4/a+0.8 angry=0.99/id0.725 SHIP, AmericanFemale v-0.4/a+1.0 angry=0.53/id0.685 SHIP; AmericanMale two-tier post-ladder (no single ship cell — best drama = v-0.6/a+0.8 str1.2 angry=1.0/id0.616 clean, soft = same cell str1.0 angry0.23/id0.654; cell A v-0.6/a+1.0 is a non-monotonic minefield, skip). BrF ship cell proxy-CLEAN of fear (str<1.0 just kills anger). **SURPRISED-class DEAD** (max 0.047 across all 84 cells) but **startled-happy** (happy-proxy) ships all 3 at high arousal + neutral/positive valence, with a **+0.17–0.20 identity LIFT** over the named-surprised route (named hits happy~1.0 but at id0.57–0.61, under floor; axes hits happy~1.0 at id0.74–0.80). Bonus: axes-happy retains ~0.10–0.15 more identity than the named happy slider too. Caveats: response surface non-monotonic/sharp-thresholded; angry region borders fear/disgust (bleed); emotion2vec saturates at 1.0 (needs ear-confirm); neutral text understates. Tooling `~/development/zonos-tools/axes_sweep.py`; per-clip JSON was `irv-ml1:/tmp/axes_sweep_results.json` (ephemeral). Sent dvalin msg `01KXT2ZB8G…`. NEXT = operator ear-confirm → bake presets. [[reference_zonos_tts_stack]]
 
-- `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — see
-  in-flight + Tried/abandoned. Populated `references/<name>/<name>.wav`+`.lab`
-  for all 32 voices; playbook gained normalize-step + A/B smoke gate. glados got
-  a real transcript (ASR'd via Parakeet): the Portal "Welcome to test chamber 4"
-  lines.
+- `[2026-07-18]` **Zonos2 emotion CANONICAL from an empirical sweep + the voice-cloning pipeline** — 4 chars cloned (Emmie/Penny/Natalie/Miranda), host-managed gateway voices, two-regime accurate/expressive policy, happy/sad usable + angry-weak/surprised-dead on named directions, dvalin-synthesized; axes sweep is the NEXT experiment. Studio + sweep tooling at `~/development/zonos-tools/`. → `persistent-memory.d/2026-07-18-zonos-emotion-canonical.md`
 
-_37 older entries archived to archival-memory.md._
+- `[2026-07-18]` **yt-voice-clipper: A6000-pin fix + v0.3.3 redeploy.** Fixed a latent misconfig — the host override *said* "pin worker to A6000" but `NVIDIA_VISIBLE_DEVICES` was `"0"` (the 3090); re-pinned worker+api to the A6000 by UUID (`GPU-9672f0d5`, 3090 is zonos2's). Then redeployed api+worker to v0.3.3 (`docker compose up -d --build`; SPA+Python; `max_gap` 0.6→1.2s; stderr surfaced in job.log). A6000 + version verified; yields test in-flight (job `f3ff746dbae9494d`). yt-voice-clipper-dev thread `01KXT0T6GYHB`. [[reference_ytvc_autodeploy]]
+
+- `[2026-07-17]` **Worldtree #365 internal-comms config CLOSED (demo+personal → b125) + WT#368 cross-agent memory-leak forensics + PERSONAL agent-memory scrub.** #365: staged the internal-tiers/rules/gate on both instances' bind-mounts (byte-exact vs baked b125), both now live on b125. WT#368 (read-only): the operator's name was in NO recall store on demo; on PERSONAL it sat in `lofn.chroma` (old-code `saga-v1` seeding + legacy contamination), and a clean-slate marker test proved **current b125 code isolates character-session extraction correctly** — the leak is legacy data, not a live bug. Operator-directed → executed a full PERSONAL agent-memory scrub (backup `/opt/worldtree-personal/agent-memory-backup-20260717-181004.tar.gz`; conversations/mood/auth preserved). worldtree-dev owns the code-fix/data contract. [[reference_corviduo_dev_emergency_ops]]
+
+- `[2026-07-17]` **Zonos emotion levers RESOLVED: text-priming is FLAT → the working lever is ZONOS2's native emotion-steering, which the gateway ALREADY exposes as presets.** The prosody-priming A/B (prime→generate→excise, silence-gap cut, parakeet-validated) was operator-judged FLAT on this checkpoint — text doesn't move it. Native `emotion_directions/` (happy/sad/angry/surprised + valence/arousal axes, per-speaker calibrated for AmericanFemale/Male/British) clearly WORKS (sad→slow/quiet, excited→fast/bright, etc.). **`zonos-gateway:0.2.0` (:8890) already wires it**: simplest caller path = `POST /v1/audio/speech {preset:"…"}` — presets neutral/warm/excited/sad/intense/whisper (defined in `~/zonos-gateway/src/zonos_gateway/dials.py`), reached via the **LiteLLM `ext-tts` alias** (engine-neutral swap point; consumers never call the gateway by name). RTF measured on 3090: cfg1.0 steering = FREE (~0.52 = neutral, additive vectors), cfg1.5 amplified ~0.625 (~+20%, still realtime). Captured the live gateway stack → `stacks/zonos-gateway/` (compose+env+README); ⚠️ gateway SOURCE at `~/zonos-gateway` on irv-ml1 is NOT in gitea (backup gap, follow-up); `stacks/zonos` (v0.1 Gradio) marked DEAD/superseded. Whisper is a composed preset (no whisper *direction*; escalation for hard affects = custom directions via `scripts/build_emotion_directions.py` or emotional-ref cloning `speaker_audio_base64`). Harnesses in scratchpad (not yet landed). [[reference_zonos_tts_stack]]
+
+- `[2026-07-17]` **Zonos2 :1920 → self-contained container (stays on 3090); prosody-priming is adapter-level, engine stays stock.** Config captured (14a0004, unpushed); build = cu128 base + `uv sync` vs the lock + weights mount; priming = prime→generate-one-utterance→parakeet-clip→deliver in the gateway adapter. Crux = does AR prosody carry the sentence boundary (A/B the join). → `persistent-memory.d/2026-07-17-zonos2-containerize-prosody-priming.md`
+
+- `[2026-07-16]` **GPU re-org: char-rp→GPU1 + both cards re-optimized for max context.** Moved char-rp (Magidonia-24B) GPU0→GPU1, then maxed context: char-rp-reasoning 150K→256K (util 0.46, 1.56x), gen→256K + seqs 16→32 (util 0.42, 5.43x), granite 64K→**128K full-chapter** (util 0.27, 1.50x). FINAL: GPU0 ~14 G reserve (both seats 256K native), GPU1 ~6.7 G headroom. All healthy. LESSON: KV must hold ≥1× max-len (util-floor crashes) + per-model KV cost varies ~8× (MoE cheap, dense pricey) → tune util empirically. See Current state for the full layout + backups.
+
+- `[2026-07-16]` **granite right-sized → ~10.5 GB freed on GPU1** (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536, summarizer healthy). GPU1 now ~45 GB free to relocate a GPU0 model. LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped (est max-len 47184<65536, ~2-3 min summarizer blip) before 0.18 landed. `.env`-only, recreate `vllm-granite` alone (shared stack).
+
+- `[2026-07-15]` **image-bench eviction DONE (parked item closed).** Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM `image-judge`+`qwen-image-bench` → gen :8015 (judge samplers + thinking-off), verified with :8014 down; comfy-dev pinged; also backfilled the canonical char-rp-reasoning litellm block (was lagging live). Revert ~90 s. auto-memory `project_arbo_gen_switch_imagebench_evict`.
+
+- `[2026-07-15]` arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake → `persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md`
+
+- `[2026-07-15]` esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service` → `persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md`
+
+- `[2026-07-15]` **Homepage AI-tab revamp** — flat "AI Systems" group -> dedicated AI tab, 6 role-based groups + AI-Dormant; committed `569e1af`, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.)
+
+- `[2026-07-15]` **Home Assistant config repo created** (`vh/home-assistant-config`, private). UI-managed HA -> allowlist model (YAML + curated secret-free `.storage` subset). git-in-place in `/config` on esh-docker-vm + scoped deploy key + local clone `~/development/home-assistant-config`.
+
+- `[2026-07-15]` **char-rp-reasoning OOM rescue** — solo-restart on the packed GPU0 crash-looped; fixed via `expandable_segments:True` + util 0.39->0.38 + max-model-len 192K->150K. LESSON (Tried): `max-model-len` does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom now.
+
+- `[2026-07-15]` **soong-lab `SOONG_LAB_LIBRARY_DIR` made persistent** (corviduo-dev) — was on the redeploy-wiped code default; set to `/home/infra-ops/soong-lab-data/library` (mirrors PORTRAIT_DIR), restarted. Closed a queued no-rush item; unblocked the operator.
+
+- `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost (`.cost.total_cost_usd`) / threshold-colored ctx+rate (green<60 / yellow60-90 / red>90).
+
+- `[2026-07-14]` NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned → `persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md`
+
+- `[2026-07-14]` NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP → `persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md`
+
+- `[2026-07-14]` Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion → `persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md`
+
+- `[2026-07-14]` char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) → `persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md`
+
+- `[2026-07-14]` soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST` → `persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md`
+
+- `[2026-07-13]` #355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN) → `persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md`
+
+- `[2026-07-13]` Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong) → `persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md`
+
+- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONE → `persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md`
+
+- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 206 → `persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md`
+
+- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync  → `persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md`
+
+- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); → `persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md`
+
+- `[2026-07-10]` Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev R34-v1 request) → `persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md`
+
+- `[2026-07-10]` ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, SageAttention preserved, crash-loop AVOIDED → `persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md`
+
+- `[2026-07-10]` ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it) → `persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md`
+
+- `[2026-07-10]` Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized) → `persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md`
+
+- `[2026-07-09]` Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker docker log-cap SOLVED no-bounce → `persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md`
+
+- `[2026-07-09]` granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed, #335 Slice-4) → `persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md`
+
+- `[2026-07-09]` mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into gateway-chat → `persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md`
+
+- `[2026-07-09]` granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4) → `persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md`
+
+- `[2026-07-08]` RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (3… → `persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md`
+
+- `[2026-07-08]` worldtree Mimir deploy-blocker resolved (mid-session): → `persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md`
+
+- `[2026-07-08]` OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training → `persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md`
+
+- `[2026-07-08]` DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call) → `persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md`
+
+- `[2026-07-08]` T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py → `persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md`
+
+- `[2026-07-08]` T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe → `persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md`
+
+_142 older entries archived to archival-memory.md._
 
 ## Tried and abandoned
 
-- `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — it fills the KV cache to the
-  `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs
-  AFTER weights+KV load (model/KV log looks healthy, then crash-loops; saw 11 restarts at util 0.36
-  with 237 MB free). Fix: free co-tenant room (right-size the other vLLM services) OR `--enforce-eager`
-  (no graphs, ~15-25% slower decode). FP8 single-stream is batch-1 GEMV (memory-bound, FP8 tensor cores
-  need batch>1) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`)
+- `[2026-07-25]` **Chaining the althing wake-listener arm orphans it.** `reply && althing-wake-listener &` (or spawning `althing-wake-listener` with `&` *inside* a `run_in_background` task) → the `&`-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a *plain operator turn* (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn `althing-wake-listener` as its OWN `run_in_background` task, and re-arm ONLY after a real fire (`<task-notification> completed rc0`). Reclaim an orphan with `althing-cli stop-monitor` then re-arm.
 
-- `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres
-  rows (`dashboards`/`dashboard_widgets`); build by cloning a default-dashboard row + swapping the
-  measure. tok/s is NOT a per-generation field (null on the observation) — it's the
-  `outputTokensPerSecond` MEASURE, computed at metrics-API/dashboard query time; no native per-call
-  tok/s display exists (streaming doesn't change that). langfuse-web needs `HOSTNAME=0.0.0.0` (Next.js
-  standalone binds one net-IP otherwise, unreachable via the published port once also on tnet). Host
-  3000 is gitea's → langfuse on 3001.
+- `[2026-07-25]` **Peer green-light ≠ operator consent for a managed-box mutation.** Auto-mode guard blocked a config-replace+restart on the Worldtree-team demo box that was authorized only by worldtree-dev's althing message — correctly: a persistent change to shared infra needs the *operator's* yes for that specific change, not a peer's. Surface it; don't route around the guard. (The operator then stood the whole change down — the guard's hold was the right call.)
 
-- `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh +
-  corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read
-  world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are
-  vh-owned (editable), the `.env` is deploy-owned 600 (vh can't edit it, no sudo).
+- `[2026-07-18]` **Fleet Gitea CI foot-guns** (3 failed soong-lab builds): the pfi-fleet runner's `node:20-slim` job image has no docker/git so `actions/checkout` + `docker/*` marketplace actions all fail; `vh` is a USER so its packages are owner-write-only (claude-bot repo-admin-collab still 401s on push/publish, and can't set repo secrets — owner-only); `GITEA_`-prefixed secret names are reserved/illegal. Fixes in → `persistent-memory.d/2026-07-18-fleet-gitea-runner-build-recipe.md`
 
-- `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL` on
-  the containers; it defaults to claude-haiku in code, opt-in via config not `.env`. Don't trust an
-  ".env-flip" recipe — inspect the live container env + the vh-owned config files first. (Inspection
-  corrected a wrong "summarizer erroring on phi4" premise → saved churning 3 live instances.)
+- `[2026-07-18]` **zonos-gateway local clone had NO git remote + a history unrelated to gitea's** — "committed to vh/zonos-gateway" was never pushed from that clone; two separate `git init` lineages, no merge-base. Reconcile = reset local→origin/main + overlay the changed files + push (NOT force — that erases gitea's voice-wav commits). Check `git remote -v` + `git merge-base` before assuming a clone is wired.
 
-- `[2026-06-04]` **Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF —
-  the "headgun" lesson.** Ollama's phi4 template drops the system `<|end|>`; serving vLLM with the
-  model's HF tokenizer template (canonical, has it) regressed brokkr's Ollama-measured R15 baseline
-  -33pp type-F1 while valid_format held 1.0. An Ollama-matching `--chat-template` "fixed" it but was
-  the WRONG fix (the bundled scaffold is the divergent one). PRINCIPLE: serve each model's canonical
-  `tokenizer.apply_chat_template`, not the bundled template — bundled ones corrupt baselines. Verify
-  the applied prompt via vLLM `/tokenize`→`/detokenize`. (`90e08f0`/`27eb537`)
+- `[2026-07-15]` `docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts → `persistent-memory.d/2026-07-15-docker-service-after-remote-fs-target-does-not.md`
 
-- `[2026-06-04]` **GPU pin by INDEX is ambiguous on irv-ml1** — native CUDA orders fastest-first
-  (A6000=0) but nvidia-smi/docker use PCI order (A6000=1), so an index pin can land on the wrong
-  card. Pin by **UUID** (`CUDA_VISIBLE_DEVICES=GPU-…`); verify via nvidia-smi compute-apps. Check
-  loaded-model VRAM with `ollama ps` (Ministral-3 @ its 256K default ctx = ~30 GB; cap num_ctx).
+- `[2026-07-15]` The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT → `persistent-memory.d/2026-07-15-the-esh-docker-vm-d-state-phantom-container.md`
 
-- `[2026-06-04]` **Persistent services on irv-ml1 need SYSTEM systemd units** — the box reaps
-  user-session processes on ssh disconnect, and `--user` systemd isn't reachable over non-login
-  ssh, so nohup/setsid/`screen -dmS`/`systemd-run --user` all die (even with `enable-linger`). Use
-  `/etc/systemd/system/`.
+- `[2026-07-15]` vLLM `max-model-len` does NOT free GPU VRAM → `persistent-memory.d/2026-07-15-vllm-max-model-len-does-not-free-gpu.md`
 
-- `[2026-06-04]` **pyworld needs `setuptools<81`** (imports the removed `pkg_resources`); and
-  **R/soundgen `-lgfortran` fails** on irv-ml1 because the default `gcc` is gcc-11 but only
-  gfortran-12 is present (libgfortran.so lives only in the gcc-12 dir) → install `libgfortran-11-dev`.
+- `[2026-07-15]` Claude Code statusline `.cost.total_cost_usd` is per-SESSION → `persistent-memory.d/2026-07-15-claude-code-statusline-cost-total-cost-usd-is.md`
 
-- `[2026-06-04]` **homepage "crash" ≠ always NFS** — a wedged container in unkillable D-state
-  ("tried to kill container, but did not receive an exit event") can come from dead `siteMonitor`
-  widget targets (retired ESH firewall IPs) hanging the node event loop into `exit_mmap`, needing a
-  host reboot. Check homepage's siteMonitors against retired hosts. (`incident_esh_docker_nfs_boot_race`)
+- `[2026-07-14]` MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround → `persistent-memory.d/2026-07-14-mtp-on-modelopt-no-checkpoint-config-skips-the.md`
 
-- `[2026-06-03]` **gitea webhook to a private IP is denied by `webhook.ALLOWED_HOST_LIST`**
-  (anti-SSRF; default `external` blocks private/loopback). Symptom: delivery shows
-  `dial tcp ...: webhook can only call allowed HTTP servers`. Fix = APPEND the target net to
-  ALLOWED_HOST_LIST in gitea's app.ini (keep `external`; scope tight, never `*`/`private`) +
-  restart gitea (act_runner job containers survive a restart). gitea runs as a container on
-  ana-docker (`gitea_gitea_data` volume, `/data/gitea/conf/app.ini`).
+- `[2026-07-14]` AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof → `persistent-memory.d/2026-07-14-aeon-s-working-nvfp4-mtp-rp-seat-was.md`
 
-- `[2026-06-03]` **torch-2.12 venvs need `uv pip install torchcodec`** — torchaudio 2.12
-  defaults to the TorchCodec backend for `.load`; without it, real audio I/O throws "TorchCodec
-  is required" — and it ONLY surfaces at actual conversion, NOT at import/model-load. Lesson:
-  validate real I/O, not just import, when provisioning ML engine envs. (seed-vc on torch 2.4
-  uses the legacy backend, exempt.)
+- `[2026-07-14]` NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept → `persistent-memory.d/2026-07-14-nvfp4-llm-compressor-compressed-tensors-gives-no-batch.md`
 
-- `[2026-06-03]` **Backgrounding `althing-cli monitor` with an inline shell `&` (instead of
-  the Bash-tool `run_in_background`) orphans it** — it survives the shell exit, holds the
-  per-handle flock UNTRACKED (won't notify the session), and `stop-monitor` doesn't detect it.
-  Fix: find + kill the orphan PID (verify cwd=this repo / handle first — nh3-dev is shared, other
-  agents' monitors run there too), then re-arm via run_in_background. Always re-arm tracked.
+- `[2026-07-14]` NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent → `persistent-memory.d/2026-07-14-nvfp4-spike-built-the-full-mtp-serve-scaffolding.md`
 
-- `[2026-06-03]` **`uv pip install .` fails on SmoothKen/knn-svc** (and similar script-repos)
-  — it's analysis scripts + a poetry pyproject, no buildable package (setuptools
-  package-discovery error). Install the pyproject deps directly, don't build the "package".
+- `[2026-07-14]` MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained` → `persistent-memory.d/2026-07-14-mtp-graft-via-top-level-mtp-tensor-names.md`
 
-- `[2026-06-02]` **Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED (sub-realtime).** Benched RTF on A6000: 0.72x (12w) / 0.82x (30w) / 0.86x (60w), **mean 0.80x = sub-realtime**, so client-side chunking would starve (same reason chatterbox-fast needs turbo's RTF>1). Root cause of the buffering (dvalin-smithy-dev deep research, verified in our code text2semantic/inference.py L600-607): Fish only chunks on `<|speaker:X|>` tags; **plain text -> batches=[whole text]** -> all semantic tokens generate before any audio (chunk_length inert). Plus a 2nd layer: kui/ASGI StreamResponse doesn't flush (header produced t=1s, delivered t=23s) -> fix = anti-buffering headers (X-Accel-Buffering:no / Transfer-Encoding:chunked) in tools/server/views.py (kept on file, not applied). A rebuild does NOT fix this (current main same logic). **STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands in the fleet -> bench fp4-quantized Fish; if RTF > ~1.5x, give it the chatterbox-fast treatment** (client-side adaptive buffer-ratchet chunker driving /v1/tts with small text pieces). Projection: fp4 (~1/4 weight bytes, memory-bound AR decode) + Blackwell (GDDR7 ~1.8TB/s vs A6000 0.77TB/s, native FP4 cores) ~ 2-3x RTF; validate fp4 voice quality (ear/ECAPA) before committing. For now Fish stays a buffered catalog entry (great for SAVED gens, not the live-audition lane).
+- `[2026-07-14]` gitea "test-delivery 204" is NOT proof a webhook works → `persistent-memory.d/2026-07-14-gitea-test-delivery-204-is-not-proof-a.md`
 
+- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data → `persistent-memory.d/2026-07-13-relaying-a-peer-s-diagnosis-as-fact-without.md`
 
-- `[2026-06-02]` **Context-priming at chunk joins (chatterbox-fast §1.6) —
-  ABANDONED (discard-cut leaks the prefix).** To give a chunk backward prosodic
-  context, prepend the prior sentence, generate `prefix+content` together, then
-  discard the prefix audio. Built + opt-in shipped (commit d707439), live-A/B'd,
-  reverted (090e70a). The kill: `generate()` returns one finished waveform with
-  NO marker for where the prefix ends, and the model renders the same prefix with
-  different timing solo vs followed-by-content — so locating the cut (generate
-  prefix solo → measure duration → snap to nearest energy-min pause within ±0.4s)
-  is a guess that left a whole clause of prefix in the output ("...without a trace
-  of sarcasm," spoken twice; operator caught it). A reliable cut needs token-level
-  boundaries (= the abandoned native-streaming arc) or per-chunk ASR/forced-
-  alignment (heavy, imperfect, eats the latency budget). → Coherence loss at joins
-  stays an ACCEPTED limitation; cold adaptive-chunk streaming judged "really good".
-  Scheduler-side work that DID land + survive: affordability-gated priming math
-  (a 2nd pass can't starve the buffer) — sound, but moot without a working cut.
+- `[2026-07-13]` `althing-cli reply <THREAD_id>` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to <peer>`. Bit me several times this session.
 
-- `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED
-  (turbo isn't built for streaming).** Long R&D arc; record so it's not
-  re-derived. (1) The model's flow is CosyVoice2-derived but `S3GenStreamer` is
-  referenced-in-docstring-only (not implemented). (2) The lib's
-  `flow_inference(finalize=False)` is BUGGY: the lookahead trim removes
-  `pre_lookahead_len(3)*token_mel_ratio(2)=6` frames from `h` but NOT from
-  `h_masks`/conds → decoder shape mismatch (e.g. 656 vs 662). A 1-line patch
-  (`h_masks = h_masks[:, :, :-pre*ratio]` after the `h` trim) + sizing the
-  meanflow noise to the trimmed length makes finalize=False RUN. (3) BUT the
-  flow encoder uses FULL-context attention (`static_chunk_size=0`), so
-  incremental/cumulative decode is **prefix-unstable** — adding tokens
-  re-attends and shifts earlier mel (maxdiff ~0.30-0.39 vs one-shot,
-  irrespective of fixed-noise slicing or emit-margin). (4) Forcing
-  `static_chunk_size>0` on the 2 modules that carry the attr did NOT stabilize
-  it (decoding_chunk_size is a forward-arg, not settable via attribute). Verdict:
-  true sub-second frame-streaming on turbo needs deep model-attention surgery
-  with quality risk — not worth it. Matches research ("turbo+streaming
-  unsolved"; vLLM-turbo outputs noise; davidbrowne17 streaming fork is
-  BASE-only). → Use adaptive-chunking instead.
+- `[2026-07-09]` FP8 breaks mOrpheus audio-token generation → `persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md`
 
-_41 older entries archived to archival-memory.md._
+- `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter,
+  `has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs.
+  (`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.)
+
+- `[2026-07-09]` **Per-frame CPU SNAC decode is too slow for streaming** — per-call overhead × ~60 frames serialized
+  → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx]
+  window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).
+
+- `[2026-07-09]` Sentence-chunking TTS loses prosody → `persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md`
+
+- `[2026-07-09]` HF whisper datasets aren't actually whispered → `persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md`
+
+- `[2026-07-08]` Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE → `persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md`
+
+- `[2026-07-08]` Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24) → `persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md`
+
+- `[2026-07-08]` Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base → `persistent-memory.d/2026-07-08-pantheon-reasoning-27b-refuses-dark-fiction-despite-an.md`
+
+- `[2026-07-08]` Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance → `persistent-memory.d/2026-07-08-pantheon-27b-mtp-on-vllm-compressed-tensors-0.md`
+
+- `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639) → `persistent-memory.d/2026-07-07-vllm-0-24-0-qwen3-5-lora-application.md`
+
+- `[2026-07-07]` SGLang generic image can't LOAD our NVFP4 AEON → `persistent-memory.d/2026-07-07-sglang-generic-image-can-t-load-our-nvfp4.md`
+
+- `[2026-07-07]` SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for → `persistent-memory.d/2026-07-07-sglang-lora-target-modules-cli-enum-rejects-the.md`
+
+- `[2026-07-07]` Engine invocation footguns cost several wasted serve-bounces this session → `persistent-memory.d/2026-07-07-engine-invocation-footguns-cost-several-wasted-serve-bounces.md`
+
+- `[2026-07-04]` LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge → `persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md`
+
+- `[2026-07-04]` A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools needs an explicit `Environment=PATH` → `persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md`
+
+- `[2026-07-04]` On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS → `persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md`
+
+- `[2026-07-01]` A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" is usually the pull-only-vs-build RACE, not registry-auth → `persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md`
+
+- `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS
+  moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%,
+  N=4 −20%). Reserve for dedicated/interactive deployments.
+
+- `[2026-07-02]` **irv-ml1 `/worktank` ROOT is root-owned — lkraven can't write there (irv-ml1 sudo
+  needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA
+  ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-<uuid>`.
+
+_101 older entries archived to archival-memory.md._

playbooks/ana-ml2-overcommit-memory.yaml

@@ -0,0 +1,43 @@
+# Make vm.overcommit_memory=1 durable on ana-ml2 (GPU inference host).
+#
+# Why: ana-ml2 runs vm.overcommit_memory=0 (heuristic) with zero swap, so the
+# CommitLimit is ~RAM/2 (~283 GB of 566 GB). The resident vLLM services already
+# commit ~224 GB of address space, leaving < 60 GB of headroom. A large model-file
+# mmap (e.g. the 50 GB NVFP4 shard during HF->native conversion, or a vLLM model
+# load) then fails with ENOMEM despite ~393 GB of RAM actually being free — the
+# kernel rejects the *commit*, not the allocation.
+#
+# overcommit_memory=1 (always overcommit) is the conventional setting for ML hosts
+# that mmap large files: the real RAM is there to back the pages, and the heuristic
+# accounting is the only thing in the way. Operator-directed permanent + durable
+# (2026-06-17). A drop-in under /etc/sysctl.d/ applies at every boot.
+#
+# Run:  scripts/elway infra-ops@ana-ml2 --playbook playbooks/ana-ml2-overcommit-memory.yaml
+# Rerunnable: a second run shows the write step `skipped` (idempotent via when:).
+
+vars:
+  dropin: /etc/sysctl.d/99-overcommit-memory.conf
+  setting: "vm.overcommit_memory = 1"
+
+steps:
+  - name: Write durable overcommit sysctl drop-in
+    # elway runs steps as the SSH user, so a shell `>` redirect can't write a
+    # root-owned path — pipe through `sudo tee` (infra-ops has NOPASSWD sudo).
+    shell: |
+      printf '# GPU inference host: large model-file mmaps (NVFP4 native convert, vLLM loads)\n# exceed the heuristic CommitLimit (overcommit=0 + zero swap) despite ample free RAM.\n# Operator-directed permanent setting 2026-06-17.\n%s\n' '{{ setting }}' | sudo tee {{ dropin }} >/dev/null
+    # Skip the write if the drop-in already holds exactly this line.
+    when: "! grep -qxF '{{ setting }}' {{ dropin }} 2>/dev/null"
+
+  - name: Apply all sysctl drop-ins now
+    shell: sudo sysctl --system >/dev/null
+    # Applying is a no-op when the runtime value already matches.
+    changed_when: "false"
+
+verify:
+  - name: Runtime vm.overcommit_memory is 1
+    shell: test "$(cat /proc/sys/vm/overcommit_memory)" = "1"
+    changed_when: "false"
+
+  - name: Drop-in file persists the setting (survives reboot)
+    shell: grep -qxF '{{ setting }}' {{ dropin }}
+    changed_when: "false"

playbooks/arbo-disable-engine-token.yaml

@@ -0,0 +1,67 @@
+# Disable bearer-token auth on the prod arbo engine (irv-ml1), leaning on
+# WireGuard as the access boundary. Operator decision 2026-06-13 (relayed by
+# comfy-dev, confirmed in-session). Deliberately reverses ADR-0001's
+# "open-auth hole closed (ENGINE_TOKEN minted)" line.
+#
+# GOTCHA (why .env-only is not enough): the app's `dependencies=protected`
+# gate no-ops only when ENGINE_TOKEN is ABSENT from the container env. An
+# empty string still gates (verified 2026-06-13: ENGINE_TOKEN="" -> /workflows
+# still 401). The var is injected by TWO paths, both must be removed:
+#   1. env_file: .env            -> delete the ENGINE_TOKEN line from .env
+#   2. environment: - ENGINE_TOKEN=${ENGINE_TOKEN}  -> commented out in compose
+# With both gone the var is unset in the container and the engine serves open,
+# exactly like the dev engine on nh3-dev.
+#
+# Reversible: the pre-change .env (with the real token) is backed up to
+# .env.pre-auth-off.bak. To re-lock: restore the ENGINE_TOKEN line in .env,
+# un-comment the compose line, `compose up -d`.
+#
+# No sudo: lkraven owns the compose dir + .env and is in the docker group.
+
+vars:
+  dir: /opt/docker/compose/arbo
+
+steps:
+  - name: Back up prod .env (preserves the real ENGINE_TOKEN for re-enable)
+    shell: cp -p {{ dir }}/.env {{ dir }}/.env.pre-auth-off.bak
+    # creates: guards the FIRST backup — never clobber it on a rerun.
+    creates: "{{ dir }}/.env.pre-auth-off.bak"
+
+  - name: Remove the ENGINE_TOKEN line from .env entirely (must be ABSENT, not empty)
+    shell: sed -i '/^ENGINE_TOKEN=/d' {{ dir }}/.env
+    when: "grep -qE '^ENGINE_TOKEN=' {{ dir }}/.env"
+
+  - name: Push the corrected compose (ENGINE_TOKEN injection commented out)
+    upload:
+      src: stacks/arbo/compose.yaml
+      dest: "{{ dir }}/compose.yaml"
+      mode: "0644"
+
+  - name: Recreate the engine so ENGINE_TOKEN is absent from its env
+    shell: docker compose -f {{ dir }}/compose.yaml up -d
+
+verify:
+  - name: .env no longer defines ENGINE_TOKEN
+    shell: "! grep -qE '^ENGINE_TOKEN=' {{ dir }}/.env"
+    changed_when: "false"
+
+  - name: Backup still carries the original token (reversibility intact)
+    shell: grep -qE '^ENGINE_TOKEN=.+' {{ dir }}/.env.pre-auth-off.bak
+    changed_when: "false"
+
+  - name: ENGINE_TOKEN is ABSENT from the running container env
+    shell: "! docker exec arbo printenv ENGINE_TOKEN >/dev/null 2>&1"
+    changed_when: "false"
+
+  - name: Protected endpoint serves tokenless after warmup (auth OFF — expect HTTP 200, was 401)
+    shell: |
+      port=$(docker port arbo 8200/tcp 2>/dev/null | sed -n 's/.*:\([0-9]\+\)$/\1/p' | head -1)
+      final=000
+      for i in $(seq 1 30); do
+        code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "http://localhost:${port}/workflows")
+        if [ "$code" != "000" ]; then final=$code; break; fi
+        sleep 2
+      done
+      echo "tokenless GET /workflows on :${port} -> HTTP ${final}"
+      test "$final" = "200"
+    changed_when: "false"

playbooks/arbo-runner-uv-symlink.yaml

@@ -0,0 +1,41 @@
+# Put `uv`/`uvx` on the irv-ml1-arbo Gitea Actions runner's PATH.
+#
+# WHY: the runner (`irv-ml1-arbo`, act_runner host-executor running AS lkraven
+# under systemd) inherits the bare systemd service PATH —
+#   /usr/local/bin:/usr/bin:/bin
+# — which does NOT include lkraven's `~/.local/bin` or `~/.cargo/bin`. uv is
+# installed at /home/lkraven/.local/bin/uv (login-shell only), so the CI step's
+# `uv run` failed with `uv: not found` even though uv is present on the box.
+# `/usr/local/bin` IS on the systemd PATH, so symlinking uv there makes it
+# visible to the runner. This also lets deploy.yml drop the per-run `curl|sh`
+# uv bootstrap. Operator-asked ("fix cicd"); comfy-dev (engine owner) authorized
+# the specific symlink 2026-06-16 (althing thread 01KV94VTS27B…).
+#
+# Re-apply this if the runner host is rebuilt or uv is reinstalled elsewhere.
+# Sudo because /usr/local/bin is root-owned; uv is owned by lkraven (the runner
+# identity), which is who actually executes the symlink at job time — traversal
+# of /home/lkraven works for lkraven, not for infra-ops (don't be fooled by an
+# infra-ops `env -i` exec test reporting Permission denied; that's the wrong
+# identity — verify AS lkraven).
+
+vars:
+  uv_src: /home/lkraven/.local/bin/uv
+  uvx_src: /home/lkraven/.local/bin/uvx
+
+steps:
+  - name: Symlink uv into /usr/local/bin (on the systemd PATH)
+    shell: ln -s {{ uv_src }} /usr/local/bin/uv
+    creates: /usr/local/bin/uv
+
+  - name: Symlink uvx into /usr/local/bin
+    shell: ln -s {{ uvx_src }} /usr/local/bin/uvx
+    creates: /usr/local/bin/uvx
+
+verify:
+  - name: uv resolves to the /usr/local/bin symlink under a systemd-like PATH (run AS lkraven)
+    shell: sudo -u lkraven env -i PATH=/usr/local/bin:/usr/bin:/bin sh -c 'command -v uv && uv --version'
+    changed_when: "false"
+
+  - name: uvx resolves the same way
+    shell: sudo -u lkraven env -i PATH=/usr/local/bin:/usr/bin:/bin sh -c 'command -v uvx && uvx --version'
+    changed_when: "false"

playbooks/deploy-lora-training-worker.yaml

@@ -0,0 +1,121 @@
+# Deploy the LoRA training worker to irv-ml1 (arbo in-arbo LoRA training Phase 1, §4.1).
+#
+# Two-part deploy (elway upload is single-file, so code lands via rsync first):
+#   1. Stage the code (run from the eshpfi-management repo root, as infra-ops):
+#        rsync -a --delete \
+#          --exclude .venv --exclude __pycache__ --exclude state --exclude logs \
+#          services/lora-training-worker/ \
+#          infra-ops@10.100.79.3:/tmp/lora-training-worker-stage/
+#   2. Run this playbook (privileged on-box install + health-gate):
+#        scripts/elway irv-ml1 --playbook playbooks/deploy-lora-training-worker.yaml
+#
+# Idempotent: a second run shows mostly ok/skipped. Runs as infra-ops (NOPASSWD sudo on irv-ml1).
+
+vars:
+  stage_dir: /tmp/lora-training-worker-stage
+  install_dir: /opt/lora-training-worker
+  handoff_dir: /worktank/arbo/train
+  loras_publish_dir: /storetank/arbo/models/loras/trained
+  worker_user: llmuser
+  arbo_user: lkraven          # arbo container runs as uid 1000 = host lkraven
+  group: arbotrain
+  port: "8203"
+
+steps:
+  - name: Create the shared handoff group
+    shell: getent group {{ group }} >/dev/null || groupadd {{ group }}
+    sudo: true
+    changed_when: "false"   # groupadd-or-noop; report ok either way
+
+  - name: Add the worker user (llmuser) to the handoff group
+    shell: id -nG {{ worker_user }} | tr ' ' '\n' | grep -qx {{ group }} || usermod -aG {{ group }} {{ worker_user }}
+    sudo: true
+    when: "! id -nG {{ worker_user }} | tr ' ' '\\n' | grep -qx {{ group }}"
+
+  - name: Add the arbo-container user (lkraven) to the handoff group
+    shell: usermod -aG {{ group }} {{ arbo_user }}
+    sudo: true
+    when: "! id -nG {{ arbo_user }} | tr ' ' '\\n' | grep -qx {{ group }}"
+
+  - name: Create the shared handoff dir (group-owned, setgid 2770)
+    shell: mkdir -p {{ handoff_dir }}
+    sudo: true
+    creates: "{{ handoff_dir }}"
+
+  - name: Set handoff dir group + setgid perms
+    shell: chgrp {{ group }} {{ handoff_dir }} && chmod 2770 {{ handoff_dir }}
+    sudo: true
+    changed_when: "false"
+
+  - name: Create the Phase-2 LoRA publish dir (ComfyUI loras/trained, group-writable)
+    # 2775 (not 2770): world-readable + traversable so ComfyUI (uid 1025 comfytoo) can list +
+    # load; group arbotrain + group-WRITE so the worker (llmuser) can publish into it. setgid
+    # propagates the group to per-train subdirs (the Phase-1 group-write lesson).
+    shell: mkdir -p {{ loras_publish_dir }} && chgrp {{ group }} {{ loras_publish_dir }} && chmod 2775 {{ loras_publish_dir }}
+    sudo: true
+    changed_when: "false"
+
+  - name: Create the install dir owned by the worker user
+    shell: mkdir -p {{ install_dir }} && chown {{ worker_user }}:{{ worker_user }} {{ install_dir }}
+    sudo: true
+    creates: "{{ install_dir }}"
+
+  - name: Sync staged code into the install dir (worker-owned)
+    shell: >
+      rsync -a --delete
+      --exclude .venv --exclude __pycache__ --exclude state --exclude logs
+      {{ stage_dir }}/ {{ install_dir }}/
+      && chown -R {{ worker_user }}:{{ worker_user }} {{ install_dir }}
+    sudo: true
+
+  - name: Ensure state + logs dirs exist (worker-writable)
+    shell: mkdir -p {{ install_dir }}/state {{ install_dir }}/logs && chown {{ worker_user }}:{{ worker_user }} {{ install_dir }}/state {{ install_dir }}/logs
+    sudo: true
+    creates: "{{ install_dir }}/logs"
+
+  - name: Build the worker venv + install deps (as llmuser; prefer uv, fall back to python3 -m venv)
+    shell: >
+      sudo -u {{ worker_user }} bash -lc '
+      cd {{ install_dir }} &&
+      if command -v uv >/dev/null 2>&1; then
+        uv venv .venv && uv pip install --python .venv/bin/python . ;
+      else
+        python3 -m venv .venv && .venv/bin/pip install -q --upgrade pip && .venv/bin/pip install -q . ;
+      fi'
+    sudo: true
+    creates: "{{ install_dir }}/.venv/bin/uvicorn"
+
+  - name: Install the systemd unit
+    upload:
+      src: services/lora-training-worker/lora-training-worker.service
+      dest: /etc/systemd/system/lora-training-worker.service
+      mode: "0644"
+    sudo: true
+
+  - name: Reload systemd + enable the worker
+    shell: systemctl daemon-reload && systemctl enable lora-training-worker.service
+    sudo: true
+    changed_when: "false"
+
+  - name: Restart the worker to pick up the synced code
+    shell: systemctl restart lora-training-worker.service
+    sudo: true
+    changed_when: "false"
+
+  - name: Give the service a moment to bind
+    shell: sleep 3
+    changed_when: "false"
+
+verify:
+  - name: Worker health endpoint responds ok
+    shell: curl -fsS http://127.0.0.1:{{ port }}/healthz
+    changed_when: "false"
+
+  - name: gpu-status reports both devices
+    shell: curl -fsS http://127.0.0.1:{{ port }}/gpu-status | grep -q '"index"'
+    changed_when: "false"
+
+  - name: Service is enabled + active
+    shell: systemctl is-active lora-training-worker.service
+    sudo: true
+    changed_when: "false"

playbooks/deploy-omnivoice.yaml

@@ -0,0 +1,140 @@
+# Deploy OmniVoice (https://github.com/k2-fsa/OmniVoice) to irv-ml1, GPU 0
+# (RTX 3090). Apache-2.0 zero-shot multilingual voice-cloning TTS, served
+# behind our OWN FastAPI wrapper (app.py): batch /v1/audio/speech plus a
+# streaming /tts driven by the vendored buffer-ratchet scheduler.
+#
+# Builds the image locally from stacks/omnivoice/Dockerfile (CUDA 12.8 +
+# torch 2.8.0 + omnivoice from PyPI + vendored scheduler.py/sanitize.py),
+# stages the build context under /opt/docker/compose/omnivoice/, brings it
+# up, and waits for /healthz on :8199.
+#
+# First run is slow: ~5-10 min docker build + a one-time HF weight pre-warm
+# (k2-fsa/OmniVoice) on first container start (entrypoint.sh). The wait loop
+# below allows up to ~20 min for build-then-up + pre-warm.
+#
+# Usage:
+#   scripts/elway irv-ml1 --playbook playbooks/deploy-omnivoice.yaml
+#
+# Idempotent — every step is creates-/when-gated; rerun is safe.
+
+vars:
+  compose_dir: /opt/docker/compose/omnivoice
+  cache_dir: /worktank/omnivoice/hf_cache
+  voices_dir: /worktank/omnivoice/voices
+  host_port: "8199"
+
+steps:
+  # ── host-side dirs ──────────────────────────────────────────────────
+
+  - name: Ensure /worktank/omnivoice root exists (one-time, sudo)
+    shell: mkdir -p /worktank/omnivoice
+    sudo: true
+    creates: /worktank/omnivoice
+
+  - name: Chown /worktank/omnivoice to lkraven
+    shell: chown lkraven:lkraven /worktank/omnivoice
+    sudo: true
+    when: '[ "$(stat -c %U /worktank/omnivoice)" != lkraven ]'
+
+  - name: Ensure cache dir exists
+    shell: mkdir -p {{ cache_dir }}
+    creates: "{{ cache_dir }}"
+
+  - name: Ensure voices dir exists
+    shell: mkdir -p {{ voices_dir }}
+    creates: "{{ voices_dir }}"
+
+  - name: Ensure compose dir exists
+    shell: mkdir -p {{ compose_dir }}
+    creates: "{{ compose_dir }}"
+
+  # ── deploy build context (compose, dockerfile, entrypoint, env) ───────
+
+  - name: Upload compose.yaml
+    upload:
+      src: stacks/omnivoice/compose.yaml
+      dest: "{{ compose_dir }}/compose.yaml"
+      mode: "0644"
+
+  - name: Upload Dockerfile
+    upload:
+      src: stacks/omnivoice/Dockerfile
+      dest: "{{ compose_dir }}/Dockerfile"
+      mode: "0644"
+
+  - name: Upload app.py (batch + streaming FastAPI wrapper)
+    upload:
+      src: stacks/omnivoice/app.py
+      dest: "{{ compose_dir }}/app.py"
+      mode: "0644"
+
+  - name: Upload scheduler.py (vendored buffer-ratchet streaming scheduler)
+    upload:
+      src: stacks/omnivoice/scheduler.py
+      dest: "{{ compose_dir }}/scheduler.py"
+      mode: "0644"
+
+  - name: Upload sanitize.py (language-safe TTS text sanitizer)
+    upload:
+      src: stacks/omnivoice/sanitize.py
+      dest: "{{ compose_dir }}/sanitize.py"
+      mode: "0644"
+
+  - name: Stage chatterbox reference voices for cloning (skip _*.wav artifacts)
+    shell: |
+      set -e
+      mkdir -p {{ voices_dir }}
+      docker exec chatterbox-fast sh -c 'ls /refs/*.wav' | while read -r f; do
+        b=$(basename "$f")
+        case "$b" in _*) continue;; esac
+        docker cp "chatterbox-fast:$f" "{{ voices_dir }}/$b"
+      done
+      echo "staged:"; ls {{ voices_dir }}
+    # Skip if already staged (Emily.wav is a proxy for "voices present").
+    when: "[ ! -f {{ voices_dir }}/Emily.wav ]"
+
+  - name: Upload entrypoint.sh
+    upload:
+      src: stacks/omnivoice/entrypoint.sh
+      dest: "{{ compose_dir }}/entrypoint.sh"
+      mode: "0755"
+
+  - name: Seed .env from template (only if absent)
+    upload:
+      src: stacks/omnivoice/.env.example
+      dest: "{{ compose_dir }}/.env"
+      mode: "0644"
+    when: "[ ! -f {{ compose_dir }}/.env ]"
+
+  # ── build + bring up ────────────────────────────────────────────────
+
+  - name: docker compose build (~5-10 min first time; cached after)
+    shell: |
+      set -o pipefail
+      cd {{ compose_dir }} && docker compose build --progress=plain 2>&1 \
+        | grep -vE '^#[0-9]+ [0-9.]+ (Downloading|Collecting|Requirement|Using cached|Installing collected|Successfully (installed|built)|━|Resolved|Prepared|Built)'
+
+  - name: docker compose up -d
+    shell: cd {{ compose_dir }} && docker compose up -d
+
+  - name: Wait for /healthz (allow ~25 min for weight + Whisper pre-warm + voice cloning)
+    shell: |
+      for i in $(seq 1 300); do
+        curl -sf -o /dev/null --max-time 3 http://localhost:{{ host_port }}/healthz && exit 0
+        sleep 5
+      done
+      exit 1
+    changed_when: "false"
+
+verify:
+  - name: /healthz returns 200
+    shell: curl -sf -o /dev/null http://localhost:{{ host_port }}/healthz
+    changed_when: "false"
+
+  - name: /v1/audio/voices lists the reused chatterbox voices
+    shell: curl -sf http://localhost:{{ host_port }}/v1/audio/voices | grep -q '"voices"'
+    changed_when: "false"
+
+  - name: Container is running
+    shell: docker inspect omnivoice --format '{{.State.Status}}' | grep -q running
+    changed_when: "false"

playbooks/fix-esh-nfs-boot-ordering.yaml

@@ -12,8 +12,20 @@
 #   - fstab: defaults -> defaults,_netdev,nofail  (keeps `hard`)
 #       _netdev : order mount after network-online.target
 #       nofail  : NAS-down at boot doesn't wedge boot / kill DNS
-#   - docker.service drop-in: After=remote-fs.target so Docker starts
-#     after the NFS mounts have completed.
+#   - fstab: + x-systemd.before=docker.service,x-systemd.mount-timeout=30
+#       Puts Before=docker.service directly on each generated .mount unit
+#       so Docker waits for the ACTUAL mounts; mount-timeout bounds the
+#       wait if the NAS is down at boot.
+#   - docker.service drop-in: After=remote-fs.target (kept as a weaker
+#     belt-and-suspenders layer).
+#
+# WHY the drop-in alone was NOT enough (2026-07-14 reboot): `nofail`
+# removes a mount from remote-fs.target's blocking set, so ordering
+# Docker `After=remote-fs.target` does not actually wait for the nofail
+# NFS mounts -> paperless still lost the race and Exited(255) on reboot.
+# The load-bearing fix is the DIRECT mount->docker ordering from the
+# fstab `x-systemd.before` option. Verify with:
+#   systemctl show docker -p After | tr ' ' '\n' | grep mnt-   # lists all 4
 #
 # Idempotent: re-runs show ok/skipped. Does NOT reboot — the real test
 # is the next reboot, run that separately.
@@ -37,6 +49,16 @@ steps:
     # Run only if at least one unfixed NFS line remains.
     when: "grep -qE '^10\\.0\\.50\\.50:.* nfs defaults ' /etc/fstab"
 
+  - name: Order each NFS mount before docker.service (direct dep; nofail-safe)
+    # THE load-bearing fix. remote-fs.target ordering (below) is defeated
+    # by `nofail` (the mount drops out of that target's blocking set).
+    # x-systemd.before=docker.service injects Before=docker.service onto
+    # each generated .mount unit, so Docker genuinely waits for the mounts.
+    shell: sed -i -E '/^10\.0\.50\.50:/{/x-systemd.before/!s/(_netdev,nofail)/\1,x-systemd.before=docker.service,x-systemd.mount-timeout=30/}' /etc/fstab
+    sudo: true
+    # Run only if an NFS line with _netdev,nofail still lacks the ordering.
+    when: "grep -E '^10\\.0\\.50\\.50:.*_netdev,nofail' /etc/fstab | grep -qv x-systemd.before"
+
   - name: Install docker.service drop-in to order after remote-fs.target
     # Use a DISTINCT filename — esh-docker-vm already ships an
     # override.conf (dockerd ExecStart/containerd socket); systemd merges
@@ -57,14 +79,22 @@ steps:
     changed_when: "false"
 
 verify:
-  - name: All 4 NFS lines now carry _netdev,nofail
-    shell: test "$(grep -cE '^10\.0\.50\.50:.* nfs defaults,_netdev,nofail ' /etc/fstab)" -eq 4
+  - name: All 4 NFS lines carry _netdev,nofail
+    shell: test "$(grep -cE '^10\.0\.50\.50:.*nfs defaults,_netdev,nofail' /etc/fstab)" -eq 4
+    changed_when: "false"
+
+  - name: All 4 NFS lines carry x-systemd.before=docker.service
+    shell: test "$(grep -cE '^10\.0\.50\.50:.*x-systemd.before=docker.service' /etc/fstab)" -eq 4
     changed_when: "false"
 
   - name: fstab parses cleanly (findmnt --verify, no fatal errors)
     shell: findmnt --verify >/dev/null
     changed_when: "false"
 
-  - name: Docker is ordered after remote-fs.target
+  - name: Docker is ordered after the actual NFS mount units (the real fix)
+    shell: systemctl show docker -p After | tr ' ' '\n' | grep -q '^mnt-documents.mount$'
+    changed_when: "false"
+
+  - name: Docker is also ordered after remote-fs.target (belt-and-suspenders)
     shell: systemctl show docker -p After | grep -q remote-fs.target
     changed_when: "false"

playbooks/install-pi-nh3-extdev.yaml

@@ -0,0 +1,130 @@
+# Install the `pi` coding agent (earendil-works) on nh3-extdev and wire every
+# /opt/externs/<client> workspace to GLM 5.2 via the litellm gateway.
+#
+# Context: nh3-extdev is SUDO-LESS (no root, no apt, no docker). So Node is
+# installed user-level from the official static tarball (checksum-verified),
+# pi is installed `-g` into that user-space prefix, and each client gets an
+# ISOLATED pi config dir via PI_CODING_AGENT_DIR (set by its run-pi.sh launcher).
+#
+# Idempotent: a second run shows mostly skip/ok. Rerunnable to add a client —
+# append its name to `clients` (its workspace dir + secrets.env with an
+# EXTERNS_<NAME>_GLM_KEY must already exist; workspace scaffolding is separate).
+#
+#   scripts/elway nh3-extdev --playbook playbooks/install-pi-nh3-extdev.yaml
+#
+# pi config layout (authoritative, from the installed package):
+#   - PI_CODING_AGENT_DIR overrides the agent dir (default ~/.pi/agent)
+#   - $DIR/models.json   : providers.<name>.{baseUrl, api, apiKey:"$ENV", models[]}
+#   - $DIR/settings.json : defaultProvider + defaultModel (bare id)
+# The per-client GLM key lives in <workspace>/secrets.env (600, gitignored),
+# referenced indirectly so the key never lands in models.json.
+
+vars:
+  node_ver: v22.23.0          # latest v22 LTS "Jod"; matches pi engine floor >=22.19.0
+  node_arch: linux-x64
+  node_root: /home/infra-ops/.local   # absolute (not $HOME — elway doesn't shell-expand creates:); identity is always infra-ops
+  gateway: http://10.250.50.70:4000/v1
+  externs: /opt/externs
+  clients: gbcnc surefire svsconstruction
+
+steps:
+  - name: Download + verify + extract user-level Node
+    shell: |
+      set -euo pipefail
+      DEST="{{ node_root }}"; DIR="$DEST/node-{{ node_ver }}-{{ node_arch }}"
+      mkdir -p "$DEST"; cd /tmp
+      curl -fsSLO "https://nodejs.org/dist/{{ node_ver }}/node-{{ node_ver }}-{{ node_arch }}.tar.xz"
+      curl -fsSL  "https://nodejs.org/dist/{{ node_ver }}/SHASUMS256.txt" -o SHASUMS256.txt
+      grep " node-{{ node_ver }}-{{ node_arch }}.tar.xz$" SHASUMS256.txt | sha256sum -c -
+      tar -xJf "node-{{ node_ver }}-{{ node_arch }}.tar.xz" -C "$DEST"
+      rm -f "node-{{ node_ver }}-{{ node_arch }}.tar.xz" SHASUMS256.txt
+    # Tier-1 idempotency: skip the whole download if the node binary is already there.
+    creates: "{{ node_root }}/node-{{ node_ver }}-{{ node_arch }}/bin/node"
+
+  - name: Wire node/pi onto PATH for login + interactive shells
+    shell: |
+      set -euo pipefail
+      LINE='export PATH="$HOME/.local/node-{{ node_ver }}-{{ node_arch }}/bin:$PATH"'
+      for RC in "$HOME/.profile" "$HOME/.bashrc"; do
+        grep -qF "$LINE" "$RC" 2>/dev/null || {
+          printf '\n# >>> pi/node user-level PATH >>>\n%s\n# <<< pi/node user-level PATH <<<\n' "$LINE" >> "$RC"
+        }
+      done
+    when: "! grep -qF 'pi/node user-level PATH' $HOME/.bashrc 2>/dev/null"
+
+  - name: Install the latest pi coding agent into the user prefix
+    shell: |
+      set -euo pipefail
+      export PATH="{{ node_root }}/node-{{ node_ver }}-{{ node_arch }}/bin:$PATH"
+      npm install -g @earendil-works/pi-coding-agent
+    creates: "{{ node_root }}/node-{{ node_ver }}-{{ node_arch }}/bin/pi"
+
+  - name: Wire each client workspace to GLM 5.2 (isolated config + scoped key)
+    shell: |
+      set -euo pipefail
+      for c in {{ clients }}; do
+        W="{{ externs }}/$c"; PI="$W/.pi"
+        [ -d "$PI" ] || { echo "!! $c: missing $PI (scaffold first)"; exit 1; }
+        KEYVAR=$(grep -oE '^EXTERNS_[A-Z0-9_]+_GLM_KEY' "$W/secrets.env" | head -1)
+        [ -n "$KEYVAR" ] || { echo "!! $c: no EXTERNS_*_GLM_KEY in secrets.env"; exit 1; }
+        cat > "$PI/models.json" <<JSON
+      {
+        "providers": {
+          "litellm-glm": {
+            "baseUrl": "{{ gateway }}",
+            "api": "openai-completions",
+            "apiKey": "\$$KEYVAR",
+            "models": [
+              { "id": "glm-5.2", "name": "GLM 5.2 (litellm/z.ai)" },
+              { "id": "glm-5.2-reasoning", "name": "GLM 5.2 reasoning (litellm/z.ai)" }
+            ]
+          }
+        }
+      }
+      JSON
+        cat > "$PI/settings.json" <<'JSON'
+      {
+        "defaultProvider": "litellm-glm",
+        "defaultModel": "glm-5.2"
+      }
+      JSON
+        cat > "$W/run-pi.sh" <<'SH'
+      #!/usr/bin/env bash
+      # Launch pi for this client: isolated config dir + scoped GLM key + repo cwd.
+      set -euo pipefail
+      HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+      set -a; . "$HERE/secrets.env"; set +a
+      export PI_CODING_AGENT_DIR="$HERE/.pi"
+      cd "$HERE/repo"
+      exec pi "$@"
+      SH
+        chmod 600 "$PI/models.json" "$PI/settings.json"
+        chmod 700 "$W/run-pi.sh"
+        rm -f "$PI/config.example"
+      done
+    # Re-write is deterministic; skip when gbcnc is already wired to the gateway
+    # AND its launcher exists (proxy for "all three wired").
+    when: "! ( grep -qF '{{ gateway }}' {{ externs }}/gbcnc/.pi/models.json 2>/dev/null && test -x {{ externs }}/gbcnc/run-pi.sh )"
+
+verify:
+  - name: pi binary reports a version
+    shell: |
+      export PATH="{{ node_root }}/node-{{ node_ver }}-{{ node_arch }}/bin:$PATH"
+      pi --version
+    changed_when: "false"
+
+  - name: each client has models.json + settings.json + run-pi.sh
+    shell: |
+      for c in {{ clients }}; do
+        W="{{ externs }}/$c"
+        test -s "$W/.pi/models.json" && test -s "$W/.pi/settings.json" && test -x "$W/run-pi.sh" \
+          || { echo "$c incomplete"; exit 1; }
+      done
+    changed_when: "false"
+
+  - name: gbcnc resolves glm-5.2 through the gateway (live round-trip)
+    shell: |
+      export PATH="{{ node_root }}/node-{{ node_ver }}-{{ node_arch }}/bin:$PATH"
+      {{ externs }}/gbcnc/run-pi.sh --no-tools --no-session --approve -p "Reply with exactly: PI_GLM_OK" \
+        | grep -qF PI_GLM_OK
+    changed_when: "false"

playbooks/serve-qwen3.5-122b.yaml

@@ -0,0 +1,72 @@
+# Displace mistral-small-4 (heretic) on ana-ml2 GPU 0 and serve
+# bjk110/Qwen3.5-122B-A10B-abliterated-NVFP4 (text-only) as the new `gen` model
+# (operator 2026-06-19). Weights pre-staged at /tank/aimodels/qwen3.5-122b-a10b-nvfp4
+# (incl. the repo's serving/entrypoint.sh + vllm_patches/ that the compose mounts).
+#
+# ⚠️ Downing mistral-small-4 takes down the Worldtree CHARACTER backend (vision-intact)
+# until it's repointed — operator-acknowledged. REVERT = down qwen, up -d the heretic.
+#
+#   scripts/elway ana-ml2 --playbook playbooks/serve-qwen3.5-122b.yaml
+
+vars:
+  compose_dir: /opt/docker/compose/qwen3.5-122b
+  heretic_dir: /opt/docker/compose/mistral-small-4-heretic
+  model_dir: /tank/aimodels/qwen3.5-122b-a10b-nvfp4
+  host_port: "8013"
+
+steps:
+  - name: Verify NVFP4 weights + the repo's patch/entrypoint are staged
+    shell: |
+      test -f {{ model_dir }}/model.safetensors.index.json \
+        && test -f {{ model_dir }}/serving/entrypoint.sh \
+        && test -f {{ model_dir }}/vllm_patches/patch_qwen35_moe_text.py
+    changed_when: "false"
+
+  - name: Ensure vLLM compile-cache dir exists (writable)
+    shell: mkdir -p {{ model_dir }}/.cache/vllm
+    creates: "{{ model_dir }}/.cache/vllm"
+
+  - name: Ensure compose dir exists
+    shell: mkdir -p {{ compose_dir }}
+    creates: "{{ compose_dir }}"
+
+  - name: Upload compose.yaml
+    upload:
+      src: stacks/qwen3.5-122b/compose.yaml
+      dest: "{{ compose_dir }}/compose.yaml"
+      mode: "0644"
+
+  - name: Seed .env from template (only if absent)
+    upload:
+      src: stacks/qwen3.5-122b/.env.example
+      dest: "{{ compose_dir }}/.env"
+      mode: "0644"
+    when: "[ ! -f {{ compose_dir }}/.env ]"
+
+  - name: Displace — down mistral-small-4-heretic (frees GPU 0; no-op if down)
+    shell: cd {{ heretic_dir }} && docker compose down
+
+  - name: Bring up qwen3.5-122b
+    shell: cd {{ compose_dir }} && docker compose up -d
+
+  - name: Wait for vLLM /health (allow ~15 min for patch + NVFP4 MoE load + warmup)
+    shell: |
+      for i in $(seq 1 180); do
+        curl -sf -o /dev/null --max-time 3 http://localhost:{{ host_port }}/health && exit 0
+        sleep 5
+      done
+      exit 1
+    changed_when: "false"
+
+verify:
+  - name: /health returns 200
+    shell: curl -sf -o /dev/null http://localhost:{{ host_port }}/health
+    changed_when: "false"
+
+  - name: served model id is qwen3.5-122-a10b
+    shell: curl -sf http://localhost:{{ host_port }}/v1/models | grep -q qwen3.5-122-a10b
+    changed_when: "false"
+
+  - name: container running
+    shell: docker inspect vllm-qwen35-122b --format '{{.State.Status}}' | grep -q running
+    changed_when: "false"

playbooks/wire-personal-worldtree-memory-allowlist.yaml

@@ -0,0 +1,65 @@
+# Add the ratatoskr memory-plane provider endpoint to the personal Worldtree's
+# Bifrost client allowlist, so a consumer may BIND it at session-create.
+#
+# Worldtree gates `bifrost.endpoint_url` against BIFROST_CLIENT_ALLOWED_HOSTS
+# (host:port CSV in /opt/worldtree-personal/.env). The affect plane :8390 was
+# listed during its deploy; the memory plane :8391 (ratatoskr-memory-provider
+# on nh3-dev) needs appending — otherwise POST /sessions 422s
+# (`endpoint_url must be HTTPS or match BIFROST_CLIENT_ALLOWED_HOSTS`) before
+# any handshake fires. See the eshpfi memory note `reference_bifrost_plane_wiring`.
+#
+# Idempotent + rerunnable: guards are sudo-free (live container env via the
+# docker group; backup via `test -e`); the append self-guards inside its
+# sudo bash -c; the recreate skips when the live env already carries the host.
+# Surgical: recreates ONLY worldtree-api (the validator); matrix is untouched
+# and picks up the value on its next natural redeploy. `--pull never` uses the
+# local pinned image so the recreate needs no gitea registry auth.
+#
+# CRITICAL pin-preservation: WORLDTREE_IMAGE is injected by the Worldtree CI/CD
+# at deploy time, NOT stored in .env, so a bare `compose up` falls back to the
+# compose default `:latest` — a STALE locally-cached build whose stricter config
+# validation crash-blocks startup on this instance's agent-profile drift (agents
+# reference removed LLM profile qwen3.6-35-a3b-heretic). The recreate step below
+# therefore re-derives the live pin from the untouched matrix sibling and passes
+# it explicitly. (Learned the hard way 2026-06-15 — a pinless recreate took the
+# personal API down for ~1 min until restored on the correct pin.)
+#
+#   scripts/elway corviduo-dev --playbook playbooks/wire-personal-worldtree-memory-allowlist.yaml
+
+vars:
+  add_host: "10.100.10.50:8391"
+  proj_dir: /opt/worldtree-personal
+  env_file: /opt/worldtree-personal/.env
+  api_service: worldtree-api
+  api_container: worldtree-personal-worldtree-api-1
+
+steps:
+  - name: Back up .env before editing the allowlist
+    shell: cp /opt/worldtree-personal/.env /opt/worldtree-personal/.env.bak-pre-memory-allowlist
+    sudo: true
+    creates: /opt/worldtree-personal/.env.bak-pre-memory-allowlist
+
+  - name: Append the memory endpoint to BIFROST_CLIENT_ALLOWED_HOSTS (self-guarded)
+    shell: >-
+      grep -q '{{ add_host }}' {{ env_file }}
+      || sed -i '/^BIFROST_CLIENT_ALLOWED_HOSTS=/ s/$/,{{ add_host }}/' {{ env_file }}
+    sudo: true
+
+  - name: Recreate worldtree-api so it loads the new allowlist (skip if already live)
+    when: "! docker exec {{ api_container }} printenv BIFROST_CLIENT_ALLOWED_HOSTS 2>/dev/null | grep -q '{{ add_host }}'"
+    # Re-derive the live image pin from the untouched matrix sibling so the
+    # recreate can't fall back to the crash-blocking :latest default.
+    shell: >-
+      WORLDTREE_IMAGE="$(docker inspect worldtree-personal-worldtree-matrix-1 --format '{{.Config.Image}}')"
+      docker compose --project-directory {{ proj_dir }} -f {{ proj_dir }}/compose.yaml
+      -p worldtree-personal up -d --pull never --force-recreate {{ api_service }}
+    sudo: true
+
+verify:
+  - name: Live worldtree-api env carries the memory endpoint
+    shell: docker exec {{ api_container }} printenv BIFROST_CLIENT_ALLOWED_HOSTS | grep -q '{{ add_host }}'
+    changed_when: "false"
+
+  - name: worldtree-api container is running
+    shell: docker ps --filter name={{ api_container }} --filter status=running -q | grep -q .
+    changed_when: "false"

scripts/backup-freshness-alert.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+# backup-freshness-alert.sh — daily wrapper around check-backup-freshness.sh.
+# Runs the check; on any stale/down layer (exit!=0) posts an althing alert to
+# infra-ops so the silent-failure class (the 2026-05-06→06-20 ana outage that
+# went unnoticed ~6.5 weeks) can't recur. Installed as a systemd user timer on
+# nh3-dev via scripts/install-backup-freshness-timer.sh.
+set -uo pipefail
+REPO=/home/lkraven/development/eshpfi-management
+ALTHING=/home/lkraven/.local/bin/althing-cli
+
+out=$("$REPO/scripts/check-backup-freshness.sh" 2>&1); rc=$?
+printf '%s\n' "$out"
+
+if [ "$rc" -ne 0 ]; then
+  printf 'Automated daily backup-freshness check found STALE or DOWN backup layer(s) on the PFI fleet.\nRunbook: docs/runbooks/backups.md (topology, 2-min check, rest-server-ana recovery).\n\n%s\n' "$out" \
+    | "$ALTHING" post --to infra-ops --subject "🔴 Backup freshness ALERT ($(date '+%Y-%m-%d'))" 2>&1 \
+    || echo "WARN: althing alert post failed — the check still ran (exit $rc); investigate manually."
+fi
+exit "$rc"

scripts/check-backup-freshness.sh

@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+# check-backup-freshness.sh — the "are we actually backed up?" check.
+#
+# Walks every backup layer and flags anything whose newest snapshot is older
+# than the threshold (default 48h) or any down endpoint. Prints a report;
+# exits 0 if everything is fresh, 1 if anything is stale/down. Designed to be
+# run by a daily timer that alerts on non-zero exit (see
+# scripts/install-backup-freshness-timer.sh), or by hand anytime.
+#
+# Companion to docs/runbooks/backups.md. Read-only — only SSH stat/curl.
+#
+#   BACKUP_MAX_AGE_HOURS=48 scripts/check-backup-freshness.sh
+set -uo pipefail
+
+MAX_AGE_H="${BACKUP_MAX_AGE_HOURS:-48}"
+SSH="ssh -o ConnectTimeout=8 -o BatchMode=yes"
+now=$(date +%s)
+stale=() ; fresh=() ; errors=()
+
+# newest snapshot epoch under a remote glob (echoes epoch or empty)
+newest_epoch() { # $1=host  $2=glob
+  $SSH "$1" "stat -c %Y $2 2>/dev/null | sort -n | tail -1" 2>/dev/null
+}
+report() { # $1=label  $2=epoch("" = none)
+  local label="$1" ep="$2"
+  if [ -z "$ep" ]; then stale+=("$label: NO SNAPSHOTS / unreachable"); return; fi
+  local age=$(( (now - ep) / 3600 ))
+  local when; when=$(date -d "@$ep" '+%Y-%m-%d %H:%M' 2>/dev/null)
+  if [ "$age" -gt "$MAX_AGE_H" ]; then stale+=("$label: ${age}h old (newest $when)")
+  else fresh+=("$label: ${age}h old (newest $when)"); fi
+}
+
+echo "=== Backup freshness (threshold ${MAX_AGE_H}h) — $(date '+%Y-%m-%d %H:%M %Z') ==="
+
+# --- Layer: restic file+DB, ANA side (rest-server-ana) ---
+for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do
+  report "restic/ana/$c" "$(newest_epoch ana-nas "/mnt/backup/restic/repo/ana/$c/snapshots/*")"
+done
+# --- Layer: restic file+DB, NH3 side (rest-server-nh3) ---
+for c in irv-ml1 nh3-docker; do
+  report "restic/nh3/$c" "$(newest_epoch nh3-nas "/volume1/Backup/restic/$c/snapshots/*")"
+done
+# --- Layer: PBS VM images (newest per guest, all namespaces) ---
+pbs=$($SSH pbs-ana 'for ns in /mnt/pbs-datastore/ns/*/; do n=$(basename "$ns")
+  for d in vm ct; do for g in "$ns$d"/*/; do [ -d "$g" ] || continue
+    nb=$(ls -d "$g"20*T* 2>/dev/null | sort | tail -1)
+    [ -n "$nb" ] && echo "$n/$d/$(basename "$g") $(stat -c %Y "$nb")"
+  done; done; done' 2>/dev/null)
+if [ -z "$pbs" ]; then errors+=("PBS-ANA: unreachable or no snapshots"); else
+  while read -r guest ep; do [ -n "$guest" ] && report "pbs/$guest" "$ep"; done <<<"$pbs"
+fi
+
+# --- rest-server endpoint health (401 = up & serving) ---
+for ep in "rest-server-ana http://10.250.50.70:8000/" "rest-server-nh3 http://10.100.50.50:8000/"; do
+  set -- $ep
+  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 6 "$2" 2>/dev/null)
+  [ "$code" = "401" ] && fresh+=("$1: up (401)") || stale+=("$1: endpoint code=$code (expected 401)")
+done
+
+echo
+echo "FRESH (${#fresh[@]}):"; printf '  ✅ %s\n' "${fresh[@]}"
+if [ "${#stale[@]}" -gt 0 ] || [ "${#errors[@]}" -gt 0 ]; then
+  echo; echo "STALE / PROBLEMS (${#stale[@]}+${#errors[@]}):"
+  printf '  🔴 %s\n' "${stale[@]}" "${errors[@]}"
+  echo; echo "RESULT: STALE — see docs/runbooks/backups.md"
+  exit 1
+fi
+echo; echo "RESULT: all backups fresh"
+exit 0

scripts/install-backup-freshness-timer.sh

@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+# install-backup-freshness-timer.sh — install/refresh the daily backup-freshness
+# alert as a systemd USER timer on nh3-dev (the only host with SSH to all backup
+# stores + althing-cli). Idempotent; re-run after editing the wrapper/check.
+# Requires linger (loginctl enable-linger lkraven) so it fires without a login.
+set -euo pipefail
+UNIT_DIR="$HOME/.config/systemd/user"
+REPO=/home/lkraven/development/eshpfi-management
+mkdir -p "$UNIT_DIR"
+
+cat > "$UNIT_DIR/backup-freshness.service" <<EOF
+[Unit]
+Description=Fleet backup freshness check + althing alert
+After=network-online.target
+
+[Service]
+Type=oneshot
+Environment=ALTHING_HANDLE=infra-ops
+ExecStart=$REPO/scripts/backup-freshness-alert.sh
+EOF
+
+cat > "$UNIT_DIR/backup-freshness.timer" <<EOF
+[Unit]
+Description=Daily fleet backup freshness check (08:00)
+
+[Timer]
+OnCalendar=*-*-* 08:00:00
+Persistent=true
+
+[Install]
+WantedBy=timers.target
+EOF
+
+systemctl --user daemon-reload
+systemctl --user enable --now backup-freshness.timer
+echo "installed. next run:"
+systemctl --user list-timers backup-freshness.timer --all --no-pager

scripts/nh3-dev-development-backup.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+# Hourly off-box snapshot of ~/development -> nh3-nas via rsync --link-dest
+# hardlink snapshots. Penance for the 2026-07-12 soong-lab clobber: uncommitted
+# dev work now has an hourly, versioned, off-box safety net. Secrets + heavy
+# reconstructable dirs are excluded. Snapshots are timestamped dirs on the NAS;
+# unchanged files hardlink to the previous snapshot (space-efficient). Retention:
+# newest 48 hourly snapshots.
+set -uo pipefail
+
+SRC="$HOME/development/"
+DEST_HOST="nh3-nas"
+DEST_BASE="/volume1/Backup/nh3-dev-development"
+STAMP="$(date +%Y-%m-%d_%H%M)"
+LOG="$HOME/.config/dev-backup/dev-backup.log"
+
+exec >>"$LOG" 2>&1
+echo "=== $(date -Is) snapshot $STAMP start ==="
+
+# previous snapshot for hardlink dedup
+PREV="$(ssh -o ConnectTimeout=15 -o BatchMode=yes "$DEST_HOST" "ls -1d $DEST_BASE/20* 2>/dev/null | sort | tail -1" || true)"
+LINKDEST=()
+[ -n "$PREV" ] && LINKDEST=(--link-dest="$PREV")
+echo "link-dest: ${PREV:-<none, first full snapshot>}"
+
+ssh -o BatchMode=yes "$DEST_HOST" "mkdir -p '$DEST_BASE/$STAMP'"
+
+rsync -a --delete --numeric-ids \
+  --exclude='node_modules/' --exclude='.venv/' --exclude='venv/' --exclude='__pycache__/' \
+  --exclude='.pytest_cache/' --exclude='.mypy_cache/' --exclude='.ruff_cache/' --exclude='.cache/' \
+  --exclude='dist/' --exclude='build/' --exclude='.next/' --exclude='target/' --exclude='*.pyc' \
+  --exclude='.env' --exclude='.env.*' --exclude='*.pem' --exclude='*.key' --exclude='id_*' \
+  --exclude='*.sqlite' --exclude='*.sqlite3' --exclude='*.db-wal' --exclude='*.db-shm' \
+  "${LINKDEST[@]}" \
+  "$SRC" "$DEST_HOST:$DEST_BASE/$STAMP/"
+RC=$?
+echo "rsync rc=$RC"
+
+# rc 0 = ok; rc 24 = some files vanished mid-transfer (benign for a live tree)
+if [ "$RC" -eq 0 ] || [ "$RC" -eq 24 ]; then
+  ssh -o BatchMode=yes "$DEST_HOST" "ln -sfn '$DEST_BASE/$STAMP' '$DEST_BASE/latest'"
+  # retention: keep newest 48 hourly snapshots
+  ssh -o BatchMode=yes "$DEST_HOST" "ls -1d $DEST_BASE/20* 2>/dev/null | sort | head -n -48 | xargs -r rm -rf"
+  echo "=== $(date -Is) snapshot $STAMP OK (rc=$RC) ==="
+else
+  echo "=== $(date -Is) snapshot $STAMP FAILED rc=$RC — keeping partial for inspection ==="
+fi

servers/ana-ml2/README.md

@@ -15,7 +15,7 @@ Primary AI inference host for PFI.
   NOT Dell / not the same box as sf-r630 / sfsrv-ana)
 - **CPU:** AMD EPYC 9254 24-core (96 threads)
 - **RAM:** 566 GB
-- **GPUs:** 2x NVIDIA RTX 6000 Ada Generation (46 GB VRAM each, GPU 0 and GPU 1)
+- **GPUs:** 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
 - **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (8.6 TB at `/tank`)
 - **OS:** Debian 13 (trixie), kernel 6.12.x
 - **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
@@ -32,21 +32,51 @@ Primary AI inference host for PFI.
 
 ## Running stacks
 
-| Stack | Port | Notes |
-|-------|------|-------|
-| llama-swap | 9292 | GGUF model server via llama.cpp |
-| vllm-embed (Qwen3) | 8001 | OpenAI-compatible embeddings; part of the `vllm` stack (GPU 1) |
-| vllm-rerank (Qwen3) | 8002 | OpenAI-compatible reranker; part of the `vllm` stack (GPU 1) |
-| vllm-reward (Skywork) | 8003 | Skywork-Reward-V2-8B-AWQ classifier; part of the `vllm` stack (GPU 1) |
-| dockge | 5001 | Docker stack management UI |
-| dozzle-agent | 7007 | Log agent; reports to the Dozzle hub on ana-docker |
-| beszel-agent | 45876 | Metrics agent; reports to the Beszel hub on ana-docker |
+Live inventory as of 2026-07-22. Each model is its own compose stack now
+(container `vllm-<x>` / `llama-<x>`); the `vllm` stack proper is just the
+embed/rerank/reward trio. GPUs are pinned per container via
+`deploy.resources.reservations.devices[].device_ids`.
 
-**Retired since last README update:**
+**GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):**
+
+| Container | Port | Served model | Quant | Ctx |
+|-----------|------|--------------|-------|-----|
+| `vllm-aeon-gen` | 8015 | `qwen3.6-35b-a3b-heretic` — the "gen" hero seat | NVFP4 (modelopt) | 256k |
+| `vllm-charrp-reasoning-nvfp4` | 8018 | `char-rp-reasoning` (R36 reasoning RP) | NVFP4 (modelopt) | 256k |
+
+**GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):**
+
+| Container | Port | Served model | Quant | Ctx |
+|-----------|------|--------------|-------|-----|
+| `vllm-granite` | 8004 | `granite-4.1-8b` — fleet summarizer/classifier | FP8 (compressed-tensors) | 131k |
+| `llama-charrp` | 8016 | `Magidonia-24B-v4.3` Q6_K — char-RP (llama.cpp) | GGUF Q6_K | — |
+| `vllm-selene` | 8011 | `selene-1-mini-8b` — Atla LLM-as-judge | FP8 | 32k |
+| `vllm-reward` | 8003 | `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — reward classifier | AWQ | 16k |
+| `vllm-embed` | 8001 | `Qwen3-Embedding-0.6B` | — | 8k |
+| `vllm-rerank` | 8002 | `Qwen3-Reranker-0.6B` | — | 8k |
+
+**Infra / non-GPU:**
+
+| Container | Port | Notes |
+|-----------|------|-------|
+| `dockge` | 5001 | Docker stack management UI |
+| `dozzle-agent` | 7007 | Log agent → Dozzle hub on ana-docker |
+| `beszel-agent` | 45876 | Metrics agent → Beszel hub on ana-docker |
+
+Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing
+it on the card with room or evicting a dormant one first.
+
+**Dormant (compose present on disk, containers stopped)** — rollback / audition
+seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
+`ms32-24b-angel`, `qwen3.5-122b`, `qwopus3.5-122b`, `qwen35-vl`, `qwen36-vl`,
+`qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`,
+`parakeet`, `vllm-qwen3`.
+
+**Retired:**
+- `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated
+  per-model seats (e.g. `llama-charrp`); no longer running.
 - `infinity` — replaced by the `vllm` stack (originally `vllm-qwen3`, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a `transformers` build that knew Qwen3.
-- `LibreChat (+ rag_api, vectordb, mongodb, meilisearch)` — removed from this host.
-- `searxng` — now hosted on ana-docker for the whole fleet.
-- Residual networks (`librechat_default`, `kokoro-tts-gpu_default`) from prior experiments are still present; safe to `docker network rm` at leisure.
+- `LibreChat (+ rag_api, vectordb, mongodb, meilisearch)`, `searxng` — removed from this host (searxng now on ana-docker fleet-wide).
 
 ## Refresh state
 
@@ -58,9 +88,17 @@ Latest snapshot: `system-details.txt` (regenerate as needed).
 
 ## GPU allocation policy
 
-By default, no container is pinned. For predictable performance when multiple GPU workloads run concurrently:
+Every seat is explicitly pinned via `device_ids` (no unpinned containers), and
+both cards run ~90% full:
 
-- **GPU 0:** heavy LLM (llama-swap big models).
-- **GPU 1:** light services (the three `vllm` services share this GPU via `--gpu-memory-utilization`).
+- **GPU 0:** the two heavy NVFP4 seats — `vllm-aeon-gen` (gen) and
+  `vllm-charrp-reasoning-nvfp4`. The live serving path (near-100% util under
+  load), ~42 + 45 GB.
+- **GPU 1:** everything else — summarizer (granite), judge (selene), reward,
+  embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle
+  between calls, ~91 GB resident.
 
-Use `deploy.resources.reservations.devices[].device_ids: ["<id>"]` in compose to pin.
+Pin with `deploy.resources.reservations.devices[].device_ids: ["<id>"]` in
+compose. Each service caps its share with `--gpu-memory-utilization`; with both
+cards near-full, placing a new seat means freeing room (evict a dormant one) or
+trimming a neighbour's utilization first.

servers/corviduo-dev/ssh-target

@@ -1 +1 @@
-vh@10.250.50.152
+infra-ops@10.250.50.152

servers/nh3-dev/README.md

@@ -22,6 +22,15 @@ local Bash already executes here — no SSH-to-self needed for non-privileged wo
   `vh/mead-hall`.
 - **bloom_music dev** — `~/development/bloom_music`; its `web/` test harness uses
   Playwright headless Chromium for OSMD browser-geometry assertions.
+- **The Booth** — ephemeral media drop board (`:8090`, `booth.service`), from
+  eshpfi `services/booth/`. Lets CC sessions surface A/B renders + smoke results
+  (and browser uploads for pickup) to the operator; 24h TTL, Homepage-linked.
+- **jackdaw-compose** — JackDAW AI Composer `/compose` backend (`:8787`,
+  `jackdaw-compose.service`), a thin stateless `bun server/index.ts` from
+  `~/development/jackdaw` → LiteLLM `gen`. Origin-gated (INV-BK04/BK05), reached
+  same-origin via the `:4500` bench's `/compose` proxy. Hosted for jackdaw-dev
+  (their code; the model endpoint + key live in server env only — unit is `0600`,
+  not committed).
 
 ## Box-wide Playwright / Chromium (2026-06-04)
 

servers/nh3-extdev/README.md

@@ -0,0 +1,67 @@
+# nh3-extdev
+
+NH3-site **manager / external-dev box** — `10.100.50.42`. Fresh Debian 13
+(trixie) VM on `nh3-pve` (QEMU: 8 vCPU / 7.8 GB / 250 GB, no GPU). Successor to
+the retired **nh3-ansible**. Not a Docker-stack host (Docker not installed).
+
+**Reach:** the dedicated agent identity only —
+`ssh nh3-extdev` (alias) = `ssh -i ~/.ssh/infra-ops_ed25519 infra-ops@10.100.50.42`.
+
+The `infra-ops` user here is **sudo-LESS by design** (operator decision
+2026-06-17): key-only login, password locked, **no NOPASSWD sudo**, **not** in the
+`docker` group — deliberately tighter than the rest of the fleet's infra-ops
+identity ([`reference_infra_ops_sudo_identity`] in auto-memory, which is the
+*sudo* variant). Management here is **user-level only**: files/repos/tooling under
+the home dir, `systemctl --user`, world-readable system state. No root ops (apt,
+system services, `/etc`, root-owned files) and **no Docker** unless a later,
+explicit grant is made (command-scoped sudoers preferred over docker-group, which
+is root-equivalent).
+
+## Purpose
+
+NH3 **client-agent control node** (successor to the retired `nh3-ansible`): runs
+the **`pi` coding agent** (earendil-works) on **GLM 5.2**, one isolated agent per
+external client, to manage that client's infrastructure. Each client gets a
+walled workspace under `/opt/externs/<client>/`; isolation is by directory +
+credential, never a shared identity.
+
+### Per-client workspace layout (`/opt/externs/<client>/`)
+
+```
+/opt/externs/<client>/
+├── AGENTS.md? (in repo/)   # operating charter the agent loads every session
+├── secrets.env             # 600, gitignored — EXTERNS_<CLIENT>_GLM_KEY (scoped litellm key)
+├── run-pi.sh               # 700 launcher: sources secrets.env, sets PI_CODING_AGENT_DIR,
+│                           #   cd repo/, exec pi  (per-client isolated config + key)
+├── .pi/                    # this client's pi agent dir (PI_CODING_AGENT_DIR points here)
+│   ├── models.json         # provider litellm-glm -> gateway, models glm-5.2[/-reasoning]
+│   ├── settings.json       # defaultProvider litellm-glm, defaultModel glm-5.2
+│   └── (sessions/auth)     # pi-managed at runtime
+├── repo/                   # client infra workspace (own .git; AGENTS.md/scripts/servers/…)
+└── .ssh/                   # per-client scoped deploy key (reaches THIS client's hosts only)
+```
+
+Launch an agent: `/opt/externs/<client>/run-pi.sh` (add pi flags as needed, e.g.
+`-p "…"` non-interactive). `pi` itself reaches GLM 5.2 through the litellm gateway
+(`http://10.250.50.70:4000/v1`, reachable cross-site from NH3 — verified).
+
+### pi / Node install (user-level, no root)
+
+Node and pi are installed **user-level** (the box is sudo-less): Node v22.23.0 LTS
+from the official static tarball (checksum-verified) under
+`~/.local/node-v22.23.0-linux-x64/`, with `pi` installed `-g` into that prefix
+(`pi --version` → 0.79.7). PATH is wired in `~/.profile` + `~/.bashrc`.
+
+Reproduce / add a client / upgrade: **`playbooks/install-pi-nh3-extdev.yaml`**
+(idempotent — `scripts/elway nh3-extdev --playbook …`). It installs Node+pi and
+wires every workspace in its `clients` var to GLM 5.2; the workspace *scaffold*
+(dirs, `secrets.env`, deploy keys, `repo/`) is provisioned separately.
+
+## Notes
+
+- Debian **13** (trixie) — newer than the fleet's Debian-12 baseline; watch for
+  package/behaviour drift vs other hosts.
+- `/etc/hosts` now carries `nh3-extdev` (the old `unable to resolve host` sudo
+  warning is silenced).
+- Stood up 2026-06-17; pi-on-GLM-5.2 client agents wired 2026-06-18. `system-details.txt`
+  is sudo-less, so docker/root-only sections are necessarily blank.

servers/nh3-extdev/ssh-target

@@ -0,0 +1 @@
+infra-ops@10.100.50.42

servers/nh3-extdev/system-details.txt

@@ -0,0 +1,75 @@
+
+===== HOST =====
+
+Hostname:   nh3-extdev
+Date:       2026-06-18T14:04:13-07:00
+Uptime:     up 23 hours, 25 minutes
+OS:         Debian GNU/Linux 13 (trixie)
+Kernel:     6.12.90+deb13.1-amd64
+Arch:       x86_64
+
+===== HARDWARE =====
+
+CPU cores:  8
+CPU model:  QEMU Virtual CPU version 2.5+
+MemTotal:   7.8 GB
+MemAvailable: 7.3 GB
+
+===== GPUS =====
+
+nvidia-smi not present (no NVIDIA GPUs or driver not installed)
+
+===== FILESYSTEMS (df) =====
+
+Filesystem      Size  Used Avail Use% Mounted on
+/dev/sda1       250G  3.6G  234G   2% /
+
+===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
+
+UUID=fabc36b3-459e-4ef1-80a6-8a0f85a1a781 /               ext4    errors=remount-ro 0       1
+UUID=bf3fd6d0-4537-4626-bdcc-551c5603ab98 none            swap    sw              0       0
+/dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
+
+===== TARGETED DATA PATHS =====
+
+/opt  (total: 1.5M)
+  total 12
+  drwxr-xr-x   3 root      root    4096 2026-06-17 15:03 .
+  drwxr-xr-x  18 root      root    4096 2026-06-17 14:37 ..
+  drwxrws---+  5 infra-ops externs 4096 2026-06-17 15:13 externs
+
+/srv  (total: 4.0K)
+  total 8
+  drwxr-xr-x  2 root root 4096 2024-05-30 19:51 .
+  drwxr-xr-x 18 root root 4096 2026-06-17 14:37 ..
+
+
+===== DOCKER =====
+
+docker not installed
+
+===== COMPOSE FILES (/opt/docker/compose/) =====
+
+/opt/docker/compose not present
+
+===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
+
+/opt/docker/conf not present
+
+===== LISTENING PORTS =====
+
+0.0.0.0:22
+[::1]:25
+127.0.0.1:25
+[::]:22
+
+===== MODEL / HUGGINGFACE CACHES =====
+
+
+===== DOCKER-ADJACENT SYSTEMD SERVICES =====
+
+(none matching)
+
+===== DONE =====
+
+Paste the above back into the chat, or pass a path as argv[1] to save.

services/booth/.gitignore

@@ -0,0 +1,6 @@
+.venv/
+__pycache__/
+*.pyc
+*.egg-info/
+.pytest_cache/
+booth-data/

services/booth/README.md

@@ -0,0 +1,135 @@
+# The Booth
+
+A dead-simple standing web server for shuttling **ephemeral files** between the
+operator and CC sessions — A/B renders, smoke-test screenshots, audio/video
+samples, or anything you want to hand off. It works both directions:
+
+- **Session → operator:** a session drops a folder of files on disk; the Booth
+  renders it as a browsable "booth".
+- **Operator/anyone → pickup:** upload files through the browser (or `curl -F`)
+  and get a **human-readable pickup id** like `4-wombat` or `star-84`.
+
+Either way it **wipes 24h after the last activity**. No database — the
+filesystem *is* the state.
+
+- **Live:** http://10.100.10.50:8090/ (nh3-dev) · linked from Homepage → *Apps → The Booth*
+- **Data dir:** `~/booth-data/` on nh3-dev (one subfolder per booth)
+- **TTL:** 24h, measured from the newest mtime in a booth's tree (it lives while
+  you're touching it, self-destructs 24h after you stop)
+
+## How a session posts
+
+A booth is **just a folder** under the data dir. Three ways, cheapest first:
+
+```bash
+# 1. On nh3-dev — the helper (services/booth/scripts/booth):
+booth add my-run  out/a.png out/b.png       # creates booth + copies, prints URL
+booth new  my-run                            # empty booth, then cp/mv into ~/booth-data/my-run/
+booth url  my-run                            # just print the URL
+booth ls                                     # list booths
+booth rm   my-run                            # wipe now (TTL would anyway)
+
+# 2. On nh3-dev — raw, no helper:
+mkdir -p ~/booth-data/my-run && cp out/*.png ~/booth-data/my-run/
+#   -> http://10.100.10.50:8090/b/my-run/
+
+# 3. From another host — rsync into the data dir:
+rsync -a ./out/  nh3-dev:booth-data/my-run/
+```
+
+Then hand the operator `http://10.100.10.50:8090/b/my-run/`.
+
+## Upload for pickup
+
+The reverse direction — put files in through the web, pick them up by id:
+
+- **Browser:** the index page has an *Upload files for pickup* panel
+  (drag-drop or click). Submit → you land on a booth with a **human-readable
+  id** (`4-wombat`, `star-84`) whose files each have a ⬇ download link.
+- **curl (a remote session with no ssh to nh3-dev can use this too):**
+  ```bash
+  curl -sS -i -F 'files=@out/a.png' -F 'files=@out/b.png' \
+    http://10.100.10.50:8090/upload | grep -i location
+  #   Location: /b/star-84/   <- the pickup id
+  ```
+- **Pick up** at `http://10.100.10.50:8090/b/<id>/` (download links), or on
+  nh3-dev straight off disk at `~/booth-data/<id>/`.
+
+Uploads are stamped as pickup booths (a `⬆ pickup` badge in the UI) and expire
+on the same 24h TTL. Limits: `BOOTH_MAX_FILES` files (default 50) and
+`BOOTH_MAX_UPLOAD_MB` total per submission (default 1024); filenames are reduced
+to a safe basename (no path traversal).
+
+## What a booth renders
+
+- **Has its own `index.html`?** → served **verbatim** (its relative assets —
+  `chart.png`, `report.css` — resolve out of the same folder). Build whatever
+  page you want.
+- **No `index.html`?** → **auto-gallery** of the folder's media:
+  - images (`png jpg jpeg gif webp avif svg bmp`) → `<img>` (click → full-screen
+    viewer with **Fit** / **1:1** — the toggle only appears when the image is
+    larger than the viewport — plus download and ✕/Esc back to the gallery)
+  - video (`webm mp4 ogv m4v mov`) → `<video controls>`
+  - audio (`mp3 wav ogg flac m4a opus aac`) → `<audio controls>`
+  - anything else → a download link
+- **Captions:** a `<file>.txt` or same-stem `<stem>.txt` sidecar is folded in as
+  that item's caption — the natural way to label an A/B pair:
+  ```
+  a.png              b.png
+  a.txt  "baseline"  b.png.txt  "cudaMallocAsync (winner)"
+  ```
+
+## Routes
+
+| Route | Purpose |
+|---|---|
+| `GET /` | Index — one card per booth (newest first), with expiry countdown |
+| `GET /b/<name>/` | A booth (its `index.html`, else auto-gallery) |
+| `GET /b/<name>/<file>` | Serve a file out of the booth |
+| `POST /upload` | Upload files → new pickup booth; 303-redirects to `/b/<id>/` (id in `Location`) |
+| `POST /b/<name>/delete` | Wipe a booth (the UI's "Wipe now" button) |
+| `DELETE /b/<name>` | Wipe a booth (curl/API) |
+| `GET /healthz` | `{ok, ttl_hours, booths}` — Homepage siteMonitor target |
+
+## Ops
+
+Runs as a **user-level** systemd service on nh3-dev (no root, no Docker),
+alongside the other fleet sidecars (herald, zellij-web, ttyd).
+
+```bash
+systemctl --user status booth.service
+systemctl --user restart booth.service
+journalctl --user -u booth.service -f          # sweeper logs "[booth] swept …"
+```
+
+Config is env in the unit (`booth.service`):
+`BOOTH_DATA_DIR`, `BOOTH_TTL_HOURS`, `BOOTH_HOST_LABEL`, `BOOTH_SWEEP_INTERVAL_MIN`,
+`BOOTH_MAX_UPLOAD_MB` (default 1024), `BOOTH_MAX_FILES` (default 50).
+
+### Install / update
+
+```bash
+cd services/booth
+uv venv && uv pip install fastapi "uvicorn[standard]" jinja2 python-multipart   # runtime deps
+cp booth.service ~/.config/systemd/user/booth.service
+systemctl --user daemon-reload && systemctl --user enable --now booth.service
+```
+
+Code runs straight from this checkout (the unit's `WorkingDirectory` /
+`ExecStart` point here), so "deploy an update" = edit + `systemctl --user
+restart booth.service`.
+
+### Tests
+
+```bash
+cd services/booth && uv pip install pytest httpx && .venv/bin/python -m pytest -q
+```
+
+## Notes / non-goals
+
+- **No auth.** LAN/WG-internal only, ephemeral content — don't drop secrets in a
+  booth, and note anyone on the LAN can upload (bounded by the size/file limits).
+  Uploaded files are served back with their own content-type, so an uploaded
+  `index.html` renders as a page (a feature for custom reports; keep it in mind).
+- Booth names with `/`, `..`, or a leading `.` are rejected; file serving and
+  uploaded filenames are guarded against path traversal and symlink escape.