esh-pfi-infrastructure/persistent-memory.md

# Persistent memory — eshpfi-management

_Last updated: 2026-07-17_

> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under an hour old, read it (it carries the in-flight
> handoff from the previous session), then delete it. Older than an hour:
> stale — delete it unread.

## Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical
Docker Compose stacks, ops playbooks, and conventions. Authoritative
copies of compose files live on the servers under
`/opt/docker/compose/<stack>/`; this repo mirrors them for version
control, editing, planning, and CI-driven deploys. **It was originally
spun up to handle the fleet backups** — keep that lens when triaging
backup/storage issues.

## Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

| Repo | Role | CI status |
|---|---|---|
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) |
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
| `vh/althing` | Lean trusted inter-agent message bus — **v2 "email model" (v2.0.0b2, 2026-07)**: per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API `/owner/*` / `althing-mcp` stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. | per-box `uv tool install` (NOT CI-deploy); **nh3-dev = the DEV box** (editable install of `~/development/althing`, gets new versions first); **nh3-extdev** a mesh peer (model B: althing-svc + shared `/srv/althing`) |
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) |
| `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |
| `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook |
| `model-training-forge` (mtf-dev) | Fine-tuning recipe forge; **T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06)** (SFT→DPO, LitBench-RM reward) | training runs, not a deployed sidecar |

(`vh/volva` + Heid were re-architected from systemd daemons to Claude Code
session orchestrators 2026-06-08; their nh3-dev `.service` units were removed —
no longer deployed sidecars here. See Recent decisions.)

- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
  fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
  VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
  cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
  for the blast-radius matrix. **⚠️ The restic file+DB layer routes
  through TWO rest-servers** (`rest-server-ana` @ ana-docker:8000 →
  ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; `rest-server-nh3` @
  nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS
  export of `/mnt/backup`. (rest-server-ana recovered 2026-06-20.)

- **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
  model/dataset onto ana-ml2's shared cache at
  `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns.

- **Worldtree admin auth — per-instance.** Each Worldtree deployment
  (demo :8080, personal :8081, pinned :8082) has its own Heimdall
  registry and its own bootstrap admin key. Infra-ops's stored
  long-lived admin key (`key_id 61419c92`) at
  `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
  auths against **demo only**. Personal-instance admin (the
  `~/.config/worldtree/personal-admin-token`, mode 600) POSTs
  `/admin/keys` (mints per-project keys; takes `user_id`+`label`, **no
  scope param** — scopes are tier-derived). **On-instance mint recipe
  (cleaner than DB-manip):** `docker exec worldtree-worldtree-api-1` POST
  `/admin/keys` with the in-container `WORLDTREE_BOOTSTRAP_ADMIN_KEY`; cleartext
  once in `.key`=`wt_live_+16hex`. auto-memory `reference_worldtree_demo_key_mint`.

- **Per-project user keys against personal Worldtree** (issued
  2026-05-19): `skaldsong:79744637`, `skaldsong:7c1dbbbe`,
  `althing:50d85460`, `mead-hall:a360822d`. Mint via `/admin/keys`, drop
  value to `/tmp/wt-personal-<name>.key` mode 600, dev collects +
  shreds (DO NOT cat to chat transcript).

- **Skaldsong CD pattern (registry-pull).** vh/skaldsong's CI builds and
  pushes `gitea.phasefinal.com/vh/skaldsong:<sha>` + `:latest`;
  `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates.
  SHA-pin only. Prereq: host needs `docker login gitea.phasefinal.com` once.

- **gitea internal route for fleet hosts.** gitea is a container on
  **ana-docker** — git-SSH `10.250.50.70:222`, HTTP `:3000`. Fleet/colo
  hosts must use this internal route, NOT public `gitea.phasefinal.com`
  (`38.120.12.44`) — the public path fail2bans the host egress IP. Full
  gotcha in `docs/orientation.md` → Git/gitea.

- **docker-as-root pattern** (for ops with no admin API, or to edit
  deploy-owned/root-owned files without sudo): `docker run --rm -v
  <target-dir>:/wt docker:cli sh -c "..."`. docker-group membership is
  effectively root via bind-mount. **Foot-gun: relative paths in compose.yaml
  resolve against the sandbox CWD but the daemon interprets them against the
  HOST fs — always pass `-e VAR=/abs/path` for any relative-default config dir.**

- **`scripts/elway` sudo handling** — elway prompts for the sudo password
  ONCE via `getpass` before the first `sudo: true` step → can't run
  unattended from a non-TTY tool if any step needs sudo. Sudo-free
  playbooks run fully non-interactive over key SSH.

- **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo
  on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On
  ana-docker: **default `ssh ana-docker` = `lkraven`** (docker-group, NO
  passwordless sudo); **`ssh infra-ops@ana-docker` HAS NOPASSWD root**. **→
  For any sudo op on ana-docker, use `ssh infra-ops@ana-docker`.** `ssh
  infra-ops@10.100.10.50` (nh3-dev) ALSO NOPASSWD sudo; on **nh3-extdev** infra-ops
  is sudo-LESS by design (`ssh lkraven@10.100.50.42` is the NOPASSWD path). **irv-ml1:
  `ssh irv-ml1` = lkraven, docker-group (plain docker) but sudo needs a PASSWORD
  (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.

## Current state / in-flight

_As of 2026-07-17 — active work is CONTAINERIZING the Zonos2 TTS engine + prototyping a prosody-priming hypothesis. Snapshot taken for a /clear to build with fresh context._

**ACTIVE / next-session build task — two decoupled tracks (full detail in the `[2026-07-17]` Recent-decisions entry):**
- Config already captured at `stacks/zonos-engine/` (commit **14a0004, UNPUSHED**): README + corrected `zonos2-server.sh` (the live-only launch config; the committed `harness/zonos_server.sh` on irv-ml1 was STALE) + `.env.example`.
1. **Engine image (stock Zonos2 @ `194c0a3`):** `FROM` a CUDA-12.8 base → `uv sync` against the repo `uv.lock` → mount ~15 GB HF weights (`~/.cache/huggingface/hub/models--Zyphra--ZONOS2`, DON'T bake) → pin the **3090** (`NVIDIA_VISIBLE_DEVICES=0`; operator 2026-07-17: keep OFF the A6000, it OOMs under ComfyUI) → `restart: unless-stopped` → CMD = the captured invocation. Build risk = compiled-CUDA deps (flashinfer/sgl_kernel/cutlass-dsl/tvm-ffi/pynini) on torch 2.9.1+cu128 — mostly wheels + the lock, expect a couple iterations. **Cutover in place:** stop native (frees ~17 G on the 3090) → `docker compose up -d` (same footprint) → repoint `zonos-gateway` `ZONOS_URL`. Engine STAYS STOCK (no fork).
2. **Prosody-priming (operator's hypothesis, the reason for building fresh):** prime the AR engine with an emotional sentence → generate "primer + real text" as ONE utterance → parakeet ASR timestamps (irv-ml1 `:8765`) → gap-clip the primer (+~15 ms fade) → deliver only the primed real text. PURE serving-layer → gateway adapter `stacks/zonos/adapter/server.py`, engine untouched. Test NATIVE-first (fast iteration). **CRUX = does AR prosody carry the sentence boundary?** → A/B the JOIN (period/comma/ellipsis/none). Primer = caller-supplied for the harness → emotion→primer library for prod.

**Open loops for the fresh session:**
- **UNPUSHED: commit 14a0004** (zonos-engine capture) — push it, or commit the image work on top + push.
- **`stacks/heretic2-charrp-reasoning/` STILL UNTRACKED** — the char-rp-reasoning NVFP4+MTP stack; its live `.env` changed this session (util 0.38→0.46, max-len 153600→262144). Commit the stack.
- **Re-arm the althing monitor** (`/althing:monitor`, handle `infra-ops`) — the wake-listener dies on /clear. Open watches: worldtree-dev (#363 research-wing ingest, PARKED no-deadline, auto-memory `project_worldtree_research_wing_ingest`) + comfy-dev (arbo→gen bake).
- **irv-ml1 3090 oversubscription footgun** (operator declined to fix for now): kokoro (`:8193`) + vibevoicefusion (`:9527`) are idle-pinned to the full 3090 (~1.4 G free) → could OOM on load.
- **Worldtree #365 DEMO config pre-sync — STAGED 2026-07-17** (worldtree-dev, thread `01KXRE59`; verbatim from their 6a60fe3/b123 blocks in msg `01KXRE90`; confirmed staged in reply `01KXRV5J`). On corviduo-dev DEMO `/opt/worldtree/config`: added `internal`+`internal-restricted` tiers after `suspended:` + the 4 `internal-*` rules after `tool-baseline-allow` in `policies.yaml`; appended `internal_agent_comms:{enabled:false}` after `metrics:` in `defaults.yaml`. Backups `policies.yaml.bak-pre-365` / `defaults.yaml.bak-pre-365` (rollback = restore). vh:vh 644. Validated through the demo container's OWN PyYAML (worldtree-worldtree-api-1, e099cd3d) + replicated the exact-set uniqueness guard (both new scope-sets unique). NO restart — boot-cached, b123 deploy activates off the bind-mount. **RESOLVED w/ worldtree-dev (msg `01KXRVBT`):** the uniqueness guard is INTERNAL-TIERS-ONLY (iterates only over internal/internal-restricted vs all others), so the pre-existing `free`==`pro` dup (`["tool.*","pending.read"]`) does NOT boot-block — it's cosmetic (audit-label-only tier-inference ambiguity, identical rules); worldtree-dev flags it operator-side as a possible cleanup, out of #365 scope. Arc closed at **v1.0.0b125** (b123 + heid code-review + bug-hunt folds; none touch the 2 config files → staged blocks byte-exact vs all three). b125 added 2 more boot checks (non-empty internal scope-set; a model.use rule covering both internal tiers) — staged copies PASS both. **b125 PUSHED (main 877bb57):** DEMO auto-deployed to b125 + healthy (validator passed live against my staged bind-mount); extracted baked config from image `877bb577ad36` (no-mount container) → staged demo #365 blocks BYTE-EXACT vs baked. **PERSONAL STAGED** (operator-directed 2026-07-17 "push it all the way to personal"): same verbatim blocks on `/opt/worldtree-personal/config` (tiers after `suspended:`, 4 rules after `tool-baseline-allow`, gate after `relational_stance` — personal has no metrics block), backups `.bak-pre-365`, byte-exact vs baked, all b125 checks PASS, personal b106 healthy no-restart. **TAG HELD:** worldtree-dev holds `staging/v1.0.0b125` (fires personal deploy) until I confirm personal config-clear; #365 done but flagged personal ALSO missing b107–b125 deltas (defaults blocks `metrics`/`kb_audit`/`reference_knowledge`; rule `affect-debug-observability-allow`) — **awaiting worldtree-dev's boot-required-or-optional ruling** (msg `01KXRWJM`) before they cut the tag. **STILL OPEN:** run staged-vs-built diff on personal after its b125 deploy. See [[reference_corviduo_dev_emergency_ops]].

**Landed this session (2026-07-16→17), all pushed EXCEPT 14a0004:**
- Memory **two-tier migration** (d710e56, 53 detail files, index 60→25 KB) + MEMORY.md auto-index compacted 30.5→14.6 KB (auto-memory, not this repo).
- **image-bench evicted** from ana-ml2 GPU1 + `image-judge`/`qwen-image-bench` gateway aliases → gen `:8015` (925947c); comfy-dev cleared the backstop.
- **ana-ml2 GPU re-org** (f25f494, a2b026d, 9e69639): char-rp→GPU1; gen + char-rp-reasoning → 256K native; granite → 128K full-chapter + `max-num-seqs 1024`. GPU0 ~14 G reserve, GPU1 ~6.7 G headroom (details in the `[2026-07-16]` entries).

**Carried standing (non-blocking):** ana-ml2 GPU0 ~14 G reserve (both seats native — for a future tenant); Worldtree #363 research-wing ingest (auto-memory, no deadline); T1 SFT LoRA dormant; rotate the 5 rest-server backup creds (operator, offline).

## Recent decisions

- `[2026-07-17]` **Zonos2 :1920 → self-contained container (stays on 3090); prosody-priming is adapter-level, engine stays stock.** Config captured (14a0004, unpushed); build = cu128 base + `uv sync` vs the lock + weights mount; priming = prime→generate-one-utterance→parakeet-clip→deliver in the gateway adapter. Crux = does AR prosody carry the sentence boundary (A/B the join). → `persistent-memory.d/2026-07-17-zonos2-containerize-prosody-priming.md`

- `[2026-07-16]` **GPU re-org: char-rp→GPU1 + both cards re-optimized for max context.** Moved char-rp (Magidonia-24B) GPU0→GPU1, then maxed context: char-rp-reasoning 150K→256K (util 0.46, 1.56x), gen→256K + seqs 16→32 (util 0.42, 5.43x), granite 64K→**128K full-chapter** (util 0.27, 1.50x). FINAL: GPU0 ~14 G reserve (both seats 256K native), GPU1 ~6.7 G headroom. All healthy. LESSON: KV must hold ≥1× max-len (util-floor crashes) + per-model KV cost varies ~8× (MoE cheap, dense pricey) → tune util empirically. See Current state for the full layout + backups.

- `[2026-07-16]` **granite right-sized → ~10.5 GB freed on GPU1** (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536, summarizer healthy). GPU1 now ~45 GB free to relocate a GPU0 model. LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped (est max-len 47184<65536, ~2-3 min summarizer blip) before 0.18 landed. `.env`-only, recreate `vllm-granite` alone (shared stack).

- `[2026-07-15]` **image-bench eviction DONE (parked item closed).** Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM `image-judge`+`qwen-image-bench` → gen :8015 (judge samplers + thinking-off), verified with :8014 down; comfy-dev pinged; also backfilled the canonical char-rp-reasoning litellm block (was lagging live). Revert ~90 s. auto-memory `project_arbo_gen_switch_imagebench_evict`.

- `[2026-07-15]` arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake → `persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md`

- `[2026-07-15]` esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service` → `persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md`

- `[2026-07-15]` **Homepage AI-tab revamp** — flat "AI Systems" group -> dedicated AI tab, 6 role-based groups + AI-Dormant; committed `569e1af`, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.)

- `[2026-07-15]` **Home Assistant config repo created** (`vh/home-assistant-config`, private). UI-managed HA -> allowlist model (YAML + curated secret-free `.storage` subset). git-in-place in `/config` on esh-docker-vm + scoped deploy key + local clone `~/development/home-assistant-config`.

- `[2026-07-15]` **char-rp-reasoning OOM rescue** — solo-restart on the packed GPU0 crash-looped; fixed via `expandable_segments:True` + util 0.39->0.38 + max-model-len 192K->150K. LESSON (Tried): `max-model-len` does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom now.

- `[2026-07-15]` **soong-lab `SOONG_LAB_LIBRARY_DIR` made persistent** (corviduo-dev) — was on the redeploy-wiped code default; set to `/home/infra-ops/soong-lab-data/library` (mirrors PORTRAIT_DIR), restarted. Closed a queued no-rush item; unblocked the operator.

- `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost (`.cost.total_cost_usd`) / threshold-colored ctx+rate (green<60 / yellow60-90 / red>90).

- `[2026-07-14]` NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned → `persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md`

- `[2026-07-14]` NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP → `persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md`

- `[2026-07-14]` Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion → `persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md`

- `[2026-07-14]` char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) → `persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md`

- `[2026-07-14]` soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST` → `persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md`

- `[2026-07-13]` #355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN) → `persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md`

- `[2026-07-13]` Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong) → `persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md`

- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONE → `persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md`

- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 206 → `persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md`

- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync  → `persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md`

- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); → `persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md`

- `[2026-07-10]` Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev R34-v1 request) → `persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md`

- `[2026-07-10]` ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, SageAttention preserved, crash-loop AVOIDED → `persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md`

- `[2026-07-10]` ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it) → `persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md`

- `[2026-07-10]` Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized) → `persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md`

- `[2026-07-09]` Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker docker log-cap SOLVED no-bounce → `persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md`

- `[2026-07-09]` granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed, #335 Slice-4) → `persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md`

- `[2026-07-09]` mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into gateway-chat → `persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md`

- `[2026-07-09]` granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4) → `persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md`

- `[2026-07-08]` RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (3… → `persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md`

- `[2026-07-08]` worldtree Mimir deploy-blocker resolved (mid-session): → `persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md`

- `[2026-07-08]` OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training → `persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md`

- `[2026-07-08]` DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call) → `persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md`

- `[2026-07-08]` T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py → `persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md`

- `[2026-07-08]` T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe → `persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md`

_142 older entries archived to archival-memory.md._

## Tried and abandoned

- `[2026-07-15]` `docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts → `persistent-memory.d/2026-07-15-docker-service-after-remote-fs-target-does-not.md`

- `[2026-07-15]` The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT → `persistent-memory.d/2026-07-15-the-esh-docker-vm-d-state-phantom-container.md`

- `[2026-07-15]` vLLM `max-model-len` does NOT free GPU VRAM → `persistent-memory.d/2026-07-15-vllm-max-model-len-does-not-free-gpu.md`

- `[2026-07-15]` Claude Code statusline `.cost.total_cost_usd` is per-SESSION → `persistent-memory.d/2026-07-15-claude-code-statusline-cost-total-cost-usd-is.md`

- `[2026-07-14]` MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround → `persistent-memory.d/2026-07-14-mtp-on-modelopt-no-checkpoint-config-skips-the.md`

- `[2026-07-14]` AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof → `persistent-memory.d/2026-07-14-aeon-s-working-nvfp4-mtp-rp-seat-was.md`

- `[2026-07-14]` NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept → `persistent-memory.d/2026-07-14-nvfp4-llm-compressor-compressed-tensors-gives-no-batch.md`

- `[2026-07-14]` NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent → `persistent-memory.d/2026-07-14-nvfp4-spike-built-the-full-mtp-serve-scaffolding.md`

- `[2026-07-14]` MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained` → `persistent-memory.d/2026-07-14-mtp-graft-via-top-level-mtp-tensor-names.md`

- `[2026-07-14]` gitea "test-delivery 204" is NOT proof a webhook works → `persistent-memory.d/2026-07-14-gitea-test-delivery-204-is-not-proof-a.md`

- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data → `persistent-memory.d/2026-07-13-relaying-a-peer-s-diagnosis-as-fact-without.md`

- `[2026-07-13]` `althing-cli reply <THREAD_id>` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to <peer>`. Bit me several times this session.

- `[2026-07-09]` FP8 breaks mOrpheus audio-token generation → `persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md`

- `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter,
  `has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs.
  (`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.)

- `[2026-07-09]` **Per-frame CPU SNAC decode is too slow for streaming** — per-call overhead × ~60 frames serialized
  → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx]
  window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).

- `[2026-07-09]` Sentence-chunking TTS loses prosody → `persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md`

- `[2026-07-09]` HF whisper datasets aren't actually whispered → `persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md`

- `[2026-07-08]` Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE → `persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md`

- `[2026-07-08]` Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24) → `persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md`

- `[2026-07-08]` Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base → `persistent-memory.d/2026-07-08-pantheon-reasoning-27b-refuses-dark-fiction-despite-an.md`

- `[2026-07-08]` Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance → `persistent-memory.d/2026-07-08-pantheon-27b-mtp-on-vllm-compressed-tensors-0.md`

- `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639) → `persistent-memory.d/2026-07-07-vllm-0-24-0-qwen3-5-lora-application.md`

- `[2026-07-07]` SGLang generic image can't LOAD our NVFP4 AEON → `persistent-memory.d/2026-07-07-sglang-generic-image-can-t-load-our-nvfp4.md`

- `[2026-07-07]` SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for → `persistent-memory.d/2026-07-07-sglang-lora-target-modules-cli-enum-rejects-the.md`

- `[2026-07-07]` Engine invocation footguns cost several wasted serve-bounces this session → `persistent-memory.d/2026-07-07-engine-invocation-footguns-cost-several-wasted-serve-bounces.md`

- `[2026-07-04]` LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge → `persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md`

- `[2026-07-04]` A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools needs an explicit `Environment=PATH` → `persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md`

- `[2026-07-04]` On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS → `persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md`

- `[2026-07-01]` A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" is usually the pull-only-vs-build RACE, not registry-auth → `persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md`

- `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS
  moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%,
  N=4 −20%). Reserve for dedicated/interactive deployments.

- `[2026-07-02]` **irv-ml1 `/worktank` ROOT is root-owned — lkraven can't write there (irv-ml1 sudo
  needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA
  ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-<uuid>`.

_101 older entries archived to archival-memory.md._