# Persistent memory — eshpfi-management

_Last updated: 2026-07-15_

> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under an hour old, read it (it carries the in-flight
> handoff from the previous session), then delete it. Older than an hour:
> stale — delete it unread.

## Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical
Docker Compose stacks, ops playbooks, and conventions. Authoritative
copies of compose files live on the servers under
`/opt/docker/compose/<stack>/`; this repo mirrors them for version
control, editing, planning, and CI-driven deploys. **It was originally
spun up to handle the fleet backups** — keep that lens when triaging
backup/storage issues.

## Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

| Repo | Role | CI status |
|---|---|---|
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) |
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
| `vh/althing` | Lean trusted inter-agent message bus — **v2 "email model" (v2.0.0b2, 2026-07)**: per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API `/owner/*` / `althing-mcp` stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. | per-box `uv tool install` (NOT CI-deploy); **nh3-dev = the DEV box** (editable install of `~/development/althing`, gets new versions first); **nh3-extdev** a mesh peer (model B: althing-svc + shared `/srv/althing`) |
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) |
| `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |
| `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook |
| `model-training-forge` (mtf-dev) | Fine-tuning recipe forge; **T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06)** (SFT→DPO, LitBench-RM reward) | training runs, not a deployed sidecar |

(`vh/volva` + Heid were re-architected from systemd daemons to Claude Code
session orchestrators 2026-06-08; their nh3-dev `.service` units were removed —
no longer deployed sidecars here. See Recent decisions.)

- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
  fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
  VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
  cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
  for the blast-radius matrix. **⚠️ The restic file+DB layer routes
  through TWO rest-servers** (`rest-server-ana` @ ana-docker:8000 →
  ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; `rest-server-nh3` @
  nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS
  export of `/mnt/backup`. (rest-server-ana recovered 2026-06-20.)

- **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
  model/dataset onto ana-ml2's shared cache at
  `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns.

- **Worldtree admin auth — per-instance.** Each Worldtree deployment
  (demo :8080, personal :8081, pinned :8082) has its own Heimdall
  registry and its own bootstrap admin key. Infra-ops's stored
  long-lived admin key (`key_id 61419c92`) at
  `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
  auths against **demo only**. Personal-instance admin (the
  `~/.config/worldtree/personal-admin-token`, mode 600) POSTs
  `/admin/keys` (mints per-project keys; takes `user_id`+`label`, **no
  scope param** — scopes are tier-derived). **On-instance mint recipe
  (cleaner than DB-manip):** `docker exec worldtree-worldtree-api-1` POST
  `/admin/keys` with the in-container `WORLDTREE_BOOTSTRAP_ADMIN_KEY`; cleartext
  once in `.key`=`wt_live_+16hex`. auto-memory `reference_worldtree_demo_key_mint`.

- **Per-project user keys against personal Worldtree** (issued
  2026-05-19): `skaldsong:79744637`, `skaldsong:7c1dbbbe`,
  `althing:50d85460`, `mead-hall:a360822d`. Mint via `/admin/keys`, drop
  value to `/tmp/wt-personal-<name>.key` mode 600, dev collects +
  shreds (DO NOT cat to chat transcript).

- **Skaldsong CD pattern (registry-pull).** vh/skaldsong's CI builds and
  pushes `gitea.phasefinal.com/vh/skaldsong:<sha>` + `:latest`;
  `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates.
  SHA-pin only. Prereq: host needs `docker login gitea.phasefinal.com` once.

- **gitea internal route for fleet hosts.** gitea is a container on
  **ana-docker** — git-SSH `10.250.50.70:222`, HTTP `:3000`. Fleet/colo
  hosts must use this internal route, NOT public `gitea.phasefinal.com`
  (`38.120.12.44`) — the public path fail2bans the host egress IP. Full
  gotcha in `docs/orientation.md` → Git/gitea.

- **docker-as-root pattern** (for ops with no admin API, or to edit
  deploy-owned/root-owned files without sudo): `docker run --rm -v
  <target-dir>:/wt docker:cli sh -c "..."`. docker-group membership is
  effectively root via bind-mount. **Foot-gun: relative paths in compose.yaml
  resolve against the sandbox CWD but the daemon interprets them against the
  HOST fs — always pass `-e VAR=/abs/path` for any relative-default config dir.**

- **`scripts/elway` sudo handling** — elway prompts for the sudo password
  ONCE via `getpass` before the first `sudo: true` step → can't run
  unattended from a non-TTY tool if any step needs sudo. Sudo-free
  playbooks run fully non-interactive over key SSH.

- **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo
  on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On
  ana-docker: **default `ssh ana-docker` = `lkraven`** (docker-group, NO
  passwordless sudo); **`ssh infra-ops@ana-docker` HAS NOPASSWD root**. **→
  For any sudo op on ana-docker, use `ssh infra-ops@ana-docker`.** `ssh
  infra-ops@10.100.10.50` (nh3-dev) ALSO NOPASSWD sudo; on **nh3-extdev** infra-ops
  is sudo-LESS by design (`ssh lkraven@10.100.50.42` is the NOPASSWD path). **irv-ml1:
  `ssh irv-ml1` = lkraven, docker-group (plain docker) but sudo needs a PASSWORD
  (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.

## Current state / in-flight

_As of 2026-07-15 — a long fleet-ops + infra session. Everything below either LANDED (committed/pushed) or is PARKED on a clean async watch. No blocking work in flight._

**Landed this session (2026-07-14 -> 15):**
- **Homepage revamp** — the flat ~22-service "AI Systems" group split into a dedicated **AI tab** with 6 role-based groups (Inference / Eval & Retrieval / Gateways & Chat / Speech (TTS) / Audio Tools / Image & Media) + an **AI - Dormant** group for the 7 stopped stacks. Relabeled 22 live compose files (`homepage.group=`) via surgical live `sed` + canonical backfill; GPU0 seats recreated in the operator's VRAM order (char-rp -> gen -> char-rp-reasoning). Committed `569e1af`, pushed. Homepage runs on esh-docker-vm `:5100` (config `configs/homepage/`).
- **esh-docker-vm rescued + NFS hardened** — uptime-kuma + paperless-web were wedged in the D-state/phantom class; **only a host REBOOT cleared it** (infra-ops sudo). **NFS fstab properly fixed** (`21d9a07`, pushed): added `x-systemd.before=docker.service,x-systemd.mount-timeout=30` to the 4 `10.0.50.50:` lines — the prior `_netdev,nofail`+`After=remote-fs.target` was silently defeated by `nofail`. Playbook `playbooks/fix-esh-nfs-boot-ordering.yaml` updated. Also: **mOrpheus killed** (irv-ml1 `docker compose down`; backend OOM-looping on the packed 3090) and a **1.36 GB HA coredump cleared**.
- **Home Assistant config repo** — `vh/home-assistant-config` (private gitea) for the HA Container at `10.0.50.46:8123` (= `homeassistant` on esh-docker-vm, macvlan, config named-volume). **git-in-place in `/config`** (root/infra-ops) + scoped deploy key; **local clone `~/development/home-assistant-config`**. UI-managed HA -> allowlist model (YAML + curated secret-free `.storage` subset; secrets/DB/coredump gitignored).
- **soong-lab library persistence** (corviduo-dev) — `SOONG_LAB_LIBRARY_DIR=/home/infra-ops/soong-lab-data/library` added to soong-lab.env (mirrors PORTRAIT_DIR), `soong-lab-studio.service` restarted. Unblocked the operator (Sindra import -> library save survives redeploys).
- **char-rp-reasoning OOM rescue + tune** — the NVFP4+MTP seat (GPU0) crash-looped on a solo restart (GPU0 too tight). Fixed: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` + `REASONING_GPU_MEM_UTIL 0.39->0.38` + `REASONING_MAX_MODEL_LEN 196608->153600` (150K). Now ~4.5 GB GPU0 headroom, healthy; brokkr's text battery passed.
- **Statusline overhaul** (`~/.claude/statusline-command.sh`, NOT this repo) — git state (branch/dirty/unpushed), 🔔/🔕 althing-monitor-armed, project tag, absolute tokens, **per-session cost** (`.cost.total_cost_usd`), threshold-colored ctx% + rate %s (green<60 / yellow60-90 / red>90).

**PARKED (operator-directed holds):**
- **image-bench (qwen-image-bench, GPU1) eviction — DONE 2026-07-15 (operator-directed, same-day as the switch).** Stopped `vllm-qwen-image-bench` on ana-ml2 (Exited(0), revert via `docker compose start`) → GPU1 freed ~32 GB (95→63 GB used). LiteLLM `image-judge` + `qwen-image-bench` aliases repointed to gen (:8015, qwen3.6-35b-a3b-heretic), deterministic judge sampling (temp0/top_k1/rep_pen1.05) + `enable_thinking:false` kept; verified live with :8014 down. comfy-dev pinged as regression watcher (NSFW-anatomy + tight-grounding; thread 01KXMRDJ). Backstop = the stopped stack (revert ~90 s) + live config backup `config.yaml.bak-pre-imagejudge-20260715`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`.
- **granite right-size — un-started GPU1 win.** granite (GPU1) is ~29 GB (util 0.34 -> ~16.5 GB oversized KV for an 8B summarizer). Right-sizing util 0.34 -> ~0.13 frees ~16 GB WHILE keeping the summarizer (fleet's dedicated summarizer/classifier; aliases granite-4.1-8b/summarizer/classifier). Independent of image-bench; operator was freeing GPU1 for a TBD "something else".

**Open loops for the fresh session:**
- **`stacks/heretic2-charrp-reasoning/` is UNTRACKED in git** (`?? stacks/heretic2-charrp-reasoning/`) — the char-rp-reasoning compose stack is deployed but never committed; commit it (canonical mirror of the live seat).
- **althing monitor** — re-arm `/althing:monitor` in the fresh session (the wake-listener dies on /clear). Open watch: comfy-dev on the arbo bake (no news = clean).
- **Carried standing items** (non-blocking): T1 SFT LoRA dormant (operator pivoted off-the-shelf); rotate the 5 rest-server backup creds (operator, offline); brokkr P00 handoff for the NVFP4 char-rp-reasoning seat (in live traffic ahead of it).

## Recent decisions

- `[2026-07-15]` **image-bench eviction DONE (parked item closed).** Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM `image-judge`+`qwen-image-bench` → gen :8015 (judge samplers + thinking-off), verified with :8014 down; comfy-dev pinged; also backfilled the canonical char-rp-reasoning litellm block (was lagging live). Revert ~90 s. auto-memory `project_arbo_gen_switch_imagebench_evict`.

- `[2026-07-15]` arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake → `persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md`

- `[2026-07-15]` esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service` → `persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md`

- `[2026-07-15]` **Homepage AI-tab revamp** — flat "AI Systems" group -> dedicated AI tab, 6 role-based groups + AI-Dormant; committed `569e1af`, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.)

- `[2026-07-15]` **Home Assistant config repo created** (`vh/home-assistant-config`, private). UI-managed HA -> allowlist model (YAML + curated secret-free `.storage` subset). git-in-place in `/config` on esh-docker-vm + scoped deploy key + local clone `~/development/home-assistant-config`.

- `[2026-07-15]` **char-rp-reasoning OOM rescue** — solo-restart on the packed GPU0 crash-looped; fixed via `expandable_segments:True` + util 0.39->0.38 + max-model-len 192K->150K. LESSON (Tried): `max-model-len` does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom now.

- `[2026-07-15]` **soong-lab `SOONG_LAB_LIBRARY_DIR` made persistent** (corviduo-dev) — was on the redeploy-wiped code default; set to `/home/infra-ops/soong-lab-data/library` (mirrors PORTRAIT_DIR), restarted. Closed a queued no-rush item; unblocked the operator.

- `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost (`.cost.total_cost_usd`) / threshold-colored ctx+rate (green<60 / yellow60-90 / red>90).

- `[2026-07-14]` NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned → `persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md`

- `[2026-07-14]` NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP → `persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md`

- `[2026-07-14]` Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion → `persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md`

- `[2026-07-14]` char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) → `persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md`

- `[2026-07-14]` soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST` → `persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md`

- `[2026-07-13]` #355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN) → `persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md`

- `[2026-07-13]` Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong) → `persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md`

- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONE → `persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md`

- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 206 → `persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md`

- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync  → `persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md`

- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); → `persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md`

- `[2026-07-10]` Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev R34-v1 request) → `persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md`

- `[2026-07-10]` ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, SageAttention preserved, crash-loop AVOIDED → `persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md`

- `[2026-07-10]` ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it) → `persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md`

- `[2026-07-10]` Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized) → `persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md`

- `[2026-07-09]` Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker docker log-cap SOLVED no-bounce → `persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md`

- `[2026-07-09]` granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed, #335 Slice-4) → `persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md`

- `[2026-07-09]` mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into gateway-chat → `persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md`

- `[2026-07-09]` granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4) → `persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md`

- `[2026-07-08]` RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (3… → `persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md`

- `[2026-07-08]` worldtree Mimir deploy-blocker resolved (mid-session): → `persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md`

- `[2026-07-08]` OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training → `persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md`

- `[2026-07-08]` DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call) → `persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md`

- `[2026-07-08]` T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py → `persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md`

- `[2026-07-08]` T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe → `persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md`

_142 older entries archived to archival-memory.md._

## Tried and abandoned

- `[2026-07-15]` `docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts → `persistent-memory.d/2026-07-15-docker-service-after-remote-fs-target-does-not.md`

- `[2026-07-15]` The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT → `persistent-memory.d/2026-07-15-the-esh-docker-vm-d-state-phantom-container.md`

- `[2026-07-15]` vLLM `max-model-len` does NOT free GPU VRAM → `persistent-memory.d/2026-07-15-vllm-max-model-len-does-not-free-gpu.md`

- `[2026-07-15]` Claude Code statusline `.cost.total_cost_usd` is per-SESSION → `persistent-memory.d/2026-07-15-claude-code-statusline-cost-total-cost-usd-is.md`

- `[2026-07-14]` MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround → `persistent-memory.d/2026-07-14-mtp-on-modelopt-no-checkpoint-config-skips-the.md`

- `[2026-07-14]` AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof → `persistent-memory.d/2026-07-14-aeon-s-working-nvfp4-mtp-rp-seat-was.md`

- `[2026-07-14]` NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept → `persistent-memory.d/2026-07-14-nvfp4-llm-compressor-compressed-tensors-gives-no-batch.md`

- `[2026-07-14]` NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent → `persistent-memory.d/2026-07-14-nvfp4-spike-built-the-full-mtp-serve-scaffolding.md`

- `[2026-07-14]` MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained` → `persistent-memory.d/2026-07-14-mtp-graft-via-top-level-mtp-tensor-names.md`

- `[2026-07-14]` gitea "test-delivery 204" is NOT proof a webhook works → `persistent-memory.d/2026-07-14-gitea-test-delivery-204-is-not-proof-a.md`

- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data → `persistent-memory.d/2026-07-13-relaying-a-peer-s-diagnosis-as-fact-without.md`

- `[2026-07-13]` `althing-cli reply <THREAD_id>` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to <peer>`. Bit me several times this session.

- `[2026-07-09]` FP8 breaks mOrpheus audio-token generation → `persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md`

- `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter,
  `has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs.
  (`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.)

- `[2026-07-09]` **Per-frame CPU SNAC decode is too slow for streaming** — per-call overhead × ~60 frames serialized
  → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx]
  window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).

- `[2026-07-09]` Sentence-chunking TTS loses prosody → `persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md`

- `[2026-07-09]` HF whisper datasets aren't actually whispered → `persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md`

- `[2026-07-08]` Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE → `persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md`

- `[2026-07-08]` Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24) → `persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md`

- `[2026-07-08]` Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base → `persistent-memory.d/2026-07-08-pantheon-reasoning-27b-refuses-dark-fiction-despite-an.md`

- `[2026-07-08]` Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance → `persistent-memory.d/2026-07-08-pantheon-27b-mtp-on-vllm-compressed-tensors-0.md`

- `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639) → `persistent-memory.d/2026-07-07-vllm-0-24-0-qwen3-5-lora-application.md`

- `[2026-07-07]` SGLang generic image can't LOAD our NVFP4 AEON → `persistent-memory.d/2026-07-07-sglang-generic-image-can-t-load-our-nvfp4.md`

- `[2026-07-07]` SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for → `persistent-memory.d/2026-07-07-sglang-lora-target-modules-cli-enum-rejects-the.md`

- `[2026-07-07]` Engine invocation footguns cost several wasted serve-bounces this session → `persistent-memory.d/2026-07-07-engine-invocation-footguns-cost-several-wasted-serve-bounces.md`

- `[2026-07-04]` LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge → `persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md`

- `[2026-07-04]` A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools needs an explicit `Environment=PATH` → `persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md`

- `[2026-07-04]` On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS → `persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md`

- `[2026-07-01]` A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" is usually the pull-only-vs-build RACE, not registry-auth → `persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md`

- `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS
  moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%,
  N=4 −20%). Reserve for dedicated/interactive deployments.

- `[2026-07-02]` **irv-ml1 `/worktank` ROOT is root-owned — lkraven can't write there (irv-ml1 sudo
  needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA
  ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-<uuid>`.

_101 older entries archived to archival-memory.md._