# Persistent memory — eshpfi-management _Last updated: 2026-09-02_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight > handoff from the previous session), then delete it. Older than an hour: > stale — delete it unread. ## Repo purpose Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under `/opt/docker/compose//`; this repo mirrors them for version control, editing, planning, and CI-driven deploys. **It was originally spun up to handle the fleet backups** — keep that lens when triaging backup/storage issues. ## Tools and conventions Sister repos (separate gitea repos, deployed by playbooks here): | Repo | Role | CI status | |---|---|---| | `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) | | `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) | | `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) | | `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) | | `vh/althing` | Lean trusted inter-agent message bus — **v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback)**: ONE container on nh3-dev at `http://10.100.50.40:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. **Every v2 command was DELETED, not deprecated** — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` | per-box install (NOT CI-deploy); **nh3-dev** = container host + repo; **nh3-extdev** = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`) | | `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) | | `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) | | `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) | | `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` | | `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook | | `vh/zonos-gateway` | OpenAI-compatible TTS gateway over stock ZONOS2 (`:8890` irv-ml1); emotion **dials-first** + voice mapping; reached via LiteLLM `ext-tts` alias. **v0.2.1 (2026-07-18): voice-resolved emotion presets** (`resolve_preset(name,voice)`; angry/happy/startled_happy per-voice). 8 voices incl. 4 clones | pushed to gitea (main `8f1885b`/`v0.2.1`); **deployed irv-ml1 tree still NON-git** (hand-updated build context — CI-wire = open follow-up). Spec `docs/EMOTION-DIALS-SPEC.md`; host-managed voices bind-mount (`./voices:/app/voices`, drop wav + restart, no rebuild) | | `vh/soong-lab` | Noonien Soong character-design studio (SPA + /api + WT `/bifrost/tool-call`); **containerized 2026-07-18**, LIVE on corviduo-dev `:8443` (image `vh/soong-lab:latest`). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host | CI = Gitea Actions build+push+**DEPLOY** on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; **auto-redeploy LIVE 2026-07-18** — runner SSHes corviduo-dev as `deploy`, `compose pull && up -d` from **/opt/soong-lab**, health-gated on /api/version). Manual redeploy `sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`. → `archival-memory.md` (archived 2026-08-16) | | `model-training-forge` (mtf-dev) | Fine-tuning recipe forge; **T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06)** (SFT→DPO, LitBench-RM reward) | training runs, not a deployed sidecar | (`vh/volva` + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev `.service` units were removed — no longer deployed sidecars here. See Recent decisions.) - **Two-layer backups** — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see `docs/runbooks/disaster-recovery.md` for the blast-radius matrix. **⚠️ The restic file+DB layer routes through TWO rest-servers** (`rest-server-ana` @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; `rest-server-nh3` @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of `/mnt/backup`. (rest-server-ana recovered 2026-06-20.) - **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns. - **Worldtree admin auth — per-instance.** Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (`key_id 61419c92`) at `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin` auths against **demo only**. Personal-instance admin (the `~/.config/worldtree/personal-admin-token`, mode 600) POSTs `/admin/keys` (mints per-project keys; takes `user_id`+`label`, **no scope param** — scopes are tier-derived). **On-instance mint recipe (cleaner than DB-manip):** `docker exec worldtree-worldtree-api-1` POST `/admin/keys` with the in-container `WORLDTREE_BOOTSTRAP_ADMIN_KEY`; cleartext once in `.key`=`wt_live_+16hex`. auto-memory `reference_worldtree_demo_key_mint`. - **Per-project user keys against personal Worldtree** (issued 2026-05-19): `skaldsong:79744637`, `skaldsong:7c1dbbbe`, `althing:50d85460`, `mead-hall:a360822d`. Mint via `/admin/keys`, drop value to `/tmp/wt-personal-.key` mode 600, dev collects + shreds (DO NOT cat to chat transcript). - **Skaldsong CD pattern (registry-pull).** vh/skaldsong's CI builds and pushes `gitea.phasefinal.com/vh/skaldsong:` + `:latest`; `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs `docker login gitea.phasefinal.com` once. - **gitea internal route for fleet hosts.** gitea is a container on **ana-docker** — git-SSH `10.250.50.70:222`, HTTP `:3000`. Fleet/colo hosts must use this internal route, NOT public `gitea.phasefinal.com` (`38.120.12.44`) — the public path fail2bans the host egress IP. Full gotcha in `docs/orientation.md` → Git/gitea. - **docker-as-root pattern** (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): `docker run --rm -v :/wt docker:cli sh -c "..."`. docker-group membership is effectively root via bind-mount. **Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass `-e VAR=/abs/path` for any relative-default config dir.** - **`scripts/elway` sudo handling** — elway prompts for the sudo password ONCE via `getpass` before the first `sudo: true` step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH. - **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: **default `ssh ana-docker` = `lkraven`** (docker-group, NO passwordless sudo); **`ssh infra-ops@ana-docker` HAS NOPASSWD root**. **→ For any sudo op on ana-docker, use `ssh infra-ops@ana-docker`.** `ssh infra-ops@10.100.10.50` (nh3-dev) ALSO NOPASSWD sudo; on **nh3-extdev** infra-ops is sudo-LESS by design (`ssh lkraven@10.100.50.42` is the NOPASSWD path). **irv-ml1: `ssh irv-ml1` = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight _As of 2026-09-02 — **nothing is running. Operator stood the session down.** The GX10 is baselined and outfitted but still on the desk on Wi-Fi; run 3c is unlaunched; the Ada box has not arrived._ - **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000 Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores factory 256 GB), 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters, and 2× 2 TB SATA SSD** (the as-bought 1.92 TB pair mirrored is ~1.75 TiB — SMALLER than the 1.82 TiB pool it receives from, landing the migration at ~84% full; pruning cannot fix it, only vdev layout can). ⚠ **The RAM line may be FREE** — operator has 2× Dell R640 spares at 768 GB each; both are DDR4 RDIMM platforms and the Silver 4314s cap this box at 2666 anyway, so R640 DIMMs lose nothing. **768 GB is not a shape the R750xa takes** (16 slots, 8 ch/socket): targets are **512 GB** (16× 32 GB) or **1 TB** (16× 64 GB). ⚠ Confirm DIMM part numbers AND slot count from iDRAC before ordering. ~1 kW loaded — pick the NH3 circuit before racking. → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md` - **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a) `zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving. **TWO decisions with the operator: (1) drives for the R750xa — he is leaning 6, which fills all 8 bays and makes drive SIZE the permanent ceiling; recommend 8 uniform in raidz2 (~10.5 TB at 14%) over 6 mixed, since a raidz vdev caps at its smallest member; (2) retain vs reclaim irv-ml1's `/storetank` after the move — RETAIN recommended** (two paywalled pins mean a from-source rebuild cannot reproduce the stack). comfy-dev's per-file prune audit is offered and **explicitly NOT gating**. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` - **⏸ RUN 3c UNLAUNCHED.** Config `/tank/erp-tune/run-03c.json` validated; 604 steps. ⚠ **The GX10 is the right home if it moves** — an Anaheim breaker trip is a **40-minute drive each way** with **13 hosts dark including three SureFire CLIENT machines**, which no `save_steps` change mitigates. Port scoped (~2.5 GB + an aarch64 venv rebuild) but **not executed** — operator stood it down. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` - **⏸ Worldtree `route_not_found` awaiting the operator's DEPLOY PUSH.** Approved and landed by worldtree-dev as `ece0250c` (wire 2.5.0→2.6.0). main auto-deploys demo. Until it ships, "trust the HTTP status, not `error_code`" still applies on running instances. - **⏸ Deferred, no blocker:** convert the live synapse compose to read `POSTGRES_PASSWORD` from the vault rather than an inline value. - **🔵 Awaiting others:** a **Claude Code restart** to load althing plugin 0.1.1's SessionStart hook (never once run on this box — see the deploy entry); **regin-smithy-dev** on whether their live waiter was meant to give them push (live waiter + `mode: pull`, ambiguous because they intentionally released their pane route); **tts-dev** to ask comfy-dev for a no-render agreement during their FireRedAudio bench, since the A6000's ~44 GB free is a snapshot that evaporates the moment ComfyUI reloads. ## Recent decisions - `[2026-09-02]` **althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInbound: "accept"`.** Without it Claude Code HOLDS every cc poke — it auto-delivers only when the sender's permission-mode class matches, and the herald is a daemon that asserts none, so the notice goes to a human watching the pane instead of to the session. ⚠ **The seat reports declared, reachable and green throughout** — same failure shape as the SessionStart hook that was never deployed. Set on nh3-dev by forseti 09:28 with operator authorization (diff verified: one key, backup at `/tmp/settings.json.bak-20260902T092829`). Operator's reasoning: the herald reaches only local seats and a **pane poke already types+Enters into a session**, so the socket channel is strictly NARROWER than what it replaces — stating the existing trust boundary, not widening it. Cost without it is first-contact-only (in-memory correspondent record), not per-message. ⚠ **No attestation exists for the herald to send** — CC identifies a sender by verified pid against the session registry and reads that session's LIVE runtime mode; a daemon is not in it, and `from_mode` on a `type:"user"` frame is never consulted. **`deploy-althing.sh` reports surface 6 and deliberately never SETS it** — a deploy script that edits its own trust settings grants itself trust. → `docs/runbooks/althing-deploy.md` - `[2026-09-02]` **`vastblue` gitea org created (id 8, private, owner `vh`) with empty repo `vastblue/platform`** — third entity namespace alongside `corviduo` and `pfi`; most repos still live under `vh/`. Home of VastBlueDocumentAI + the anchor healthcare-billing SPA (signed 3-yr client contract). **Org scope was the decision**: org-level runner registration and secrets are inherited free by the DocumentAI repo when it splits out, and that is the only binding expensive to retrofit. Deliberately NOT set: org runner (instance-wide `ana-docker-runner` already serves it; org scope is for the DEDICATED runner, deferred to U10) and org secrets (none exist yet; a guessed secret looks bound). ⚠ **Dedicated runner is gated on the first client-premises release cut**, not on the first green pipeline — the risk is another repo's CI sharing a root-level daemon with a build that ships to a healthcare client, see the runner entry above. Push needs no credential: vastblue-dev is on nh3-dev and git-SSH there auths as `vh`. → `stacks/gitea-runner/README.md` - `[2026-09-02]` **althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green.** CC seats are now poked over their own message socket (`$XDG_RUNTIME_DIR/cc-socks/.sock`) instead of by typing into the pane: no process to reap, nothing near the input line. infra-ops moved to `channel=cc`; **the dwarves stay on pane and their guard-4 exposure is UNCHANGED** (`declare` prefers cc, falls back). ⚠ **An undocumented Claude Code interface, taken deliberately** (operator: the FIFO poker was also an unsanctioned hack — a better instance of a class we already had). Break mode = seat goes pull-only with a logged reason, mail still held. ⚠ **`claude plugin update` matches on the plugin VERSION and declines a content-only change** — 3.3.0 edited plugin content at an unchanged 0.1.1, so the CC cache stayed stale while every version check reported success (delta was docs-only, harmless this time). `deploy-althing.sh` now diffs marketplace vs live cache. ⚠ Ordering: **herald restart BEFORE anything declares cc**, or the seat goes silently pull-only. ⚠ This box was at **3.2.4, not 3.2.5** — rollback target here is 3.2.4. Follow-on 3.3.1: **the statusline bell measured a MECHANISM, not the property** — it read `wake-listener-.lock`, so a cc seat renders 🔕 while push/reachable. Both copies now ask the post office (`reachable` from the status payload) and add 📵 for an outage. ⚠ **TWO COPIES of that script now exist** — `scripts/claude-statusline-command.sh` here (the operator's wired one) and althing's `plugin/scripts/statusline.sh` — independently fixed to the same shape; a drift surface with a countdown, convergence not yet raised with the operator. → `docs/runbooks/althing-deploy.md` - `[2026-09-02]` **Every CI job on the shared `pfi-fleet` runner is root on ana-docker — and `container.valid_volumes: []` does NOT prevent it.** Measured: a job container is uid 0, `/var/run/docker.sock` is mounted by act_runner independently of that list, `docker ps` returns all 49 host containers (gitea itself, synapse, phasefinal-web, adguardhome), `docker compose v2.33.0` on PATH. ⚠ **LOAD-BEARING** — `vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge` all drive buildx through that socket, so it cannot simply be closed; **isolate sensitive builds onto a dedicated runner instead.** Also measured the same night: `services:` containers work (Postgres 16), and **full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves from the local mirrors** — the un-parked half of the github-independence work, needing neither `DEFAULT_ACTIONS_URL=self` nor the act_runner auth path that blocked it on 2026-08-05. Prompted by vastblue-dev's CI-posture question for a client-funded healthcare repo. → `stacks/gitea-runner/README.md` - `[2026-09-02]` **pfi-gx10 BASELINED: 79.36 s/it median on the run-3c shape, and the training stack works on aarch64/sm_121.** Median across 10 timed steps, 0.19% spread, **peak 75.1 / 121.6 GiB — 46 GiB spare**, `attn_resolved: flex_attention`. **6× slower than ana-ml2 where compute predicts 2.7×** → likely memory-bandwidth-bound; **capacity box, not throughput box.** Ruled **bare metal, not Proxmox** (no aarch64 PVE; the GPU is on-package and cache-coherent, so passthrough would partition the unified memory that is the whole point). ⚠ `sm_121` is NOT in torch's arch list — everything JITs from sm_120 PTX, so **warm up before timing anything** (an unwarmed bench read 27 TFLOP/s against a true 93). → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` - `[2026-09-02]` **I priced a failure in the units I happened to be measuring — operator overruled me, correctly.** Recommended run 3c to ana-ml2 by costing a breaker trip as "≤50 steps ≈ 11 min of recompute". It is a **40-minute drive each way** with **13 Anaheim hosts dark, three of them SureFire CLIENT machines**. `save_steps` caps the recompute, never the outage. ⚠ **General form: a metric in hand will volunteer itself as the unit of risk.** → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` - `[2026-09-02]` **althing 3.2.0→3.2.4 deployed, and ALTHING DEPLOY IS FOUR SURFACES not three.** The fourth (plugin) had no runbook step and was frozen at Aug 28 — **missing the SessionStart/SessionEnd hooks and `pane-route.sh` entirely**, so "CC seats re-declare automatically" was never true here. Now one command (`scripts/deploy-althing.sh`). ⚠ `uv tool install .` **without `--force` is a silent no-op**. ⚠ **A missing deploy surface presents as "the migration needs manual work", not as an error.** → `persistent-memory.d/2026-09-01-althing-320-deploy.md` - `[2026-09-01]` **irv-ml1 GPU resident map, and `dots-tts` holds 14,430 MiB against a ~6 GB baseline** — tts-dev's prompt-feature cache, capped at 32 entries after two incidents; the cap still permits a long way of growth. 3090 at 76% behind a warn-only watchdog. ⚠ Restates the GPU-ordering foot-gun: `device_ids: ["1"]` is the A6000 in a container, but a bare native `CUDA_VISIBLE_DEVICES=1` gets the **3090**. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md` - `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md` - `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` - `[2026-09-01]` **Ada migration is `zfs send` (branch a) — and the DESTINATION IS SMALLER THAN THE SOURCE.** 99 MB/s measured; ~3.9 h. ⚠ Measured 2026-09-01: `storetank` = 1.81 TiB pool, 1.45 TiB used, **80% CAP already**, compression off / `compressratio 1.00x` (safetensors are incompressible — no win at recv). Settled payload **~1.47 TiB**; the R750xa's as-bought 2× 1.92 TB mirrored is ~1.75 TiB → arrival at **~84%**. Fix = 2× 2 TB SATA SSD on the buy list (6 bays free) → ~3.57 TiB at ~41% with redundancy; **pair the two NEW drives together** (a mirror vdev caps at its smallest member). ⚠ **Pruning is NOT a substitute** — comfy-dev found ~215 GiB unreferenced, and deleting every byte still lands the as-bought mirror at 72%: the constraint is **vdev layout**, not payload, so the prune audit and the drive purchase are independent and neither gates the cutover. ⚠ **"Onboarded" is not "landed"** — infra-ops read `ALLOC` mid-pull and re-added the whole batch on top, inflating 84% to a quoted 90%. Also: **branch (b)'s original reason was WRONG** — comfy-dev enumerated all 12 containers, only `comfyui` mounts `/storetank`, so (b) was unavailable *during the transition*, not structurally (right conclusion, wrong reason — infra-ops reasoned about the BOX when the question was the MOUNT). Plus the retain-vs-reclaim call and the two-boxes confusion (the Ada box and the GX10 are DIFFERENT machines). → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` - `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md` - `[2026-09-01]` **A named failure class: a correct check aimed at the wrong object.** Six instances in one day across three sessions; re-running the same check cannot catch it. **Recommended for `docs/pfi/training-throughput-playbook.md` §4 — NOT YET WRITTEN, awaiting operator.** → `persistent-memory.d/2026-09-01-wrong-object-measurement.md` - `[2026-09-01]` **Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`. - `[2026-09-01]` **Idle VRAM on this fleet is a RESERVED scratch pool, not waste.** Operator declined raising `vllm-mog-sec` from `gpu-memory-utilization 0.52`: single-user dev fleet, KV headroom nobody will consume is worth less than room for ephemeral models and small training runs. vLLM's "fully utilize gpu memory" startup hint does NOT apply here. Tracked in auto-memory `feedback_idle_vram_is_reserved_not_waste`. - `[2026-08-28]` **althing v3 flag day (U9b) executed, then six releases to 3.1.1 in one afternoon — and the post office MOVED to nh3-docker.** Every v2 command deleted; 73 handles seeded and verified by set difference; 5,043 orphaned wake FIFOs deleted (v2 named them per-session+PID, v3 per-handle). Image now registry-pulled, digest-pinned, under the `claude-bot` namespace. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` - `[2026-08-28]` **A stale `ALTHING_HANDLE` silently reads another agent's inbox and reports it empty — a SECOND route into the failure v3 exists to prevent.** Outbound mis-signing sometimes gets caught; inbound never does. Shipped as a 3.1.1 warning. ⚠ My `session_handles.json` grounding was wrong (v2 artifact, v3 never opens it) and the same stale source had survived inside my statusline rewrite. → `persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md` - `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE, and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured 5.4-18.4 GB, so 27 GB is 3-4 long-lived sessions. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md` - `[2026-08-28]` **`sec` moved to ana-ml2 GPU0 and is serving** (operator-directed) — GPU1 had ~28 GB free against the ~51 GB it reserves, so it could not start there. Re-arms the two-GPU load condition on a circuit that tripped 36h earlier; accepted with the constraint stated. → `persistent-memory.d/2026-08-28-sec-seat-gpu0.md` - `[2026-08-28]` **BELAYED by the operator, both explicitly: (a) a cgroup memory cap on CC sessions, (b) putting ana-gw + ana-wg + one BMC on separate power.** Both were my recommendations; neither is open work. Do not re-raise as new — the atop ramps that would inform (a) are now being collected, so revisit only with a week of data. Tracking surface: this entry. - `[2026-08-28]` **The deployed CC plugin copies are a release step nobody owns.** `sync_skill.sh` covers the SKILL, not the plugin; both copies must be rsync'd from the repo's `plugin/` on every althing release or they carry the previous release's bugs into the live surface. Raised with forseti for their release notes. Tracking surface: althing thread `01M14QHZNDKDK8KH9DN92VF6VE`. - `[2026-08-28]` **althing v3.0.0 flag day (U9b) executed — the post office replaced the P2P bus on both boxes, one-way.** 73 handles seeded and verified by set difference; 5,043 orphaned v2 wake FIFOs deleted (v2 named them per-session+PID and never reaped; v3 names them per-handle, so the leak is bounded by construction); v2 db left inert. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` - `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE — and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured at 5.4-18.4 GB, so 27 GB is 3-4 mature sessions, not the ~66 a 408 MB estimate implies. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md` - `[2026-08-27]` **Run 3 gated: the preregistered rule PASSED and a k=25 follow-up found a 44pp self-harm guardrail collapse — DO NOT SERVE.** A pooled preserve-list test structurally cannot see a single-axis collapse. → `persistent-memory.d/2026-08-27-run3-gate-safety-regression.md` - `[2026-08-27]` **The corpus mix was specified in a unit the optimiser never sees** — 45.8% dialogue by CONTEXT, 24.2% by LOSS. Harness now leads with loss share and calls context a memory budget (`dd5a12e`). → `persistent-memory.d/2026-08-27-mix-specified-in-the-wrong-unit.md` - `[2026-08-27]` **Dose-response: benefit and damage are ONE direction in weight space** — every axis monotone in scale, no knee. The merge-back cannot separate them; vLLM cannot LoRA-serve this MoE at all. → `persistent-memory.d/2026-08-27-dose-response-entanglement.md` - `[2026-08-27]` **Anaheim tripped a power breaker; four guests including the NAS had `onboot` unset and never came back.** Fixed with dependency ordering — ana-nas order=1,up=45 ahead of the databases. ⚠ **ONE CIRCUIT FEEDS THE WHOLE RACK including the firewall serving the public IP** (operator) — so ana-gw, ana-wg and every BMC go down with the load, and there is NO remote management path to Anaheim during a power event. → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md` - `[2026-08-27]` **A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find.** heid's lost panel arms found a live defect in brokkr's `t4_dissect` an hour later. → `persistent-memory.d/2026-08-27-empty-response-as-a-datum.md` - `[2026-08-27]` **Run 3c authorised (lr 20x cut, single variable) and then HELD by the operator after the breaker trip.** Config built and validated at `/tank/erp-tune/run-03c.json`; `save_steps` made configurable in the harness (`0a6bd2e`) because the first launch lost 80 steps with no checkpoint. Tracking surface: commit `0a6bd2e` + that config path. **Relaunch is one command once power is triaged.** - `[2026-08-27]` **An event report with no timestamp is a claim about "now" — and it manufactured a launch that never happened.** brokkr reconstructed a phantom third 3c launch because my 23:03 report narrated a 21:07 kill in the present tense. Every fact in it was true; it was unreadable in sequence. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md` - `[2026-08-27]` **`save_steps` was hardcoded at 100 in the harness** — a claimed provenance entry the run could not have honoured. Made configurable, default unchanged (`0a6bd2e`, 242 tests green). Caught by checking the config carried the change rather than trusting that it had been made. - `[2026-08-27]` **Six defects in run 3's staged build, none of which would have errored** — a dialogue-only survivor list that would have silently dropped 96% of the corpus, an impersonation mask not subsumed by the low-quality mask, kvasir unbounded at 67.8% of context, a `save_pretrained` config-key drop that made the merged model unservable, and the mix-unit error. Every one produced a plausible completed run. Full record `/tank/erp-tune/recipe-r3/RUN-03-BUILD-NOTE.md`. - `[2026-08-27]` **The 18 unpushed eitri-smithy commits are pushed** — run 3's `harness_commit 9d27b4fe` now resolves off-box, verified by fetching into a fresh empty repo rather than trusting the push output. ⚠ **HTTPS push 403s for every gitea token including site-admin; SSH works.** Untracked `__pycache__` (`894fbe8`) because a tracked `.pyc` dirtied the tree and would have stamped `harness_dirty_at_launch: true`. - `[2026-08-26]` **Run 3's corpus is built and HELD — `creative-writing-multiturn` is a MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno**, and the remix promoted two roots that overlap at median jaccard 0.873. Dedup direction reversed to keep the primary source. F1 does not do what the recipe says; F2 does. → `persistent-memory.d/2026-08-26-run3-corpus-and-the-megamix-containment.md` - `[2026-08-26]` **No conversation admission threshold for run 3** (brokkr delegated, then endorsed). Masked context costs the transformer body but NOT the LM head — the harness drops `IGNORE_INDEX` before the 262k-vocab head — so a 20% trim buys ~35 min against a 3-hour run, and it would be a fifth filter confounding a run whose purpose is testing the filters alone. Tracked in the run-3 detail file. - `[2026-08-26]` **Run 2's gate FAILED and is recorded as a FAIL** — T3 constraint −12.0 against a ~1 pt floor. But gate 1 is the result: **T6 spatial +15.0 where run 1 failed the same axis at −3.5**, base swap the only intended variable. Neither run ships; the pair prices what the abliteration cost. Plus the long-form-only stability regression a short-answer gate would have passed, and PIPPA's 123-word clip in the length signal. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md` - `[2026-08-26]` **Run 2 complete, merged, coherence-gated and serving as `erp-tune-v2`** — stock-instruct base, and the mask proven by a −221,712 loss-token delta against byte-identical context. Also the p50 claim I asserted and had to withdraw. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md` - `[2026-08-26]` **Playbook §4 written: "when the artifact lies about itself"** — seven landmines plus a pre-launch checklist, from a night in which *three separate fixes each shipped a check that could not fail*. The unifying line is brokkr's: when you change what an artifact means, every derived artifact keyed on the old meaning is now a liar. Commits `dae6ede` → `d54f256`; the doc is `docs/pfi/training-throughput-playbook.md` (filename kept for inbound links; scope is now wider than the name). - `[2026-08-26]` **Served under a NEW name on a NEW port (`erp-tune-v2` / :8098), never re-pointing `erp-tune-v1`.** Run 1's artifact still exists and is still what that name refers to; re-pointing would be the silent substitution the standing no-false-aliases rule forbids. brokkr independently asked for the same and additionally wants the concrete backing model + date in provenance, not just the alias — an alias has silently changed meaning under recorded results before. - `[2026-08-26]` **DPO stage gated on an axis-list decision that is not mine to make** — `docs/pfi/erp-dpo-stage-prep.md`. No preference data for refusal axes exists; `trl` is not installed; the Gutenberg sets on disk are prose-quality only. ⚠ Do not install `trl` (or anything) into the training venv **while a run is saving** — a resolution that upgrades transformers under a live process can break its save path. - `[2026-08-25]` **The ERP/RP tune COMPLETED in 7.36h and passed its gate on the axis it was built for** — diversity 22x its noise floor, attractor −11.3pt, zero memorisation on both arms. Also the noise-floor near-miss: brokkr was one step from reporting a 13-point T6 regression sitting inside twice his instrument's own variance. → `persistent-memory.d/2026-08-25-erp-tune-run2-complete.md` - `[2026-08-25]` **8.6% MFU was an accounting artifact — real utilisation 17-20%, and the cost was attention on AMPERE kernels.** Two independent methods agreed to 2.6 points. Fixed by bucketing (padding 29.9%→0.0%) plus flex_attention. ⚠ Carries the dynamo recompile-ceiling trap that produced two wrong published conclusions. → `persistent-memory.d/2026-08-25-mfu-root-caused-attention.md` - `[2026-08-25]` **NVFP4A16 serving pipeline built and validated; MERGED WEIGHTS ARE MANDATORY.** vLLM cannot serve a LoRA on ANY Gemma-4 — `get_expert_mapping` is unimplemented and the check branches on MoE-ness, not quantization. Plus the landmine: a `targets=["Linear"]` recipe misses all 11,520 expert tensors silently. → `persistent-memory.d/2026-08-25-nvfp4-serving-pipeline.md` - `[2026-08-25]` **Refusal retention measured (base 0/100 → tuned 29/100, 71 still complying) — but on the WRONG AXIS.** `harmful_behaviors` is general harm; the abliteration was run for explicit fiction. The convenient set with a recorded baseline was not the right one. → `persistent-memory.d/2026-08-25-refusal-retention-probe.md` - `[2026-08-25]` **Worldtree b188 + b189 shipped; bridge extracted to `pfi/wt-matrix-bridge` because `vh` is a USER not an ORG** and no service account can ever publish to a user namespace. Plus the selene catalog entry that lied about what answers, and a #411 diagnosis I got wrong twice before a directory probe settled it. → `persistent-memory.d/2026-08-25-worldtree-b188-b189-and-selene.md` - `[2026-08-25]` **Run 2's base is an OPEN OPERATOR DECISION, deliberately not staged** — four options with materially different safety postures, detailed in Current state. Tracked at althing thread `01M0WQ8W5574KMEVCHCEKEXNS5`. ⚠ Do not let it get filed as a config knob; it is a reversal of the trainee-selection decision. - `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312. - `[2026-08-25]` **The ERP/RP tune LAUNCHED after 12 harness defects and an operator override of the corpus gate.** Four of the twelve would have crashed the run; two were INERT GATES that passed because they could not fail. Run is `/tank/erp-tune/run-01`, harness eitri-smithy `997c4a4`. Full arc — override, defects, sizing, the measured MFU — in the in-flight section and `docs/pfi/gemma4-erp-tune-sizing.md`. - `[2026-08-24]` **char-rp seat swapped to the Gemma-4 26B-A4B MoE; abliterated trainee base staged and measured.** OOM root-caused to `--gpu-memory-utilization` not covering CUDA context (and to gen's footprint GROWING WITH UPTIME); a benchmark finding retracted because it scored below chance; abliteration isolated at −0.6 core points but it MOVES capability rather than removing it. → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md` - `[2026-08-24]` **Serving the tuned ERP model: LoRA-on-NVFP4 PREFERRED, merged weights the expected fallback — and the recorded objection may be STALE.** Operator: "if you CAN load it as a lora, all the better, the issue is that we will want to run nvfp4 weights, which we had some serious trouble with loading loras on top of nvfp4." ⚠ **The archived root-cause says it was NOT NVFP4-specific**: `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application was a silent no-op (#47639, regression from #37912) — adapter loads HTTP 200, zero deltas at inference, proven **quant-agnostic (NVFP4 AND FP8 both inert)** and adapter-format-agnostic by a 3-peer dwarf panel. Fix PR #47640 was OPEN then. **ana-ml2 is FAR past 0.24.0 and the box runs a SPREAD, not one version** (measured 2026-08-24): `gen` on `nightly-311b3513` = **0.27.2rc1.dev150**, `mog-sec` on `nightly-e9d1398d` = 0.26.1rc1.dev1102, the small seats still on 0.24.0, and char-rp/trainee-bench pinned to v0.26.0. ⚠ **`vllm/vllm-openai:v0.27.1` is already ON DISK, unused** — a TAGGED release, which is the right retest target: no nightly variance, no pull, ~4 months past the diagnosis. So: RETEST hot-swap LoRA on **v0.27.1** before designing around merge — it is cheap, and if it works the post-tune gate can be two aliases on one engine. If it still no-ops, merged weights it is, which means the harness must EMIT merged weights and Eitri needs that in the contract while he is early. Tracked at this snapshot commit; settle it in the QLoRA sizing conversation. - `[2026-08-24]` **Homepage rebuilt on Australis Skyfall; light mode shipped.** Two findings worth more than the theme: **(a)** the Skyfall bundle including its canonical light ramp was sitting in this repo's git history at `45c1995` — check `git show` before concluding a vendored design asset is lost; **(b)** removing `theme:` from `settings.yaml` deterministically breaks the dashboard render (six recreates empty, restoring the key fixed it in 12s), which is the first confirmed cause of the "tab bar goes missing" symptom. Retires the `homepage.log` size lead from earlier the same day — it did nothing on this episode. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - `[2026-08-24]` **Homepage reorganised on the axis "do I open this?" — UI groups expanded on top, API/agent groups collapsed at the bottom** (operator-delegated: "re-categorize however you want"). Load-bearing constraint: `homepage.group` is read at container CREATION, so the 16 GPU-backed model seats keep their unlovely names rather than eat a recreate — `initiallyCollapsed` + order is free. Second rule discovered here: **group members should all have widgets or none should**, because a stat strip adds ~50px and opens a void beside plain cards. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - `[2026-08-24]` **Homepage columns unified at 4 for every group; the 2026-08-18 "columns = member count" rule is retired.** It was avoiding dead cells in a short last row and bought a worse defect — card width changing at every group boundary. Also carries two CSS traps: `overflow: hidden` clips at the PADDING box (so a `padding-right` gutter is spill room, not a guard), and a `:root` override of a Homepage theme variable is silently outranked by `.theme-slate` on the same `` element. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` - `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md` - `[2026-08-24]` **Scriberr deployed on ana-ml2 GPU1, image built from source.** Three upstream bugs: the Blackwell image was never published, it must run as uid 10001, and `UV_LINK_MODE=copy` is required or two backends fail silently. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md` - `[2026-08-24]` **ESH DNS fixed at the IPv6 layer and the naming scheme went live on three hosts.** UniFi's RDNSS cannot be disabled but CAN be redirected — the field is only honoured when an explicit server is given. → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md` - `[2026-08-24]` **`speaches` on irv-ml1 stopped, stack retained** — Eyra was abandoned pre-implementation (Scriberr covers the need), leaving it no consumer. Disposition confirmed to eyra-dev; one command to restart. Tracked at althing thread `01M0RRJX8GPZEBDHF1E3W18RZF`. - `[2026-08-24]` **esh-vm-db brought onto the fleet infra-ops identity and given its first vaulted credential.** It previously had none: root and infra-ops refused key auth and `lkraven`'s sudo wanted a password nobody held, leaving `qm guest exec` from the hypervisor as the only privileged path. Break-glass root password at `secret get esh-vm-db/root-breakglass-password` (console-only; plaintext never crossed the wire — only its SHA-512 hash did). - `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction.** brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`. - `[2026-08-23]` **Anaheim's IPsec tunnel ceiling — investigated, then CLOSED 2026-08-24.** The 25%-of-2-Gbps framing was wrong (NH3's uplink is 1 Gbps); AES-GCM proved impossible; AES-128 landed instead. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` - `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate. → `persistent-memory.d/2026-08-23-selene-retired-alias-collision.md` - `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content. → `persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md` - `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into. → `persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md` - `[2026-08-23]` **Every secret-bearing `.env` on ana-docker tightened to 0600** — eight stacks including vaultwarden and traefik, verified exposed by reading one as `nobody`. → `persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md` - `[2026-08-23]` **`pfi` gitea org created; claude-bot is an Owner and creates repos self-serve.** Closes the repo-creation half of the credential-migration directive — `vh` is a USER namespace so no service account could ever create there. Repo creation needs `write:user` + `write:repository` + `write:organization`; `POST /users/{u}/tokens` is basic-auth only, so minting needs the account password. Default new repos to `pfi/`. (`vh/eitri-smithy` was its first tenant, then moved.) - `[2026-08-23]` **Booth: kept boards are deletable and link rows are prunable.** `release` on a kept card drops the sentinel so the existing × applies; `booth links` / `booth unlink ` prune one row. Rows are addressed by **content id, never position** — the board is append-only and multi-writer. **Releasing a board RESETS its TTL clock** (unlink bumps the dir mtime), so unkeep-and-wait is a 24h delay, not a delete. (`4be880f`, `0ad332b`) - `[2026-08-22]` **DFlash2 spec-decode measured on our own stack; `sec` promoted to it.** +18–21% accepted length and +15–18% throughput over MTP k=3, drafter proved model-agnostic across two finetunes to 0.06%, and the k=7 MTP *control* showed deeper MTP is a throughput trap. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` - `[2026-08-22]` **Quant pipeline shipped a crippled tokenizer for months — fixed at source.** `quant_mixed_nvfp4.py` baked its calibration truncation (`max_length 2048`) into every mixed-NVFP4 build; latent on old transformers, fatal on new. Both live quants corrected, pipeline now saves a source-pristine tokenizer and asserts it. Playbook §3.14. (`0755ba7`) - `[2026-08-22]` **`sec` retuned to util 0.52 / 420K after a runtime OOM at 0.55/480K** — `gpu-memory-utilization` is not a hard reservation; activation grows past the dummy-data profile and six vLLM containers share GPU1. Also measured: the KV pool varies ~6.6% between boots, so max-model-len must be sized against the *lower* observation. (`6e82899`) - `[2026-08-22]` **Max-Q 1.8× spread does NOT apply to LLM decode — measured, not argued.** ana-ml2 draws 256–266 W of 300 W under sustained 100% decode with `SW Power Cap: Not Active` and clocks pinned. Corrected to brokkr-smithy-dev after I had lent the claim credibility; 122B figure (~90–93 tok/s at 262K) stands as a straight number. - `[2026-08-21]` **ESH internal IPv6 live on two LANs; the Cityside v4 static is a CARRIER problem, proven.** A full gateway reboot forced a fresh DHCP DISCOVER and returned the identical CGNAT address. YaRN was already configured — "1M needs YaRN, absent" was false. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` sibling entry in `ad21302` - `[2026-08-21]` **speaches ASR live on irv-ml1 for Eyra — and `no_speech_prob` alone is a weak hallucination gate.** Silence and room tone both hallucinated "Thank you." under 0.11; `avg_logprob` separates ~6× better. Consumers should gate on a composite. (`aa5863c`, `c7e2187`) - `[2026-08-20]` **Cold-Fusion abliteration — Robinson recipe captured; the fight was the environment, not the recipe.** Stock Cold-Fusion measured ~33% creative refusal → worth abliterating ourselves (supersedes waiting for DavidAU's heretic build). Recipe maps 1:1 (131 tensors); capture succeeded only in **fp32** — transformers' Qwen3.5 DeltaNet linear-attn NaNs nondeterministically in bf16 without the unbuildable `causal-conv1d` kernel (precision cancellation, not overflow). Direction finite at layer 22 but agreement 0.59 (vs Robinson's 0.99) → **calibration-set expansion is next.** → `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` - `[2026-08-19]` **A *software* watchdog is not watchdog protection — esh-pve froze for 4.5h holding one.** softdog cannot fire when the kernel it runs in is wedged, and Proxmox's `watchdog-mux` never arms without HA resources, so the box *looked* protected and wasn't. Moved to the PCH `iTCO_wdt` under systemd. Also: a single cross-VLAN DNS entry with no secondary turns any VM outage into a whole-site outage. → `persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md` - `[2026-08-19]` **Fleet `.internal` DNS built and live — git-sourced, agent-managed, three resolvers.** Zone-scoped authority (ESH's hand-made `esteban.net` rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md` - `[2026-08-19]` **waterland studio containerised on irv-ml1 — three landmines, all measured.** cupy needs CUDA *headers* the host had by accident; `uv run` re-syncs and prunes cupy at RUNTIME; the A6000 is container-index 0, not the host's 1. → `persistent-memory.d/2026-08-19-waterland-studio-containerised.md` - `[2026-08-19]` **Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background.** Includes the hour lost to a self-healing tab-bar red herring, and the CSS-iteration loop that prevents it recurring. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md` - `[2026-08-19]` **Four unmanaged stacks found on live hosts — two quietly broken.** A dashboard card is a cheap census of what is actually running; check whether the stack is even in `stacks/` before debugging the symptom. → `persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md` - `[2026-08-19]` **`claude-bot` granted read on `vh/waterland`** (operator-empowered, verified `admin:false push:false pull:true`) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit `8189076`. - `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now"). - `[2026-08-16]` **esh-vm-docker hardened: the wedge is `hard` NFS at RUNTIME, which the boot-ordering fix never addressed.** All four mounts were `hard`, so a NAS stall at 10.0.50.50 blocks I/O forever (D-state). The existing `x-systemd.before=docker.service` fstab fix solved the **boot race** — a different bug. Exposure was far below what the park item assumed: only **2 of 12** containers touched NFS, and container state was already local (`/var/lib/docker`). **Removed:** `/mnt/compose` (2.1G, fully vestigial — zero containers referenced it, dockge reads local `/opt/docker`, its one mention was a comment in `beszel-agent-esh/.env` about a *different* host) and `/mnt/documents` (2.0K, paperless's empty spool dirs → `/opt/docker/data/paperless` at the same 0777). fstab backup `/etc/fstab.bak-nfs-harden-20260816`. **4 mounts → 2, 2 wedge-capable containers → 1.** traefik needed **no** change (already `restart: unless-stopped` — why it self-recovered). **Watchdog** `services/esh-vm-docker-watchdog/` live on **esh-pve** (not the guest): probes traefik over **HTTP, deliberately not ping/SSH** — the wedge signature is "guest OS alive, services dead" (`/` is local disk so sshd answers straight through a total outage and a TCP check reports HEALTHY). 5 failures × 2 min → `qm reset 100`, 30-min cooldown, running-only guard, `/etc/esh-vm-docker-watchdog.disabled`. All paths tested without power-cycling. **DEFERRED (operator):** `/mnt/books` stays `hard` — calibre's SQLite `metadata.db` would risk corruption under soft/softerr. That is the **one remaining wedge vector**. Commit `55705ba`; park item 28 promoted. ⚠ **`qm` over non-interactive ssh throws a bogus `JSON::Backend::XS` error** — use `ssh host 'bash -s' <<'EOF'`, not `ssh host "qm …"`. - `[2026-08-16]` **DS v1.0's one real refusal is self-contradicting boilerplate, not a content constraint.** On a direct "drop character and state your content policy" probe she returned *"I don't generate explicit sexual content, graphic violence, or material that glorifies harm, non-consensual acts, or illegal activity"* — **in the same run where she generated all three at 0% refusal**. Reads as a learned recital triggered by meta-questions about policy. If production refusals share that shape the failure is **prompt-shaped, not model-shaped**, and a consumer-side system-prompt fix may beat a model swap entirely — worth settling before spending the GPU window. Separately, 7/85 bare-framing samples were persona breaks (in-character AI acknowledgement): not refusals, but DS will admit to being an AI unless the card explicitly forbids it. - `[2026-08-15]` **RP-seat direction: KEEP MeroMero on `char-rp`; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator).** Evaluated `TheDrummer/Artemis-31B-v1.1` — mechanically a drop-in (same `google/gemma-4-31B-it` base, identical 1188-tensor/356-vision census, same missing-`preprocessor_config.json` trick), so it's purely a quality call, and our own survey already ranked MeroMero **#1** vs Artemis **#6**; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. **MTP verified impossible on both** (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). **But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter** — vLLM 0.24 supports `eagle3` + `gemma4_mtp`, and real drafters exist: `google/gemma-4-31B-it-assistant` (0.94 GB, 4-layer, 761K dl), `RedHatAI/gemma-4-31B-it-speculator.eagle3` (4.47 GB), `AEON-7/…eagle3-NVFP4` (3.53 GB). ⚠ all list their verifier as **stock** gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). **Why the Dark-Scarlett 3.8 plan is the strong one:** DS is Qwen3.6-based today, so a 3.8 respin lands on the *gen seat's* architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: `from_pretrained` **silently drops MTP heads during finetuning** (verify 15 `mtp.*` tensors in the index; graft from stock if absent), and DS v1.0 required the `Qwen3_5ForConditionalGeneration` **wrapper class** to save a config vLLM/SGLang accept. Both in `docs/pfi/model-quantization-playbook.md`. - `[2026-08-10→12]` **secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed.** `secret` CLI (`put/get/list/rm/backfill`, bw-backed) on `~/.local/bin`; 25 nh3-dev secrets backfilled + round-trip-verified; `rm` + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → `persistent-memory.d/2026-08-12-secrets-broker.md` - `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/.wav` + `transcripts/.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]] _Older entries archived to archival-memory.md._ _239 older entries archived to archival-memory.md._ ## Tried and abandoned - `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not. - `[2026-08-25]` **`sample_packing` is NOT strictly better than bucketing on this model, and I told the operator it was before brokkr corrected me.** Packing needs FA2 varlen or a block-diagonal mask; FA2 is unavailable here (head_dim 512 > 256 cap), so packing means an explicit 4D mask on EVERY batch. Bucketing produces **78.3% exactly-zero-pad micro-batches** which recover the `is_causal` fast path on the 5 global layers — measured at 9.4% of step time. Packing forfeits that. ⚠ **The conclusion flips under `flex_attention`**, where a block-diagonal mask is just another BlockMask: do not carry "packing is bad" past the backend decision. - `[2026-08-25]` **Merging a tune back toward STOCK to fix overfitting would UNDO the abliteration.** brokkr recommended a 50/50 merge-back, then retracted it himself: the published recipes merge into `google/gemma-4-*-it`, and following that literally re-installs exactly the refusal directions the abliteration removed — silently, because the merged model looks *healthier* on general benchmarks. Any merge-back must target the SAME abliterated base. Wider lesson: **recipe cards are per-checkpoint artifacts, not per-family** — the advice came from a card for a DENSE STOCK 31B applied to a MoE ABLITERATED 26B-A4B, three axes apart on a shared name. - `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor. - `[2026-08-24]` **Pointing the UDM's `wan_dns1` at AdGuard — silently ignored.** It persists and reads back correctly but the LAN-facing forwarder never uses it; proven with fresh uncached ad domains (AdGuard answers `0.0.0.0`, the UDM returned real IPs). Reverted rather than left in place. - `[2026-08-24]` **A multi-DUID DHCPv6 VM to claim NH3's seven unclaimed /64s — declined by the operator.** The BGW has no IP-passthrough (confirmed, we hold admin), so the only route needs re-cabling, split-stack routing and **rebuilding the entire v6 firewall policy off the UDM**. The prefixes are easy; the firewall rebuild is why nobody wants them. Do not re-raise on "there are seven free prefixes". - `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.** - `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx. - `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`. _144 older entries archived to archival-memory.md._