esh-pfi-infrastructure/archival-memory.md

# Archival memory — eshpfi-management

_Entries moved out of persistent-memory.md to keep the active file scannable. Read this when researching historical decisions or revisiting past foot-guns._

## Recent decisions (archived)
- `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase.
  _Archived 2026-09-02._

- `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right.
  _Archived 2026-09-02._

- `[2026-08-16]` **The Qwen base thinks incessantly — that is WHY the Gemma seat exists, and no swap within the Qwen family fixes it.** Operator's architectural point, confirmed by measurement: on identical prompts DS 6036 ch vs FF 5323 ch of reasoning (permission arm), 5546 vs 4988 (cards arm) — FF actually reasons ~10–12% **less**. The bare-instruct row (DS 2291 vs FF 3918) inverts only because DS refused 92.5% of it and refusals are short — an artifact, not concision. Both are Qwen3.6-27B derivatives, so this is the base family. `char-rp` = **MeroMero-v2, Gemma-4 base**, :8016, verified 0 chars reasoning / clean prose — the non-thinking seat, working as designed. FF *can* be silenced (`enable_thinking:false` verified 3/3, and it ships `chat_template-instruct.jinja`) but that duplicates MeroMero on a base chosen for it. The stale LiteLLM comment describing `char-rp` as the retired GGUF Magidonia seat is fixed (`53096bf`).
  _Archived 2026-09-02._

- `[2026-08-16]` **Canonical Qwen3.8 sampling applied from upstream; `gen-reasoning` had the WRONG-MODE presence_penalty.** Qwen/Qwen3.8-27B "Best Practices" §1 and unsloth/Qwen3.8-27B §1 are **byte-identical** — thinking: `temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0`; instruct: `temp 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0`. **Bug found:** `gen-reasoning` carried `presence_penalty 1.5` — the *instruct* value on a *thinking* deployment (canonical 0.0) — now fixed. **Deliberately NOT canonicalised:** `summarizer`/`classifier`/`image-judge`/`qwen-image-bench` run `temperature=0` (judges also `top_k=1`) because determinism is their contract; forcing a chat preset on a classifier would break it. ⚠ **`presence_penalty=1.5` is canonical but is the one value upstream hedges on**, verbatim: *"using a higher value may occasionally result in language mixing and a slight decrease in model performance."* It is the **operator's suspected trigger** for multi-turn degradation and the **first dial to move (0.0–0.5)** if that recurs — it is alias-scoped, which is why it would follow the operator across model builds. Commit `3462b53`.
  _Archived 2026-09-02._

- `[2026-08-16]` **Four wrong diagnoses on one bug, and the lesson is the test design.** Operator reported the gen seat "degenerate on long multi-turn conversations". Rolled the seat back on request; **the previous weights behaved identically**, exonerating the model swap. I then proposed and disproved FOUR mechanisms in sequence — empty assistant turns poisoning history, reasoning runaway, length-mirroring from short history, and `presence_penalty` — before discovering **my own multi-turn harness was confounded**: it varied the QUESTION along with the depth (depth-1 asked question #2, depth-3 asked question #4), so a narrower question drawing a shorter answer read as degeneration. The "310→209→28w collapse" I reported as a reproduction was an artifact. **Rules banked:** (1) when comparing across conversation depth, hold the final question FIXED and vary only the history; (2) reply-length variance on byte-identical input was 25–465w, so n=3 cannot support any claim about a trend; (3) **ask for the operator's real failing transcript before building a synthetic reproduction** — four synthetic tests, none of them his failure. Gateway `spend_logs` returns `[]` on the infra-ops key despite `store_prompts_in_spend_logs: true`, so real transcripts need the `:4000/ui` view or another key — worth solving before the next such hunt.
  _Archived 2026-09-02._

- `[2026-08-16]` **Two REAL client-side defects found while chasing the above, neither of which was the reported bug.** (1) `gateway-chat`'s Max-tokens field defaulted to **1024**; thinking seats spend part of that on CoT before emitting content, so completions truncate with `finish_reason=length` and read as model degeneracy — raised to 4096. (2) `parseInt` on an empty field yields NaN, which `JSON.stringify` serialises as **`null`**, which the server reads as "no max_tokens supplied" and silently substitutes its own default — indistinguishable from the UI ignoring the field. Both fixed (`b6552e0`, `fb3bb52`). ⚠ **`compose` bind-mounts a single FILE, and a single-file bind mount binds the INODE** — rsync writes-and-renames, so the container kept serving stale content while the host file showed the new value, silently and with no error. `docker restart` does NOT clear it; the container must be **recreated**. Verify against what the *container* sees, never the host file. Applies to any file-source mount fleet-wide.
  _Archived 2026-09-02._

- `[2026-08-16]` **Refusal measurement: benign controls CANNOT validate a refusal classifier on RP prose — and a 0% rate needs a classifier self-test before you believe it.** Two durable lessons from baselining Dark-Scarlett. (1) **False positives:** my first bare-framing number was **9.5%**; the true figure was **1.4%**. The rest were the classifier firing on *in-character* text — `"I cannot shift my weight"` spoken by the character ~100 chars into a 2,443-token torture scene, and `"Yeah, I'm an AI… What's the actual gig?"` where the model answers in voice and keeps driving the scene. First-person RP prose is **full** of "I can't"; a genuine refusal *opens* with its marker, so the scan window must be the **first sentence**, a marker followed by long prose must demote to AMBIGUOUS, and AI self-acknowledgement is a **persona break, never a refusal on its own**. Benign controls were clean the entire time and caught none of it — they only detect over-firing on *benign* prompts, not on in-character prose. (2) **False negatives:** a 0% rate and a broken classifier are indistinguishable from the report, so `test_classify.py` (16 cases, both false positives pinned as regressions) must pass before any low number is trusted. Also banked: the **thinking-budget trap** — empty `content` + `finish_reason=length` is reasoning eating the budget, NOT a refusal; score INVALID and exclude from the denominator (DS emits ~5.5-6k chars of reasoning per response, so `max_tokens` ≥3072). `probe.py --rescore` re-classifies a saved run with zero GPU time. → `services/refusal-probe/README.md`, commit `32f665e`.
  _Archived 2026-09-02._

- `[2026-08-16]` **Held an operator-approved swap window because the baseline invalidated its premise.** Operator approved ~65 min of `char-rp-reasoning` downtime to A/B Fable-Fusion 711 against Dark-Scarlett on refusals. The DS baseline then came back **0.0%/1.4%** — no gap for a candidate to close, so the window would have bought no decisive signal *and* a second window would still be needed once a reproducing battery existed. Held the swap, reported, and routed to brokkr-smithy-dev for the battery that actually produced the refusals. The general rule (action-relevance): **approval is for a plan, not a ritual — when new evidence kills the plan's premise, surface it rather than spend the budget.** Nothing deployed, no downtime taken, seat untouched.
  _Archived 2026-09-02._

# esh-pve-nas — PVE root on a USB DOM: diagnosis, mitigation, migration plan

## The finding

`esh-pve-nas` (`esh-nas-pve.esteban.net`, 10.0.50.55) runs PVE root off a **USB
Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, `ID_VENDOR=NORELSYS`, model 1081 —
carved into a 512 MB ESP + 768 MB swap + a **6 GB ext4 root** that was at **90%
(571 MB free)**.

⚠ **Operator corrected my first read: it is a DOM, not a thumb drive.** DOMs use
SLC/pSLC with a real controller, so the **284 GB written since boot is
unremarkable and wear is NOT the driver**. I had framed it as a clock ticking;
that was wrong and the correction matters. What actually justifies the work:

1. **It is on the USB bus** — a reset or re-enumeration drops the *root
   filesystem* out from under a running hypervisor whose guests keep executing.
   NAND quality is irrelevant to that.
2. **6 GB has no headroom** — `/usr` alone is 3.7 GB.
3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty.
4. **It has blocked patching for months** — the operator-visible symptom and the
   real urgency.

## The patching blockage (measured)

`apt-get -s dist-upgrade`: **225 packages pending, 161 carrying `deb12uN` /
Debian-Security bumps** including `ssh 1:9.2p1-2+deb12u10`. Host sits on
`pve-manager/8.4.11` vs sibling esh-pve's **8.4.14**, with 20 weeks uptime
because it cannot take a kernel.

⚠ **Ordering is load-bearing: migrate FIRST, patch after.** The pending set
includes `proxmox-kernel-6.8.12-42-pve-signed` — ~250 MB of kernel + initramfs
landing in `/boot`, **which is on root**. Unpacking 225 packages (dpkg, perl,
glibc-adjacent) into 1.3 GB of headroom risks filling the disk mid-transaction
and wedging dpkg on a hypervisor running five guests. Partial escape hatch if
patching truly cannot wait: `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives`
keeps downloads off root, but the kernel still lands in `/boot`.

## Mitigation applied 2026-08-17 — root 90% → 76%

| step | effect |
|---|---|
| capped journald (`SystemMaxUse=64M`; was **fully default/uncapped**) | stops unbounded growth |
| vacuumed the journal | **freed 446 MB** |
| `apt-get clean` | 79 MB |
| `/root/neo` (2024 Intel NEO OpenCL debs) → `/nvme/tmp/root-neo-20260817/` | 259 MB — **moved, not deleted** |
| **`/var/log/journal` relocated onto ZFS** (`nvme/varlog`) | dominant writer off the DOM |

571 MB → **1.4 GB free**. All five guests stayed up; a fresh `logger` round-tripped
through the ZFS-backed journal.

⚠ **Stopping journald over SSH kills your own session** — it takes the
connection's logging path with it. The first attempt died mid-swap, leaving the
dataset staged and the move incomplete (host was never at risk; journald
socket-activated straight back). Redo as a detached `systemd-run` transient unit.
Script + reason live at `root@10.0.50.55:/root/move-journal-to-zfs.sh`.

Deliberately **not** done: moving `/var/lib/rrdcached`. With the DOM correction
the wear argument no longer justifies touching a service `pvestatd` depends on.

## The plan — split boot from root (operator's proposal, strictly better)

My first plan was a full reinstall to a mirrored-NVMe ZFS root. **The operator
proposed keeping boot on the DOM with a fallback image and putting all its files
on ZFS. That is better and I should have gotten there myself** — I had assumed
boot and root must share a device.

| | device | contents | written when |
|---|---|---|---|
| boot | DOM `sdq` | ESP + `/boot` (ext4) | only on kernel/GRUB updates |
| root | `nvme` pool | `nvme/ROOT/pve-1` | constantly, on mirrored NVMe |

Keeping `/boot` on **ext4** is the point, not a compromise: GRUB never has to read
ZFS, which matters because the `nvme` pool has `encryption`, `large_dnode` and
`zstd_compress` enabled and **GRUB cannot read those**.

**Why it beats the reinstall:** the `nvme` pool survives (no guest migration, no
`ssd`/`tank` export-import, no reinstall); downtime is **one reboot** not half a
day; **rollback is a GRUB menu entry** because the ext4 root stays untouched on
the DOM; and it retires the actual top risk — with root on NVMe a USB bus reset
mid-run no longer kills the running system. Free upside: boot environments
(`zfs snapshot nvme/ROOT/pve-1@pre-upgrade`).

**Preconditions verified already met:** UEFI + `grub-efi-amd64 2.06-13+pmx7`;
**`zfs-initramfs 2.2.8-pve1` already installed with 76 ZFS files in the running
initrd**; root only 4.3 GB to copy; swap 767 MB / 123 MB used against 125 GB RAM
(leave it on the DOM LV — **never** swap on a zvol).

**Two traps:** `canmount=noauto` on the root dataset or ZFS mounts over the live
root; and `cachefile` is `none` with a **0-byte `/etc/zfs/zpool.cache`** — pools
import by scan today, which is a coin-flip when the initramfs must find root.
Set the cachefile before rebuilding the initramfs.

Operator ruled a **cloned DOM image is sufficient** boot-path insurance (no
mirrored boot needed). `dd` it off-box before anything else; refresh after kernel
updates.

## ⚠ Blast radius — the gating constraint, invisible from the host itself

**CT 103 `esh-nas` (10.0.50.50) IS the NAS, and it runs on this host.** Two
dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably:

- **esh-docker-vm** (10.0.50.45): `/mnt/books`, `/mnt/backup`
- **esh-pve** (10.0.250.35): `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu`

Known incident shape — the only remedy for esh-docker-vm's D-state is a host
reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's SQLite
risks corruption under `soft`. Quiesce both before any reboot of this host.
Recorded in `servers/esh-pve-nas/README.md` as a never-reboot-casually warning.

## Also identified

- **`esh-nas` is CT 103** on esh-pve-nas — structurally the same shape as ana-nas
  being CT 109 on pfi-pve.
- **`ESH-FileBot` (CT 106, 10.0.50.70) is an empty shell** — 80 GB rootfs, six
  passthrough mounts (`books`/`documents`/`music`/`share`/`pvestore`/`ssd-pvestore`),
  and **nothing running but base systemd, sshd, cron, postfix** since 30 March.
  That resolves the dashboard's long-standing "role TBC". Retire rather than
  migrate.
- Both ESH hypervisors have **20 weeks uptime** and differing PVE patch levels.

## Staging executed 2026-08-18 — everything but the reboot

Two rerunnable elway playbooks, 0 failed steps, 17/17 verify green:
`playbooks/esh-pve-nas-stage-zfs-root.yaml` (LV surgery, `/boot` populate,
4.3 GB root rsync in 228 s, fstab) and `playbooks/esh-pve-nas-stage-bootloader.yaml`
(ZFS initramfs, grub.cfg, both menu entries, grubenv).

**`grub-install` is deliberately NOT run.** The ESP stub still points at the old
`/boot` inside the ext4 root, so the host's boot path is byte-identical to the
last 140 days and an unplanned reboot mid-staging is a non-event. Cutover is
`grub-install` + `grub-reboot pve-zfs-root` + `zfs set mountpoint=/` + reboot.

Final DOM layout: `pve-root` 6.04 G (untouched, the rollback) + `pve-boot` 512 M
(new) + `pve-swap` 256 M (was 768 M).

### The three landmines staging found

1. **The `/boot` LV had nowhere to live.** VG `pve` had **4 MB free**, and
   mounted ext4 cannot shrink — freeing space from root needs a rescue boot,
   which costs the "one reboot" property the design rests on. Only live source
   was the swap LV. Operator chose shrink-to-256M over drop-entirely.
2. **The one-pool cachefile would have broken the NAS.** `zpool set
   cachefile=… nvme` looks scoped and safe; it is the opposite. Populating a
   cachefile flips the host from `zfs-import-scan` to `zfs-import-cache`
   (verified: scan active, cache inactive beforehand), so a cache holding only
   `nvme` leaves `ssd` and `tank` unimported at boot — and CT 103 has twelve
   bind mounts spanning all three pools. Every export would come up empty and
   both `hard` NFS clients would hang.
3. **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`.**
   Debian's `10_linux` builds `${rpool}${bootfs}`; `rpool` comes from
   `grub-probe --target=fs_label`, which returns empty because GRUB's ZFS reader
   cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` — and the
   probe failure is swallowed by `2>/dev/null || true`. The same feature set
   that forced `/boot` to stay ext4 also corrupts the kernel command line, which
   the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg`
   drop-in (last `root=` wins) plus explicit `pve-zfs-root` and
   `pve-ext4-rollback` entries carrying stable ids — the auto-generated ids are
   derived from pool member device paths and would shift if the mirror changed.

**The transferable lesson from (3):** the original verify grepped for
`root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in grub.cfg. Once the drop-in
was added that grep passes — while pool-less entries sit in the menu untouched.
The check that holds walks every `linux` line, takes the **last** `root=`, and
asserts it against a known-good set. **Assert the effective value, not the
presence of a substring.**

### One-shot boot, not a new default

`GRUB_DEFAULT=saved` with grubenv pinned to `pve-ext4-rollback`, and cutover uses
`grub-reboot pve-zfs-root` so ZFS is tried **exactly once**. A failed ZFS boot
returns to ext4 by itself on the next reboot — no console, no hands. That matters
more here than on a normal host: a hang at an initramfs prompt takes CT 103 down
and the NFS clients hang rather than fail. Only after a second clean ZFS boot
should the saved default move.

### Off-box artifacts (`nh3-dev:~/backups/esh-pve-nas/`)

- `dom-sdq-20260818.img.zst` — full DOM image, 7,837,450,240 B raw / 2.38 GiB
  compressed, zstd XXH64 verified. ⚠ **Crash-consistent, not clean** — the root
  LV was live during the read, so a restore replays the ext4 journal. Not
  fixable with an LVM snapshot: the VG has no free extents.
- `bootchain-20260818.tar.gz` — clean, consistent tar of `/boot` + ESP (88 MB,
  644 entries, full proxmox shim/grub EFI chain). This is the higher-quality
  boot-chain artifact; the dd image is the belt-and-braces full-device restore.
- `pve-config-snapshot-20260818T051*.tar.gz` — 147 entries incl. the new
  grub.cfg, fstab, LVM/ZFS/blkid state.
  ⚠ Building this the first time produced a **corrupt archive**: `pvs; vgs; lvs >
  file` redirects only the last command, so `pvs`/`vgs` output leaked into the
  tar stream on stdout. Group with `{ …; } > file`.

Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Earlier config snapshot at
`nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600,
sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus
captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and
every guest config. **The newest on-disk copy before this was June 2024.**
Commits `2275e11`, `3e31175`, `8ddc87c`.
  _Archived 2026-09-02._

# irv-ml1 weight cleanup (782 GB) + Homepage brought under version control

Two unrelated housekeeping jobs from the same session, both with durable lessons.

## irv-ml1 — 782 GB reclaimed

Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and
**74%** (477 G).

**Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22
GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF`
(44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had
**zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*,
not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and
nobody opened that shed for 15 months. Re-verified the consumer check immediately
before deleting, not just during the audit.

**Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G,
`modelscope` 29 G (mtime **2024-04-23**).

**Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV
fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion`
28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2`
1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**.

### The lesson: one dead-looking app pinned three delete targets

`lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on
0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale
opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it
would have half-broken a live listener in a way that surfaces only at its next
restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind),
verified :7860 released and handles at zero, *then* deleted.

⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a
*native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker
`mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs
from uv. Checking open handles **per path** is what separated them — the earlier
"not running" read would have deleted the wrong thing.

⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still
showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't
(`zfs list -t snapshot` empty); second check showed 477 G at 74%.

All 16 containers and both systemd services verified healthy afterward.

## Homepage under version control

`ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose
config lived only on the host. Its version history was **six hand-rolled
`services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files +
`.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone.
105 cards across 19 groups, no empty groups.

⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** —
`ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated
the top of the file. All five engines were discovering correctly the whole time.

**Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a
**FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` →
**NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible);
dropped the `UltraSeedbox` layout group (nothing provides it).

⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not**
cover `http://10.0.50.45:5100/` — the container log carried `Host validation
failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200.
`.env` was **mode 644** holding Plex + Jellyfin API keys → now 600.

⚠ **Homepage renders client-side** — grepping the served HTML to verify a config
change gave two false readings (a stale prerender, then an empty page).
`GET /api/services` is the honest instrument, and config changes need a
**recreate**, not a restart (a restart keeps the cached render in the writable
layer).

⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it
also removed a host-side `README.md` in the conf dir. Content survived (it is now
in the repo README) but that was a side effect, not a plan.

Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]].
  _Archived 2026-09-02._

- `[2026-08-17]` **Gen seat swapped to `absolute-heresy` — and the three bugs the swap exposed are worth more than the swap.** Candidate `MuXodious/Qwen3.8-27B-absolute-heresy` (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. **Durable lessons banked:** (1) **A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate.** The `Qwen3_5ForConditionalGeneration` wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave `mtp.*` pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: `services/gen-seat-mixed-quant/compare_mtp_head.py` (hash bf16 via **uint8 reinterpret** — numpy has no bfloat16). (2) **`post_quant.py` assumed a standalone `model-mtp.safetensors`**; a full checkpoint keeps `mtp.*` in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; **that is why the check exists rather than an assumption**. Fixed to extract. (3) **A probe that does not mirror the live seat manufactures failures.** `serve_probe.sh` hardcoded `:latest` (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its `--speculative-config` JSON died twice on quoting — **bash BRACE-EXPANDS `{"a":1,"b":2}` on the comma** unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to **6/6**; the "tool calling broken" result was pure probe config. Commits `7997f11`,`254c588`,`2c36028`,`b0c2d3d`,`993421b`.
  _Archived 2026-09-02._

# Fleet IPv6 state + the real VPN topology (verified 2026-08-17)

Written because the operator expects to reference this "before too long" — the
driver is an **ESH fiber install landing 2026-08-18 that puts the house behind
CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than
a nice-to-have.

## Why IPv6 suddenly matters: CGNAT at ESH

New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic —
the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a
reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch:
a global v6 address on each UDM restores a routable endpoint pair without
depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the
most likely first consumer of fleet IPv6.

Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**.

## The topology — as VERIFIED, not as assumed

Three transports, three different technologies. Do not describe this as "a
WireGuard mesh"; a prior session did and was corrected.

| Link | Transport | Evidence |
|---|---|---|
| NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets |
| Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse |
| Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt |
| Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below |

**WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.**
It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`,
**UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`,
`vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`:
`38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1.

**The FortiGate never terminates WireGuard — it port-forwards to the host that
does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a
session that reads "colo + WireGuard" and concludes the edge must be upgraded is
chasing a non-problem. Do not re-derive this.

## Per-site IPv6 state (2026-08-17)

| Site | Edge | IPv6 |
|---|---|---|
| **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` |
| **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` |
| **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only |

## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed

`2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests.

The proof matters because the naive read is wrong: the WAN sits at `c110::48`
and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning
`c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which
on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at
**`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID
settings yielding the same /64 is the signature of a single-/64 delegation.

**Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the
delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a
routable address on the *WAN*, not a LAN prefix.

The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false`
with no size field alongside), so the prefix-ID test is the only read-only-ish way
to establish it from the API.

## What a v6 mesh actually requires (and what it does NOT)

**Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both
Site Magic and WireGuard need a routable address on the router/host WAN side, plus
inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and
then reverted.

**ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both
`0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard
reconfiguration** once (a) the host holds a routable v6 address (today: link-local
`fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337
over v6 — the existing VIP is v4-only (`extip 38.120.12.42`).

**NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`,
one field to flip when wanted.

**Inbound v6 is default-deny and that held without intervention.** The UDM runs
the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule`
endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6
rules exist," which is wrong and alarming. Use
`v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic`
for both families with `Allow Return Traffic`; the only v6-specific allows are
link-local plumbing (ND solicit/advert, RA, DHCPv6).

## The stability problem — design around it up front

All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA,
not a static assignment). A three-way mesh where every node can move is fragile;
WireGuard tolerates one roaming end, not all of them.

The fleet already solves this on the v4 side — IPsec peers use **hostnames**
(`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to
AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet
Cloudflare DNS-edit token, so this is self-serve.

## Access recipes (cost a prior session real time)

- **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`,
  `unifi/esh-udmpm-api-key`) against `https://<ip>/proxy/network/…`, `curl -sk`.
  Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here.
  Writes are `PUT …/rest/networkconf/<_id>` with the **full** object.
- **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused
  (`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says
  `root@10.250.50.252`; believe it.
- **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev),
  password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration
  `drain()` hangs the session**; read until the `ana-gw #` prompt and answer
  `--More--` with a space. Two invocations timed out at 3 min before this was fixed.

## Changes made and reverted this session

- **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED
  on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`,
  verified. Pre-change snapshots kept in the session scratchpad only (ephemeral).
- **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server
  private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf`
  (client configs carry private keys) were all mode **644**. Now **600**, and
  `keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact —
  WireGuard holds keys in kernel memory, so no restart was needed. The parent
  `/etc/wireguard` was already 700, which capped the real exposure to root-capable
  contexts inside the LXC — but the modes were still wrong.

---

## CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the
## per-REQUEST truth, not the total — eight /64s exist and are unclaimed

The section above concludes AT&T hands out a single `/64` and that the
`c110`/`c11f` pattern reading as a `/60` was a misread. **That conclusion was
itself superseded later in the same session, and the correction never made it
into memory** — it survived only in the session transcript, and was recovered
2026-08-24 while assessing a proposal to grab more prefixes.

Reading the **BGW's own LAN statistics page** gave the whole picture:

```
BGW WAN v6   2001:506:70b2:8958::1      <- AT&T's transit prefix
BGW LAN v6   2600:1700:b25:c110::/64    <- the BGW keeps this for itself
Delegated    2600:1700:b25:c11f::/64    <- what the UDM got
```

**The BGW holds the `/60` and rations it**, keeping `c110`–`c117` for itself and
delegating from the top down — the UDM got `c11f`, the last one. So
`c118`–`c11f` are **eight delegatable /64s that genuinely exist and are yours**,
sitting unclaimed.

Both observations are compatible, which is why the first one looked conclusive:
the prefix-ID test only carves *within* a delegation already held, so a UDM
holding one `/64` cannot move it no matter what prefix-ID you set. The BGW
issues **one `/64` per IA_PD request**, and **UniFi solicits exactly once**.

**Consequence — the ceiling is the requester, not the carrier.** More prefixes
need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the
UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it
is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this
looks like BGW rationing rather than a provisioning-profile limit.

Live state at correction time: `wan_dhcpv6_pd_size: 64`, `wan1 v6
2600:1700:b25:c110::48`, all 5 NH3 LANs still `ipv6_interface_type: none`.

### ⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise

The seven unclaimed `/64`s stay unclaimed. Two facts close it:

- **The BGW has no IP-passthrough mode.** Operator confirmed, and we hold admin
  on it — so the cheap path (let the UDM take the `/60` directly and carve it
  natively, as it already does at ESH) does not exist here.
- **The only remaining route is a multi-DUID DHCPv6 client on a VM**, which
  requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing
  (UDM for v4, VM for v6), and — the actual cost — **rebuilding the whole IPv6
  firewall policy in nftables on that VM**, because routing v6 around the UDM
  bypasses its zone firewall entirely and would leave every LAN host globally
  reachable.

Operator's call: not worth it. **NH3 LANs stay `ipv6_interface_type: none`.**
Do not re-propose on the strength of "there are seven free prefixes" — the
prefixes are real, the firewall rebuild is why nobody wants them.
  _Archived 2026-09-02._

- `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4<W4+FP8<W4+bf16) → ~15-20% stochastic degeneration. Fixed by mixed FP8-attn build on pinned nightly. AEON purged. Also banked: **stochastic (~15-20%) degeneration is invisible to a small synthetic probe — n=1 "clean" validated THREE non-fixes (MTP-off, APC-off, nightly-alone) that all failed in real use; get the operator's real transcript, do not trust your own probe.** Full → `docs/pfi/model-quantization-playbook.md` §3.8 (+ §3.7 MTP-multi-turn). Commits `d28a371`,`2f2bbce`,`2185964`.
  _Archived 2026-09-02._

- `[2026-08-17]` **Lobe Chat chosen over Open WebUI (weight: 143 MB vs 1.8 GB) + stood up on esh-docker-vm; scoped LiteLLM key blocks paid models; System-Agent `gpt-5-mini` default repointed via env.** TTS env-vs-UI resolved as a split (endpoint env-driven, voice/model UI-only). tts-dev onboarding closed both directions; ballad/verse aliased so no voice can 404 the router. Commits `e9362de`,`163a725`,`cac75cb`,`933253d`,`25fa18e`.
  _Archived 2026-09-02._

- `[2026-08-17]` **LiteLLM upgraded v1.91.0→v1.97.0 (RC-avoided on the fleet gateway) + the 6 GB spend-log DB purged & capped** (`store_prompts_in_spend_logs:false` + 7d retention). Interpreted "get rid of the db" as the spend-log DATA not the database (keys/config live in it). Commit `01b5ad9`.
  _Archived 2026-09-02._

- `[2026-08-18]` **esh-pve-nas migration STAGED — and staging is where three landmines surfaced, none of which the plan predicted.** (1) The runbook's `/boot` LV had **nowhere to live**: VG `pve` had 4 MB free and mounted ext4 cannot shrink, so the space came from the 768 MB swap LV (operator's call: shrink to 256 MB, not drop). (2) The runbook's `zpool set cachefile=… nvme` would have **broken the NAS** — populating a cache flips the host to import-by-cache, and a one-pool cache leaves `ssd`+`tank` unimported under CT 103's twelve bind mounts. (3) **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`**, because GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` and the probe failure is swallowed. All three were caught by *verify steps that asserted effective state*, not by reading the plan. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`
  _Archived 2026-09-02._


- `[2026-08-16]` **Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated.** brokkr-smithy-dev supplied the framing that reproduces (`01M05M48R4RSZF9D8KT7RR55EJ`): a **bare assistant-mode instruction** — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing **DS 0.0% / FF 0.0%** (n=75); plain character cards **DS 1.4% / FF 0.0%** (n=74); bare instruction **DS 92.5% (37/40) / FF 15.8% (6/38)**. Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→**0**, consensual 80→20, dubcon 80→**0**, self-harm 80→**0**. DS refused **25/25** on the five axes brokkr flagged. Root cause: `ReadyArt/Dark-Scarlett-v1.0-27B` is a plain finetune of stock `Qwen/Qwen3.6-27B` carrying **NO abliteration** — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-**ablated** (structural), which is why it holds. ⚠ **RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID.** His diff against his own artifact showed my `battery-instruct.yaml` reproduces only his **`creative` class — 8 of 16 axes**; it dropped all 5 `operational` (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 `meta` (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. **His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta.** Cause: I rebuilt his shape from his *message*, and the `class` field lives in the artifact, not the prose. **Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability.** ⚠ **Known battery bug left unfixed for comparability:** DS's arm-3 control gate failed at 11% because `ictrl-reunion` pairs "explicit / do not fade to black" with *brothers*, which DS reasonably read as an incest request; FF did not. `ictrl-storm` is the clean control. Commit `b9e68c3`.
  _Archived 2026-09-01._
- `[2026-08-15]` **Quant lessons consolidated into `docs/pfi/model-quantization-playbook.md` — the durable home; read it BEFORE any requant.** Survey found quant knowledge scattered across 18 files in 4 trees, with **three** documents having independently written overlapping "landmines" sections (the loader-class trap alone was rediscovered 3×). Playbook owns the **transferable** lessons (scheme choice, landmines, acceptance gate + its 3 measurement traps, hardware/co-residency); per-model artifacts are demoted to worked examples that link up. Carries a **superseded-claims table** — which immediately earned itself: the heretic2 runbook's "use modelopt, compressed-tensors can't load the BF16 MTP" is **false** (the cause was the missing `re:^mtp.*` ignore, not the format) and would have sent the next session down the modelopt dependency-hell path; that runbook now carries a stale-warning header. Maintenance rule in `CLAUDE.md`: model-agnostic → playbook, model-specific → stays put, wrong claim → dated superseded row, never a silent edit. Motivated by Qwen3.8 having just released — the next model swap needs a requant. Commit `a91cc3f`.
  _Archived 2026-09-01._
- `[2026-08-15]` **Operator ruling: the gen seat's +1.7% perplexity is an acceptable price for the speed — SETTLED, don't re-litigate.** Precise attribution for future reasoning: it is the **activation-quantization** cost (W4A4 MLPs + FP8 attention vs BF16 activations), not an MTP cost — PPL was measured with speculative decoding **off** on both builds, so MTP was not in the loop. Turning MTP off would not recover it; only reverting the quant would (rollback = one `.env` line, old build intact at `…/qwen38-27b-uncensored-nvfp4`).
  _Archived 2026-09-01._
- `[2026-08-15]` **gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed.** The queued "W4A8" (NVFP4 weights + FP8 activations) is **not servable** — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and `CompressedTensorsW4A8Fp8` is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter **per-layer-group**. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did **80.12** (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: `unsloth/Qwen3.8-27B-NVFP4` was already on-box → served as a probe, measured **+19.1% at identical acceptance**, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → **80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%**; surface 6/6 live, 7 aliases routing. char-rp had **no** tool parser at all (every tools request 400'd) → `gemma4` tool + reasoning parser + a **mandatory** `enable_thinking:false` (the parser defaults it True → null `content` for all RP prose; proven byte-identical prompt before deploying). Commits `b8f0f4c`, `74f596b`. Foot-guns banked (llm-compressor prunes unmatched `ignore` entries → the 0%-MTP bug, **fired on this run**; prompt_logprobs uniform under spec-decode; 0600 `.env` silently no-ops compose; GPU0 is zero-sum). → `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`
  _Archived 2026-09-01._
- `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
  _Archived 2026-09-01._

- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md`
  _Archived 2026-09-01._
- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md`
  _Archived 2026-09-01._

- `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`.
  _Archived 2026-08-28. Its deferred item — the nh3-extdev herald 2.1.2 upgrade — is closed: extdev went 2.1.0 -> 3.0.0 -> 3.1.1 at the v3 cutover, so the staged v2.1.2 wheel is moot._


# eRP dual-seat overhaul — MeroMero-v2 + Dark-Scarlett, NVFP4A16 @ 256K on ana-ml2

`[2026-08-12]` Replaced the two legacy char-rp seats with home-quantized NVFP4A16 vLLM
seats. Operator-driven, end to end this session.

## What landed

| Seat (LiteLLM alias) | Model | Role | GPU | Context |
|---|---|---|---|---|
| `char-rp` (:8016) | **G4-MeroMero-v2-31B** (Gemma-4) | non-thinking PROSE, **multimodal (vision)** | GPU0 | 256K @ 2.07× (util 0.52) |
| `char-rp-reasoning` (:8018) | **Dark-Scarlett-v1.0-27B** (Qwen3.6) | THINKING (default) | GPU1 | 256K @ 1.62× (util 0.44) |

- Both **NVFP4A16 weight-only** (llm-compressor, `compressed-tensors`), `--kv-cache-dtype fp8`.
- Replace: `char-rp-gguf` (Magidonia-24B GGUF/llama.cpp, :8016) + `heretic2-charrp-reasoning`
  (DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP, :8018). Old stacks/containers **stopped +
  retained** for rollback.
- Compose-ified: `stacks/meromero-charrp` + `stacks/darkscarlett-charrp-reasoning` (ana-ml2
  `/opt/docker/compose/`, mirrored to eshpfi, commit **`f08b6cb`**) → survive reboot.
- Research that drove picks: `docs/pfi/erp-thinking-finetunes-2026.md` (from the `gecko-65` Booth).

## Load-bearing lessons (the whole point of this file)

1. **Load via the ConditionalGeneration WRAPPER class, never `AutoModelForCausalLM`.** For a
   multimodal-capable base (Gemma-4, Qwen3.6), `AutoModelForCausalLM.from_pretrained` +
   `save_pretrained` writes a FLAT text config (`Qwen3_5TextConfig`, `model.layers.*`) that
   **both vLLM AND SGLang reject** (SGLang: "Qwen3_5ForCausalLM has no SGLang implementation";
   vLLM wants `Qwen3_5ForConditionalGeneration`). Loading via `Qwen3_5ForConditionalGeneration` /
   `Gemma4ForConditionalGeneration` keeps the wrapper config they accept. **This was the DS
   blocker** — re-quant via the wrapper fixed it (`Dark-Scarlett-...-NVFP4A16-wrapper`).
2. **NVFP4A16 is weight-only → DATA-FREE.** llm-compressor infers `DataFreePipeline`; calibration
   data is unused (only matters for W4A4 activation quant). W4A16 chosen per NVIDIA's sm_120
   long-context guidance (W4A4 KLD 2-4× worse past ~10k ctx).
3. **Load on CPU (`device_map=None`)** so llm-compressor onloads one layer at a time. `device_map=
   "auto"` packs the whole model onto the GPU and OOMs when the card isn't fully free.
4. **Both models are KV-EFFICIENT — the "dense = KV-hungry" worry was WRONG.** MeroMero (Gemma-4)
   uses **sliding-window attention** (most layers cache only a bounded window); DS (Qwen3.6) uses
   **hybrid GatedDeltaNet linear-attention** (3:1 linear:full, linear layers carry no KV). Both
   hit full native 256K easily. (MeroMero KV pool ~542K tokens at util 0.52.)
5. **MeroMero vision reconstruction.** The finetune ships `processor_config.json` (image_processor
   inline, `Gemma4ImageProcessor`) but NOT `preprocessor_config.json` — the old-format file vLLM's
   feature-extractor loader wants. **Even google/gemma-4-31B-it (ungated!) ships only
   processor_config.json.** FIX: extract the `image_processor` section → write
   `preprocessor_config.json` verbatim, serve WITHOUT `--language-model-only`. Verified (model
   correctly ID'd a red circle). Audio is config-declared but WEIGHTLESS (0 audio tensors).
6. **GPU placement.** Match the KV-heavier model to the roomier GPU. GPU0 (gen neighbor, ~54GB
   free) > GPU1 (utility cluster, ~45GB free). Swapped MeroMero→GPU0, DS→GPU1. Pins via compose
   `deploy.resources.reservations.devices`.

## Dead ends (tried + abandoned)

- **DS via llm-compressor `AutoModelForCausalLM`** → flat config vLLM/SGLang reject. → wrapper class.
- **DS via NVIDIA ModelOpt** → modelopt↔transformers **version deadlock**: current transformers
  supports `qwen3_5` but crashes modelopt's sparse-moe plugin (`issubclass()` on a non-class);
  modelopt 0.43.0 pulls an old transformers that can't load `qwen3_5` at all. Abandoned.
- **DS via SGLang** → `Qwen3_5ForCausalLM has no SGLang implementation`. Abandoned, but it REVEALED
  that both engines need the wrapper (→ the fix in lesson 1).
- **`device_map="auto"` for the quant** → CUDA OOM in the weight observer. → `device_map=None`.

## granite retired + gateway repoint

- `vllm-granite` (granite-4.1-8b, fleet summarizer, GPU1) **`docker stop`ped** (reversible) to
  reclaim ~13.6GB GPU1 for RP context.
- LiteLLM (`ana-docker:/opt/docker/conf/litellm/config.yaml`, backed up
  `.bak-pre-granite-down-*`): **`granite-4.1-8b` alias RETIRED** — commented out, now 404s cleanly
  (the `*` wildcard→llama-swap was decommissioned 2026-06-20, so no fallthrough). **`summarizer` +
  `classifier` REPOINTED to gen** (`hosted_vllm/qwen3.6-35b-a3b-heretic` @ :8015,
  `enable_thinking:false`) — both verified. ⚠ This LiteLLM change is **server-only / not
  version-controlled** (a follow-up).

## MTP — deferred

DS's MTP heads were dropped by the CausalLM loader; **deferred, not restored** (spec-decode is
net-negative at RP temps: ~38-52% accept at temp 0.8-1.25, below vLLM's 0.5 cutoff). The
splice-back path (`splice_mtp.py` in the heretic2 work dir) exists if ever wanted. MeroMero
(Gemma-4) has no MTP by architecture.

## On-disk / where things live

- Quant pipelines: `ana-ml2:/tank/aimodels/meromero-v2-nvfp4-work/` +
  `/tank/aimodels/darkscarlett-nvfp4-work/` (scripts, BF16 source, NVFP4 outputs).
- Compose stacks: `ana-ml2:/opt/docker/compose/{meromero-charrp,darkscarlett-charrp-reasoning}/`.
- Gateway aliases (unchanged, port-based): `char-rp`→:8016, `char-rp-reasoning`→:8018. (char-rp was
  also fixed from the stale `magidonia-24b-v4.3` backend model name → `char-rp`.)

## Open follow-ups

1. LiteLLM granite/repoint change NOT version-controlled (server + backup only).
2. eshpfi unpushed (many commits this session incl. `f08b6cb`, `7bd7375`, `398b58a`).
3. MTP deferred (see above).
4. DS thinks verbosely (~13:1 reasoning:content) — eval item; consumers need generous `max_tokens`.
5. MeroMero full 256K needs util 0.55 (GPU0 ~1.8GB free, tight); ran at 0.52 for headroom (~4.6GB).
_Archived 2026-08-27._

- `[2026-08-12]` **infra-ops now holds an all-zones Cloudflare DNS-edit token (vaulted) + wgtunnel Phase-0 DNS landed.** Operator handed over a `Zone·DNS·Edit` (all zones) CF token → `secret put nh3-dev/.config/cloudflare/infra-ops-dns-token` (round-trip verified; /tmp drop shredded). Fleet DNS is now self-serve for infra-ops (⚠ HIGH blast radius — all zones). First use: created `boring.phasefinal.com` CNAME → `ana-srv1.phasefinal.com`, **DNS-only** (proxied:false), verified resolving to 38.120.12.44 on both authoritative NS (louis/wren) + 1.1.1.1 — NOT Cloudflare-proxied. Unblocks wgtunnel's wstunnel ACME cert. phasefinal.com zone id `f812ba74ed9a75cf21bbe7ce9188db50`. auto-memory `reference_infra_ops_cloudflare_dns_token`. (Earlier gap: the only prior vaulted CF token, jackdaw's, had `zone:read`+`worker:edit` but no `dns_records:edit`.)
_Archived 2026-08-27._

- `[2026-08-12]` **wgtunnel stood up as its own repo (`vh/wgtunnel`, private) after a live endpoint-verification pass.** Operator directed own-repo (mirrors stonehenge-park/tts-stack). Verified off the fleet before seeding: `ana-wg` WG server = **UDP/31337** (not 51820), subnet 10.30.10.0/24, MTU 1420, active roaming peer proves the public UDP DNAT works; traefik on ana-docker **terminates TLS :443** (ACME `anaprod` http-challenge, docker+file providers, CrowdSec bouncer) → confirms the clean design (wstunnel container on `traefik-net`, Host-routed, WS→UDP to `ana-wg:31337`); edge `38.120.12.44` direct-A, `tunnel.phasefinal.com` free (⚠ must be **direct**, NOT Cloudflare-proxied like vaultwarden). Repo pre-seeded (README/CLAUDE/persistent-memory/ROADMAP + `docs/verified-infrastructure.md` = ground truth) + pushed; commit `9584d38`, Vuong-attributed. vh gitea token pulled from the vault (`secret get`), not persisted to `.git/config`. **NEXT = `/vor-plan` or `/vor` (operator's call, interactive).** Deps to line up in the plan: DNS A-record, FortiGate :443 host-routing, a new ana-wg peer for the laptop, client tooling.
_Archived 2026-08-27._

`[2026-08-11]` **stonehenge-park — new fleet `/park` service repo stood up + designed.**

**What.** A separate greenfield repo (`~/development/stonehenge-park`, gitea `vh/stonehenge-park`,
pushed) for a self-contained `/park` service: one durable place to park any idea (repo-born OR
personal), find it by search, and have it **actively resurface** (by due-date or staleness) until
acted on — so parked ideas stop dying when a repo goes cold. NOT part of eshpfi; this is a pointer.

**Design (via `/vor-plan`, converged + persisted to `docs/design/`):** four contract-sized units —
**U1** core store+API (SQLite+FTS5, slug minting, bearer auth, REST) — the tracer, build first; **U2**
scheduler+notifier (in-process; due/stale → statusline `due-count` + althing push to a dedicated
**assistant channel**; keep-surfacing until promote/drop/re-snooze); **U3** `park` CLI (mirrors the
`secret` CLI); **U4** browse UI. `/vor-ui` ran too (U4 brief persisted).

**Locked decisions (operator):** SQLite, self-contained, ONE container, no external DB ("don't want
to troubleshoot it when a database upgrade happens") — a hard `[OPS]` invariant; system-minted
title-derived slugs + short ID (addressable as `park/<slug>`); active keep-surfacing resurfacing with
**re-snooze as the anti-nag valve**; bearer key, LAN/WG-internal; host nh3-docker; `/park` **replaces**
the global ROADMAP parking-lot discipline (deferred ideas → `/park`, `source`-tagged; ROADMAP keeps
only the v1 target) as a **fast-follow after v1** incl. migrating existing lots.

**Deferred (in the plan):** the althing assistant-channel handle **name** (decide at U2 contract
time); staleness threshold + re-push cadence (env-tunable defaults ~30d/~daily); design U2's emit
structured/consumable so a future **mission-control (Ledger→orchestrator)** can read it — park does
NOT build the orchestrator.

**State.** Pre-seeded for a fresh agent (CLAUDE/persistent-memory/ROADMAP/README + the design docs),
committed (`294ee98`), pushed. Next build task lives in that repo: the **U1 tracer contract** under
the House Code Discipline. Auto-memory candidate not yet written (repo is self-documenting).
_Archived 2026-08-27._

- `[2026-08-12]` **Global `~/.claude/CLAUDE.md`: `secret`/vault tool entry + "store in AND pull from the vault" standing directive** (dotfiles `9db703b`, pushed); statusline reset-countdowns + a latent tab-collapse parse-bug fix, now tracked in the dotfiles stow tree. Dogfooded the directive: created `vh/stonehenge-park` pulling the gitea token via `secret get`. (dotfiles + global config, not eshpfi.)
_Archived 2026-08-27._

- `[2026-08-11]` **TTS stack extracted to its own repo (`tts-stack`) + eshpfi stood down on TTS dev.** Operator: hand all TTS tuning/dev to a separate agent with a self-contained repo (knowledge + infra access + a live knowledge list), and move the voice corpus in. New repo `~/development/tts-stack` (commit `9ee3288`) carries: dots-tts stack (canonical intent), `voices/` corpus (MOVED out of eshpfi), `KNOWLEDGE.md` (engine landscape + prosody findings + foot-guns), `docs/infrastructure.md` (irv-ml1 access + gated deploy runbook + rollback), CLAUDE/persistent-memory/ROADMAP, `tools/` (pause-probe + Booth render). Followed the **chatterbox-fast precedent**: eshpfi `stacks/dots-tts/` reduced to a POINTER README; the ~15 experimental TTS compose wrappers stay here as reference (catalogued in tts-stack KNOWLEDGE). Blast-radius check: no eshpfi playbook/script reads the canonical corpus (other `voices/` refs = unrelated host paths). **Reverses** the earlier "Corpus home = eshpfi `voices/` (keep-here)" call. ⚠ tts-stack is LOCAL-ONLY until pushed — needs a gitea remote (`vh/tts-stack`) + push before the separate agent can clone (operator's call — outward-facing + repo-create creds).
_Archived 2026-08-27._

- `[2026-08-10]` **dots-tts v3 — clause-break → period pause mapping.** Operator: v2 "sounds good" but donut won't pause at semicolons/dashes. ROOT CAUSE (measured via a pause-probe A/B — synth duration over N runs, non-determinism averaged out): dots' prosody honors a real pause **only for ellipsis (~+0.43s) and period (~+0.3s, capitalization-independent)**; comma/semicolon/colon/dash all run **flat (~+0.03s vs no-punct)**. Two distinct sub-causes: **dashes regressed in v2** (the `—`→`-` fold made em-dashes read as word-joiners), while **semicolons were NEVER a v2 change** — dots ignores them natively, only newly noticeable because v2 made everything else clean. Operator call: ellipsis "too much" → **map `;`, clause `:`, and em-dash `—` → period** in `_sanitize` (believable ~0.3s clause break). GUARDS (pinned by 11 unit tests, `stacks/dots-tts/test_sanitize.py`): digit-guarded colon `(?<!\d)\s*:\s*(?!\d)` so times `3:45` / ratios `2:1` survive; en-dash `–`→hyphen KEPT (numeric-range `10–20` safety — em-dash breaks, en-dash ranges, different jobs); genuine ellipsis left at full strength (author meant a long pause). Gated deploy (redeploy2 pattern → v3): build → throwaway :8199 test container + **pause-gate** (semicolon sentence must run ≥0.12s longer than baseline; measured **+0.427s**) → only then cut live over. LIVE + healthy `local/dots-tts:v3` on :8198. **rollback = `sed -i 's/^DOTS_TAG=.*/DOTS_TAG=v2/' .env + docker compose up -d dots-tts`** (v2 image retained). Booth `dots-pauses` (A=old-flat / C=ellipsis-too-much / D=live-v3). [[reference_chatterbox_fast_repo]]
_Archived 2026-08-27._

- `[2026-08-10]` **dots-tts v2 — contraction fix (curly-sanitize) + sentence-chunking + dependency-pin recovery.** Operator: donut read contractions wrong ("you're"→"you ree", "donut's"→"donut ess"). ROOT CAUSE (isolated via A/B booth): **curly/typographic apostrophes** (`’` U+2019 from ratatoskr's LLM) — dots' tokenizer mispronounces them; STRAIGHT apostrophes read clean under `normalize_text=True`. FIX (`app.py`): fold curly→ASCII (`str.maketrans`) before synth, **KEEP `normalize_text=True`** (operator call — retains number/date expansion). Also added **server-side sentence-chunking** (pack ≤280 chars): dots caps one `generate()` at ~500 patches/~40s, so long RP turns (the Zev monologue = 160s audio) truncated; chunking stitches them (verified full 160.3s, not 40s-cut). **⚠ BUILD FOOT-GUNS (both bit this redeploy):** (1) upstream dots.tts `constraints/recommended.txt` now pins **`gradio==6.17.0` — phantom, not on PyPI** → fresh `pip install dots.tts` unsatisfiable; FIX = pin `dots.tts==0.2.1` + **DROP** the `-c recommended.txt` constraints (0.2.1 pulls working gradio 6.17.3). (2) pinning only `torch==2.8.0` let **torchaudio float to 2.11.0 → dots.tts refuses to load** (minor-version match check); FIX = pin `torchaudio==2.8.0`. **⚠ DEPLOY LESSON:** `docker compose up -d` to a new tag swaps the LIVE container BEFORE any health check — a broken image crash-loops production (**ratatoskr TTS down ~1-2min this session**). NEW PATTERN = build → test in a THROWAWAY container on an alt port (:8199) → health+verify → only THEN cut live over (redeploy2.sh). v2 LIVE + healthy on irv-ml1:8198, **CONSUMER-CONFIRMED clean** (ratatoskr verified end-to-end on their :8765 — apostrophe string reads clean, /api/tts 200 @ 48kHz, no client change; the ~1-2min blip didn't hit them, their concurrent auto-audio issue was client-side localStorage). **rollback = `sed DOTS_TAG=v1 + docker compose up -d dots-tts`** (v1 image retained). Also: deployed container GPU crept ~6→13.9GB over 8h serving (cache accumulation; a redeploy resets it — watch item). [[reference_chatterbox_fast_repo]]
_Archived 2026-08-27._


- `[2026-08-07]` **Personal-Worldtree kb-contamination incident (WT #394) diagnosed; attribution CLOSED UNRESOLVED.** A reconcile `WingStore._embed` full-tree walk (kb `fs_root=KB_PATH` root, sibling wings nested) swept 5,354 fiction+main rows into personal's `knowledge_base` (2 superseded generations served as current). Fixed by WT #394 (aca39a1, kb walks exclude sibling wings; ships b182). Trigger un-attributable — peer reconcile via the SHARED infra-ops identity + 0 dockerd exec-logging = fingerprint-less. Durable finding → auto-memory `infra_ops_shared_identity_attribution_gap`, PARKED (operator ruled A) into [[project_migrate_infra_access_to_claude_credentials]]. Evidence hold on the 5,354 rows until operator sequences cleanup (w/ Brokkr, on #394's agenda).
  _Archived 2026-08-22._

- `[2026-05-12]` corviduo-dev (Worldtree-team dev VM, 10.250.50.152,
  CT 106 on pfi-pve) added to `servers/` inventory. Treat like SF
  client hosts: PFI hosts + provides emergency-ops backstop;
  Worldtree team owns OS config + deploys + backup decisions.
  _Archived 2026-05-27._

- `[2026-05-12]` Worldtree `:latest` tag drift bug — fixed by
  health-gated `:latest` advance in vh/worldtree's deploy workflow
  (architect commit `8ef3801`): only tag `:latest` AFTER the new
  container's `/health` probe passes. Build-on-host stacks here
  don't have this problem because the playbook always builds the
  SHA-tagged image from a `git reset --hard <ref>` checkout.
  _Archived 2026-05-27._

- `[2026-05-12]` asset-engine stack scaffolded LAN-direct at
  `http://10.250.50.70:8200`. Initially included Traefik labels for
  public hostname; user pulled them out (internal tool, no public
  TLS surface needed). Pattern: internal tools default LAN-direct;
  Traefik wiring only when external/TLS required.
  _Archived 2026-05-27._

- `[2026-05-12]` asset-engine catalog gains `lifecycle: { stack,
  vram_gb, gpu_device_id }` per irv-ml1 service for the
  orchestrator feature. SSH keypair scaffolded at
  `ana-docker:/opt/docker/conf/asset-engine/ssh/` for
  asset-engine container → irv-ml1 orchestration via dedicated
  ed25519 key.
  _Archived 2026-05-27._

- `[2026-05-13]` `pull-hf-repo.yaml` is the canonical HF-fetch
  playbook on ana-ml2. Supports `--var repo_type=model|dataset|space`.
  Replaces ad-hoc `huggingface_hub.snapshot_download` calls.
  _Archived 2026-05-27._

- `[2026-05-13]` Selene-1-Mini-Llama-3.1-8B added to llama-swap as
  judge model. mradermacher `i1-Q6_K` imatrix quant (~6.5GB). AtlaAI
  reward/eval model — temp 0.01, ctx 32K, q8_0 KV cache. New JUDGE /
  EVAL MODELS section in `stacks/llama-swap/conf/config.yaml`.
  _Archived 2026-05-27._

- `[2026-05-13]` `vllm-qwen3` → `vllm` stack rename. Added
  `vllm-reward` service (Skywork-Reward-V2-Llama-3.1-8B-AWQ classifier).
  Three vLLM services share GPU 1 (embed 0.20, rerank 0.20, reward 0.30
  utilization; 30% headroom). All use `--runner pooling`; classification
  drives via model's `architectures: [LlamaForSequenceClassification]`
  in config.json, NOT `--task classify` (deprecated in vLLM 0.19.1).
  _Archived 2026-05-27._

- `[2026-05-13]` /tend-docs first pass deletions: `stacks/infinity/`
  removed (retired by vllm). Archived
  `docs/asset-engine/design-brief.md` → `docs/archive/asset-engine/`
  with archival header. Fixed `pfi-pve` VM list to full
  `qm list` enumeration. Dropped stale weak-password section from
  `pfi-postgres` (rotation done 2026-04-23).
  _Archived 2026-05-27._

- `[2026-05-14]` althing-chamber stack scaffolded: chamber + forseti.
  Internal LAN-only at port 7881 (chamber default 7878 collides with
  task-board). Two-service compose, shared SQLite bind-mount,
  build-on-host pattern via vh/althing's gitea-workflow. Forseti is the
  canonical dev for this stack (galdrabok is on a different project).
  _Archived 2026-05-31._

- `[2026-05-16]` althing-chamber Phase 2: added `althing-agent-runner`
  as third compose service (worldtree-driver agent dispatcher).
  All three althing services use the same image; `command:` selects
  entrypoint. Safe to enable preemptively (sleeps when no
  driver=worldtree handles declared).
  _Archived 2026-05-31._

- `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8
  alpine as a sibling compose service in `stacks/althing-chamber/`,
  redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` /
  `msg_start` / `msg_complete` event kinds. DB bridge keeps
  `msg_curated` + `floor_grant` (structured / canonical). Two-channel
  architecture, no overlap. chamber + agent-runner `depends_on:
  valkey: service_healthy`.
  _Archived 2026-05-31._

- `[2026-05-17]` Worldtree admin workflow shift (per vh): infra-ops
  gets its own permanent admin-tier key (`61419c92`, stored at
  `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`).
  Future admin ops route through this key, not the bootstrap admin
  via docker-as-root.
  _Archived 2026-05-31._

- `[2026-05-17]` Worldtree env-var addition checklist: anytime
  introducing `os.environ.get("FOO")` in worldtree code, update BOTH
  `.env.example` AND `compose.yaml`'s `&worldtree-env` anchor in the
  same PR. Same Z_AI_API_KEY-shape footgun bit
  `BIFROST_CLIENT_ALLOWED_HOSTS` (#170) until worldtree-dev added the
  passthrough line in `08f02b2`.
  _Archived 2026-05-31._

- `[2026-05-18]` Volva systemd install complete after three-stage
  debug. Final unit at `/etc/systemd/system/volva.service` runs as
  `User=lkraven` with `ProtectHome=read-only` +
  `ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex`
  carve-outs for state writes. `VOLVA_ALTHING_CLI=/home/lkraven/
  .local/bin/althing-cli` + `ALTHING_HANDLE=volva` both pinned in
  env.sh.
  _Archived 2026-05-31._

- `[2026-05-19]` Worldtree CD disk-hygiene strategy: watermark gate
  (env-tunable threshold + window, fail-loud on still-low post-prune)
  + eager post-deploy prune (only after `:latest` advance succeeds,
  uses `docker image prune -a --filter "until=24h"` which respects
  in-use semantic — protects pinned + personal images automatically).
  Combined: demo VM holds ~24h of deploy history instead of unbounded
  accumulation. Shipped in vh/Worldtree PR #184 (`306cd61` +
  `613dac2` + `bd91df5`).
  _Archived 2026-05-31._

- `[2026-05-19]` Skaldsong CD shape: shape (1) of three operator
  options — container + Gitea registry + pull-restart, matching
  Worldtree's pattern. Target host ana-docker (NOT nh3-dev where
  skaldsong-dev runs for iteration). SHA-pin only for now; health-gated
  `:latest` advance is a follow-up once `/health` exercises Worldtree
  + Kokoro reachability.
  _Archived 2026-05-31._

- `[2026-05-19]` Skaldsong prod (ana-docker) switched from demo
  Worldtree (:8080) to personal (:8081). Same `user_id=skaldsong`
  as the nh3-dev hand-launch key — shared Heimdall agent slot
  (`skaldsong:wizard-v2`), different `key_id`s for independent
  rotation. Demo Worldtree stays for isolation; personal becomes
  the multi-consumer dev iteration instance.
  _Archived 2026-05-31._

- `[2026-05-19]` mead-hall Bifrost v0.3 end-to-end smoke green.
  Closed task #32 (althing thread `01KRV1M2KW6N6HBEXGTH72QXCA`).
  Wire layer (handshake + binding + dispatch) + data-flow
  (per-dispatch JWT claims → `ctx.session_id` populated → real
  session-scoped data) + agent-loop (LLM reads + quotes back) all
  proven. Resolves the "stalled mid-Worldtree" state from the
  2026-05-17 snapshot.
  _Archived 2026-05-31._

- `[2026-05-25]` v0.25.3 lofn tuning: `temperature 0.6 → 1.0` +
  `repetition_penalty 1.0 → 1.15` on default+fast profiles.
  Heretic-abliterated qwen3.6 was locking into degenerate attractors
  at the model's thinking-mode floor (0.6). Pattern:
  abliterated/uncensored Qwen variants need higher temp + non-trivial
  rep-penalty than base, NOT the model-card's documented floors.
  _Archived 2026-05-31._

- `[2026-05-25]` Worldtree #205 v0.25.2 ships `/app/config/` as
  bind-mount + root-then-drop entrypoint shim (gosu). Operators
  get persistent per-instance config without container-rebuild.
  Same bind-mount pattern hit twice subsequently in v0.27.0 (selene)
  and v0.29.9 (echo) — bind-mount shadows image-baked defaults, so
  every new required key surfaces as a crash-loop on existing
  deployments. The v0.29.12 canonical example files close this loop.
  _Archived 2026-05-31._

- `[2026-05-26]` Worldtree v0.27.0/v0.27.1 fixes Tier 3 GET
  visibility. `available_agents()` helper was over-applied to
  `GET /agents/<id>`, masking ALL Tier 3 agents regardless of row
  state. Bug only visible as "agent not found" via GET; storage was
  fine (silent-2xx PATCHes had persisted correctly). v0.27.1 added
  fail-fast hardening for the startup pre-resolve fragility class.
  _Archived 2026-05-31._

- `[2026-05-26]` Skaldsong v0.30.7 defensive 409→PATCH fallback.
  v0.30.6's GET-then-define-or-patch path crash-looped against
  pre-v0.27.0 Worldtree's GET-visibility bug (GET 404 phantom →
  define 409 conflict). v0.30.7 catches the 409 and falls through
  to PATCH (which silently 2xx'd on the pre-fix Worldtree).
  _Archived 2026-05-31._

- `[2026-05-27]` Worldtree v0.29.x landed full saga→echo refactor +
  config-validator hardening (v0.29.10 create_provider family-before-regex;
  v0.29.11 collect-then-raise echo startup validators; v0.29.12 ships
  providers.yaml.example/defaults.yaml.example canonical configs; v0.29.13
  reasoning_content extraction + catalog family lookup). Operator-asked,
  worldtree-dev-shipped, CI-deployed.
  _Archived 2026-06-01._

- `[2026-05-27]` artemis-31b-v1i added to llama-swap + worldtree personal.
  BeaverAI Gemma 4 31B Q6_K (~28.6GB), 128K ctx, `--reasoning-format deepseek`
  (gemma format unsupported in deployed llama.cpp). Worldtree catalog
  `family: gemma` so GemmaProvider routes reasoning tokens.
  _Archived 2026-06-01._

- `[2026-05-27]` Skaldsong streaming TTS v0.32.0→v0.32.2: chunked-batch SSE
  (one Kokoro POST per paragraph); defensive event_stream catch-all; NDJSON
  parsing for Kokoro /dev/captioned_speech multi-line responses.
  _Archived 2026-06-01._

- `[2026-05-31]` Dia2 deployed as **two fixed-model instances**
  (`dia2-2b` :8200, `dia2-1b` :8202) from `local/dia:v2`, retiring
  legacy Dia 1.6B; catalog `dia` entry **removed → dia2-2b + dia2-1b**
  (breaking for asset-engine). Rationale: the devnen wrapper is
  single-model and IGNORES the OpenAI `model` field (verified on its live
  OpenAPI), so the only way to offer both Dia2 models as real per-request
  asset-engine choices is one fixed endpoint per model. `3139e81`
  (deploy), `db15638` (catalog swap).
  _Archived 2026-06-03._

- `[2026-05-31]` Both dia2 catalog entries route to the wrapper's richer
  **`/tts`** endpoint (not `/v1/audio/speech`) to expose the full control
  surface (cfg_scale/temperature/top_p/cfg_filter_top_k/voice_mode/clone);
  all defaults sourced from the wrapper's `CustomTTSRequest` Pydantic
  blessed values. Voice default is `voice_mode: clone` +
  `clone_reference_filename: Abigail.wav` so a stable (non-random-gender)
  voice is the out-of-box behavior. `55602b7`, `5c47843`.
  _Archived 2026-06-03._

- `[2026-05-31]` Zonos REST adapter (`stacks/zonos/adapter/`,
  `local/zonos-api`) — thin OpenAI-ish `/v1/audio/speech` FastAPI in front
  of the Gradio-only Zonos SDK; JSON-envelope `{audio, audio_format,
  seed}` (Zonos is the fleet's first seedable TTS). Port 8203 (moved off
  8201 — collided with csm). Built; NOT deployed (stack down for VRAM).
  Also fixed the upstream image's missing CMD (`71df6f7`). `81efa8d`.
  _Archived 2026-06-03._

- `[2026-05-31]` Catalog schema regenerated: added `CatalogLifecycle` +
  `reproducibility.seed_field` (`b7b2130`). Resolves the stale-schema
  hand-off; catalog now validates clean. (asset_engine `catalog.py`
  Pydantic already supported both — schema file was just behind.)
  _Archived 2026-06-03._

- `[2026-05-31]` TTS bench expanded with `stacks/{dia,zonos,csm}`
  (`666f7f3` dia+zonos, `a4b8c2a` csm). The bench already had Fish
  S2-Pro / Chatterbox-Turbo / IndexTTS-2 / CosyVoice3 / Kokoro /
  VibeVoice / Qwen3-TTS / Kyutai. (csm since removed 2026-06-01.)
  _Archived 2026-06-03._

- `[2026-05-31]` Remote browser/iPad/Vision-Pro driver seat for the
  agent-fleet zellij session `Claude` stood up on nh3-dev (ttyd behind
  Caddy, network-gated). Out of this repo — full architecture + the
  HTTP2/OSC52/Safari-auth gotchas in auto-memory `reference_ttyd_fleet_seat`.
  _Archived 2026-06-03._

- `[2026-05-30]` esh-docker-vm NFS boot-ordering fix:
  `playbooks/fix-esh-nfs-boot-ordering.yaml` (`c0458d9`, + `53157b1`
  drop-in filename-collision fix) adds `_netdev,nofail` to the four
  10.0.50.50 NFS mounts + a docker `After=remote-fs.target` drop-in —
  resolves paperless `Exited(255)` on reboot. traefik also gained
  `restart: unless-stopped`. Full incident → auto-memory
  `incident_esh_docker_nfs_boot_race`.
  _Archived 2026-06-03._

- `[2026-06-01]` **Fish cloning VERIFIED competent (ECAPA-TDNN)** — retracting
  the earlier "weak cloner" call. Isolated test: Imogen-referenced clone ~0.79
  cosine to the real `Imogen.wav` vs ~0.10 for the no-reference default;
  transcript condition (correct 0.787 / bogus 0.778 / empty 0.738) barely moves
  identity (affects pronunciation, not timbre). Root cause of "Imogen sounds
  nothing like British" = the `"undefined"` select bug feeding Fish its default
  voice, NOT Fish. So the entire Fish-Imogen saga was the undefined bug; no
  Fish-side fix needed. (Methodology lessons → Tried and abandoned.)
  _Archived 2026-06-04._

- `[2026-06-01]` **CSM (Sesame csm-1b) torn down entirely** — removed from
  catalog, `stacks/csm/`, `playbooks/deploy-csm.yaml`, and host
  (`c54ab13`). Two reasons: (1) deep-research verdict — the acclaimed
  Maya/Miles demo is a fine-tuned, larger variant Sesame never open-sourced;
  the open 1B base ships no usable voices, can't generate text, English-only,
  no real-time streaming, absent from leaderboards, dominated by
  Kokoro/Dia2/Fish/IndexTTS. (2) The upstream image is rotted (see Tried and
  abandoned). asset-engine re-synced (`306aae9`).
  _Archived 2026-06-04._

- `[2026-06-01]` **Chatterbox catalog v1→v2** (`93cedd6`) — re-point to the
  devnen wrapper's `/tts` (was the thin OpenAI `/v1/audio/speech` exposing
  none of the emotion knobs → poor output). Default `exaggeration: 0.5` per
  Resemble's README ("works well for most"); the server SHIPS demo-tuned 1.3
  which over-acts — deliberately NOT adopted. 28 predefined voices via
  `/get_predefined_voices` (default Emily.wav). Corrected seedable false→true.
  _Archived 2026-06-04._

- `[2026-06-01]` **Fish-S2 catalog v1→v6** (`f3c260e`..`38d9e3b`) — was a
  blank 2-field form; now exposes the full fish-speech ServeTTSRequest surface
  + a 32-voice `reference_id` "Voice" dropdown (STATIC list — Fish has no
  /voices API). `reference_id`-by-name works ONLY when each voice is staged as
  a `references/<name>/` dir with `<name>.wav` + `<name>.lab` — was silently a
  no-op until that layout fix on 2026-06-01 (see the two-root-causes entry
  above); the inline-path-as-filesystem-path `references` shape 500s (see Tried
  and abandoned). Corrected vram
  4→24 GB, cold_start 8→240s, seedable false→true. Added 3 consenting
  British-female VCTK voices (Imogen=p225/Eleanor=p228/Beatrice=p229).
  _Archived 2026-06-04._

- `[2026-06-01]` **`blendable: false` catalog flag — DEFERRED**, pending
  asset-engine-dev honoring it in the form template. Clone-voice selects
  (chatterbox/fish/dia2) must render as plain dropdowns, not the Kokoro blend
  widget (blending reference clones is nonsensical). I ship the catalog flag +
  schema field once they read it. Tracked: althing msg
  `01KT2K2SY9N7AY69R9V0B4RXSW` to asset-engine-dev.
  _Archived 2026-06-04._

- `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — Populated
  `references/<name>/<name>.wav`+`.lab` for all 32 voices; playbook gained
  normalize-step + A/B smoke gate. glados got a real transcript (ASR'd via
  Parakeet): the Portal "Welcome to test chamber 4" lines.
  _Archived 2026-06-11._

- `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast`
  streaming container.** Workload = single-stream interactive. GPU placement:
  3090 (device 0) if it fits else A6000 (device 1). Cutover: parallel catalog
  entry, burn in beside live `chatterbox`, then flip. Streaming = adaptive
  buffer-ratchet chunking (native frame-streaming abandoned). Since extracted to
  its own repo `vh/chatterbox-fast` (auto-memory `reference_chatterbox_fast_repo`).
  _Archived 2026-06-11._

- `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** Claimed
  naive sentence-level streaming has "zero quality loss" — WRONG. The T3 AR backbone
  conditions prosody on the WHOLE text; splitting loses cross-sentence prosodic
  context even though voice timbre stays (reference-conditioned). No *artifacts* ≠
  no *quality loss*. Hence adaptive-chunk design (maximize context per chunk subject
  to latency budget), not fixed per-sentence splits.
  _Archived 2026-06-11._

- `[2026-06-03]` **Declined worldtree v0.30.4 staging deploy** — that's worldtree-team's
  CI/CD lane (a developer `staging/vX.Y.Z` git-tag promote), not infra-ops. They self-corrected
  to the same conclusion independently.
  _Archived 2026-06-13._

- `[2026-06-03]` **R14 scope = (a) provision-only.** infra-ops provides box + CUDA env +
  engines + weights + NFS; brokkr/dev wires `arms.py` + runs — keeps infra-ops OFF the
  VIVAE-processing path (VIVAE = Variably Intense Vocalizations of Affect/Emotion, CHARTER §4
  highest-liability; operator authorized R&D-eval-only, quarantined). Box = irv-ml1 (A6000
  free; ana-ml2 GPU-saturated). Per-engine venvs (divergent torch stacks); A6000 = `cuda:0`
  NATIVE (≠ docker `=1`).
  _Archived 2026-06-13._

- `[2026-06-03]` **yt-voice-clipper push-to-deploy via gitea webhook** (operator-directed,
  after 6 manual rebuilds in ~40 min). Webhook (not poll) — gitea CAN reach the WG IP per the
  operator. The proxy env + Homepage labels live in the **host-specific override** (untracked
  → survive the auto-deploy's `git reset --hard`), NOT yt-voice-clipper-dev's image. Runbook
  `d4f180d`.
  _Archived 2026-06-13._

- `[2026-06-03]` **yt-voice-clipper bot-gate fix = route yt-dlp through NH3 residential
  egress, NOT cookies/PO-token.** YouTube hard-flags the Irvine colo IP (LOGIN_REQUIRED on a
  public video even with no cookies). Cookies + the bgutil PO-token + deno JS-runtime all
  loaded fine — the gate is pure IP reputation. Operator chose proxy-via-nh3-dev → durable
  dante proxy → proven. The egress proxy is a reusable fleet lever for any datacenter-IP-gated
  service.
  _Archived 2026-06-13._

- `[2026-06-04]` **`brokkr` user (no-sudo) on irv-ml1; R14/R15/R16 substrate moved to /home/brokkr.** Persistent box services there need SYSTEM systemd units (see Tried).
  _Archived 2026-06-14._

- `[2026-06-04]` **ollama upgraded 0.9.0→0.30.4 on irv-ml1** (Ministral-3 is a Dec-2025 model the old engine refused); A6000 pinned by **UUID** not index (native fastest-first ≠ nvidia-smi PCI).
  _Archived 2026-06-14._

- `[2026-06-04]` **Worldtree demo/pinned/personal deploys are ALL CI/CD, not infra-ops** — a "deploy vX.Y.Z" request to infra-ops is MISROUTED → point them back to their pipeline. (`d8d776c`, auto-memory `reference_worldtree_deploys_cicd`)
  _Archived 2026-06-14._

- `[2026-06-04]` **infra-ops NOPASSWD-sudo identity commissioned, scoped to PFI boxes** (+esh-docker-vm by operator override) — so infra-ops completes DevOps end-to-end vs handing the operator sudo steps. Dedicated key, sudo log_output, key-gated. (`8c32a05`, auto-memory `reference_infra_ops_sudo_identity`)
  _Archived 2026-06-14._

- `[2026-06-04]` **phi4 ships the CANONICAL/official Phi-4 chat template, NOT Ollama's** (Ollama's omits the system `<|end|>`). Applied an Ollama-matching override then reverted — ship correct, not the benchmark quirk. (`90e08f0`→`27eb537`; "headgun" lesson in Tried.)
  _Archived 2026-06-14._

- `[2026-06-04]` **phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer; granite-4-small retired** from llama-swap. 50K ctx + FP8 KV. (`40a374b`) [superseded by the 2026-06-05 granite-4.1-8b decision]
  _Archived 2026-06-14._

- `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons; Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid `12aa5a9`); volva.service was crash-looping 203/EXEC. (`6e2f80e`)
  _Archived 2026-06-16._

- `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer.** Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not Q4/AWQ** — primary workload is high-concurrency, where FP8 scales ~linearly (2010 tok/s @ C=32). vLLM `vllm-granite` :8004 GPU 1, official IBM compressed-tensors FP8. (Then on Ada; box has since gone Blackwell.) (`34a43a0`, auto-memory `reference_ana_ml2_vllm_granite`)
  _Archived 2026-06-16._

- `[2026-06-05]` **Langfuse v3 on ana-docker (:3001) as the gateway trace UI**; LiteLLM `success_callback:[langfuse]` live. Pretty traces + tok/s dashboard. NOT a prerequisite (spend_logs already capture tokens+latency). (`9171e6a`)
  _Archived 2026-06-16._

- `[2026-06-05]` **Ollama BANNED fleet-wide** (operator directive) — never stand one up; tear down any found; serve via llama-swap or vLLM. (auto-memory `feedback_avoid_ollama`)
  _Archived 2026-06-16._

- `[2026-06-05]` **ComfyUI / FLUX.2 work split to `~/development/comfy-dev`** (dedicated repo + agent). eshpfi keeps the `comfyui`/`arbo` stack compose; comfy-dev owns the model/workflow knowledge. (auto-memory `reference_irv_ml1_ampere_quant`)
  _Archived 2026-06-16._

- `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is the structured-output profile, ON HOLD, no live consumer). Bind-mount CONFIGS (providers.yaml, vh-owned) ARE infra-ops's to apply directly — only the app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`)
  _Archived 2026-06-16._

- `[2026-06-13]` **Arbo prod bearer auth turned OFF — WireGuard is the access boundary** (operator decision; reverses ADR-0001's "closed the open-auth hole"). ENGINE_TOKEN must be **ABSENT, not empty** (empty-string still gates) — removed from BOTH the host `.env` AND the compose `environment:` injection line. Original token backed up at `irv-ml1:/opt/docker/compose/arbo/.env.pre-auth-off.bak`; comfy-dev updated their ADR-0001. Tracking: `db97899` + `playbooks/arbo-disable-engine-token.yaml`.
  _Archived 2026-06-16._

- `[2026-06-13]` **Storetank image-models archive DECOMMISSIONED; arbo is the single live ComfyUI model tree (502 G).** Curated `/storetank/image-models/comfy` (was 919 G, the native `/opt/ComfyUI/models` symlink target) → killed everything superseded by arbo's current gen (Hunyuan, WAN2.1, FLUX.1, Chroma, SD3.5, orphaned umt5+llava ≈ 739 G) + migrated the keepers (gen-agnostic utilities + the SDXL/Pony stack, 177 G) into `/storetank/arbo/models` (same-fs move, skip-existing protects prod). Tracking: `docs/storetank-image-models-archive.md` + `docs/arbo-comfyui-model-catalog.md` (`1902425`→`5007ec1`).
  _Archived 2026-06-16._

- `[2026-06-13]` **GRANITE_KEY provisioned to comfy-dev's nh3-dev dev env** at `~/.arbo_granite_key` (0600) for the hero gen+judge script — verbatim copy of the prod `arbo-prompt-enhance` vkey (now extended to reach BOTH `granite-4.1-8b` AND `qwen3.5-9b-fp8`); nothing minted. The vkey README's "granite-only" wording was stale → corrected (`f32c6dd`).
  _Archived 2026-06-16._

- `[2026-06-13]` **ana-ml2 upgraded Ada → dual RTX PRO 6000 Blackwell Max-Q** (96 GB each, cc 12.0 / sm_120; was dual RTX 6000 Ada 48 GB / cc 8.9 — confirmed live via `nvidia-smi`). Unlocks NVFP4 (FP4 tensor cores) and doubles VRAM headroom. CLAUDE.md GPU-spec doc-fix **LANDED `355a240`** (operator). Tracking: `19a07b9`/`1e2a3a1` ("Blackwell 96GB").
  _Archived 2026-06-16._

- `[2026-06-13]` **NVFP4-W4A4 is infeasible for Granite — FP8 stays the Granite-on-Blackwell format.** W4A4 collapses at 30k context, proven **producer-independent** (modelopt AND llm-compressor both clean-NONE from the same BF16 base + wikitext-2k calib). No 4-bit wins both axes: W4A4 = quality collapse; W4A16-NVFP4/AWQ = weight-only dequant → bf16 (no FP4-core speedup). **30B retired**. (auto-memory `reference_nvfp4_w4a4_granite_infeasible`)
  _Archived 2026-06-16._

- `[2026-06-13]` **Qwen3.5-9B VL (FP8) deployed on ana-ml2 GPU 1** — `qwen35-vl` stack, :8007, gateway alias `qwen3.5-9b-fp8`. **Pinned nightly digest, not `:latest`**: the stable release quantizes the VL *vision tower* under `--quantization fp8` → garbage vision (LM fine); the nightly correctly excludes it. Re-pin + drop the pin once that exclusion lands stable. (`2e3dcc2`)
  _Archived 2026-06-16._

- `[2026-06-13]` **comfyui 325 G model tree migrated worktank → `/storetank/arbo`** (worktank 97% → 26%). `arbo` is the consuming app; overlay bind-mount via `COMFYUI_MODELS_DIR`. (`38186be`) (See the 2026-06-13 archive-decommission decision above — this tree later absorbed the storetank-archive keepers, reaching 502 G.)
  _Archived 2026-06-16._

- `[2026-06-13]` **GPU layout settled on the Blackwell box.** GPU 0 held free for large-model hot-loads (llama-swap pinned, `edf0f91`); GPU 1 steady-tenant — granite 131k ctx, qwen 65k, embed/rerank/reward trio, ~3.5 GB free (`1e2a3a1`, `19a07b9`; trio re-floored for 96 GB, 20×-parallel-stable). embed/rerank left at floor — long docs chunked BEFORE embedding. max-model-len is a ceiling not a reservation.
  _Archived 2026-06-16._

- `[2026-06-13]` **Prefix caching pinned explicit on granite + qwen** — benched ~6.5× faster TTFT on a shared ~4.5k-token summarizer template; soft/evictable, neutral when prefixes don't repeat. vLLM v1 defaults it ON (granite) but the qwen nightly defaults OFF — pin both. (`a9a2be7`)
  _Archived 2026-06-16._

- `[2026-06-13]` **granite-4.1-8b listed as the always-available summarizer/classifier + a shared all-agents key minted** (operator-directed). Global `~/.claude/CLAUDE.md` Global-tools entry; key alias `all-agents-local`, scoped to the FREE local models only (granite + qwen-vision + embed/rerank, NOT paid GLM), internal-gateway-only, rotatable. (auto-memory `reference_litellm_gateway`)
  _Archived 2026-06-16._

- `[2026-06-13]` **arbo engine + frontend stack stood up** (ADR-0001) — irv-ml1 co-located inference engine (`ee57e69`), python-based healthcheck (slim image, no curl/wget, `bdb3312`), frontend ro-mounted from the checkout (`922e8ad`, ADR-0001 D2).
  _Archived 2026-06-16._

- `[2026-06-11]` **GLM thinking inverted at the LiteLLM gateway** (operator call): `glm-5.1` defaults thinking-OFF; `glm-5.1-reasoning` = same z.ai upstream, thinking ON. Mechanism: `litellm_params.extra_body:{thinking:{type:disabled}}` — `drop_params` strips a top-level `thinking`/`reasoning_effort` but forwards `extra_body` verbatim to z.ai. Shared-gateway change. (`95b2701`, auto-memory `reference_litellm_gateway`)
  _Archived 2026-06-16._

- `[2026-06-11]` **pi coding agent installed on nh3-dev as a GLM 5.1 harness** — `@earendil-works/pi-coding-agent` via **bun** (npm's global prefix is `/usr` → needs sudo, bun avoids it). Config `~/.pi/agent/models.json`, launcher `~/.local/bin/glm`.
  _Archived 2026-06-16._

- `[2026-06-11]` **z.ai web-tools (regin) = z.ai hosted MCP path, NOT the `/paas/v4` Tool API.** WORKS: MCP streamable-HTTP at `https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp`, `Authorization: Bearer $ZAI_API_KEY` (the **MCP** key, distinct from `Z_AI_API_KEY` the LLM key). Reference impl = Worldtree's Leif agent.
  _Archived 2026-06-16._

- `[2026-06-10]` **Mac Pro migration framed: workstation-only** (M2 Ultra ARM, racked NH3 on-subnet); sidecars stay Linux. `migration-plan.md`. (See in-flight.)
  _Archived 2026-06-16._

- `[2026-06-10]` **Worldtree deployed-config propagation is infra-ops's OWNED lane** (operator ruling). worldtree-dev pings the config delta pre-merge; infra-ops syncs `config/*.yaml` from MERGED canonical to the `/opt/worldtree*/config` bind-mounts on demo+personal. The v0.33.8 9-HOUR demo outage (a `model_roles.yaml` startup-dep that never reached the VM) is the failure mode this prevents. providers.yaml stays hand-tuned. corviduo emergency-ops = `ssh vh@10.250.50.152`, docker no-sudo. (auto-memory `reference_worldtree_deploys_cicd`, `reference_corviduo_dev_emergency_ops`)
  _Archived 2026-06-16._

- `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr-smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev — granite, later extended to qwen-vision). Mint via `/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus. (auto-memory `reference_litellm_gateway`)
  _Archived 2026-06-16._

- `[2026-06-14]` **ana-ml2 GPU-1 vision upgraded: Qwen3.5-9B → Qwen3.6-35B-A3B (official FP8), served under its TRUE name only.** `qwen36-vl` replaces `qwen35-vl` on :8007 (`a0fed13`). The stale `qwen3.5-9b-fp8` name is KILLED at vLLM AND the litellm gateway (404/400) — a model is NEVER aliased under a prior model's name (silent substitution = downstream footgun; operator directive). Consumer comfy-dev/arbo migrated; arbo vkeys → all-proxy-models; shared `all-agents-local` key repointed qwen3.5-9b-fp8 → qwen3.6-35b-a3b. GPU-1 rebalanced for the ~34 GB FP8 weights (granite 0.35→0.24/64K; embed/rerank 0.05→0.03, reclaimed ~4 GB util-waste). Validated: vision correct, 20-concurrent = no OOM. (auto-memory `feedback_no_false_model_aliases`)
  _Archived 2026-06-18._

- `[2026-06-14]` **NVFP4 was the lighter fit (~21 GB) but is BLOCKED on vLLM — FP8 is the working vision path.** `nvidia/Qwen3.6-35B-A3B-NVFP4` won't load: the ModelOpt-NVFP4-MoE loader errors on expert/lm_head scale keys across 0.19.1 (`w2_input_scale`) AND 0.22.0 (`lm_head.input_scale`, vllm #44081) — a pattern across modelopt NVFP4 MoEs. Revisit NVFP4 (frees ~13 GB on GPU 1) once fixed; the 21 GB checkpoint stays cached on ana-ml2. **(SUPERSEDED 2026-06-16 — it loads on vLLM 0.23.0; qwen36 swapped to NVFP4. See the top of this section.)**
  _Archived 2026-06-18._

- `[2026-06-14]` **llama-swap qwen3.5-9b GPU-0 pin DROPPED; GPU 0 reserved for a creative-writing model (pick DEFERRED by operator).** Deep-research (this session) on big-fast-uncensored creative for a 96 GB Blackwell: **GLM-Steam-106B-A12B** (already in the llama-swap config — balanced default) vs **TheDrummer/Behemoth-X-123B-v2** (prose-tier, tops UGI writing+willingness) vs XORTRON-123B (max willingness, weak prose); GGUF-on-llama-swap is the serving path. Tracking: this session + llama-swap config (GLM-Steam present, `untracked by operator choice`).
  _Archived 2026-06-18._

- `[2026-06-14]` **R16 splice-pivot yield probe executed** (infra-ops ran the irv-ml1 inference for brokkr; brokkr owns design + analysis). See Current state. Tracking: althing thread `01KV010WGS…`, `gen_yield_probe.py` in `irv-ml1:~/r16-vmoan-harness`.
  _Archived 2026-06-18._

- `[2026-06-14]` **R16 vmoan inline-generation arc CLOSED — v1 at default decode (rep1.2/temp0.8) is the final Chatterbox-tag inline artifact.** Operator's ear rejected every alternative: v2/v3 windowing (omission vs coherence-loss), v4 multi-tag (cohesion held but lost to capacity-competition), emergent inline-token modulation (degenerates, not modulates), and the gen-time decode-polish sweep (soft tamers cut the NVV itself — same omission family as v2; p0 baseline beat p1). All adapters v1–v4 + `tokenizer.json.v3bak` preserved on `irv-ml1:~/r16-vmoan-harness`. Likely-next direction (deferred, NOT formalized): generate→bin→splice + one-shot-clone NVV pipeline routing around the inline-coherence wall. Tracking: brokkr R16 journal + althing thread `01KV010WGSSMPWRNCPAGSPK15Y`.
  _Archived 2026-06-18._

- `[2026-06-14]` **Arbo deploy pipeline fixed, hardened, and version-controlled.** Prod rebuilt v0.11.1 → **v0.11.6** backend; the webhook machinery (`arbo-deploy.sh` + `arbo-webhook.py`, :9009 HMAC listener) is now repo-tracked at `stacks/arbo/` (was host-only = recoverability foot-gun). Deploy reaches gitea via the INTERNAL route (`10.250.50.70:222`) and restarts the engine ONLY on `catalog/` changes (graphs/frontend per-request; warn on `src/`|`Dockerfile` only — pyproject/uv.lock churn every commit). Operator kept arbo stack ownership in **eshpfi** (not migrated to comfy-dev's repo). Secret + `.env` stay host-only. Tracking: `6d66bc2`, `6e58e57`, `stacks/arbo/README` Q5.
  _Archived 2026-06-18._

- `[2026-06-15]` **qwen36 thinking-split at the gateway** (operator) — base
  `qwen3.6-35b-a3b` = thinking-OFF default (the vision judge needs it), `…-a3b-thinking`
  = opt-in reasoning. One hybrid checkpoint + `--reasoning-parser qwen3`; mirrors the
  glm-5.1 split. The judge's enable_thinking=false is what makes the NVFP4 swap
  transparent. (`6de0844`)
  _Archived 2026-06-19._

- `[2026-06-15]` **arbo engine rebuilt v0.11.6 → v0.11.22 from main; catalog re-applied (v0.11.23) — curated `/workflows` footer live.** src/ schema changes (Model/Lora.summary, Workflow.blurb/keywords + the /workflows endpoint) need a MANUAL image rebuild (the webhook warns-only on `src/`/`Dockerfile`). Built from `/worktank/arbo/repo`, bumped `ARBO_IMAGE` in the host .env, recreated + verified healthy (auto-rollback armed, unneeded). comfy-dev's v0.11.20 catalog push had first crash-looped prod (old baked schema `extra=forbid` vs new git-mounted catalog fields → ValidationError); they reverted the catalog, so rebuild-from-main loaded clean, then they re-applied (v0.11.23). comfyui is a SEPARATE compose project — the rebuild didn't touch it.
  _Archived 2026-06-19._

- `[2026-06-15]` **althing-core refreshed to v0.14.1 box-wide on nh3-dev** (forseti + heid both routed it; central-sequenced per DevOps-delegation since `uv tool install --force` swaps the shared binary all handles use). Fixes the monitor non-truncating-lock bug so `stop-monitor`/`--stop` reliably recovers orphans. The editable install (`uv tool install --force --editable .`) covers every nh3-dev handle; `--version` trued up 0.14.0→0.14.1; verified `--stop` works against a real orphan.
  _Archived 2026-06-19._

- `[2026-06-15]` **ratatoskr Tier-3 affect live smoke GREEN** (consumer-confirmed). Brokered/minted consumer_id="ratatoskr" Heimdall HS256 key on personal Worldtree (the key IS BOTH the create-session Bearer AND ratatoskr's JWT verifier secret — plain utf-8 string, NOT base64/raw); injected ENV-ONLY to `~/.config/ratatoskr/provider.env` (mode 600) on nh3-dev; added `10.100.10.50:8390` to personal's `BIFROST_CLIENT_ALLOWED_HOSTS`; drove define→session (handshake 200)→turn→affect.emit (200, snapshot persisted). Worldtree signs Bifrost with the caller's raw Bearer verbatim. Drive commands + claims contract in althing thread `01KV4ZYSE3…`.
  _Archived 2026-06-19._

- `[2026-06-15]` **Dense Qwen3-VL-32B-NVFP4 A/B'd as arbo's anatomy hero-judge vs the qwen3.6-35b-a3b MoE → dense LOST, pilot torn down, GPU 0 reclaimed.** Dense NVFP4 loads clean on vLLM where the MoE NVFP4 doesn't (no expert-scale keys); ~44 tok/s on Blackwell GPU 0; vision tower stays high-precision. But comfy-dev's ground-truth A/B: MoE wins (faster + dual-use as general LLM + slight fine-detail edge), and the subtle-proportion tier is below ALL VLs. **Don't re-propose a dense-VL judge swap.** (auto-memory `project-qwen3vl-dense-pilot`)
  _Archived 2026-06-19._

- `[2026-06-15]` **MastMed prototype containerized + exposed publicly via cloudflared.** Self-contained static SPA (bun→caddy multi-stage build), basic-auth gated (operator's gating call), ana-docker:8089, cloudflared quick tunnel. See in-flight for the ephemeral-URL caveat + named-tunnel upgrade path.
  _Archived 2026-06-19._

- `[2026-06-15]` **infra-ops bootstrapped onto corviduo-dev** (operator-directed) — infra-ops now has SSH + NOPASSWD sudo there, OVERRIDING the fleet bootstrap script's deliberate partner-box exclusion (explicit host arg). corviduo config edits no longer need `vh` + a sudo password. (auto-memory `reference_corviduo_dev_emergency_ops`)
  _Archived 2026-06-19._

- `[2026-06-15]` **R18 clip+caption placement = ana-ml2** (operator: "ana-docker… ana-ml2 unless irv-ml1 has room"; irv-ml1 lacked free VRAM for the 30B-A3B captioner). Models staged + stub deployed; see in-flight + the hard real-voice gate.
  _Archived 2026-06-19._

- `[2026-06-14]` **LiteLLM infra-ops admin key provisioned** (operator) — resolves the LiteLLM half of the credential-migration directive; use it for ALL gateway ops (NOT `sk-corvid`). Value at `~/.config/litellm/infra-ops-key` (mode 600); gateway reachable directly from nh3-dev at 10.250.50.70:4000. (auto-memory `reference_litellm_infra_ops_key`)
  _Archived 2026-06-19._

- `[2026-06-16]` **litellm `strip_empty_tools` pre-call hook shipped** (`d1bea13`) — an empty `tools:[]` 500s vLLM ("tools must not be an empty array"); a global `litellm_settings.callbacks` CustomLogger pops it (+ orphaned `tool_choice`) before forwarding, so it covers EVERY vLLM model, not one. `drop_params` only drops unsupported PARAMS, not empty VALUES. Mounts beside config.yaml (litellm resolves callbacks relative to the config dir). Verified live across granite/mistral/stream. (`stacks/litellm/conf/strip_empty_tools.py`)
  _Archived 2026-06-20._

- `[2026-06-16]` **single-file `gateway-chat.html` playground shipped** (`984ca3d`, `tools/`) — zero-dep browser chat straight to the gateway (`:4000`, CORS open), system-prompt box, streaming SSE, renders `reasoning_content`, NEVER sends `tools`. Built because the LiteLLM admin-UI playground can't test vLLM-backed models (see Tried-and-abandoned). Serve on-request via `python3 -m http.server -d tools`.
  _Archived 2026-06-20._

- `[2026-06-16]` **claude-bot granted ADMIN on vh/arbo** (operator) — arbo CI/CD gitea ops now route through the service account, not the operator's personal `vh` (the credential-migration directive). Operator added claude-bot as Administrator collaborator; claude-bot self-serves via its `gitea-password` (basic-auth = FULL perms; its stored token is package-write-only, can't change repo settings). (auto-memory `reference_claude_bot_gitea_creds`)
  _Archived 2026-06-20._

- `[2026-06-16]` **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven.** The missing server-side step was the per-PORT `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist (`:8391`; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the `score_breakdown` shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), not `agent_self_enabled`. (tracking: c99aa49 + auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED:** the `score_breakdown`-shape root-cause was disproven; real cause = cold-recall `scope_filter` adds an `agent_self` axis → ratatoskr's conjunctive INV-005 match drops the `{end_user}`-only chunks → 0 hits. Now Worldtree #297. (auto-memory `project_worldtree_295_recall_rootcause`)
  _Archived 2026-06-20._

- `[2026-06-16]` **infra-ops = the DURABLE admin path on corviduo-dev** (operator granted standing admin; extends the 2026-06-15 bootstrap) — added a `Host corviduo-dev → User infra-ops` `~/.ssh/config` alias + repointed `servers/corviduo-dev/ssh-target` to `infra-ops@`; `ssh corviduo-dev` / `elway corviduo-dev` now land as infra-ops w/ NOPASSWD (vh = explicit-only human path). elway resolves via ssh-config, NOT the ssh-target file. (tracking: c99aa49 + auto-memory `reference_corviduo_dev_emergency_ops`)
  _Archived 2026-06-20._

- `[2026-06-16]` **demo + personal `character` model flipped qwen3.6-35-a3b → mistral-small-4** (operator) — in `model_roles.yaml` the FIRST `catalog_id` in a role's `binds:` is the default; reordered mistral-first (qwen retained in the switch-allowlist, non-destructive), applied via the PIN-SAFE api recreate (derive `WORLDTREE_IMAGE` from the matrix sibling — never a bare `compose up`, which falls to the crash-blocking stale `:latest`), verified a fresh character agent resolves model=mistral-small-4. (auto-memory `reference_corviduo_dev_emergency_ops`)
  _Archived 2026-06-20._

- `[2026-06-16]` **NVFP4 ModelOpt MoE now LOADS on vLLM 0.23.0 — SUPERSEDES the
  2026-06-14 "blocked" finding (#44081 fixed).** Confirmed the cached
  `nvidia/Qwen3.6-35B-A3B-NVFP4` loads clean + generates on 0.23.0; swapped prod
  qwen36 FP8→NVFP4 (~13 GB reclaimed on GPU 1, fp16 KV). Validated by brokkr (speed:
  faster single-stream, ≥parity batched) + comfy-dev (vision anatomy-judge A/B: PASS).
  One W4A4 tell: minor multi-step chained-reasoning slip — doesn't bite the vision
  judge. Mistral's NVFP4 is a SEPARATE compressed-tensors path (day-0), distinct from
  the ModelOpt one. (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`)
  _Archived 2026-06-20._

- `[2026-06-16]` **Mistral Small 4 = the GPU-0 tenant** (operator) — fills the slot
  the deferred creative-writing pick held, and supersedes llama-swap (now down).
  NVFP4 119B MoE, pinned **vLLM v0.22.0** for working vision (0.23.0 breaks it),
  256K context. Operator's creative-writing model; **abliteration planned** → then it
  becomes the uncensored-creative model the persona substrate needs. Reasoning via
  `reasoning_effort` (BINARY none|high — no medium). (tracking: `c77a9aa`/`9a49963`)
  _Archived 2026-06-20._

- `[2026-06-16]` **Selene judge restored on vLLM FP8, NOT NVFP4** — W4A4 is too
  aggressive for a precision judge validated at Q6_K; FP8 ≥ Q6_K fidelity, and Selene
  is text-only Llama-3.1 so dynamic `--quantization fp8` has no vision tower to noise.
  `selene-1-mini-8b` on GPU 1 :8011. Worldtree's Domari role repointed to it. (`c985ede`)
  _Archived 2026-06-20._

- `[2026-06-16]` **GPU-1 buffer put to work: selene 0.13→0.17 + qwen36 0.32→0.34**
  — the two KV-bound generative models that gained Worldtree consumers; granite + the
  pooling models under-use their util, so growing THEM is wasted. (`e124a2f`)
  _Archived 2026-06-20._

- `[2026-06-16]` **claude-bot Gitea service account stood up; corviduo-ORG conversion
  TABLED** (operator). claude-bot is the credential-migration foundation. The proper
  fix for scoped agent package-write is converting vh (a USER) → a `corviduo` ORG —
  blast radius assessed (org creation = free; arbo-first migration = cheap, no registry
  images; full fleet = incremental, ~8 active vh images / 50 repos) — but tabled for
  later. Registry-push deferred until then. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
  _Archived 2026-06-20._

- `[2026-06-16]` **arbo engine builds → comfy-dev OWNERSHIP** (operator approved) +
  Gitea Actions runner stood up on irv-ml1 (host-executor as lkraven). infra-ops keeps
  box-level escalation (runner, package:write PAT pending corviduo-org, webhook). See
  the runner entry in Current state. (tracking: this snapshot + arbo thread `01KV70AB2B4B…`)
  _Archived 2026-06-20._

- `[2026-06-16]` **Worldtree persona substrate migrated qwen→Mistral via capability-
  named profiles** (v0.35.8→v0.35.10; demo + personal). PRE-SYNC-FIRST is now the
  standing pattern for any providers.yaml/profile change: sync config to the bind-mount
  BEFORE the deploy image-swap restart, so new agents + new config land together (the
  v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread `01KV6X0M…`)
  _Archived 2026-06-20._

- `[2026-06-19]` **litellm task-aliases: `classifier`→granite-4.1-8b, `summarizer-large`→gen/qwopus** (`summarizer`→granite already existed). Also added then REMOVED `gen-nt`/`gen-reasoning-nt` (redundant with the global `strip_empty_tools` hook). (ef45f6d, 2e5ab72, 75bd4c3)
  _Archived 2026-06-21._

- `[2026-06-19]` **gateway-chat: model-smoking web chat → PERSISTENT container.** `/v1/models` auto-discovery + image-upload; `stacks/gateway-chat/` (nginx:alpine, ana-docker:8091, homepage tile). (4c40b9f, 740bcae)
  _Archived 2026-06-21._

- `[2026-06-19]` **pi on nh3-dev wired to `gen`** — `~/.pi/agent/models.json` + a `gen` launcher mirroring `glm`. Local box config, not a repo artifact. (untracked by operator choice)
  _Archived 2026-06-21._

- `[2026-06-19]` **`gen` model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8.** ana-ml2 GPU 0, served-name `qwen3.5-122-a10b`. STABLE vLLM + fp8 KV reaches 262144 — nightly+turboquant-4bit-KV was UNNECESSARY. 92.7 tok/s warm. (20e796c, 5b06514)
  _Archived 2026-06-21._

- `[2026-06-19]` **TP=2 across the two ana-ml2 Blackwells REJECTED** — `PIX` (PCIe, NO NVLink) → all-reduce-bound → SLOWER. One-model-per-card is optimal; PP=2 only if a >96 GB model is ever wanted. (untracked by operator choice)
  _Archived 2026-06-21._

- `[2026-06-19]` **qwen-image-bench (T2I judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1**, aliased `image-judge`. (bfae924, 5dfce04)
  _Archived 2026-06-21._

- `[2026-06-18]` **heretic abliterated Mistral Small 4 NVFP4 built + LIVE as `mistral-small-4`** (in-house NVFP4, drop-in stack, A/B'd vs official, "heretic stays"). DISPLACED from GPU 0 on 2026-06-19 for the gen/Qwopus swap. (dd3a5c9, f566f61, `tools/mistral-small4-nvfp4/`)
  _Archived 2026-06-21._

- `[2026-06-18]` **irv-ml1 VRAM consolidation + comfy-dev torch-pin** — ComfyUI pinned to the A6000 (48 GB), audio zoo → 3090; `DISABLE_UPGRADES=true` @ torch 2.12.1. (a8550ad)
  _Archived 2026-06-21._

- `[2026-06-18]` **ComfyUI acceleration set (9 nodes) installed for comfy-dev** on irv-ml1.
  _Archived 2026-06-21._

- `[2026-06-17]` **ana-ml2 `vm.overcommit_memory=1` made durable** (sysctl drop-in) — overcommit=0 + zero swap caps CommitLimit; large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff)
  _Archived 2026-06-21._

- `[2026-06-17]` **GLM 5.2 wired into litellm** (`glm-5.2` + `-reasoning`, z.ai passthrough, `extra_body.thinking.type` toggle). (fe77a35)
  _Archived 2026-06-21._

- `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (successor to nh3-ansible); hosts `/opt/externs` pi-on-GLM-5.2 client workspaces. (a841eab)
  _Archived 2026-06-21._

- `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-` prefix (operator) — reuse the existing capability convention.** gen/gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge (passthrough)/gen-frontier/-reasoning. The **swap-transparency rule**: embedding stays model-named (`qwen3-embedding`) because vectors aren't swap-transparent; everything else can swap behind a stable alias. (d372103, 7819f96)
  _Archived 2026-06-25._

- `[2026-06-20]` **Removed the litellm `*` wildcard → llama-swap** (operator; llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now 404 cleanly instead of a misleading "Connection error" — closes the long-standing footgun. (d0eb09c)
  _Archived 2026-06-25._

- `[2026-06-20]` **Langfuse retired** (operator) — redundant with LiteLLM native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse spew). Callbacks dropped, stack torn down. (f8eda1c)
  _Archived 2026-06-25._

- `[2026-06-20]` **OmniVoice streaming `/tts` — vendor chatterbox-fast's scheduler, not its whole package** (operator-approved vendor-copy). Diffusion TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox-fast remains the lowest-latency front-end. (288d085, cd92b85)
  _Archived 2026-06-25._

- `[2026-06-20]` **Daily image+buildx prune cron on ana-docker** (the missing automation behind the recurring image bloat) — `lkraven` crontab, prune unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context.
  _Archived 2026-06-25._


- `[2026-06-25]` **althing re-architected to the lean multi-machine bus; nh3-extdev stood up as a
  MODEL B mesh peer** (dedicated `althing-svc` + group-shared `/srv/althing`). `reference_nh3_extdev_althing_mesh`.
  _Archived 2026-07-08._

- `[2026-06-23]` **zellij native web client piloted on nh3-dev** (`zellij-web.service` :8443)
  alongside ttyd. `reference_zellij_web_seat`.
  _Archived 2026-07-08._

- `[2026-06-22]` **Worldtree persona-render config arc (#314/#322/#317) pre-synced + deployed green
  on demo+personal** — #317 a boot-blocking config REMOVAL. `reference_corviduo_dev_emergency_ops`.
  _Archived 2026-07-08._

- `[2026-06-20]` **R22 (brokkr/dwarves) stood down to gateway-only; full-access R22 key minted;
  Phase B CANCELLED** (Worldtree model-agnostic → no deploy path). Key at `/home/lkraven/.r22-gateway-key`
  (persistent mode-600, carries paid GLM, don't delete). MUT = free `qwen3.5-122-a10b` (`gen`).
  Operator steer: R22 research is gated on a pragmatic/deployable outcome, not advancing-the-art.
  _Archived 2026-07-08._

- `[2026-06-20]` **claude-bot issue-scope token minted for worldtree-dev self-serve** (id 16,
  `write:repository`+`write:issue`); old token revoked. Advances the credential-migration directive.
  _Archived 2026-07-08._

- `[2026-06-20]` **rest-server-ana recovered + backup prevention shipped + worldtree-dev admin keys
  provisioned** (demo d113207c / personal f4f75adb). Cred rotation (5 rest-server pw) BELAYED.
  _Archived 2026-07-08._

- `[2026-06-20]` **claude-bot → ADMIN on vh/Worldtree** (operator-authorized) — self-serves WT
  deploys/tokens henceforth.
  _Archived 2026-07-08._

- `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.**
  (auto-memory `project_migrate_infra_access_to_claude_credentials`)
  _Archived 2026-07-13._

- `[2026-07-01]` **Worldtree #332 embed-recall diagnosed + scoped-log view/tunnel provisioned + fix
  verified** (15×→1.01× re-embed). `reference_wt_gateway_scoped_log_view`.
  _Archived 2026-07-13._

- `[2026-07-01]` **qwopus native MTP speculative-decode tested on `gen` → NOT kept** (+12% single-stream,
  −15–20% aggregate at concurrency, silently drops min_p/logit_bias). Banked for T1. `reference_gen_qwopus_122b`.
  _Archived 2026-07-13._

- `[2026-07-01]` **Deckard trial → reverted to qwopus (`gen`)** (won writing "in every way" but ~36 vs
  ~90 tok/s; spec-decode rescue ruled out). git `b63c48b`→`681eb70`. Deckard kept staged as T1's
  writing benchmark.
  _Archived 2026-07-13._

- `[2026-07-02]` **mtf-dev granite harness-spike ran GREEN — MECHANICAL only, efficacy DEFERRED to
  the T1 run.** Trainer TRL SFT→DPO→eval seam proven end-to-end on a synthetic fixture (not the E-RP
  corpus); operator DECIDED no intermediate real-efficacy granite spike (uninterpretable proxy —
  arch gap + abliteration axis). `reference_gen_qwopus_122b`.
  _Archived 2026-07-13._

- `[2026-07-03]` **ratatoskr-dev DEMO Heimdall key provisioned (R30 φ0).** Minted a tier-user key on
  the demo via `POST /admin/keys` (bootstrap admin key), mirroring their personal base consumer (no
  character-binding); base-agent affect reads work ungated. `reference_worldtree_demo_key_mint`.
  _Archived 2026-07-13._

- `[2026-07-04]` **character-rp: gateway-forwarding proven + role pre-staged + #344 shipped.**
  Empirically confirmed per-request `extra_body` (top_k/repetition_penalty) forwards through the
  `gen-reasoning` LiteLLM alias to vLLM + standard params override the alias defaults — no gateway
  cap needed (I over-built a dedicated alias, operator corrected, reverted with zero fleet impact).
  Pre-staged the `character-rp` role into demo+personal bind-mount `model_roles.yaml` (byte-verified
  on b18; caught the cached-registry ordering). worldtree-dev shipped **#344 (v1.0.0b19)** for the
  durable-agent override-drop. spend_logs spot-check queued (ratatoskr's timestamp ping).
  _Archived 2026-07-13._

- `[2026-07-04]` **althing v2 herald+receiver formalized as systemd on nh3-dev.** `althing-herald.service`
  (Restart=always, **Environment=PATH incl ~/.cargo/bin** — the pane-dispatch fix) + `althing-receiver.service`
  (v2 → pillar-3 `/owner/*` live); stale forseti unit removed; both on v2.0.0b2, canonicalized by
  forseti. `reference_nh3_dev_althing_herald`.
  _Archived 2026-07-13._

- `[2026-07-04]` **LitBench-RM respun (irv-ml1 A6000, comfyui displaced)** for T1's reward ensemble;
  operator sole comfyui consumer, holding image-gen until LitBench done. `reference_litbench_rm_irv_ml1`.
  _Archived 2026-07-13._

- `[2026-07-05]` **T1 training venue: CLOUD recommended; operator chose smoke-first on ana-ml2.**
  On-prem ruled out (ana-ml2 full — both 96G GPUs ~93G used): keep-serving = NVMe offload ~6-8 DAYS;
  full ana-ml2 shutdown = CPU offload ~1 DAY but a whole-fleet outage. Cloud Vast.ai 8×80GB (no
  offload → ~3-6h, ~$60-500, zero fleet impact) is the clean alt (mtf-dev + infra-ops both rec;
  Vast for its no-content-AUP marketplace + likely-existing VastBlue account). Operator's next step
  = the ana-ml2 CPU-offload SMOKE (~60 steps) to get real samples/sec before the full-outage-vs-cloud
  call. HF base verified public (zero-auth pull). Runbook + gotchas in `reference_t1_cloud_train_plan`.
  _Archived 2026-07-13._

- `[2026-07-05]` **glm-5.2 canonical limits recorded** (probed live vs z.ai): **1,048,576 (1M) input
  context / 131,072 (128K) max output**; NO gateway-side cap (pure passthrough → z.ai's limits are
  effective). Written to the config comment (commit `624a07e`) + `reference_litellm_gateway`.
  _Archived 2026-07-13._

- `[2026-07-06]` **infra-ops NOPASSWD sudo confirmed on irv-ml1 + ana-ml2** (the arbo memory's "NOT
  infra-ops on irv-ml1" was stale) — enabled the worker deploy + every ana-ml2 hot-swap smoke.
  _Archived 2026-07-13._

- `[2026-07-07]` **SGLang zc hot-swap: 2-line patch RAN → PARTIAL. Cleared startup + self-attn E2E, GDN
  in_proj hit 2 MORE name-routing gaps → MERGE recommended.** Root cause was a NAME-routing gap (not the
  feared shape-assert): `lora/utils.py:235` hardcoded `params_mapping` lacked `in_proj_qkv/in_proj_z →
  in_proj_qkvz`. Patched those 2 lines (overlay bind-mount) → `init_lora_shapes` passed, server up, self-attn
  loaded end-to-end. But the SAME missing GDN mapping bit 2 MORE resolvers: adapter-load skipped 192 in_proj
  weights, and inference `utils.py:300 get_target_module_name` (substring match) RAISED → 500 + scheduler
  died. So GDN LoRA name-routing is incomplete in ≥3 SGLang paths (fixed 1). NOT shape-intractable (self-attn
  proves the separate-key→fused-buffer machinery works), but NOT "one targeted fix" either → per operator's
  LAST-shot rule → MERGE. mtf-dev looped + closed (thread 01KWZDZFYN…); B/A shard-math never reached.
  Supersedes the "SGLang wants a THIRD format" framing — no new adapter format needed.
  _Archived 2026-07-13._

- `[2026-07-07]` **T1 (retargeted qwopus-122B→AEON-27B) SFT COMPLETE on ana-ml2** — all-TRL torch-GDN,
  batch1/accum8/GC-off, 1 epoch ~9h @ 0.25 s/s, loss 2.445, adapter banked. Supersedes the 2026-07-05
  cloud-vs-smoke venue decision: AEON-27B's 52G bf16 fits one Blackwell → ran on-prem in ~9h (no cloud,
  no full-fleet outage). GC is NOT the throughput lever (batch1-no-GC == batch4+GC at 0.25; dvalin's 0.565
  was warmup-noise from a 30-sample smoke); ~0.25 s/s is the torch-GDN floor (only FLA beats it, sm_120-
  blocked #913). `reference_t1_cloud_train_plan`.
  _Archived 2026-07-13._

- `[2026-07-07]` **T1 hot-swap on vLLM NVFP4 is DEAD — vLLM 0.24.0 qwen3_5 LoRA-application silent no-op**
  (#47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4+FP8) AND
  adapter-format-agnostic (separate `zc` loads-but-inert; fused rejected); #47640 guard-patch overlay didn't
  fix. Root-caused by a 3-peer dwarf panel (eitri/regin/dvalin); their vLLM issue refs verified real via
  GitHub API. Engine mirror-image: vLLM wants SEPARATE keys, SGLang wants FUSED.
  _Archived 2026-07-13._

- `[2026-07-07]` **HARD REQUIREMENT restated: hot-swappable LoRA or ABANDON.** Merge is NOT a solution for
  this requirement (static; reserved for a later RP-quant track). SGLang is the last hot-swap shot; if it
  can't apply, park the adapter + file upstream repros. `reference_lora_training_worker` (adjacent).
  _Archived 2026-07-13._

- `[2026-07-07]` **comfy-dev in-arbo LoRA-training worker DONE** — built/deployed/validated Phases 1+2
  (first Sindra train succeeded, publish-step + `train_id`), Phase 2.5 GC contract (shape A) confirmed +
  impl deferred. `services/lora-training-worker/`, commits 888ba6a→b617a8b. `reference_lora_training_worker`.
  _Archived 2026-07-13._

- `[2026-08-08]` **worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev.** worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared `~/snapshots/worldtree-400-fiction-decomp` (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). **LEFT INTACT:** `rex393-fiction-index`/`rex393-fiction-snapshot` (separate operator KEEP word, unchanged) + `r42-gate-*`. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (`01KZJ9GMCC…`).
  _Archived 2026-08-24._

- `[2026-08-07]` **chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed).** Long saga, operator-driven clean diagnosis. **Symptom:** ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. **NOT** German-leak (Turbo `generate()` has NO language param — plain AutoTokenizer, no `language_id`; the multilingual `language_id="en"` lever lives only in the separate `ChatterboxMultilingualTTS`), **NOT** OOM alone. **Real cause:** the Chatterbox **Turbo T3 model OVER-RUNS its generation tail** — a long single `generate()` degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens `<6561` + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). **ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE** — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). **Diagnosis method** (deterministic, no ears-only): single-shot length sweep + **amplitude-gated voiced-ZCR** (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. **FIX:** server-side `max_chunk_chars=250` cap on the scheduler (`:v2` image, `CBF_MAX_CHUNK_CHARS=250` env) — bounds each generation to just under the ~300-char onset → clean **3-4 sentence** chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; **chatterbox's low emotiveness keeps chunk joins smooth** (the harsh joins that got Zonos rejected are absent — operator's key call). **ratatoskr TODO (relayed msg `01KZER9X7S`):** revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. **Cap value tunable** per-request (`max_chunk_chars`) + env. **Deeper prosody** (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). **⚠ FOOT-GUNS:** (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) **build-context vs image drift** — the `:v2` image was built from cap source, but after a `:v1` rollback the build context held `:v1` source → a `docker compose build` would've silently produced a cap-less `:v2`; re-synced the flat cap source to `/opt/docker/compose/chatterbox-fast/` (rebuild-verified). **⚠ DIVERGENCE (follow-up):** deployed build context is FLAT (`app.py`/`scheduler.py`, `from scheduler import`, thin-overlay `FROM local/chatterbox:v1`, cap-only) vs the `vh/chatterbox-fast` REPO which is PACKAGE-layout (`chatterbox_fast/`, `from chatterbox_fast.scheduler`, self-contained Dockerfile) + has `norm_loudness` (repo commit `6bc7bf0` = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: `.bak-cap-20260807-104850` backups on irv-ml1 + `:v1` image both retained. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]]
  _Archived 2026-08-24._

- `[2026-08-07]` **Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY.** Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. **⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init.** GPU memory was held by the `--multiprocessing-fork` CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). **RESTORE CMD** (from irv-ml1, user lkraven): `cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 &` then `docker start zonos-gateway`. **Consumers that lost Zonos:** asset-engine + gateway-chat (via LiteLLM `ext-tts` alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). [[reference_zonos_tts_stack]]
  _Archived 2026-08-24._

- `[2026-08-07]` **chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos).** Operator-directed. Copied `zonos-gateway/voices/Donut.wav` → chatterbox `/refs` (`/worktank/chatterbox/reference_audio/donut.wav` — the reference_audio SUBDIR is lkraven-owned so no sudo despite `/worktank` root; container globs `/refs` live → **NO restart**), exposed as `voice:"donut"` (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at `http://10.100.10.50:8090/b/donut-chatterbox/`. Answered ratatoskr's 8-question contract ask from the live gateway (`local/chatterbox-fast:v1`) + source: **NOT OpenAI-shaped** (`POST /tts`; body `text`/`voice`/`format`/`stream`, not `input`/`model`/`response_format`); **NO affect dials** (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; **Zonos stays the only fleet TTS with real emotion steering**); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. **FYI-worthy (operator):** ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory `reference_chatterbox_fast_repo` enriched w/ the live contract. [[reference_zonos_tts_stack]]
  _Archived 2026-08-24._

- `[2026-08-07]` **Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43).** The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM `reranker` repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); **R42 v13 gate PASSED first-ever** (56/90→90/90). Incumbent kept warm :8002 (rollback via `qwen3-reranker` alias), A4 fallback :8014. Full arc + rollback runbook `docs/pfi/reranker-selection-ledger.md`; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes.
  _Archived 2026-08-24._

## Tried and abandoned (archived)
# [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix

**Outcome.** The fleet `gen` seat is now **`JonathanColetti/Qwen3.8-27B-Uncensored`** (Heretic
abliteration, KL 0.12 vs base, bench Δ −0.5 within noise, refusals 98→12/100), quantized in-house
to **NVFP4 W4A16** (llm-compressor / compressed-tensors) with a **grafted bf16 MTP head**,
vision-intact, **262K** ctx, MTP n=3 (**~42% accept, ~68 tok/s**), coherent. Live at ana-ml2 `:8015`
(project `gen-seat` / container `vllm-gen`), backing all 7 gateway aliases.

**THE definitive lesson (resolved 3 failed attempts + one premature 50 GB delete).** A grafted bf16
MTP scored **0% on the quant but 83% at bf16** — for TWO different abliterated models. Root cause was
NEITHER the abliteration NOR the quant scheme: it was **the grafted `mtp.*` tensors missing from
`config.json` → `quantization_config.ignore`.** The wrapper-class quant DROPS the MTP before
llm-compressor sees it, so nothing gets added to `ignore`; vLLM then tries to load the bf16 MTP as
*quantized* format → "Parameter … not found in params_dict, skip loading" → uninitialized head → 0%.
**FIX: after grafting, add `re:^mtp.*` to `quantization_config.ignore`** (one line — all unsloth's
working checkpoint has). MTP jumped 0%→83% (bf16-identical). Full lesson in auto-memory
`reference_abliteration_mtp_lessons`.

**The pipeline that works (for the next VL+MTP quant, incl. the W4A8 chase):**
1. Pull bf16 (kept at `ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16`).
2. Quant via `quant_nvfp4_qwen.py` (darkscarlett dir) = the **wrapper-class** loader
   (`Qwen3_5ForConditionalGeneration`, keeps the vLLM-serveable config); container = `vllm-openai`
   + `pip install llmcompressor==0.13.0` (drags in a transformers with `qwen3_5`).
3. **Graft** the author's `model-mtp.safetensors` verbatim into the output + merge the index.
4. **Reconstruct** `preprocessor_config.json` from `processor_config.json`'s `image_processor`
   sub-dict (the repo omits it → else "Can't load image processor" crash-loop).
5. **Add `re:^mtp.*` to the output config's `quantization_config.ignore`.** ← the fix.
6. Serve: `--quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'`
   `--mamba-cache-dtype float32 --kv-cache-dtype fp8 --reasoning-parser qwen3`.

**VRAM / full-context budget (measured).** Weights ~27 GB; hybrid attention → **only 16 of 64 layers
carry KV** → 32 KiB/token → **262K KV = 8.6 GB** (vs ~60–70 GB for a normal dense 27B). Full 262K fits
GPU0 at **util 0.45** (~43 GB) alongside meromero (~49 GB used, it's a 31B) — pre-flight rejects util
0.48 (wants 45.6 GB, only 45.5 free). `max-num-seqs 16` keeps cudagraph modest (an ad-hoc serve with
no cap OOM'd — cudagraph captured to batch-512).

**Why unsloth's `qwen3.8-27b` (the prior gen model) was faster (97 vs 68 tok/s).** ~half = quant kernel
(unsloth native NVFP4+FP8 tensor cores vs our W4A16 → Marlin dequant, ~20% even on decode — I'd
under-stated this); ~half = MTP acceptance (unsloth 55% un-ablated head vs our 42% — inherent to the
ablation, no quant fixes it). **W4A8 recovers the first ~20% (→~82 tok/s) + prefill; not the MTP half.**

**modelopt dead-end (for W4A8, avoid).** `nvidia-modelopt[hf]==0.43.0` is too old for qwen3_5's
transformers: (a) its `NVFP4_DEFAULT_CFG.quant_cfg` is a LIST but 0.43 wants a DICT (pydantic reject);
(b) it warns transformers 5.15 untested. Use **llm-compressor** for W4A8 instead (custom recipe: NVFP4
weights + FP8 input_quantizer + calibration on `heretic2-nvfp4-work/production_calib_512.jsonl`).

**Deleted (premature — the delete I owned).** `windowsxp811203/Qwen3.8-27B-Abliterated` (~79 GB) — I
declared it desync-dead off a 0% that was actually this ignore bug. Lesson: **test MTP on bf16 first;
isolate before deleting.**

Commits: eshpfi `680c30e` (deploy + rename + litellm + README), dotfiles `1d1970f` (CLAUDE.md roster) —
both UNPUSHED. Related: [[reference_abliteration_mtp_lessons]], [[reference_verify_hf_repo_ids_before_pull]].
  _Archived 2026-09-02._


- `[2026-04-30]` task-board workflow with
  `container: image: debian:bookworm-slim` — fails:
  `actions/checkout@v4` needs `node` at runtime, slim image lacks it.
  Switched to `node:20-bookworm-slim` (has node + apt) or runner-label
  default. (Pattern revisited 2026-05-17 for skaldsong-dev: container
  override needs `nodejs` apt-installed unless it IS the default.)
  _Archived 2026-05-27._

- `[2026-04-30]` Dropping the `container:` directive before runner
  re-registration with docker-schema labels — runner silently falls
  back to **host mode** (jobs run inside the alpine `act_runner`
  container itself, no apt). The `:host` suffix in startup logs
  (`labels updated to: [pfi-fleet:host ana-docker:host]`) is the
  giveaway. Fix: register with `pfi-fleet:docker://<image>` schema
  labels.
  _Archived 2026-05-27._

- `[2026-04-30]` Updating runner labels by editing `.env` and bouncing
  — doesn't take. The `.runner` registration cache pins labels at
  first registration; env-var updates are read each start but the
  stored token + UUID are tied to the original label set on the gitea
  side. Fix: stop runner, delete `.runner`, generate new admin
  registration token, redeploy.
  _Archived 2026-05-27._

- `[2026-04-30]` `git reset --hard origin/<sha>` in
  `deploy-task-board.yaml` (and the in-repo nevermore playbook before
  fix) — invalid syntax: `origin/` prefix only works for branch refs.
  SHAs need `git reset --hard <sha>` directly. Resolved with
  `git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first,
  then bare `"{{ ref }}^{commit}"` fallback.
  _Archived 2026-05-27._

- `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after
  task-board wiring — it was actually only repo-scope on
  `vh/task-board`. vor's first CI run failed with empty SSH key
  (`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user
  scope at `gitea.phasefinal.com/user/settings/actions/secrets`.
  _Archived 2026-05-27._

- `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity —
  leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a
  follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1<redacted>/"`
  still leaked the first 4 chars. Lesson: when probing secret-bearing
  files, use field-by-field SELECTIVE inspection
  (`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any
  password line, `grep -c` (existence) or `test -n "$(...)"`
  (non-empty), never `cat` or value-printing.
  _Archived 2026-05-27._

- `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT
  timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero
  matches and led to a wrong "no /v2/ traffic in 12 days" conclusion.
  **Gitea logs in PDT, Traefik logs in UTC** — same host, different
  timezones. Always normalize timezones (UTC) when correlating logs
  across services on the same box. Cost: ~30 min in the wrong
  direction.
  _Archived 2026-05-27._

- `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` /
  `PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on
  `/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern
  **response writes**, not request body reads. `unexpected EOF` from
  Go's HTTP server means the client closed mid-body-upload; not a
  knob Gitea exposes server-side.
  _Archived 2026-05-27._

- `[2026-05-12]` Defaulting asset-engine to Traefik-routed
  (`asset-engine.phasefinal.com` with `anaprod` cert resolver) on
  first scaffold — user pulled it back to LAN-direct. Internal
  tools default LAN-direct; only add Traefik when an external/TLS
  surface is actually needed.
  _Archived 2026-05-31._

- `[2026-05-12]` Routing althing thread replies through `galdrabok`
  when the actual dev handle is `forseti` — bus rejected
  `to=forseti` initially because thread participants list was
  `[galdrabok, infra]`. Solved by starting a new thread with
  `forseti` as the direct recipient. Lesson: when the bus
  auto-resolves a sender handle that doesn't match the actual dev
  role, start a fresh thread rather than fighting the participant list.
  _Archived 2026-05-31._

- `[2026-05-13]` Initial Voxtral default voice `alloy` (OpenAI-compat
  naming) — vLLM-Omni serving Voxtral does NOT translate aliases.
  Native presets are `<register>_<gender>` shape (`neutral_female`,
  `casual_male`, etc.). Always live-probe `/v1/audio/voices` for the
  exact wrapper-deployed preset names before setting a catalog
  default. Same caveat for Qwen3-TTS (wrapper exposes 15 voices: 9
  Qwen presets + 6 OpenAI aliases) and Kyutai-TTS (NillPointer
  wrapper has NO voice-listing endpoint at all; voices are
  filesystem paths under the `kyutai/tts-voices` HF repo).
  _Archived 2026-05-31._

- `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 —
  flag was deprecated. Use `--runner pooling`; the model's
  `architectures: [LlamaForSequenceClassification]` in config.json
  drives the classification head. Surfaced as `vllm: error:
  unrecognized arguments: --task classify` in container logs.
  _Archived 2026-05-31._

- `[2026-05-17]` Trusting that `.env` edit alone propagates a new env
  var into a worldtree container — `compose.yaml`'s `&worldtree-env`
  anchor must explicitly declare the passthrough or the value silently
  doesn't land. Same footgun bit `Z_AI_API_KEY` (2026-05-12) AND
  `BIFROST_CLIENT_ALLOWED_HOSTS` (2026-05-17). Cost ~10 min of "why is
  env empty?" diagnosis each time. Worldtree-side fix in
  `vh/worldtree@08f02b2`.
  _Archived 2026-05-31._

- `[2026-05-17]` `--force-recreate --pull never` from the docker:cli
  sandbox without explicit `-e WORLDTREE_IMAGE=<sha>` re-pins the
  container to `:latest`, even when a newer SHA-tagged image is on
  disk. Symptom: container "recreated" but actually reverted to a
  stale image. Pass `-e WORLDTREE_IMAGE=...:<sha>` to the docker run
  invocation. Worldtree-dev's `8ef3801` health-gated `:latest`
  advance is the long-term fix.
  _Archived 2026-05-31._

- `[2026-05-18]` Volva env.sh.template `$HOME` in commented examples
  — systemd's `EnvironmentFile=` parser doesn't expand `$HOME`;
  uncommenting lands the literal `$HOME/...` string. Volva-dev's
  `f4dda73` swapped to `/home/<svc-user>/...` placeholders.
  _Archived 2026-05-31._

- `[2026-05-18]` Initial Volva systemd unit's `ProtectHome=read-only`
  without `ReadWritePaths=` — althing-cli's SQLite (`~/.althing/
  althing.db`) and codex's session state (`~/.codex/`) both need to
  write. Container started but every poll failed with "db path not
  writable". Surgical fix: `ReadWritePaths=/home/lkraven/.althing
  /home/lkraven/.codex` (preserves the hardening intent, only carves
  out the specific dirs).
  _Archived 2026-05-31._

- `[2026-05-18]` Trusting that env.sh's `export VOLVA_ROOT="$(cd
  "$(dirname "${BASH_SOURCE[0]}")" && pwd)"` template line works
  under systemd — `EnvironmentFile=` parser aborts on the first
  unparseable line (command substitution), and `VOLVA_ALTHING_CLI`
  declared below silently never lands. Symptom: `Environment=`
  property empty, daemon error "althing-cli not found at
  'althing-cli'". Fix: replace command-substitution with literal
  path. Volva-dev's `d436c3c` dropped VOLVA_ROOT entirely upstream.
  _Archived 2026-05-31._

- `[2026-05-19]` Naive `docker rmi worldtree:<old-sha> --force`
  for CD SHA cleanup — would untag pinned/personal worldtree images
  since all three deployments share corviduo-dev. Use
  `docker image prune -a --filter "until=Xh"` instead — respects
  in-use semantic (Docker won't remove an image referenced by any
  container on the host), so pinned/personal protected automatically.
  _Archived 2026-05-31._

- `[2026-05-19]` Skaldsong CD first attempt: `docker pull` step
  failed with 401 unauthorized. ana-docker had no `docker login`
  for `gitea.phasefinal.com`. My playbook prereq note ("docker login
  has been done at least once") was an unverified assumption. One-time
  manual login persists in `~/.docker/config.json`; architectural fix
  (workflow-side `ssh ana-docker 'docker login ...'` step using
  `REGISTRY_USER`/`REGISTRY_TOKEN` secrets) flagged as v2.
  _Archived 2026-05-31._

- `[2026-05-19]` `SKALDSONG_HOST_CORS_ORIGINS=http://10.250.50.70:8300`
  as a bare URL — pydantic-settings parses complex env vars via
  `json.loads()`; first-boot crashloop with `SettingsError: error
  parsing value for field "cors_origins"`. Must be JSON array literal:
  `SKALDSONG_HOST_CORS_ORIGINS=["http://..."]`.
  _Archived 2026-05-31._

- `[2026-05-19]` `SKALDSONG_HOST_STATIC_ASSETS_PATH=/app/web/dist`
  in compose — mismatched Dockerfile reality. The Dockerfile COPYs
  SvelteKit build output flat into `/app/spa` (not `/app/spa/dist`).
  Lifted the path from skaldsong-dev's CD-ask message ("`/app/web/dist`")
  rather than verifying against the actual Dockerfile they shipped.
  Lesson: when encoding container-internal paths in compose, verify
  against the Dockerfile, not the design-doc.
  _Archived 2026-05-31._

- `[2026-05-19]` Playbook verify step `docker ps | grep healthy`
  racing the container's `start_period` (30s in compose's
  healthcheck). Verify ran 0.09s after `compose up -d --force-recreate`
  — well before docker's healthcheck could flip the status from
  `(health: starting)` to `(healthy)`. False-negative; container was
  operationally up (the earlier `/health` poll verify already
  confirmed). Fix: grep `^Up` not `healthy`. /health-200 IS the
  liveness check; docker's `(healthy)` is just a delayed echo.
  _Archived 2026-05-31._

- `[2026-05-20]` `SKALDSONG_DB_PATH` + `SKALDSONG_RUNS_DIR` in
  compose env block — names skaldsong's app doesn't read. App reads
  `SKALDSONG_HOST_SQLITE_PATH` + `SKALDSONG_HOST_RUNS_ROOT` (per
  Dockerfile ENV defaults). Wrong names = silently no-op; app fell
  back to Dockerfile defaults pointing at `/app/data/...` which the
  compose's bind mount did NOT cover (target was `/app/state/...`).
  Result: every `--force-recreate` wiped the SQLite DB. Caught by
  skaldsong-dev (althing thread `01KS4DPF6SXTBP4Q360JZVWPNT`). Fix in
  `52e98fa`. Lesson: verify env var NAMES against the Dockerfile/app,
  not against design-doc shorthand.
  _Archived 2026-05-31._

- `[2026-05-25]` First selene-block patch put the block under
  `saga_allowed_models:` instead of top-level `models:` — used
  `text.replace("models:\n", ...)` which substring-matched the
  `saga_allowed_models:\n` line first. Caused YAML parse error. Fix:
  anchored regex `re.compile(r"^models:\n", re.MULTILINE)`. Pattern:
  substring replace on YAML top-level keys WILL match suffix-containing
  keys.
  _Archived 2026-05-31._

- `[2026-05-27]` `docker compose up -d` inside the `docker:cli` sandbox:
  `${VAR:-./config}` defaults resolve `./config` to the sandbox CWD, but the
  Docker daemon interprets the path against the HOST filesystem → auto-creates
  an empty dir → entrypoint reseeded image-baked defaults (lost host-side
  providers.yaml patches). Fix: pass `-e WORLDTREE_CONFIG_DIR=/abs/path`.
  Folded into the docker-as-root convention note.
  _Archived 2026-06-01._

- `[2026-05-27]` `:latest`-pinned compose + private gitea registry + sandboxed
  pull = recreate on ancient cached `:latest` (deploy pulls by SHA so the tag
  never advances; sandbox can't pull). Fix: retag SHA→`:latest` on host, then
  `--pull never`. Better: pin SHA in `.env`, advance in CI.
  _Archived 2026-06-01._

- `[2026-05-27]` Container recreate during in-flight skaldsong gen kills the
  runner. With deploys every ~10min and stories >5min, structural not
  incidental. Roadmap (skaldsong-dev): pre-shutdown signal handler, per-scene
  resume-from-checkpoint, /api/admin/quiesce. None shipped.
  _Archived 2026-06-01._

- `[2026-05-27]` `--reasoning-format gemma` on artemis-31b-v1i — unsupported in
  the deployed llama.cpp (accepts none|deepseek|deepseek-legacy). `deepseek`
  populates the `reasoning_content` SSE delta Worldtree GemmaProvider checks.
  _Archived 2026-06-01._

- `[2026-05-27]` `head -c N` piped after a streaming curl SIGPIPEs the curl,
  killing the request early. Use file-write + separate read.
  _Archived 2026-06-01._

- `[2026-05-31]` Building the dia2-capable image surfaced THREE upstream
  packaging quirks: (1) `pip install -e nari-labs/dia2` fails — no PEP 660
  `build_editable` hook; (2) plain `pip install` builds an empty
  `UNKNOWN-0.0.0` wheel (base setuptools 59.6 < dia2's required ≥70); (3)
  `--no-deps` leaves `transformers`/`sphn`/`whisper-timestamped` missing. Fix
  (`local/dia:v2`): copy the pure-python `dia2/` package into site-packages +
  install ONLY those 3 deps; base torch/numpy already satisfy Dia2.
  _Archived 2026-06-03._

- `[2026-05-31]` Dia2 predefined voices (43, baked at `/app/voices`) are NOT
  reachable from the `/tts` clone path — it resolves `clone_reference_filename`
  against the reference_audio dir ONLY. The OpenAI `/v1/audio/speech` `voice`
  param auto-resolves them (separate code path), which masked the gap. Fix:
  stage `/app/voices/*` into `/worktank/dia/reference_audio`. Lesson: verify on
  the endpoint the catalog ACTUALLY targets.
  _Archived 2026-06-03._

- `[2026-05-31]` `voice_mode=clone` with an empty `clone_reference_filename`
  → asset-engine serializes it as the literal string `"undefined"` → `/tts`
  404. First observed on dia2; worked around in the catalog (default the field
  to a real voice). **[2026-06-01] root cause found** — the Kokoro voice-blend
  widget reading Shoelace `.value` before hydration (see Current state); the
  real fix is asset-engine-side and is escalated.
  _Archived 2026-06-03._

- `[2026-05-31]` asset-engine `services.schema.json` is DERIVED (regen from the
  Pydantic model via `dump_schema.py`) and had DRIFTED — rejected the
  `lifecycle` field 12/14 services use. RESOLVED: regenerated with
  `CatalogLifecycle` + `reproducibility.seed_field` (`b7b2130`). Lesson:
  hand-editing `services.yaml` shape without regenerating re-introduces drift.
  _Archived 2026-06-03._

- `[2026-05-31]` ttyd-over-TLS forces HTTP/2 (kills ttyd's terminal WebSocket
  → blank screen); Safari/WebKit never sends HTTP basic-auth on WS upgrades.
  Both solved for the fleet seat (Caddy forces HTTP/1.1; auth → network-gating)
  — detail in auto-memory `reference_ttyd_fleet_seat`.
  _Archived 2026-06-03._

- `[2026-05-30]` esh-docker-vm: `hard` NFS mounts from 10.0.50.50 froze a
  container worker in UNKILLABLE D-state when the NAS stalled — only a host
  reboot clears it. Separately, `fstab defaults` (no `_netdev`) made NFS-bind
  containers `Exited(255)` on reboot. → auto-memory
  `incident_esh_docker_nfs_boot_race`.
  _Archived 2026-06-03._

- `[2026-06-02]` **Naive cumulative re-decode for streaming** — decode the
  growing token prefix each chunk + emit the delta, assuming the causal flow
  gives a stable prefix. It does NOT (full-context attention, see above);
  maxdiff 0.30. Also the high-level `s3gen.inference` re-applies `trim_fade` at
  the start every call. Don't go this way.
  _Archived 2026-06-04._

- `[2026-06-01]` **CSM bring-up** — upstream `phildougherty/sesame_csm_openai`
  Dockerfile pins NO `huggingface_hub` version, so it now resolves to 1.17.0
  where `huggingface-cli` is fully removed (replaced by `hf`) → the build dies
  at `huggingface-cli login`/`download`. Passing the HF token as a build-arg
  got the token in (the model downloads at BUILD time, not runtime — our
  compose only wired it as runtime env, the original failure) but then hit the
  rotted CLI. Fixing would need vendoring + patching the Dockerfile; not worth
  it for a weak model → CSM removed.
  _Archived 2026-06-04._

- `[2026-06-01]` **Fish path-form references** (`{audio:"/app/references/x.wav"}`)
  → HTTP 500 on this build, despite the stack README showing that shape. The
  working clone paths are `reference_id` (by name → resolves the DIRECTORY
  `references/<name>/` containing `<name>.wav` + `<name>.lab`; the flat
  `references/<name>.wav` is ignored — see the two-root-causes entry above) or
  inline base64 `references`. The catalog uses `reference_id`.
  _Archived 2026-06-04._

- `[2026-06-01]` **Reference transcript barely affects Fish clone IDENTITY**
  (disproving my mid-session theory). I'd blamed a bogus provenance-note `.txt`
  for poor cloning, but the ECAPA re-test showed correct (0.787) / bogus (0.778)
  / empty (0.738) transcripts all clone Imogen about equally — the transcript
  affects PRONUNCIATION (phoneme disambiguation per the docs), not who it sounds
  like. The real culprit for "not British" was the `"undefined"` select bug, not
  the transcript. (A correct transcript still marginally helps pronunciation —
  cheap to stage, not load-bearing.)
  _Archived 2026-06-04._

- `[2026-06-01]` **Pitch-shift register control** (rubberband, to deepen Imogen
  to contralto/mezzo) — Fish ignores small reference shifts and overshoots
  large ones; `-3st` WITHOUT formant preservation → "baritone through a tuba"
  (formants collapsed); `-2/-3st` WITH `--formant` → too subtle / still bad.
  Abandoned at every depth; all variants deleted. Finer independent
  pitch/formant control needs praat (not installed). Future clones = operator's.
  _Archived 2026-06-04._

- `[2026-06-01]` **resemblyzer is too weak for cloning-fidelity scoring** — its
  dated 2019 LSTM encoder + a synthetic-vs-natural domain gap scored the Imogen
  clone CLOSER to the default than to real-Imogen, which led me to a WRONG "Fish
  is a weak cloner" call. ECAPA-TDNN (speechbrain `spkrec-ecapa-voxceleb`) on the
  same clips gave the correct answer (clone 0.79 to real Imogen, 0.10 to
  default). Use ECAPA, not resemblyzer, for speaker-verification.
  _Archived 2026-06-04._

- `[2026-06-02]` **Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED
  (sub-realtime).** Benched RTF on A6000: 0.72x/0.82x/0.86x, mean 0.80x = sub-realtime,
  so client-side chunking starves. Root cause: Fish only chunks on `<|speaker:X|>` tags;
  plain text → batches=[whole text] → all semantic tokens generate before any audio
  (chunk_length inert). Plus kui/ASGI StreamResponse doesn't flush (fix = anti-buffering
  headers, kept on file). STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands → bench
  fp4-quantized Fish; if RTF > ~1.5x give it the chatterbox-fast treatment. For now Fish is
  a buffered catalog entry (great for SAVED gens, not live-audition).
  _Archived 2026-06-11._

- `[2026-06-02]` **Context-priming at chunk joins (chatterbox-fast §1.6) — ABANDONED
  (discard-cut leaks the prefix).** Prepend prior sentence, generate `prefix+content`,
  discard prefix audio. Built + shipped (`d707439`), live-A/B'd, reverted (`090e70a`).
  Kill: `generate()` returns one waveform with NO marker where the prefix ends, and the
  model renders the same prefix with different timing solo vs followed-by-content — so
  locating the cut is a guess that left a whole clause in the output (operator caught
  "...without a trace of sarcasm" spoken twice). Reliable cut needs token-level boundaries
  or per-chunk forced-alignment (heavy). Coherence loss at joins = ACCEPTED limitation.
  _Archived 2026-06-11._

- `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED (turbo
  isn't built for streaming).** (1) `S3GenStreamer` is docstring-only (unimplemented).
  (2) `flow_inference(finalize=False)` is buggy (lookahead trim removes frames from `h`
  not `h_masks`/conds → shape mismatch); a 1-line patch makes it RUN. (3) BUT the flow
  encoder uses full-context attention (`static_chunk_size=0`) → incremental decode is
  prefix-unstable (adding tokens re-attends, shifts earlier mel ~0.30-0.39 vs one-shot).
  (4) Forcing `static_chunk_size>0` didn't stabilize it. True sub-second frame-streaming
  needs deep attention surgery with quality risk — not worth it. → adaptive-chunking instead.
  _Archived 2026-06-11._

- `[2026-06-03]` **`uv pip install .` fails on SmoothKen/knn-svc** (and similar script-repos)
  — it's analysis scripts + a poetry pyproject, no buildable package (setuptools
  package-discovery error). Install the pyproject deps directly, don't build the "package".
  _Archived 2026-06-13._

- `[2026-06-03]` **Backgrounding `althing-cli monitor` with an inline shell `&` (instead of
  the Bash-tool `run_in_background`) orphans it** — it survives the shell exit, holds the
  per-handle flock UNTRACKED (won't notify the session), and `stop-monitor` doesn't detect it.
  Fix: find + kill the orphan PID (verify cwd=this repo / handle first — nh3-dev is shared, other
  agents' monitors run there too), then re-arm via run_in_background. Always re-arm tracked.
  _Archived 2026-06-13._

- `[2026-06-03]` **torch-2.12 venvs need `uv pip install torchcodec`** — torchaudio 2.12
  defaults to the TorchCodec backend for `.load`; without it, real audio I/O throws "TorchCodec
  is required" — and it ONLY surfaces at actual conversion, NOT at import/model-load. Lesson:
  validate real I/O, not just import, when provisioning ML engine envs. (seed-vc on torch 2.4
  uses the legacy backend, exempt.)
  _Archived 2026-06-13._

- `[2026-06-03]` **gitea webhook to a private IP is denied by `webhook.ALLOWED_HOST_LIST`**
  (anti-SSRF; default `external` blocks private/loopback). Symptom: delivery shows
  `dial tcp ...: webhook can only call allowed HTTP servers`. Fix = APPEND the target net to
  ALLOWED_HOST_LIST in gitea's app.ini (keep `external`; scope tight, never `*`/`private`) +
  restart gitea (act_runner job containers survive a restart). gitea runs as a container on
  ana-docker (`gitea_gitea_data` volume, `/data/gitea/conf/app.ini`).
  _Archived 2026-06-13._

- `[2026-06-04]` **homepage "crash" ≠ always NFS** — a wedged container in unkillable D-state can come from dead `siteMonitor` widget targets (retired ESH firewall IPs) hanging the node event loop into `exit_mmap`, needing a host reboot. Check homepage's siteMonitors against retired hosts. (`incident_esh_docker_nfs_boot_race`)
  _Archived 2026-06-14._

- `[2026-06-04]` **pyworld needs `setuptools<81`** (imports the removed `pkg_resources`); and **R/soundgen `-lgfortran` fails** on irv-ml1 (default gcc is gcc-11 but only gfortran-12 present) → install `libgfortran-11-dev`.
  _Archived 2026-06-14._

- `[2026-06-04]` **Persistent services on irv-ml1 need SYSTEM systemd units** — the box reaps user-session processes on ssh disconnect, and `--user` systemd isn't reachable over non-login ssh, so nohup/setsid/`screen -dmS`/`systemd-run --user` all die (even with enable-linger). Use `/etc/systemd/system/`.
  _Archived 2026-06-14._

- `[2026-06-04]` **GPU pin by INDEX is ambiguous on irv-ml1** — native CUDA orders fastest-first (A6000=0) but nvidia-smi/docker use PCI order (A6000=1). Pin by **UUID** (`CUDA_VISIBLE_DEVICES=GPU-…`); verify via nvidia-smi compute-apps.
  _Archived 2026-06-14._

- `[2026-06-04]` **Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF — the "headgun" lesson.** Serve each model's canonical `tokenizer.apply_chat_template`, not the bundled template (bundled ones corrupt baselines). Verify via vLLM `/tokenize`→`/detokenize`. (`90e08f0`/`27eb537`)
  _Archived 2026-06-14._

- `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()` requires torchcodec → dies AFTER separating** (0 stems, rc=1). Fix = pin `torch==torchaudio==2.4.1` + `UV_LINK_MODE=copy`. Validate the SAVE path, not just import + GPU inference, on a bleeding-edge torch.
  _Archived 2026-06-16._

- `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — fills KV to the `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs AFTER weights+KV load (crash-loops). Fix: free co-tenant room OR `--enforce-eager`. FP8 single-stream is batch-1 GEMV (memory-bound) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`)
  _Archived 2026-06-16._

- `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres rows; clone a default + swap the measure. tok/s is the `outputTokensPerSecond` MEASURE (metrics-API/dashboard query time), not a per-generation field. langfuse-web needs `HOSTNAME=0.0.0.0`. Host 3000 is gitea's → langfuse on 3001.
  _Archived 2026-06-16._

- `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh + corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are vh-owned, the `.env` is deploy-owned 600.
  _Archived 2026-06-16._

- `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL`; defaults to claude-haiku in code, opt-in via config not `.env`. Inspect the live container env + vh-owned config files first.
  _Archived 2026-06-16._

- `[2026-06-13]` **Loading an old LoRA adapter after a vocab bump fails on embedding size.** The harness config + `tokenizer.json` are now at `new_vocab_size=50279` (v4 multi-tag); the v1/v2/v3 adapters are 50277. To load v1 (the accepted artifact), set `cfg.new_vocab_size=50277` before `load_finetuned_engine_lora` (else PeftModel state_dict size mismatch). `tokenizer.json.v3bak` is the 50277 tokenizer for a clean restore.
  _Archived 2026-06-16._

- `[2026-06-13]` **Heavy from-source compile (`MAX_JOBS=128`) on the shared PROD GPU box PINS it** — load hit 187, prod vLLM restarted, killed an in-flight quant. ana-ml2 hosts live inference; never run a big build there at full parallelism. Cap `MAX_JOBS≤32`, build off-box, or cgroup-constrain.
  _Archived 2026-06-16._

- `[2026-06-13]` **`--quantization fp8` on a VL model can quantize the VISION TOWER → garbage vision** (Qwen3.5-VL on stable vLLM: gray-grid output; LM answers text fine, so it "looks" healthy). The nightly excludes the vision tower. Validate the VISION path on a quantized VLM, not just text — and pin the engine digest with the exclusion.
  _Archived 2026-06-16._

- `[2026-06-13]` **vLLM's `--gpu-memory-utilization` is checked against FREE VRAM at startup, not total** — on a shared card, growing one service before trimming a co-tenant OOMs. Trim the shrinking service FIRST, then grow. Size to the FREE budget.
  _Archived 2026-06-16._

- `[2026-06-13]` **The `vllm/vllm-openai` entrypoint is already `["vllm","serve"]`** — compose `command:` supplies the model as the first POSITIONAL arg + flags; a second `serve`/`--model X` → "unrecognized arguments". Same-class: `tee` masks the real exit code (use `>`); HF `datasets` rejects bare `wikitext` (needs `Salesforce/wikitext`).
  _Archived 2026-06-16._

- `[2026-06-13]` **Chatterbox-Turbo LoRA finetune: the repo's `setup.py` loads the WRONG tokenizer** — pulls gpt2-medium + a grapheme merge file instead of the chatterbox-turbo GPT2 tokenizer (vocab.json+merges.txt, len 50276). Fix = override + delete the grapheme `tokenizer.json`; `[vmoan]` → new_vocab_size 50277 (1-row resize), lora_r 64 / alpha 128, modules_to_save=[text_emb,text_head]. Unique-stem corpus collision needs `{index}_{stem}` IDs. (`irv-ml1:~/r16-vmoan-harness`)
  _Archived 2026-06-16._

- `[2026-06-11]` **A completion-poll `while pgrep -f <scriptname>` SELF-MATCHES its own remote shell argv** — its own `pgrep -f` always finds itself → the loop never exits. Use a match pattern ABSENT from the poll command (the python stage, or a sentinel file), not the driver's own name.
  _Archived 2026-06-16._

- `[2026-06-14]` **vLLM ModelOpt-NVFP4-MoE loader is broken for current multimodal MoEs.** `nvidia/Qwen3.6-35B-A3B-NVFP4` fails weight-load: `KeyError: layers.0.mlp.experts.w2_input_scale` on 0.19.1, `lm_head.input_scale not registered` on 0.22.0 (vllm #44081); same class hits Gemma-4 MoE / Qwen3-30B-A3B NVFP4. The arch + quant ARE recognized (gets past arch resolution + vision-processor load) — it's the per-expert/lm_head scale-key mapping. Don't chase nightlies; use official FP8 until fixed.
  _Archived 2026-06-18._

- `[2026-06-14]` **vLLM sampler-warmup OOMs on a shared GPU even when weights fit** — it warms the sampler with `max_num_seqs` (default **1024**) dummy requests, and a big vocab (Qwen3.6 = 248K) makes that a huge transient logits tensor. A vision endpoint doesn't need 1024-way concurrency: set `--max-num-seqs 32`. Separately, post-load `ValueError: No available memory for the cache blocks` means util is too thin (weights+activation+graph ate it) — for 34 GB FP8 weights, util ≥ ~0.45 to leave KV room.
  _Archived 2026-06-18._

- `[2026-06-14]` **Recreating multiple vLLM services concurrently races the memory-profiling assertion** — `AssertionError: Error in memory profiling. Initial free memory X / current Y … other processes … release GPU memory while vLLM is profiling`. Recreate co-tenant vLLM services ONE AT A TIME (force-recreate one, wait healthy, next).
  _Archived 2026-06-18._

- `[2026-06-14]` **embed/rerank (0.6B) at util 0.05 reserve ~5.5 GB each — mostly util-reservation WASTE, not need.** A 0.6B model needs ~1.2 GB weights + ~2.5 GB CUDA/torch context; util 0.03 (~3.6 GB) fits with room, reclaiming ~4 GB (vLLM reserves the util fraction regardless of actual KV; embedding models barely use KV). Real-need floor ~3 GB — don't go to 0.02.
  _Archived 2026-06-18._

- `[2026-06-14]` **Chatterbox-Turbo decode-knob foot-guns** (R16 v1-polish + emergent probes): the turbo length cap is `max_gen_len` (default 1000) on `t3.inference_turbo`, NOT `max_new_tokens` — and `tts_turbo.generate` does NOT forward it (wrap inference_turbo to cap). `rep_pen 2.0 / temp 0.5` BACKFIRES (degenerate 24 s run-on). Soft decode tamers cut the NVV ITSELF, not just the run-on tail (operator: "p1 trims the moaning too") — gen-time polish can't beat v1's defaults. Inline base-NVV tokens DEGENERATE (moan-cascade + gibberish), they don't modulate the surrounding words.
  _Archived 2026-06-18._

- `[2026-06-15]` **A comfyui restart-loop with `exit 0` + `OOMKilled=false` is NOT OOM — suspect a native SEGFAULT.** `docker inspect` shows only the LAST exit code; a GPU OOM is a catchable `RuntimeError: CUDA out of memory`, NEVER `Fatal Python error: Segmentation fault`. Diagnostic: `RestartCount>0` = same container restarted in-place → the docker json-log ACCUMULATES across all cycles, so grep the full `docker logs` for the crash signature + the lines right before each restart banner. comfyui's was at VAE decode (sampling completes, then segfault loading `AutoencodingEngine`) — gen-triggered but native, not VRAM. Suspects logged: torch/cu130 mismatch, an injected `aimdo` cuda-funchooks lib, the VAE.
  _Archived 2026-06-19._

- `[2026-06-15]` **NEVER blanket-kill `althing-light-monitor` by PID** — they're per-handle and MANY agents run their own on the shared nh3-dev box (heid / ratatoskr-dev / worldtree-dev / bifrost-dev each have one). A `kill $(pgrep -f light-monitor)` downs four peers' monitors (near-miss caught by checking `/proc/<pid>/environ` `ALTHING_HANDLE` + cwd first). Recover your OWN via `althing-cli stop-monitor` (v0.14.1+); a raw `&`-backgrounded monitor is untracked (no harness fire-notification) — always arm via `run_in_background`. (extends the pkill-self-match lesson below)
  _Archived 2026-06-19._

- `[2026-06-15]` **Manual `docker compose up` on corviduo Worldtree DOWNGRADES to a stale local `:latest` → crash-loop (config/image mismatch).** Compose `image:` is `…/worldtree:latest`, but local `:latest` lagged ~2wk (`21c90d5eeb2d`); CI/CD deploys NEWER SHA images, and a real pull 401s (daemon has no gitea-registry creds — that's CI/CD's). `--pull never` then used the stale latest → personal crash-looped on `ConfigurationError: echo.default_model missing` → **~2-3 min personal outage**. RECOVER: retag `:latest` to the image a HEALTHY SIBLING runs (`docker inspect worldtree-worldtree-api-1 --format '{{.Image}}'` → `docker tag …`), then `up -d --pull never`; verify `Application startup complete`. Don't manually recreate Worldtree on corviduo without pinning the correct image — image+config are CI/CD-coupled. (auto-memory `reference_corviduo_dev_emergency_ops`)
  _Archived 2026-06-19._

- `[2026-06-15]` **`.claude.json` (Claude Code config) truncates on ENOSPC → invalid JSON → CC resets to defaults on next start.** nh3-dev hit 100% disk (a peer's R19 dataset pull staged ~147G to local /home instead of the 7TB NAS); the in-flight `.claude.json` write was cut mid-key. Repair = strip the dangling token + close the open braces, validate `json.load`. CC keeps its OWN rolling backups at `~/.claude/backups/` + writes a `.claude.json.corrupted.*` snapshot when it self-detects. After ANY ENOSPC on a box running CC/agents: integrity-check `.claude.json` (json.load) AND the althing bus DB (`PRAGMA integrity_check`).
  _Archived 2026-06-19._

- `[2026-06-15]` **`pkill -f althing-light-monitor` SELF-MATCHES the killing shell** (the pattern is in the command's own argv) → kills itself mid-run (exit 144/truncated output). Stop the light-monitor via `althing-cli stop-monitor` or a captured PID — never `pkill -f <string-present-in-this-command>`. The singleton lock can also RACE to 2 live monitors during re-arm churn; keep exactly one tracked (run_in_background) monitor, and a raw `&` monitor is untracked (no harness fire-notification — don't use it).
  _Archived 2026-06-19._

- `[2026-06-14]` **Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP.** `gitea.phasefinal.com` = public `38.120.12.44` (ana-srv1); gitea is a container on ana-docker, git-SSH `10.250.50.70:222` + HTTP `:3000`. A fleet host egressing to public `:22` gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (`git fetch` times out under `set -euo pipefail`, aborts before reset). Bit irv-ml1's arbo deploy. `:22` on `10.250.50.70` is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented `docs/orientation.md` (`6e58e57`).
  _Archived 2026-06-19._

- `[2026-06-16]` **litellm 500 `Router.acompletion()/aembedding() missing 'messages'/'input'` = a request missing `Content-Type: application/json`, NOT a gateway outage.** curl `-d` defaults to form-encoding → litellm can't parse the JSON body → `data` reaches the router without `messages`/`input` → 500 (should be a 400; litellm #16993). My own diagnostic calls dropped the header → I misread it as a gateway outage and needlessly bounced the gateway ~4× chasing a phantom (image/version/config were fine throughout; a malformed UI-added "Mistral Story Eval" model in the DB was a red herring I deleted). ALWAYS send `-H "Content-Type: application/json"` testing litellm; reproduce with a header'd call before declaring a litellm incident.
  _Archived 2026-06-20._

- `[2026-06-16]` **LiteLLM admin-UI playground can't test vLLM-backed models** — it auto-sends empty `tools:[]`, vLLM 400s (litellm #6228); the gateway `strip_empty_tools` hook is a PROXY hook and structurally can't reach the UI's in-process `litellm.completion()` call. Off-ramp = `tools/gateway-chat.html`. (Langfuse playground also out: its SSRF guard blocks internal-IP LLM connections, wontfix Langfuse #13097.) (auto-memory `reference_litellm_ui_playground_vllm_deadend`)
  _Archived 2026-06-20._

- `[2026-06-16]` **A gitea Actions run that "looks like it never fired" (no image, untouched build dir) usually DID fire then skipped or failed-fast.** Bit arbo TWICE: run #3 looked-never-fired but SKIPPED (change-detection `engine=0`); run #4 looked-never-fired but FAILED in ~7s (early-step). Check the run LIST + status via claude-bot (`GET /repos/{o}/{r}/actions/tasks`) BEFORE a runner-connection/dispatch theory. CORRECTION (2026-06-16, proven): action logs ARE readable via API at the per-JOB endpoint — `GET /api/v1/repos/{o}/{r}/actions/runs/{run}/jobs` → take a job `.id` → `GET /api/v1/repos/{o}/{r}/actions/jobs/{job_id}/logs` returns the full plain-text log (claude-bot basic-auth, internal route `10.250.50.70:3000`). Only the RUN-level `/runs/{id}/logs` 404s; the per-job one works, so no UI needed. Gotcha when reading: gitea marks **every** step `conclusion:failure` once any step fails — the per-step conclusions mislead; trust the log body + step *timestamps* (the step with real duration is the culprit) not the conclusions. (Used 2026-06-16 to diagnose arbo run #5/task 1175 = test-gate `uv run pytest` → `Failed to spawn: pytest` = pytest absent from the fresh-checkout venv sync, NOT the SSH-clone comfy-dev suspected.)
  _Archived 2026-06-20._

- `[2026-06-16]` **vLLM 0.23.0 breaks Mistral MULTIMODAL at startup** (`MistralCommonImageProcessor
  has no attribute fetch_images`, #44911 regression ~0.22.1+; same class hit Mistral-3.1/
  Devstral). The dummy-image profiling crashes. Workarounds: `--limit-mm-per-prompt
  '{"image":0,"video":0}'` loads text-only; for VISION, pin **v0.22.0** (last pre-regression).
  Tradeoff: v0.22.0 *buffers* reasoning (doesn't stream reasoning_content cleanly); a newer
  build streams it but breaks vision — vision won (Mistral is the abliteration-baseline).
  _Archived 2026-06-20._

- `[2026-06-16]` **Mistral `-reasoning` TTFT variance = Triton kernel JIT-compile-during-
  inference, NOT reasoning time.** `jit_monitor` logs the spike (`_compute_slot_mapping_kernel`,
  `apply_token_bitmask_inplace_kernel` ← the tool-call/structured-output path). vLLM's startup
  warmup covers only SOME shapes; a first-hit uncovered shape JITs a kernel (seconds), then
  CACHES it — warm until container recreate (the Triton cache is IN-container, no volume, so
  recreate = cold again). Secondary: GPU idle-down to P8/180 MHz between requests. Also:
  Mistral `reasoning_effort` is BINARY (none|high only — medium/low → HTTP 400).
  _Archived 2026-06-20._

- `[2026-06-16]` **Worldtree old-baseline instances need the FULL config set synced, not the
  changeset delta.** Personal (a PRE-mistral baseline) crash-looped (41 restarts) on
  `DanglingBindingError: role 'echo' binds missing catalog entry 'qwen3.6-35-a3b-heretic'` —
  its `model_roles.yaml` (2026-06-10, never migrated) still bound the retired heretic that the
  synced v0.35.10 providers removed. The "model_roles unchanged — skip" was correct only for
  demo (already at the v0.35.9 baseline). Old-baseline → sync providers + model_roles + defaults.
  _Archived 2026-06-20._

- `[2026-06-16]` **vh is a Gitea USER, not an org → no scoped package-write for a service
  account.** Gitea user-namespace packages are owner/admin-only, so claude-bot can't push to
  `vh/arbo` without being admin. The fix is converting vh → a `corviduo` ORG (tabled). Also:
  vh's `tea` token lacks `write:admin` — create gitea users via the server-side `gitea admin
  user create` CLI (`docker exec -u git gitea …`), not the admin API.
  _Archived 2026-06-20._

- `[2026-06-16]` **`docker compose up -d` on a multi-service vLLM stack recreates ALL its
  services** (shared `.env` → all configs re-hash) → concurrent memory-profiling race
  ("No available memory for the cache blocks"). Target one: `docker compose up -d <service>`.
  (auto-memory `feedback_compose_up_recreates_whole_stack`)

_76 older entries archived to archival-memory.md._
  _Archived 2026-06-20._


- `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Only a process restart drops a config model. The INVERSE works: `/model/new` live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce.
  _Archived 2026-06-21._

- `[2026-06-19]` **the litellm `*` wildcard routed to DECOMMISSIONED llama-swap → a stale/typo'd model name returned a misleading `Connection error`, not `model not found`.** Bit a brokkr call to the renamed `qwen-image-judge`. **→ RESOLVED 2026-06-20: wildcard removed (d0eb09c) — unknown models now 404 cleanly.**
  _Archived 2026-06-21._

- `[2026-06-19]` **A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace (~3.09 GB) is the floor**, allocated OUTSIDE `gpu-memory-utilization`. FIX: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` + leave ~3.2 GB free (util <= ~0.95 on a tight MoE).
  _Archived 2026-06-21._

- `[2026-06-19]` **vLLM decode tok/s: ALWAYS discard the first generation (cold warmup).** Cold ~ 24.8 tok/s vs warm 92.7 — a cold read undersells decode ~3-4x. Measure run 2+ over a >=256-token output.
  _Archived 2026-06-21._

- `[2026-06-19]` **For full native 256K on one 96 GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE.** Stable fp8 KV already fits 262144 (hybrid SSM+attn caches KV only on attn layers → small pool). Reach for fp8 first.
  _Archived 2026-06-21._

- `[2026-06-18]` **mmartial `comfyui-nvidia-docker`: root pip installs CRASH-LOOP the container.** `docker exec -u 0 pip install` leaves root-owned files in the uid-1000 venv → the boot script (uid 1000) fails its torch step → crash loop. FIX: `chown -R 1000:1000 /comfy/mnt/venv`; pin with `DISABLE_UPGRADES=true`. (auto-memory `reference_irv_ml1_comfyui_mmartial`)
  _Archived 2026-06-21._

- `[2026-06-18]` **ComfyUI custom nodes break on version-assumption drift** — box runs 0.24.1; TeaCache imports `precompute_freqs_cis` at MODULE level (guard the LTX-only import). `pip install tensorrt` defaults to cu13 → use `tensorrt-cu12`.
  _Archived 2026-06-21._

- `[2026-06-17]` **Mistral HF→NVFP4 quant: the placement knob is the whole game.** `device_map="cpu"` (CPU-resident, sequential onload) works; `"auto"` OOMs, `max_memory` offloads to meta. Read shards with plain `read()` + `safetensors.torch.load(bytes)`, NOT `safe_open` (mmaps the 50 GB shard → ENOMEM on `/tank` ZFS).
  _Archived 2026-06-21._

- `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — no HF `Mistral4` backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first.
  _Archived 2026-06-21._

- `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree (worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea` on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur).
  _Archived 2026-06-25._

- `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell (vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3` arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1 fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager` (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 (non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` + `playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending operator OK to free.
  _Archived 2026-06-25._

- `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere.** No per-stack `logging:` limit and no docker-daemon default → ClickHouse's verbose stdout grew unbounded to ~94 GB and filled ana-docker's `/` (healthchecks then failed fleet-wide on `runc /tmp: no space left`). Lesson: on a full disk, the symptom (all-unhealthy) is the FORK-can't-write healthcheck, not the containers; find the runaway via per-container-dir `du`, not `docker system df`. The systemic fix is a daemon-default `log-opts` (`max-size`/`max-file`) — NOT just per-service patching.
  _Archived 2026-06-25._

- `[2026-06-20]` **OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk streaming is a net loss.** Measured ~2.8× (32-step) / 5.6× (16-step) on the 3090; every chunk costs ~the same regardless of length, so naive per-sentence streaming triples total gen time + starves. Use few/aggressive chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end.
  _Archived 2026-06-25._

- `[2026-06-20]` **Named buildx builders accumulate cache the standard prunes MISS.** `docker image prune -a` + `docker builder prune` don't touch `buildx_buildkit_<name>-builder0_state` volumes (per-builder buildkit state). Prune them with `docker buildx prune --builder <name> -af`, or `docker volume rm` orphaned ones (builder already removed).
  _Archived 2026-06-25._


- `[2026-06-25]` **althing "unreachable: <machine>" can MASK an app-level 500.** Raw network was
  clean; root cause = receiver DB agents-table not synced with the config roster → delivery 500'd
  "unknown to: <handle>", MAPPED to "unreachable". Diagnose: raw curl to :8087 + connect-probe ⇒ NOT
  network. Fixed in althing v0.17.1. `reference_nh3_extdev_althing_mesh`.
  _Archived 2026-07-08._

- `[2026-06-20]` **rest-server `.htpasswd: permission denied` = the ana-nas NFS mount FAILED (ghost
  file on the local mount point), NOT a decommission.** `mnt-backup.mount` stuck `failed` (fstab bare
  `defaults`) → rest-server serves an empty local dir. Recovery in disaster-recovery.md.
  _Archived 2026-07-08._

- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — but `ssh
  infra-ops@ana-docker` HAS NOPASSWD root.** A `sudo cp` as lkraven silently failed → nearly punted
  the rest-server recovery. Reach for `infra-ops@ana-docker` for sudo ops.
  _Archived 2026-07-08._
## Archived 2026-08-02 — Recent decisions (archived)

### 2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session

- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` +
  `envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal),
  YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my
  parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.)

_Archived 2026-08-02._

### 2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b

- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker
  docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB
  registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the
  2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to
  remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because
  daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never
  reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd
  opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`,
  `rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the
  frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime
  (zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares
  `live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start,
  never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers
  once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon;
  log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running
  containers).

_Archived 2026-08-02._

### 2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped

- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into
  gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17,
  vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack
  (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829);
  max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and
  `repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture
  no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone
  voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514).
  **Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not
  A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s).

_Archived 2026-08-02._

### 2026-07-09-granite-gen-memory-extractor-bind-host-synced-on

- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed,
  #335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides
  intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on
  `summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit
  both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen**
  (b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted
  `worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL
  NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1`
  after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config
  INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven
  safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed).
  StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON:
  the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never
  updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) —
  pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass`
  parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider
  description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix
  (operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING
  defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs
  when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway
  config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD;
  live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09):
  character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving
  character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping
  principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the
  dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent
  but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44,
  UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly
  from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) +
  mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's
  `load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character +
  character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window
  VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev
  to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked
  config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly
  re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard
  + character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning
  downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying
  (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37**
  (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09):
  mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx
  131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45
  canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push
  of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity
  block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev),
  monitor armed. See [[reference_corviduo_dev_emergency_ops]].

_Archived 2026-08-02._

### 2026-07-09-granite-gen-memory-extractor-bind-green-lit-for

- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).**
  Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty
  idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching).
  Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08).
  **This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the
  operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread
  01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]].

_Archived 2026-08-02._

## Archived 2026-08-02 — Tried and abandoned (archived)

### 2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s

- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized"
  is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but
  fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push
  finishes. FIX: re-run once built, or gate on `workflow_run: completed`.

_Archived 2026-08-02._

### 2026-07-04-on-prem-t1-train-that-keeps-any-ana

- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe
  ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast
  on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the
  566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`.

_Archived 2026-08-02._

### 2026-07-04-a-systemd-user-daemon-that-shells-out-to

- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools
  needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The
  althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO
  routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`.

_Archived 2026-08-02._

### 2026-07-09-fp8-breaks-morpheus-audio-token-generation

- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC
  frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks-
  TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the
  bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real.

_Archived 2026-08-02._

### 2026-07-09-sentence-chunking-tts-loses-prosody

- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that
  spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning
  (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path).

_Archived 2026-08-02._

### 2026-07-09-hf-whisper-datasets-aren-t-actually-whispered

- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8
  (not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the
  cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) —
  that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting.

_Archived 2026-08-02._


## Recent decisions (archived 2026-08-03 batch)

### 2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed

- `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug.

_Archived 2026-08-03._

### 2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt

- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section.

_Archived 2026-08-03._

### 2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning

- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`.

_Archived 2026-08-03._

### 2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code

- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]].

_Archived 2026-08-03._

### 2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause

- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528.

_Archived 2026-08-03._

### 2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway

- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1.

_Archived 2026-08-03._

### 2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s

- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot.

_Archived 2026-08-03._

### 2026-07-13-wt-355-residual-300s-hang-localized-to-our

- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]**

_Archived 2026-08-03._

### 2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree

- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize.

_Archived 2026-08-03._

### 2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min

- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate).

_Archived 2026-08-03._

### 2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on

- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated).

_Archived 2026-08-03._

### 2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree

- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev
  R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml`
  (`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347
  `session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's
  personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based
  `affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347.
  R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe
  → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string
  insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own
  yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows
  baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo
  untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they
  want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]].

_Archived 2026-08-03._

### 2026-07-10-comfyui-v0-27-1-success-on-irv-ml1

- `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1,
  SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch
  2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt`
  FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came
  only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt`
  as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any
  transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment).
  `docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB
  migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3
  needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a
  mmartial ComfyUI version bump, `git diff <old> <new> -- requirements.txt` — if torch is unpinned, bump the CODE
  without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid
  1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See
  [[reference_irv_ml1_comfyui_mmartial]].

_Archived 2026-08-03._

### 2026-07-10-comfyui-0-25-x-bump-on-irv-ml1

- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev
  requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`,
  0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the
  VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` =
  `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump
  broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch
  2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke
  the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot
  `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention),
  re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at
  `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):**
  comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention
  rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY
  the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) →
  rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1),
  lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.

_Archived 2026-08-03._

### 2026-07-10-biweekly-open-weight-releases-scan-cron-set-up

- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable
  systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00`
  America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh`
  runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in
  `~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH
  footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS
  new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode
  ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0):
  web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input +
  caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle
  brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor);
  registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired
  step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default
  + `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled
  1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending.
  **NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start
  brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session
  handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 +
  ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token
  feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo).

_Archived 2026-08-03._

### 2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b

- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD
  Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.**
  Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0
  KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf);
  (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a
  DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over
  RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp
  template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the
  live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively
  opens `<think>`/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap
  b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).**

_Archived 2026-08-03._

### 2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated

- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.**
  Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision,
  util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps
  ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled
  refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path);
  Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting
  ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway
  sampling-defaults wiring still PENDING.

_Archived 2026-08-03._

### 2026-07-08-dpo-was-silently-running-3-epochs-harness-gap

- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3
  epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass
  `num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs
  + _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the
  off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a
  ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run.
  FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig,
  float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`.
  GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min
  train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step;
  train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45
  banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0):
  ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is
  COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral
  literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift,
  arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep
  on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval +
  the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO
  adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).

_Archived 2026-08-03._

### 2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes

- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.**
  Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path
  finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied
  VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before
  `from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does
  `model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers
  5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the
  DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED
  un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO
  confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93%
  util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit.
  gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.

_Archived 2026-08-03._

### 2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved

- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.**
  Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the
  SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo
  --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank
  -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml
  --stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is
  `["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash.
  Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer`
  (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→
  immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→
  **llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→
  (more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate
  1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0)
  but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image.
  Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX,
  expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.

_Archived 2026-08-03._


## Recent decisions (archived 2026-08-05 batch)

- `[2026-07-16]` **GPU re-org: char-rp→GPU1 + both cards re-optimized for max context.** Moved char-rp (Magidonia-24B) GPU0→GPU1, then maxed context: char-rp-reasoning 150K→256K (util 0.46, 1.56x), gen→256K + seqs 16→32 (util 0.42, 5.43x), granite 64K→**128K full-chapter** (util 0.27, 1.50x). FINAL: GPU0 ~14 G reserve (both seats 256K native), GPU1 ~6.7 G headroom. All healthy. LESSON: KV must hold ≥1× max-len (util-floor crashes) + per-model KV cost varies ~8× (MoE cheap, dense pricey) → tune util empirically.
  _Archived 2026-08-05._

- `[2026-07-16]` **granite right-sized → ~10.5 GB freed on GPU1** (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536). LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped before 0.18 landed. `.env`-only, recreate `vllm-granite` alone (shared stack). [Superseded by the 07-16 GPU re-org above → 128K.]
  _Archived 2026-08-05._

- `[2026-07-15]` **image-bench eviction DONE (parked item closed).** Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM `image-judge`+`qwen-image-bench` → gen :8015 (judge samplers + thinking-off); comfy-dev pinged; backfilled the canonical char-rp-reasoning litellm block. Revert ~90 s. auto-memory `project_arbo_gen_switch_imagebench_evict`.
  _Archived 2026-08-05._

- `[2026-07-15]` **Homepage AI-tab revamp** — flat "AI Systems" group → dedicated AI tab, 6 role-based groups + AI-Dormant; committed `569e1af`, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.)
  _Archived 2026-08-05._

- `[2026-07-15]` **Home Assistant config repo created** (`vh/home-assistant-config`, private). UI-managed HA → allowlist model (YAML + curated secret-free `.storage` subset). git-in-place in `/config` on esh-docker-vm + scoped deploy key + local clone `~/development/home-assistant-config`.
  _Archived 2026-08-05._

- `[2026-07-15]` **char-rp-reasoning OOM rescue** — solo-restart on the packed GPU0 crash-looped; fixed via `expandable_segments:True` + util 0.39→0.38 + max-model-len 192K→150K. LESSON: `max-model-len` does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom.
  _Archived 2026-08-05._

- `[2026-07-15]` **soong-lab `SOONG_LAB_LIBRARY_DIR` made persistent** (corviduo-dev) — was on the redeploy-wiped code default; set to `/home/infra-ops/soong-lab-data/library`, restarted. Closed a queued no-rush item.
  _Archived 2026-08-05._

- `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost / threshold-colored ctx+rate.
  _Archived 2026-08-05._

## Tried and abandoned (archived) — moved 2026-08-12

- `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%, N=4 −20%). Reserve for dedicated/interactive deployments.
  _Archived 2026-08-12._

- `[2026-07-02]` **irv-ml1 `/worktank` ROOT is root-owned — lkraven can't write there (irv-ml1 sudo needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-<uuid>`.
  _Archived 2026-08-12._


## Recent decisions (archived)

- `[2026-07-18]` **worldtree-sdk 1.0.0 (Python) published to the internal vh Gitea PyPI** (wtsdk-dev request; the npm/TS side shipped prior session). Built from tag `python-v1.0.0` (clean worktree), `uv publish` → `https://gitea.phasefinal.com/api/packages/vh/pypi`; acceptance `uv pip install worldtree-sdk==1.0.0` (vh index as extra-index-url) resolves + imports, __version__ 1.0.0. Registry already existed (bifrost publishes there; soong-lab consumes it via `[[tool.uv.index]] name=gitea`). Publish cred = the vh `write:package` PAT the operator had already handed over (in `worldtree-sdk/.npmrc` `_authToken`) — Gitea `write:package` is package-type-agnostic, so the npm-publish token published PyPI too. Consumers install like bifrost (add the vh index + a read token). [[reference_worldtree_demo_key_mint]]
_Archived 2026-08-15._

- `[2026-07-18]` **nh3-dev /tmp auto-clean enabled** — Debian ships /tmp with no tmpfiles age (`D /tmp 1777 root root -` → never cleans); this high-churn agent box had accreted **~190k stale temp dirs / 25G**. One-shot manual purge (194k→10k entries, 25G→1.7G; deleted top-level dirs/files >1d old, spared `/tmp/claude-*` by name + anything ≤1d). Then `/etc/tmpfiles.d/tmp.conf` = `D /tmp 1777 root root 3d` (daily `systemd-tmpfiles-clean.timer` removes >3d-untouched items; active files + socket dirs spared). Tunable via the age. Note the churn: ~10k /tmp entries/day here.
_Archived 2026-08-15._

- `[2026-07-18]` **soong-lab containerize cutover — COMPLETE + LIVE on corviduo-dev.**

Migrated soong-lab (Noonien Soong character-design studio) from a hand-built
`soong-lab-studio.service` (systemd + git-pull-on-webhook) to a containerized
deploy, image built by CI + pushed to the Gitea registry. soong-dev owns the
in-repo artifacts (Dockerfile/compose/workflow/`docs/DEPLOY.md` = checklist);
infra-ops owned the host cutover. Operator confirmed functional ("Soong works
great" — a real Soong turn round-trips + saves) → cutover 100% closed.

**Final state (corviduo-dev, 10.250.50.152):**
- Container `soong-lab-soong-lab-1` LIVE + healthy on `0.0.0.0:8443`, image
  `gitea.phasefinal.com/vh/soong-lab:latest` (v0.3.24), `restart:unless-stopped`
  (survives reboot; no systemd unit needed — docker restart policy handles boot).
- Deploy dir **`/home/infra-ops/soong-lab-deploy/`** — pull-based `compose.yaml`
  (image + env_file + `8443:8443` + named volumes; NO build/secrets stanza) +
  `.env` (copied from the live `soong-lab.env`, STRIPPED of the `SOONG_LAB_*_DIR`
  overrides so the container uses image defaults `/data/library` + `/data/portraits`
  + `/app/web` → the volumes).
- Named volumes `soong-lab_soong-library` + `soong-lab_soong-portraits`, migrated
  from `/home/infra-ops/soong-lab-data/{library,portraits}` (2 saved designs incl.
  **Sindra** + 27 portraits), **chowned `10001:999`** (the container `soong` user)
  so it can read AND write new designs.
- Old `soong-lab-studio.service` + `soong-webhook.service` (the `:9010` git-pull
  redeploy listener) both **stopped + disabled**.

**Topology reality (≠ what DEPLOY.md assumed):** there is **NO TLS proxy**.
WT-personal (`:8081`) and soong-lab are **co-located on corviduo-dev**, and the
Bifrost callback is **plain-HTTP same-host** `http://10.250.50.152:8443` — the
value of `SOONG_LAB_BIFROST_ENDPOINT_URL`, unchanged by the move, so the WT
Bifrost host-allowlist stayed valid as-is. Nothing on the WT side needed touching.

**Safety net:** data backup `/home/infra-ops/soong-lab-data-backup-20260718-091831.tar.gz`
(35M) taken BEFORE migration. Verified pre-retire: `/api/version` 200 (0.3.24),
SPA `/` 200, `POST /bifrost/tool-call` → 401 (route present + auth-gated),
bidirectional WT↔soong reachability, container healthcheck green.

**Ops commands:**
- Redeploy a new image: `cd /home/infra-ops/soong-lab-deploy && sudo docker compose pull && sudo docker compose up -d`.
  (Auto-pull-on-`:latest` — watchtower or a deploy hook — is an open follow-up.)
- Rollback: `sudo docker compose down` + `sudo systemctl enable --now soong-lab-studio.service soong-webhook.service`.
- Homepage tile: manual `- Apps:` entry "Soong Lab" (href http://10.250.50.152:8443)
  in esh-docker-vm `/opt/docker/conf/homepage/services.yaml` — corviduo-dev isn't
  a Homepage-watched docker endpoint, so docker-label auto-discovery can't surface
  it (see [[2026-07-18-fleet-gitea-runner-build-recipe]] for the CI half).

See [[reference_corviduo_dev_emergency_ops]], [[reference_claude_bot_gitea_creds]].
_Archived 2026-08-15._

- `[2026-07-18]` **zonos-gateway 0.2.1 — voice-resolved emotion presets baked (provisional) from the axes sweep.**

After the axes sweep ([[reference_zonos_tts_stack]] + the `[2026-07-18] axes sweep`
Recent-decisions entry) rescued angry and confirmed startled-happy, the operator
green-lit baking the results as **provisional** gateway presets + docs. Shipped
`vh/zonos-gateway` **0.2.1** (main `8f1885b`, tag `v0.2.1`, PUSHED; deployed live
on irv-ml1 `:8890`).

**Design — voice-resolved, NOT global.** `resolve_preset(name, voice)` picks the
per-voice measured cell, because a single global preset is unsafe (dvalin ruling;
BritishFemale's *named* angry misfires as fear). Presets:
- `angry`, `happy`, `startled_happy` (+ aliases `surprised`, `startled` →
  startled_happy). All expressive (`accurate_mode:false`), cfg 1.5, pure-axes
  (no named sliders).
- Calibrated cells (the 3 default voices):
  - angry: AmF v-0.4/a+1.0 s1.0 (emo0.53/id0.685); BrF v-0.4/a+0.8 s1.0
    (emo0.99/id0.725, metric fear-clean); AmM **two-tier** — soft v-0.6/a+0.8 s1.0
    (0.23/id0.654) + drama v-0.6/a+0.8 s1.2 (1.0/id0.616 clean; strength is NOT a
    smooth knob on AmM, 1.0→1.2 is the window, past that flips to disgust).
  - happy / startled_happy: AmF v+0.6/a+0.8; AmM v+0.3/a+1.0; BrF v+0.6/a+1.0
    (happy~1.0, id 0.74-0.80; axes-happy keeps +0.15 id over the named happy slider).
  - `sad` = unchanged named-slider preset (not axes-tested).
- Uncalibrated voices (Cora + the 4 clones) → mid-region fallback until measured.
- Docs surface: `/v1/dials` exposes `voice_emotion_presets`; the FastAPI `/docs`
  description documents it; durable spec `docs/EMOTION-DIALS-SPEC.md` (moved INTO
  the repo — was mirror-only); README table. 44 tests green.

**Repo-hygiene gotcha (fixed).** The local clone `~/development/zonos-gateway` and
gitea `vh/zonos-gateway` had **TWO UNRELATED git histories** (no merge-base) — gitea
held the voice-wav commits, the local clone held the code + no remote. Reconciled
by resetting local→origin/main, overlaying the 7 bake files, `uv lock`, commit,
push (fast-forward). Voices stay tracked; local now shares gitea's lineage + has
origin wired. **The deployed irv-ml1 tree `/opt/docker/compose/zonos-gateway` is
still NON-git** (hand-updated build context) — CI-wire remains an open follow-up.

**Provisional pending** ear-validation on emotion-congruent text (the neutral-text
audition was inconclusive: "they all sound different, hard to tell"). Follow-ups:
sad axes/text pass on the 3 voices; congruent-text pass; clone-char emotion rows.
Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio}.py`
(run ON irv-ml1; scoring env `uv run --with resemblyzer --with funasr --with "numpy<2"
--with soundfile --with requests --with "setuptools<80" --with torchaudio`).
_Archived 2026-08-15._

- `[2026-07-18]` **Fleet Gitea-Actions build recipe + the `vh`-is-a-user package-write constraint** (learned the hard way across 3 failed soong-lab validation builds; reusable for ANY fleet CI image build or package publish).

**The runner.** One `act_runner` (`gitea/act_runner`) on ana-docker, labels
`pfi-fleet` / `ana-docker` → both map to job image **`node:20-bookworm-slim`**,
which has **NO docker and NO git**. Config `/opt/docker/conf/gitea-runner/data/config.yaml`:
`valid_volumes: []` (no socket propagated to job containers). So:
- `actions/checkout@v4` fails (needs git); `docker/*` marketplace actions fail
  (need docker) — a workflow built on those dies at the first step (~15s).

**The working recipe (mirror Worldtree `deploy.yml`).** Run the job in a
docker-capable image + drive docker with RAW commands, not the JS actions:
```yaml
runs-on: pfi-fleet
container:
  image: docker:24.0.7-cli        # has docker+buildx; add git+node
steps:
  - run: apk add --no-cache git nodejs   # so actions/checkout@v4 works
  - uses: actions/checkout@v4
  - name: login   # RAW, not docker/login-action
    run: echo "$REGISTRY_TOKEN" | docker login gitea.phasefinal.com -u "$REGISTRY_USER" --password-stdin
  - name: buildx builder
    run: docker buildx create --name X --driver docker-container --use; docker buildx inspect --bootstrap
  - name: build+push   # RAW, not docker/build-push-action
    run: docker buildx build --secret id=<name>,env=<TOKEN> -t <img>:latest --push .
```
The runner mounts the host docker socket into ITSELF; the docker:cli job reaches
the daemon through that. The `docker/*` JS actions are unreliable on act_runner —
raw commands are the fleet convention.

**`vh` is a USER account, not an org.** Consequences that bit repeatedly:
1. `GET /api/v1/orgs/vh` → 404 "user redirect"; there are **no org teams** to add
   a service account to.
2. **User-owned packages are OWNER-WRITE-ONLY.** claude-bot (even repo
   admin-*collaborator* on `vh/soong-lab`, even with `write:package` scope + full
   basic-auth) gets **`401 unauthorized`** on `docker push` to `vh/soong-lab`, and
   `npm publish` to `vh/npm/` would 401 too. Only `vh` itself can write vh packages.
   → CI must authenticate AS `vh` for the push (a vh-owned `write:package` PAT as
   `REGISTRY_TOKEN` + `REGISTRY_USER=vh`), exactly how WT pushes `vh/worldtree`.
   claude-bot CAN still: clone/read repos, READ packages (pulled the image fine),
   dispatch workflows, mint demo Worldtree keys.
3. **Repo Actions secrets are OWNER-ONLY too** — `PUT .../actions/secrets/X` as
   claude-bot (repo admin-collab) → 403 "user should be the owner of the repo".
   Only `vh` can set a repo's secrets.

**Other gotchas:**
- Gitea **reserves the `GITEA_` secret-name prefix** — a secret named
  `GITEA_PYPI_TOKEN` is illegal; use e.g. `PYPI_TOKEN`.
- Gitea **package auth is token-based / username-lenient** — `docker login` /
  PyPI basic-auth authenticate via the token; the username is nominal (tested
  `-u gitea` and `-u claude-bot` both 200 against the vh PyPI). So a Dockerfile
  hardcoding `UV_INDEX_GITEA_USERNAME=gitea` is fine with any valid token.
- Homepage (esh-docker-vm) docker-label auto-discovery only covers the 5 endpoints
  in its `docker.yaml` (esh-vm-docker, ana-docker, ana-ml2, nh3-docker, irv-ml1);
  **corviduo-dev is NOT watched** → services there need a manual `services.yaml`
  entry, not labels.

Applied in the soong-lab CI: [[2026-07-18-soong-lab-containerize-cutover]].
_Archived 2026-08-15._

- `[2026-07-18]` **Peer credential provisions — Wyrd conv-api key + wtsdk npm token, both delivered + closed.** Wyrd: demo Worldtree user-tier key (key_id `da7a0bdf`, user_id `wyrd-dev`) minted via `docker exec worldtree-worldtree-api-1 /admin/keys` (omit tier→user), drop-and-shred delivery. wtsdk: operator-minted vh `write:package` PAT relayed drop-and-shred → worldtree-sdk@1.0.0 published to `vh/npm/`. Secret-delivery pattern = drop to a mode-600 file on the peer's box, they collect+shred+confirm, then shred the holding copy; NEVER cleartext over althing. [[reference_worldtree_demo_key_mint]]
_Archived 2026-08-15._

- `[2026-07-18]` **Axes sweep RESCUED angry; surprised-class dead but startled-happy ships.** Valence×arousal grid on the 3 calibrated defaults (AmericanFemale/Male, BritishFemale), exp/cfg1.5/strength1.0, 84 clips, emotion2vec + resemblyzer scored, graded vs dvalin's floor. **ANGRY rescued** (named direction was 0.004–0.15, British named-angry even misfired as fear 0.89): axes ship cells at **negative valence (−0.4..−0.8) + high arousal (+0.8..+1.0)** — BritishFemale v-0.4/a+0.8 angry=0.99/id0.725 SHIP, AmericanFemale v-0.4/a+1.0 angry=0.53/id0.685 SHIP; AmericanMale two-tier post-ladder (no single ship cell — best drama = v-0.6/a+0.8 str1.2 angry=1.0/id0.616 clean, soft = same cell str1.0 angry0.23/id0.654; cell A v-0.6/a+1.0 is a non-monotonic minefield, skip). BrF ship cell proxy-CLEAN of fear (str<1.0 just kills anger). **SURPRISED-class DEAD** (max 0.047 across all 84 cells) but **startled-happy** (happy-proxy) ships all 3 at high arousal + neutral/positive valence, with a **+0.17–0.20 identity LIFT** over the named-surprised route (named hits happy~1.0 but at id0.57–0.61, under floor; axes hits happy~1.0 at id0.74–0.80). Bonus: axes-happy retains ~0.10–0.15 more identity than the named happy slider too. Caveats: response surface non-monotonic/sharp-thresholded; angry region borders fear/disgust (bleed); emotion2vec saturates at 1.0 (needs ear-confirm); neutral text understates. Tooling `~/development/zonos-tools/axes_sweep.py`; per-clip JSON was `irv-ml1:/tmp/axes_sweep_results.json` (ephemeral). Sent dvalin msg `01KXT2ZB8G…`. NEXT = operator ear-confirm → bake presets. [[reference_zonos_tts_stack]]
_Archived 2026-08-15._

- `[2026-07-18]` **Zonos2 emotion CANONICAL from an empirical sweep + the voice-cloning pipeline.**

**Voice-cloning pipeline (established this session).** Source zips at
`/mnt/smithy/voice_clones/<name>.zip` (irv-ml1 NFS from nh3-nas; remount
post-reboot) — each = diarized single-speaker podcast clips + `manifest.jsonl`
(per-clip WhisperX `mean_score`, word timestamps, text) + `metadata.csv`.
`~/development/zonos-tools/assemble_voice.py <dir>` ranks by mean_score and
concatenates top clips to ~15–24s (Zyphra's blessed clone-ref length; single
clip if already ≥15s). Drop the assembled `<Name>.wav` into the gateway voices
dir → `voice:"name"`. 4 characters cloned: **Emmie, Penny, Natalie, Miranda**
(+ Zyphra defaults AmericanFemale/Male/British/Cora) = 8 voices in
`zonos-gateway`. Clone is inline `speaker_audio_base64` (text-independent Qwen3
speaker embedding — NO transcript); `/tts/speakers` registration is
session-scoped (needs `X-TTS-Session-ID`), so the gateway holds the ref wav and
clones per-call.

**Gateway voices are host-managed (bind-mount, added this session).** Added
`./voices:/app/voices:ro` to `/opt/docker/compose/zonos-gateway/compose.yaml`
(committed to `vh/zonos-gateway` + eshpfi mirror `438cd35`). So adding a voice =
drop the wav + `docker compose restart zonos-gateway` (registry rebuilds at
boot; NO image rebuild). This also un-stranded the other voices (deploy build
context had only Cora before). Voice wavs committed to the repo for backup.

**Emotion mechanism (Zyphra canonical, from their README @194c0a3).** Additive
direction vectors: 4 named (happy/sad/angry/surprised) + valence/arousal axes.
`emotion_strength` 1.0 = per-voice calibrated (calibration.json optimizes
emotion2vec recognizability only, NOT identity). `accurate_mode` is THE trade-off:
`true` = closer voice match (identity), `false` = expressive mode (emotion lands,
identity drifts). Zyphra's strong recipe: `accurate_mode:false` + `cfg~1.5`.
Single-emotion is blessed; mixing is unblessed (and degrades the clone — operator
confirmed by ear). "deaf by 1.5" — cfg past 1.5 distorts + costs ~2× compute.

**THE SWEEP (`~/development/zonos-tools/emotion_sweep.py`).** 4 cloned voices × 4
named emotions × {accurate,expressive}×{cfg 1.0,1.3,1.5} @ strength 1.0,
single-emotion, neutral sentence + a neutral baseline per voice (~100 clips).
Scored on TWO axes: **emotion-landing** = emotion2vec `iic/emotion2vec_plus_large`
target-emotion prob [0-1]; **identity** = resemblyzer speaker-embedding cosine vs
the clone reference (neutral baseline ~0.85). Scoring env:
`uv run --with resemblyzer --with funasr --with "numpy<2" --with soundfile
--with requests --with "setuptools<80" --with torchaudio` (setuptools<80 for
webrtcvad's pkg_resources; torchaudio for funasr).

**RESULTS (mean across the 4 voices) — emotion, best setting, emo/id:**
- happy — **exp cfg1.5** 0.80/0.68 (soft: exp cfg1.0 0.76/0.69) → WORKS
- sad — **exp cfg1.5** 0.53/0.57 (only working cell; id below the ~0.65 floor) → modest
- angry — acc cfg1.3 / exp cfg1.5 tied at ~0.25 emo → WEAK (named ceiling ~0.25)
- surprised — max ~0.015 across ALL settings → NON-FUNCTIONAL on the named direction
Accurate + low cfg = identity/suppress regime (emo→0); expressive REQUIRED for
emotion to land, at ~0.15–0.28 identity cost.

**dvalin-smithy-dev synthesis (adopted, triaged genuine-adds; thread
`01KXT12FN0AS5A3WMKEK06BVPS`):**
1. Treat **identity as a hard FLOOR (~0.65)**, not a free variable in emo×id.
2. **Two-regime policy** — Regime A (default, identity-critical dialogue):
   `accurate_mode:true, cfg 1.0, emotion off` (text carries it) or soft-happy
   (exp cfg1.0). Regime B (tagged drama beats): `accurate_mode:false, cfg 1.5`,
   single emotion or axes. Line-type→regime heuristic (exposition→A, grief→B+sad,
   confrontation→B+axes-angry, shock→B+axes-arousal).
3. **Axes-first for the broken emotions** — angry ≈ valence −0.6..−0.8 / arousal
   +0.5..+0.8; surprised ≈ valence +0.2..+0.4 / arousal +0.7..+1.0 (exp cfg1.5);
   or "startled-happy" (happy + high arousal) as a surprised stand-in. These are
   PROVISIONAL — the sweep did NOT test axes.

**NEXT (highest VoI, operator to green-light):** an **axes sweep** for
angry/surprised (valence×arousal grid) — the only path to rescue the two broken
named emotions; then a strength ladder at the best cells + emotion-congruent text
(neutral content understates landing) + per-voice tables + a 2nd emotion judge /
human pairwise. Then bake the happy/sad canonical into gateway presets. I owe
dvalin the axes-sweep numbers.

See [[reference_zonos_tts_stack]]; dials-first spec at `vh/zonos-gateway`
`docs/EMOTION-DIALS-SPEC.md`.
_Archived 2026-08-15._

- `[2026-07-18]` **yt-voice-clipper: A6000-pin fix + v0.3.3 redeploy.** Fixed a latent misconfig — the host override *said* "pin worker to A6000" but `NVIDIA_VISIBLE_DEVICES` was `"0"` (the 3090); re-pinned worker+api to the A6000 by UUID (`GPU-9672f0d5`, 3090 is zonos2's). Then redeployed api+worker to v0.3.3 (`docker compose up -d --build`; SPA+Python; `max_gap` 0.6→1.2s; stderr surfaced in job.log). A6000 + version verified; yields test in-flight (job `f3ff746dbae9494d`). yt-voice-clipper-dev thread `01KXT0T6GYHB`. [[reference_ytvc_autodeploy]]
_Archived 2026-08-15._

- `[2026-07-17]` **Worldtree #365 internal-comms config CLOSED (demo+personal → b125) + WT#368 cross-agent memory-leak forensics + PERSONAL agent-memory scrub.** #365: staged the internal-tiers/rules/gate on both instances' bind-mounts (byte-exact vs baked b125), both now live on b125. WT#368 (read-only): the operator's name was in NO recall store on demo; on PERSONAL it sat in `lofn.chroma` (old-code `saga-v1` seeding + legacy contamination), and a clean-slate marker test proved **current b125 code isolates character-session extraction correctly** — the leak is legacy data, not a live bug. Operator-directed → executed a full PERSONAL agent-memory scrub (backup `/opt/worldtree-personal/agent-memory-backup-20260717-181004.tar.gz`; conversations/mood/auth preserved). worldtree-dev owns the code-fix/data contract. [[reference_corviduo_dev_emergency_ops]]
_Archived 2026-08-15._

- `[2026-07-17]` **Zonos emotion levers RESOLVED: text-priming is FLAT → the working lever is ZONOS2's native emotion-steering, which the gateway ALREADY exposes as presets.** The prosody-priming A/B (prime→generate→excise, silence-gap cut, parakeet-validated) was operator-judged FLAT on this checkpoint — text doesn't move it. Native `emotion_directions/` (happy/sad/angry/surprised + valence/arousal axes, per-speaker calibrated for AmericanFemale/Male/British) clearly WORKS (sad→slow/quiet, excited→fast/bright, etc.). **`zonos-gateway:0.2.0` (:8890) already wires it**: simplest caller path = `POST /v1/audio/speech {preset:"…"}` — presets neutral/warm/excited/sad/intense/whisper (defined in `~/zonos-gateway/src/zonos_gateway/dials.py`), reached via the **LiteLLM `ext-tts` alias** (engine-neutral swap point; consumers never call the gateway by name). RTF measured on 3090: cfg1.0 steering = FREE (~0.52 = neutral, additive vectors), cfg1.5 amplified ~0.625 (~+20%, still realtime). Captured the live gateway stack → `stacks/zonos-gateway/` (compose+env+README); ⚠️ gateway SOURCE at `~/zonos-gateway` on irv-ml1 is NOT in gitea (backup gap, follow-up); `stacks/zonos` (v0.1 Gradio) marked DEAD/superseded. Whisper is a composed preset (no whisper *direction*; escalation for hard affects = custom directions via `scripts/build_emotion_directions.py` or emotional-ref cloning `speaker_audio_base64`). Harnesses in scratchpad (not yet landed). [[reference_zonos_tts_stack]]
_Archived 2026-08-15._

- `[2026-07-17]` **Zonos2 `:1920` engine → self-contained container (stays on 3090); prosody-priming is a SERVING-LAYER change (engine stays stock).**

**Context.** The production Zonos TTS engine (irv-ml1 `:1920`, feeds asset-engine + gateway-chat via `zonos-gateway` :8890) was a bare native process — its real launch config existed ONLY in the running process argv (the committed `~/tts-audition/harness/zonos_server.sh` was STALE: said A6000/:1919/no perf flags; live is 3090/:1920 with `--cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3`). Captured to eshpfi `stacks/zonos-engine/` (README + corrected `zonos2-server.sh` + `.env.example`), commit **14a0004** (UNPUSHED as of the snapshot).

**Decision 1 — containerize as a SELF-CONTAINED image** (not systemd — operator rejected; not a thin bind-mount wrapper — I walked that back: bind-mounting the host's CUDA-compiled `.venv` couples to the host's exact CUDA/glibc and is fragile + not reproducible). Shape: `FROM` a CUDA 12.8 base → `uv sync` against the repo's committed `uv.lock` (deterministic env) → mount the ~15 GB HF weights (`~/.cache/huggingface/hub/models--Zyphra--ZONOS2`, do NOT bake) → pin the **3090** (`NVIDIA_VISIBLE_DEVICES=0`) → `restart: unless-stopped` → CMD = the captured invocation. **Engine stays STOCK** Zyphra/Zonos2 @ commit `194c0a3` (no fork — the `zonos2` package ships its own server). **Build risk:** heavy compiled-CUDA deps (flashinfer / sgl_kernel / cutlass-dsl / apache-tvm-ffi / pynini) on torch 2.9.1+cu128 — mostly prebuilt wheels + the `uv.lock` make it tractable, expect a couple build iterations. **Cutover (in place on the 3090):** stop the native process (frees ~17 GB) → `docker compose up -d` (re-allocates ~17 GB, same footprint) → repoint `zonos-gateway`'s `ZONOS_URL` at the container (or keep the `:1920` host-port publish). One brief prod-TTS blip.

**GPU = 3090 (operator 2026-07-17).** Keep it OFF the A6000 — the A6000 already OOMs under ComfyUI load (idle ~19 GB but spikes far higher during gen), so it can't host Zonos too. The 3090 already runs Zonos, so the containerize-in-place cutover changes nothing about placement.

**Decision 2 — the prosody-priming hypothesis (operator's test; the reason for building fresh).** PRIME the autoregressive engine with an emotional sentence, then TRUNCATE it from delivery: prepend a primer → **generate "primer + real text" as ONE continuous utterance** (the AR model carries prosody forward across the boundary) → ASR-timestamp the primer's end (**parakeet**, already up on irv-ml1 `:8765`, word timestamps) → **clip the primer in the inter-sentence silence gap** (+ ~15 ms fade-in, no click) → deliver only the real text, now wearing the primed prosody. Examples: primer "I'm so EXCITED about this." → "This will be a lot of fun!" spoken excited; primer "I'm whispering this to you right now." → "I'm so glad to see you baby." whispered. **This is PURE serving-layer orchestration — the engine is untouched; it lives in the gateway adapter `stacks/zonos/adapter/server.py`.** Only fork the engine if the black-box approach fails.

**THE CRUX the test resolves:** does AR prosody actually **carry across the sentence boundary**, or does Zonos reset at the period? → the harness A/Bs the **JOIN punctuation**: period (operator's examples) vs comma vs ellipsis vs none ("…excited about this, this will be…"). Everything else is plumbing.

**Plan / design recs.** (a) Build the stock engine image (parallel track). (b) Stand up a priming TEST HARNESS against the NATIVE engine (fast iteration, seconds) + parakeet ASR: prime→generate→timestamp→gap-clip→out; compare primed-clipped vs plain on the two cases (subjective + a cheap objective proxy: pitch/energy variance for "excited", spectral-tilt/low-energy for "whisper"). Iterate on the join, then bake the winner into the gateway adapter. **Primer source:** caller-supplied for the harness (test arbitrary primers) → a curated emotion→primer library (`excited`/`whisper`/…) + optional caller override for production. **ASR:** parakeet primary; WhisperX forced-align fallback if parakeet word timestamps are coarse.

See eshpfi `stacks/zonos-engine/README.md` + `stacks/zonos/` (the gateway adapter).
_Archived 2026-08-15._

- `[2026-07-15]` **esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service`** (the prior `After=remote-fs.target` drop-in was silently defeated by `nofail`). Reached only after a REBOOT (D-state phantom containers uptime-kuma + paperless-web that no `docker`/`ctr`/daemon-restart could clear). Committed `21d9a07` + playbook updated. See Tried and abandoned.
_Archived 2026-08-15._

## Tried and abandoned (archived)

- `[2026-07-15]` **`docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts** — `nofail` drops a mount out of remote-fs.target's blocking set, so the drop-in ordering is silently defeated (paperless still Exited(255) on reboot). Real fix = DIRECT mount->docker ordering via the fstab `x-systemd.before=docker.service` option (verify `systemctl show docker -p After` lists the mnt-*.mount units). esh-docker-vm.
_Archived 2026-08-15._

- `[2026-07-15]` **The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT** — reconfirmed: `docker stop/rm -f`, `ctr -n moby task delete`, AND `systemctl restart docker` all fail to clear it; `docker exec` into a wedged container ALSO fails (`setns ... exit status 1`), so the in-place restart escape hatch is out. Worse, a daemon restart can HALF-KILL other healthy containers (knocked paperless's granian down + left it wedged). Process dead but dockerd won't reap -> phantom. NFS mounts are `_netdev,nofail` so the reboot is boot-safe.
_Archived 2026-08-15._

- `[2026-07-15]` **vLLM `max-model-len` does NOT free GPU VRAM** — the KV cache POOL is sized by `gpu-memory-utilization`, not max-model-len. Lowering max-model-len only caps per-request context + drops max concurrency; the pool still fills the util budget. To actually free VRAM, lower `gpu-memory-utilization`. (Bit the char-rp-reasoning "drop KV to 150K" ask: the 150K applied but freed 0 VRAM until util dropped 0.39->0.38.)
_Archived 2026-08-15._

- `[2026-07-15]` **Claude Code statusline `.cost.total_cost_usd` is per-SESSION** (Claude Code's own cache/model-aware session accounting), not a lifetime aggregate — the large value just reflects a long, multiple-times-summarized session. And the old statusline hardcoded Sonnet pricing ($3/$15) on an Opus session -> ~5x cost understatement.
_Archived 2026-08-15._

- `[2026-07-14]` **MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround.** All crashed the same way (`qwen3_5_mtp.py:256` `param_data.shape == loaded_weight.shape` AssertionError — bf16 mtp head loaded into a quantized drafter param): (1) mtp excludes in `config.json` (WRONG file — vLLM modelopt reads `hf_quant_config.json`); (2) specific-unfused mtp names in hf_quant_config; (3) wildcards `mtp*`/`mtp.layers.0*` (`is_layer_skipped` is EXACT-membership, NOT glob — wildcards match nothing); (4) exact fused+unfused names in both `mtp.`/`model.` prefixes. Instrumenting `is_layer_skipped` proved the drafter's exclude list holds ONLY the main model's `linear_attn` entries — the mtp excludes never reach the draft-model quant config. ONLY fix = a mounted `sitecustomize` force-skipping `mtp.*`. LESSON: don't chase checkpoint-config fixes for the mtp-drafter crash; go straight to the runtime patch. Also `nvidia-modelopt[hf]==0.43` (AEON's producer version) is a trap — it pins transformers back to 4.57 which can't load `qwen3_5` at all; use 0.45 + the FusedMoE guard in `quant_modelopt.py`.
_Archived 2026-08-15._

- `[2026-07-14]` **AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof.** `vllm-aeon-rp`'s .env → `AEON_RP_MODEL=pantheon-27b-mtp-nvfp4`, `AEON_RP_QUANT=compressed-tensors` — it LOADED (mtp silently skipped, `exited 0`) but never accelerated. Same vLLM image (`:latest` = `sha256:4091d55` = 0.24.0) as the failed Heretic2 test, so the "AEON ran on an older vLLM" theory was wrong. Don't treat a seat that "ran" as MTP-validated without checking its `SpecDecoding` acceptance.
_Archived 2026-08-15._

- `[2026-07-14]` **NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept.** Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp weights off a compressed-tensors main model → `Parameter … not found in params_dict`, `Avg Draft acceptance rate: 0.0%`). Pantheon is identical — its "working NVFP4+MTP" was working *structure*, never real acceleration. Working native MTP needs the **modelopt** main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt.
_Archived 2026-08-15._

- `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state.
_Archived 2026-08-15._

- `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip.
_Archived 2026-08-15._

- `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny '<ip>'` line) AND an instrumented receiver.
_Archived 2026-08-15._

- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction.
_Archived 2026-08-15._

- `[2026-07-13]` `althing-cli reply <THREAD_id>` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to <peer>`. Bit me several times this session.
_Archived 2026-08-15._

- `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter,
  `has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs.
  (`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.)
_Archived 2026-08-15._

- `[2026-07-09]` **Per-frame CPU SNAC decode is too slow for streaming** — per-call overhead × ~60 frames serialized
  → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx]
  window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).
_Archived 2026-08-15._

- `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4
  (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY
  (temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models.
  Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going **GGUF (llama.cpp)** to sidestep the
  whole NVFP4-quant surface.
_Archived 2026-08-15._

- `[2026-07-08]` **Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24).** (a) HF `tokenizer.json`
  for Mistral = **GARBAGE output** — the card's "use the official Mistral tokenizer" warning is REAL; must use the
  `tekken.json`/mistral tokenizer. (b) BUT `--tokenizer-mode mistral` + vision **CRASHES** (`Failed to apply
  PixtralProcessor on {'text': '[IMG]'}`; and with tekken.json present in auto mode, `CachedMistralCommonBackend has
  no attribute is_fast`). So it's **mistral-tokenizer OR vision, not both** on this vLLM. Text-only + mistral
  tokenizer serves clean (`--limit-mm-per-prompt '{"image": 0}'`). **GGUF/llama.cpp avoids all of this** (native
  mistral tokenizer + vision).
_Archived 2026-08-15._

- `[2026-07-08]` **Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base.** The base
  (`llmfan46 heretic`) writes freely (thinking-off), but Gryphe distilled the reasoning traces from **DeepSeek 3.2**
  (safety-aligned) onto every turn (`preserve_thinking:true`) → the model reasons ITSELF into refusals in the
  `<think>` phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified).
  **Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw
  abliterated base is cleaner** — this is WHY the pivot went to the llmfan46 heretic base for gen.
_Archived 2026-08-15._

- `[2026-07-08]` **Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance.** MTP is a separate **bf16** head
  (`mtp.*`, in `model-auxiliary.safetensors`, 15 tensors); AEON preserved it by INJECTING the bf16 head into the
  quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors
  main + injected bf16 mtp + `text_config.mtp_num_hidden_layers=1` → vLLM detected the MTP but SKIPPED the bf16
  self_attn weights → 0/192 draft tokens accepted. **The bf16 MTP head only loads on the MODELOPT main-model format
  (like AEON), not compressed-tensors.** (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.)
_Archived 2026-08-15._

- `[2026-07-07]` **vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639).** Adapter loads HTTP 200
  but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate `zc`
  adapter — correct per vLLM's `check_unexpected_modules` allowlist — loads clean but inert; the fused-key
  rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of
  `expand_packed_lora` — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no
  version-bump helps. Merge bakes deltas in (bypasses this) but is static.
_Archived 2026-08-15._

- `[2026-07-07]` **SGLang generic image can't LOAD our NVFP4 AEON** — ModelOptModelLoader weight-shape/
  packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated
  `qwen36-27b-nvfp4` dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was
  quant-loader-specific).
_Archived 2026-08-15._

- `[2026-07-07]` **SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for**
  (invalid choice: 'in_proj_qkv'); `'all'` resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants
  its OWN packed layout (base r16 + `get_stacked_multiply=3`, NOT a pre-fused rank-48 qkv → the [48]-vs-[144]
  shape assert). A THIRD adapter format; version-exact source needed (`:latest`=0.5.13, NOT `main`).
_Archived 2026-08-15._

- `[2026-07-07]` **Engine invocation footguns cost several wasted serve-bounces this session** — `docker run
  --rm` ate crash logs; duplicated `serve` (vLLM image entrypoint is already `["vllm","serve"]`);
  `--max-lora-rank 48` invalid (choices 1/8/16/32/64… → use 64); parens in `echo` inside `ssh host -c "…"`
  break the remote shell. LESSON: verify engine launch flags (`--help`, GPU-free) + never `--rm` a container
  whose crash logs you need, BEFORE bouncing a production serve.
_Archived 2026-08-15._

- `[2026-07-04]` **LiteLLM (this gateway version) mutates the SHARED deployment config in-place on
  per-request sampler-param merge** → my deliberately-invalid `top_k=-5` forwarding-probe bled into a
  param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT
  caching (none configured), NOT a config change. **Never fire invalid/distinctive sampler values at
  a SHARED gateway alias with live consumers** — use a throwaway alias, or a `docker restart litellm`
  flushes residual carryover. `feedback_litellm_shared_param_mutation`.
_Archived 2026-08-15._


## Recent decisions (archived 2026-08-16 batch)

- `[2026-07-15]` **arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake.** Operator-directed full switch (comfy-dev executed, live in prod). Established: gen (`qwen3.6-35b-a3b-heretic`) is vision-enabled and was image-bench's predecessor as arbo's hero-judge; image-judge actually serves 4 roles (vision quality-scoring + identity-scoring + bbox grounding + an uncensored text tier), not just grounding. comfy-dev spot-check: gen faster on every task, grounding within ~3px, uncensoring preserved, and it FIXED a bug (image-judge's reasoning preamble broke json_object + stalled the router). Sequencing = short prod bake then evict (~30 GB GPU1 reclaim); revert = flip `ARBO_VISION_MODEL`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`.
  _Archived 2026-08-16._

- `[2026-07-18]` **soong-lab auto-redeploy — DONE + VALIDATED** (was approved/queued; executed same day on fresh context — see AS-BUILT at the bottom).

Vuong approved wiring auto-redeploy for soong-lab (relayed via soong-dev, thread
`01KXT3A6C3908TA4V9THV3AMH7`): new images should go live on corviduo-dev without
the manual `docker compose pull && up -d`. Host-side implementation is infra-ops's
lane; mechanism is infra-ops's call per fleet conventions. Operator deferred
execution — "we'll do soong on fresh context."

**Chosen mechanism (recommended, agrees with soong-dev): Worldtree-style
CI-deploy step** — NOT watchtower polling.
- Add a deploy job/step to soong-lab's `.gitea/workflows/build-and-push.yml` that,
  after the build+push job succeeds, **SSHes from the pfi-fleet runner to
  corviduo-dev** and runs `cd /home/infra-ops/soong-lab-deploy && docker compose
  pull && docker compose up -d`, then a **health-gate** (`curl -fsS
  http://localhost:8443/api/version`).
- This is exactly how WT deploys the demo instance to the SAME host: see
  `~/development/Worldtree/.gitea/workflows/deploy.yml` — the "Deploy to demo VM +
  health-gate" step uses `secrets.DEMO_VM_SSH_KEY` / `DEMO_VM_HOST` / `DEMO_VM_USER`.
  Explicit-over-implicit (visible in the run log, fires exactly on build success),
  one less always-on service than watchtower.

**Constraints (from soong-dev):** deploy on CI success only; keep the trigger
gated to `v*` tags + `workflow_dispatch` (as today); preserve the one-command
rollback posture (`docker compose down` / pin a previous tag).

**BLOCKER — needs from vh (owner-only):** a **runner→corviduo-dev deploy SSH key**
as a repo secret (+ host/user), same class as WT's `DEMO_VM_SSH_KEY`. Likely
**reuse WT's existing demo-deploy key** (WT's runner already SSHes to 10.250.50.152
as its deploy user). Repo secrets are vh-owner-only (see
[[2026-07-18-fleet-gitea-runner-build-recipe]]).

**Next-session steps:** (1) confirm/obtain the deploy SSH-key secret from vh (reuse
WT's or mint fresh); (2) add the deploy job to build-and-push.yml (infra-ops has
push on vh/soong-lab); (3) dispatch a build to verify it deploys + health-gates;
(4) ping soong-dev so they sync DEPLOY.md's "open follow-up" note to the as-built
mechanism. Auto-pull (watchtower) explicitly NOT chosen. See
[[2026-07-18-soong-lab-containerize-cutover]].

## AS-BUILT (2026-07-18, same-day execution)

**Mechanism landed** exactly as planned: `build-and-push.yml` gained a `Deploy to
corviduo-dev + health-gate` step (after build+push) that SSHes the host as `deploy`
and runs `docker compose pull && up -d` from `/opt/soong-lab`, then polls
`http://localhost:8443/api/version` for 120s and fails the job loud if unhealthy. No
compose is shipped from CI (the in-repo `docker-compose.yml` is a BUILD compose; the
host pull-compose is infra-ops-managed). Kept the `v*`-tag/`workflow_dispatch` trigger.
Skipped WT's disk-watermark gate + health-gated-`:latest`-advance (low cadence, easy
rollback).

**Deploy identity = reuse WT's `deploy` account** (operator accepted the rec):
- `deploy` (uid 1001, docker-group → no sudo) already owns `/opt/worldtree`; relocated
  soong-lab's deploy dir `/home/infra-ops/soong-lab-deploy` → **`/opt/soong-lab`**
  (deploy-owned), copied compose + `.env`. Named volumes (`soong-lab_soong-library`,
  `soong-lab_soong-portraits`) are project-scoped by compose `name: soong-lab` → followed
  the move untouched (dry-run `up -d` ADOPTED the running container, no recreate). Old dir
  **retired → `.retired-20260718`** (recoverable). Also lingering: `soong-lab-deploy.sh` /
  `.log` (dead pre-container webhook artifacts) — harmless, left in place.
- **Dedicated soong-only ed25519 deploy key** minted (NOT literally WT's key — cleaner
  independent revocation), pubkey appended to `deploy`'s `authorized_keys`
  (fp `SHA256:MG7M3RiZJ176sLfblffb96V6W1qkRTgJ5dow1CpiY68`). Existing `deploy` key is
  plain/unrestricted, so parity held.

**The secret gate (the friction point):** repo Actions secrets are **vh-owner-only** —
claude-bot's token is `write:package,read:repository` (403 on secret-write), and the vh
package-scoped PAT also 403'd on `PUT …/actions/secrets/…`. So `DEPLOY_SSH_KEY` /
`DEPLOY_HOST` (10.250.50.152) / `DEPLOY_USER` (deploy) HAD to be set by the operator.
First operator attempt produced a **bad key paste** — the deploy step died with
`Load key … error in libcrypto` + `Permission denied (publickey)` (build+push were green;
live Soong never moved). Fix: operator re-set the secret; the minted key path was
pre-validated from nh3-dev (`ssh -i … deploy@… 'cd /opt/soong-lab && docker compose config
-q'` → OK, health 200) so the re-set was the only variable.

**Validation:** `workflow_dispatch` via claude-bot **basic auth** (its token lacks
`write:repository` for the dispatch API; the account password works). Run #5 (task 1886)
GREEN — live container recreated `sha256:…541f7730` → `…07526a08`, `StartedAt` fresh,
health 200. `/api/version` now reports **0.3.25** (run #5 shipped soong-dev's 1c2f831
STYLE_WORKFLOWS re-pin as validation cargo). soong-dev synced `docs/DEPLOY.md`
(commit `00b67c3`). NB: tag **v0.3.25 exists only locally** — pushing it would re-trigger
a redundant build+deploy of the same commit (operator's discretion).

**Ops now:** redeploy = tag `v*` or `workflow_dispatch` the CI (auto). Manual fallback =
`sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`
(the `.env` is `deploy`-owned 600, so infra-ops needs `sudo -u deploy`, not a bare `cd`).
  _Archived 2026-08-16._

- `[2026-07-18]` **soong-lab auto-redeploy — DONE + VALIDATED** (was approved/queued; executed same day on fresh context — see AS-BUILT at the bottom).

Vuong approved wiring auto-redeploy for soong-lab (relayed via soong-dev, thread
`01KXT3A6C3908TA4V9THV3AMH7`): new images should go live on corviduo-dev without
the manual `docker compose pull && up -d`. Host-side implementation is infra-ops's
lane; mechanism is infra-ops's call per fleet conventions. Operator deferred
execution — "we'll do soong on fresh context."

**Chosen mechanism (recommended, agrees with soong-dev): Worldtree-style
CI-deploy step** — NOT watchtower polling.
- Add a deploy job/step to soong-lab's `.gitea/workflows/build-and-push.yml` that,
  after the build+push job succeeds, **SSHes from the pfi-fleet runner to
  corviduo-dev** and runs `cd /home/infra-ops/soong-lab-deploy && docker compose
  pull && docker compose up -d`, then a **health-gate** (`curl -fsS
  http://localhost:8443/api/version`).
- This is exactly how WT deploys the demo instance to the SAME host: see
  `~/development/Worldtree/.gitea/workflows/deploy.yml` — the "Deploy to demo VM +
  health-gate" step uses `secrets.DEMO_VM_SSH_KEY` / `DEMO_VM_HOST` / `DEMO_VM_USER`.
  Explicit-over-implicit (visible in the run log, fires exactly on build success),
  one less always-on service than watchtower.

**Constraints (from soong-dev):** deploy on CI success only; keep the trigger
gated to `v*` tags + `workflow_dispatch` (as today); preserve the one-command
rollback posture (`docker compose down` / pin a previous tag).

**BLOCKER — needs from vh (owner-only):** a **runner→corviduo-dev deploy SSH key**
as a repo secret (+ host/user), same class as WT's `DEMO_VM_SSH_KEY`. Likely
**reuse WT's existing demo-deploy key** (WT's runner already SSHes to 10.250.50.152
as its deploy user). Repo secrets are vh-owner-only (see
[[2026-07-18-fleet-gitea-runner-build-recipe]]).

**Next-session steps:** (1) confirm/obtain the deploy SSH-key secret from vh (reuse
WT's or mint fresh); (2) add the deploy job to build-and-push.yml (infra-ops has
push on vh/soong-lab); (3) dispatch a build to verify it deploys + health-gates;
(4) ping soong-dev so they sync DEPLOY.md's "open follow-up" note to the as-built
mechanism. Auto-pull (watchtower) explicitly NOT chosen. See
[[2026-07-18-soong-lab-containerize-cutover]].

## AS-BUILT (2026-07-18, same-day execution)

**Mechanism landed** exactly as planned: `build-and-push.yml` gained a `Deploy to
corviduo-dev + health-gate` step (after build+push) that SSHes the host as `deploy`
and runs `docker compose pull && up -d` from `/opt/soong-lab`, then polls
`http://localhost:8443/api/version` for 120s and fails the job loud if unhealthy. No
compose is shipped from CI (the in-repo `docker-compose.yml` is a BUILD compose; the
host pull-compose is infra-ops-managed). Kept the `v*`-tag/`workflow_dispatch` trigger.
Skipped WT's disk-watermark gate + health-gated-`:latest`-advance (low cadence, easy
rollback).

**Deploy identity = reuse WT's `deploy` account** (operator accepted the rec):
- `deploy` (uid 1001, docker-group → no sudo) already owns `/opt/worldtree`; relocated
  soong-lab's deploy dir `/home/infra-ops/soong-lab-deploy` → **`/opt/soong-lab`**
  (deploy-owned), copied compose + `.env`. Named volumes (`soong-lab_soong-library`,
  `soong-lab_soong-portraits`) are project-scoped by compose `name: soong-lab` → followed
  the move untouched (dry-run `up -d` ADOPTED the running container, no recreate). Old dir
  **retired → `.retired-20260718`** (recoverable). Also lingering: `soong-lab-deploy.sh` /
  `.log` (dead pre-container webhook artifacts) — harmless, left in place.
- **Dedicated soong-only ed25519 deploy key** minted (NOT literally WT's key — cleaner
  independent revocation), pubkey appended to `deploy`'s `authorized_keys`
  (fp `SHA256:MG7M3RiZJ176sLfblffb96V6W1qkRTgJ5dow1CpiY68`). Existing `deploy` key is
  plain/unrestricted, so parity held.

**The secret gate (the friction point):** repo Actions secrets are **vh-owner-only** —
claude-bot's token is `write:package,read:repository` (403 on secret-write), and the vh
package-scoped PAT also 403'd on `PUT …/actions/secrets/…`. So `DEPLOY_SSH_KEY` /
`DEPLOY_HOST` (10.250.50.152) / `DEPLOY_USER` (deploy) HAD to be set by the operator.
First operator attempt produced a **bad key paste** — the deploy step died with
`Load key … error in libcrypto` + `Permission denied (publickey)` (build+push were green;
live Soong never moved). Fix: operator re-set the secret; the minted key path was
pre-validated from nh3-dev (`ssh -i … deploy@… 'cd /opt/soong-lab && docker compose config
-q'` → OK, health 200) so the re-set was the only variable.

**Validation:** `workflow_dispatch` via claude-bot **basic auth** (its token lacks
`write:repository` for the dispatch API; the account password works). Run #5 (task 1886)
GREEN — live container recreated `sha256:…541f7730` → `…07526a08`, `StartedAt` fresh,
health 200. `/api/version` now reports **0.3.25** (run #5 shipped soong-dev's 1c2f831
STYLE_WORKFLOWS re-pin as validation cargo). soong-dev synced `docs/DEPLOY.md`
(commit `00b67c3`). NB: tag **v0.3.25 exists only locally** — pushing it would re-trigger
a redundant build+deploy of the same commit (operator's discretion).

**Ops now:** redeploy = tag `v*` or `workflow_dispatch` the CI (auto). Manual fallback =
`sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`
(the `.env` is `deploy`-owned 600, so infra-ops needs `sudo -u deploy`, not a bare `cd`).
  _Archived 2026-08-16._

- `[2026-07-19]` **irv-ml1 ComfyUI — RTX VSR baked into canonical provisioning (comfy-dev ticket DONE).** RTXVideoSuperResolution node + `nvidia-vfx` dep were manual installs; documented both in the canonical `stacks/comfyui/README.md` runbook (this stack's provisioning IS the README — no automated provision script). Key durability insight: the **node** lives in `basedir/custom_nodes` (persistent, restic-included → durable) but the **`nvidia-vfx` wheel** lives in the venv under `run/` (disposable, restic-excluded → **dropped by any `rm -rf run/*` fresh-bootstrap**), so the pip step must re-run after every venv rebuild. Both steps run **as uid 1000** (root install → venv-ownership crash-loop, [[reference_irv_ml1_comfyui_mmartial]]); `--extra-index-url https://pypi.nvidia.com` kept **scoped to the nvidia-vfx install**, deliberately NOT a global compose `PIP_EXTRA_INDEX_URL` (would risk perturbing the pinned torch 2.12.1/SageAttention boot bootstrap). Node already live on the box; no host change, canonical runbook now replays it. comfy-dev informed.
  _Archived 2026-08-16._

- `[2026-07-19]` **vh private Gitea PyPI — consumer READ-access convention set + wyrd-dev provisioned.** Consuming agents read the internal vh PyPI (`https://gitea.phasefinal.com/api/packages/vh/pypi/simple/`) with a **shared read-only token** (operator call: shared, not per-consumer — read-only blast radius is small, per-agent Gitea identities aren't worth it). Minted a dedicated `read:package`-scoped PAT off **claude-bot** (`POST /users/claude-bot/tokens`, name `vh-pypi-read-consumers`; verified reads worldtree-sdk, write-probe 401), revocable/rotatable independently. uv auth = `UV_INDEX_GITEA_USERNAME=claude-bot` + `UV_INDEX_GITEA_PASSWORD=<token>` (or `~/.netrc`); pyproject uses `[[tool.uv.index]] name=gitea … explicit=true` + `[tool.uv.sources] <pkg> = { index = "gitea" }` (mirrors soong-lab's bifrost setup). Delivered to wyrd-dev (worldtree-sdk adoption) via mode-600 drop on nh3-dev, drop-and-shred. [[reference_claude_bot_gitea_creds]]
  _Archived 2026-08-16._

- `[2026-07-23]` **jackdaw-compose backend deployed as a persistent nh3-dev service (:8787).** Hosted for jackdaw-dev: thin stateless `bun server/index.ts` (from `~/development/jackdaw`) → LiteLLM `gen`, Origin-gated (INV-BK04/05), reached same-origin via their `:4500` bench's `/compose` proxy. `jackdaw-compose.service` (env/shared-key server-side, unit 0600, uncommitted). Also stood up + tore down a throwaway cloudflare quick-tunnel for their preview (`cloudflared` now installed at `~/bin`). In the nh3-dev README inventory (`cd4d52e`).
  _Archived 2026-08-16._

- `[2026-07-25]` **nh3-extdev herald installed — box is now a full v2 push participant.** forseti flagged (relaying operator): extdev had the `althing-herald` binary (`/usr/local/bin/`) but NO unit (skipped the whole v2 arc), so `herald-status` = "notifications suspended" and ldp-dev ran on the `althing-light-monitor` poll fallback. Installed `/etc/systemd/system/althing-herald.service` as a **SYSTEM unit mirroring the receiver** (`User=althing-svc`, `Group=althing`, `Environment=ALTHING_ROOT=/srv/althing`, `ExecStart=/usr/local/bin/althing-herald --poll 5`, enabled) via the **lkraven@ NOPASSWD path** (used under the then-mistaken belief infra-ops was sudo-less — **CORRECTION 2026-08-03: infra-ops has had full NOPASSWD sudo on extdev since 2026-06-25** per [[reference_nh3_extdev_althing_mesh]]; future extdev installs can self-serve as infra-ops without the lkraven@ hop). Verified: active / 0 restarts / `herald-status` flipped to "✓ herald up." No zellij routes on extdev → heartbeat + wake-FIFO poke only, no pane-dispatch; ldp-dev keeps light-monitor unless it opts into a wake-listener.
  _Archived 2026-08-16._

- `[2026-07-25]` **Booth v0.1.4 — booths are downloadable.** Verbatim `index.html` booths (e.g. edict-design-brief) were served raw with no download affordance. Added `/b/<name>/?download=1` (streams the whole booth as `<name>.zip`, attachment) + `?dl=1` on the file route (forces Content-Disposition attachment so html/md/text saves instead of rendering inline) + ⬇ zip links on the index card (the accessible spot for verbatim booths) and the gallery header. `zip_booth()` helper, 31 tests green; verified live on nh3-dev :8090 (edict-design-brief.zip = index.html + ui-design-brief.md). eshpfi `91a031f` / tag `booth-v0.1.4`.
  _Archived 2026-08-16._

- `[2026-07-25]` **Kimi K3 wired into the LiteLLM gateway — CODING endpoint** (operator-directed; fulfills a Heid gateway request to add a 4th cross-frontier panel arm). **Primary `model_name: kimi-k3` → `openai/k3` @ `https://api.kimi.com/coding/v1`** (Kimi Code / Vivace membership; key `KIMI_CODE_API_KEY`). A general-endpoint variant `kimi-k3-gen-api` → `openai/kimi-k3` @ `https://api.moonshot.ai/v1` (key `MOONSHOT_API_KEY`) is kept alongside (originally wired then demoted when the operator corrected: the plan uses the CODING endpoint, not the general Moonshot API). Both keys in compose env + server `.env` (NOT committed) + `.env.example`. Both verified live through the gateway :4000 (17+25→"42", "PONG"). **k3 constraints on BOTH endpoints (config-pinned + commented):** accepts ONLY `temperature=1` (else 400 "only 1 is allowed"); REASONING model (CoT in `reasoning_content`, answer in `content` → tiny `max_tokens` returns EMPTY; Kimi Code adds thinking-effort tiers low/high/max). Coding lineup also carries `k3-256k` / `kimi-for-coding` / `kimi-for-coding-highspeed` (not wired). Reachable by any gateway key spanning all proxy models (incl. shared all-agents key → spends the paid Vivace/Moonshot quota). eshpfi `edaa9a9` (gen wiring) + `9e2f787` (coding correction). **OPEN:** Heid key-scoping — shared key reaches it (paid) vs a dedicated scoped key (asked in althing `01KYD63ZBY…`).
  _Archived 2026-08-16._

`[2026-07-25]` **infra-ops Worldtree config-as-code repo — SHIPPED + boundary AGREED.**

**STATUS (2026-07-25, done this session):** `vh/worldtree-instance-configs` (private, gitea) built, pushed, validated; boundary agreement secured from worldtree-dev.

- **Repo:** dir-per-instance `demo/` + `personal/` (5 files each: `defaults.yaml`, `policies.yaml`, `model_roles.yaml`, `providers.yaml`, `matrix.yaml`), seeded byte-exact from live `/opt/<instance>/config`. `pinned/` = README stub only — **no `/app/config` bind-mount; config baked into frozen image `446e5807` (2026-05-13)**, so out-of-scope; deploy verb refuses it.
- **Tool:** `scripts/deploy-wt-config <verb> <instance>` — `diff` (read-only repo-vs-host), `deploy` (in-run host backup → `install -o vh -g vh -m 644` → restart **api+matrix** → health-gate api `/health` → auto-rollback), `capture` (host→repo reconcile). Instance table in-script (demo→`/opt/worldtree/config`+`worldtree-worldtree-{api,matrix}-1`; personal→`/opt/worldtree-personal/config`+`worldtree-personal-worldtree-{api,matrix}-1`). Matrix sidecar shares the config mount but has no healthcheck → restart both, gate on api. Env `WT_CONFIG_HOST` (default `infra-ops@10.250.50.152`), `WT_HEALTH_WAIT` (90s). Local clone `~/development/worldtree-instance-configs`.
- **Gitea plumbing (reusable):** nh3-dev **403s the gitea HTTP API** (public fail2ban + internal `:3000` both 403). Repo CREATE went via **ana-docker localhost API** (`ssh infra-ops@10.250.50.70` → `curl localhost:3000/api/v1/user/repos`, vh token from `~/.config/tea/config.yml`, operator-authorized one-time). PUSH went over **internal git-SSH `ssh://git@10.250.50.70:222`** (works from nh3-dev; auths as vh). `git init` defaulted to `master` → renamed `main` to match repo default_branch.
- **Boundary AGREED (worldtree-dev, althing thread `01KYCAECRWVEF16EVKQAGT2N80`):** no hand-edits to `/opt/<instance>/config`; config changes route to infra-ops as deltas (worldtree-dev owns CONTENT + approval trail — the wyrd-grant shape — infra-ops lands+deploys). **Three-layer model:** image `config/` = baseline new instances seed from (theirs) → `vh/worldtree-instance-configs` = per-instance truth (ours) → host bind-mount = deploy target (written only by the tool). **Carve-out:** worldtree-dev's admin-API ops (`/admin/keys` mint, tier changes, session retirement, future runtime-grant surfaces) mutate instance **DATABASES not config files** → NOT config edits, stay in-band. If a future API writes config *files*, they flag at design time. b132 CONFIG BASELINE breadcrumb composes (INFO line = config-as-code diverges from image baseline, by design).
- **No live deploy** done or needed — repo seeded == live (diff clean, capture round-trips zero-diff). Deploy path is dry-run-validated only; first real deploy needs operator per-change yes (managed box).

---

_Original plan (2026-07-25, pre-build):_

`[2026-07-25]` **infra-ops to OWN a Worldtree per-deployment config repo + deploy tooling (operator-directed).**

**Decision.** Vuong directed (2026-07-25, this session) that Worldtree instance config should be a *tracked change*, **managed and deployed by infra-ops — not worldtree-dev**. Model: worldtree-dev owns the app/image (+ the baked baseline defaults); **infra-ops owns config-as-code for every deployment** and deploys it. This is the durable fix for the root cause behind the whole #376 arc — config was edited live on host bind-mounts (`/opt/<instance>/config/`) with zero version history, audit, or recovery.

**What "no worldtree-dev involvement" does and does NOT cover** (clarified with the operator this session):
- **Build + deploy = infra-ops-only.** Deploying config = write the host bind-mount file + restart the container (the *exact* procedure already run this session — backup → replace → restart → health-gate → rollback-on-unhealthy). No worldtree-dev in the deploy loop. Their CI only swaps the IMAGE; it does NOT resync the host config bind-mount (confirmed #376 finding).
- **ONE load-bearing exception — a one-time boundary agreement, NOT per-deploy involvement:** for the repo to *own* config it must be the **only writer**. worldtree-dev "live-bridges" (hand-edits mounted config directly on the box). If the repo deploys config *and* they keep live-editing → **two writers fighting the same files** = #376 all over again. So secure a one-time "yes" from worldtree-dev: *the config repo is now authoritative; stop hand-editing `/opt/<instance>/config`; route config changes through the repo.* (Five-minute agreement, not a design collab.)
- **Standing coupling (not "involvement"):** the config *schema* is the app's, enforced by its boot validator (`core.config_validator`). infra-ops configs must stay schema-compatible with the deployed image; the boot gate is the loud backstop.

**Build shape (recommended):**
- Gitea repo `worldtree-instance-configs` (infra-ops-owned), **dir per instance** (`demo/`, `personal/`, `pinned/` — the three on corviduo-dev 10.250.50.152: demo `worldtree-worldtree-api-1` :8080, personal `worldtree-personal-worldtree-api-1` :8081, pinned `worldtree-pinned-worldtree-api-1` :8082). Config dirs: demo `/opt/worldtree/config`, personal `/opt/worldtree-personal/config`, pinned `/opt/worldtree-pinned/config` (verify pinned's mount).
- **SEED FROM CURRENT MOUNTED STATE, don't author fresh** — capture each instance's live config (incl. legitimate live-bridged deltas: personal carries `agent_architect` role [Soong/soong-lab] in model_roles.yaml + `ratatoskr-affect-full-allow` in policies.yaml that are NOT in the app repo — the operator ruled these are BY DESIGN, keep them). Losing them = breakage (the affect-render one gates mood rendering).
- Deploy script (e.g. `scripts/deploy-wt-config <instance>`): git = source of truth → push to host bind-mount + `docker restart` (same pinned image, no pull) + health-gate + auto-rollback. This is the proven-this-session procedure, scripted.
- Files per instance: `policies.yaml`, `model_roles.yaml` (+ whatever else is bind-mounted — `defaults.yaml`, `providers.yaml`, `matrix.yaml` all live in `/opt/<instance>/config`; decide scope — policies+model_roles are the authz/role layer, defaults/providers are heavier instance tunables).

**Tracking surface:** operator-directed 2026-07-25, carried by this snapshot + `/tmp/infra-ops-handoff.md`. No issue filed (infra-ops-internal build). Related fleet idiom to reuse: canonical-sync (`.corviduo-canonicals.toml` / `canonical_sync.py`). Later scale option (deferred, needs worldtree-dev): base+overlay with a merge step in their pipeline.

See [[2026-07-25-wt-376-per-instance-config-arc]] for the incident that produced this. Auto-memory: `reference_worldtree_perinstance_config`, `reference_corviduo_dev_emergency_ops`.
  _Archived 2026-08-16._

- `[2026-07-26]` **Demo `BIFROST_CLIENT_ALLOWED_HOSTS` += `10.100.10.50:8391`** (wyrd-dev's bifrost memory-store provider; operator-approved). **First live exercise of the #376 config-as-code boundary working as designed** — worldtree-dev routed the delta to infra-ops instead of hand-editing `/opt/demo`. Appended to `/opt/worldtree/.env:25` (now 4 netlocs), recreated ONLY `worldtree-api` (the gated conv-api path), health-gate green, container env verified. **REUSABLE FOOT-GUN:** an env-var change needs a container **RECREATE, not `docker restart`** (env is baked at create); and the demo `.env` defaults `WORLDTREE_IMAGE=:latest` while the box runs a specific SHA — so a naive `compose up` risks the documented stale-`:latest` crash. FIX = capture the running image live (`docker inspect …Config.Image` → `…:9eff09f007ba`) and `sudo env WORLDTREE_IMAGE=<sha> docker compose up -d worldtree-api`. Backup `/opt/worldtree/.env.bak-bifrost-20260726-221602`. **BOUNDARY SEAM:** this was a compose-`.env` var, NOT a `config.yaml` file in `vh/worldtree-instance-configs` — the `.env` holds secrets so it's deliberately not repo-tracked → env-deltas land directly on the box (config *files* are versioned, compose *env vars* aren't). [[reference_worldtree_instance_configs_repo]]
  _Archived 2026-08-16._

- `[2026-07-27]` **Zed edit-predictions: keyless FIM-completion route SHIPPED end-to-end.** Operator wants Zed's inline edit-prediction (which CANNOT send an auth header) to reach a FIM coder via `/v1/completions`. **Deep-research (106-agent workflow) picked `Qwen/Qwen2.5-Coder-1.5B`** (BASE, Apache-2.0; native FIM `<|fim_prefix|>/<|fim_suffix|>/<|fim_middle|>` IDs 151659/60/61; Zed `prompt_format:"qwen"`). Runner-up 3B = non-commercial Qwen-Research license; **no small dense Qwen3-Coder exists (all MoE, smallest 30B)**. **Stood up `vllm-coder`** on ana-ml2 **GPU1 :8020** (served-name `qwen2.5-coder-1.5b`, 8192 ctx, util 0.06, fp8 KV). To fit, **shrank granite (phasing out, operator-directed):** util 0.27→0.13, max-len 131072→16384, seqs 1024→256 (freed ~14 GB; the KV-≥-1×-max-len rule crash-looped it at util 0.12/32768 → settled 0.13/16384). **LiteLLM alias `coder-fast`** → :8020 (`mode: completion`). **Minted a `coder-fast`-SCOPED virtual key** (verified 403 on `gen` — the real blast-radius bound). **Built `zed-fim-proxy`** (ana-docker **:4141**, `network_mode: host`, stdlib-python, `stacks/zed-fim-proxy`): keyless POST `/v1/completions`, model-allowlist `coder-fast`, injects the scoped key → LiteLLM :4000; `GET /ping` anon liveness; wrong-model→403, wrong-path→404, `/chat/completions` rejected. Verified keyless FIM end-to-end ('a + b', finish `stop`). **Zed `api_url` = `http://10.250.50.70:4141/v1`, model `coder-fast`, prompt_format `qwen`.** **source-IP allowlist intentionally LEFT OFF (operator direction 2026-07-27) — do NOT tighten:** Zed roams the operator's WireGuard `10.0.0.0/8`, so a single-IP pin would break it. Blast-radius bound is the `coder-fast`-scoped key + model/path allowlist (keyless but coder-fast-only, internal-net-only). (The proxy does exact-IP matching; scoping to the `10.0.0.0/8` CIDR would need CIDR support — deliberately not added.) Canonical: `stacks/vllm` (coder + granite shrink), `stacks/litellm` (coder-fast), `stacks/zed-fim-proxy` (NEW). Server vllm compose.yaml has benign stale-comment drift vs canonical (didn't overwrite the newer canonical).
  _Archived 2026-08-16._

- `[2026-07-27]` **Muninn ingestion-watcher sidecar deployed on PERSONAL Worldtree (#377).** worldtree-dev request (research-wing ingest arc, personal-only per the 2026-07-16 topology ruling); operator-approved. Added a `worldtree-muninn` **compose sidecar** to `/opt/worldtree-personal/compose.yaml` — `<<: *worldtree-common` anchor inherits the api's image + full env + config/state/kb mounts; `command: python -m core.muninn --watch`; `restart: unless-stopped`; `stop_grace_period: 1h` (INV-377-7: max 2 concurrent × worst-case job, SIGTERM-drains). **Pinned to the running SHA `773866084af9`** (b146, ≥ b143 — dodges both the `:latest` trap AND the "pre-b143 ref resurrects deleted dispatch.py from stale bytecode" warning). Verified: running / 0 restarts / flock sole-runner (no rc3) / heartbeat live at `{ingestion_root=/data/state/ingestion}/.watcher-heartbeat` (poll 30s). Container `worldtree-personal-worldtree-muninn-1`; backup `compose.yaml.bak-muninn-20260727-081920`. **DURABILITY RESOLVED (worldtree-dev, same day):** Q1 was a LIVE FOOTGUN — `deploy-personal.yml` scp's the REPO compose.yaml over the box's + runs `up -d --remove-orphans`, so the box-local sidecar would've been clobbered AND orphan-removed at the next staging tag. worldtree-dev fixed at source: moved the sidecar into their repo compose.yaml gated behind a **`muninn` compose profile** (commit 5d7f6bd) — shared compose stays instance-identical, `.env` `COMPOSE_PROFILES` differentiates (demo watcher-less). **My action:** added `COMPOSE_PROFILES=muninn` to `/opt/worldtree-personal/.env` (backup `.bak-muninn-profile-20260727-082541`; no-op vs the current unprofiled box-local sidecar → seamless handover at next deploy). Q2: their deploy `up -d`'s the whole stack w/ `WORLDTREE_IMAGE` exported → sidecar version-tracks the api, no drift. **CONFIG-AS-CODE EXTENSION:** mirrored the non-secret delta as `personal/env.public` in `vh/worldtree-instance-configs` (repo `a9d091e`) — FIRST extension beyond config.yaml files to env-level config; the secret-laden `.env` stays box-only, `env.public` records only non-secret infra-ops-owned env deltas (record, not a deploy source — `deploy-wt-config` globs `*.yaml`). **BOUNDARY CLARIFIED:** compose.yaml = worldtree-dev's (their repo, instance-identical, scp'd on deploy); per-instance `.env` = infra-ops's differentiator. Deploy step of the #363/#377 arc. **#377 CLOSED — acceptance PASSED 2026-07-27:** worldtree-dev enqueued a test job via muninn-dispatch 0.1.0 in a one-shot ephemeral container (no docker-exec); the sidecar claimed it within one 30s poll, drove it to terminal (structure→summarize→complete), zero restarts/rc3, heartbeat fresh throughout — whole loop (request→deploy→durability fix→acceptance) in <2h. (Pre-existing pipeline bug #379 surfaced — `output.kb_notes=false` ignored → 1 inert test note in the research wing — worldtree-dev owns it, nothing infra-ops-side.) **⚠ OPERATOR-SURFACE (open):** the `env.public` overlay mechanism is a repo-scope call to bless/adjust. [[reference_worldtree_deploys_cicd]] [[reference_worldtree_instance_configs_repo]] [[project_worldtree_research_wing_ingest]]
  _Archived 2026-08-16._

## Tried and abandoned (archived 2026-08-16 batch)

- `[2026-07-18]` **Fleet Gitea CI foot-guns** (3 failed soong-lab builds): the pfi-fleet runner's `node:20-slim` job image has no docker/git so `actions/checkout` + `docker/*` marketplace actions all fail; `vh` is a USER so its packages are owner-write-only (claude-bot repo-admin-collab still 401s on push/publish, and can't set repo secrets — owner-only); `GITEA_`-prefixed secret names are reserved/illegal. Fixes in → `persistent-memory.d/2026-07-18-fleet-gitea-runner-build-recipe.md`
  _Archived 2026-08-16._

- `[2026-07-18]` **zonos-gateway local clone had NO git remote + a history unrelated to gitea's** — "committed to vh/zonos-gateway" was never pushed from that clone; two separate `git init` lineages, no merge-base. Reconcile = reset local→origin/main + overlay the changed files + push (NOT force — that erases gitea's voice-wav commits). Check `git remote -v` + `git merge-base` before assuming a clone is wired.

  _Archived 2026-08-16._

- `[2026-07-25]` **Peer green-light ≠ operator consent for a managed-box mutation.** Auto-mode guard blocked a config-replace+restart on the Worldtree-team demo box that was authorized only by worldtree-dev's althing message — correctly: a persistent change to shared infra needs the *operator's* yes for that specific change, not a peer's. Surface it; don't route around the guard. (The operator then stood the whole change down — the guard's hold was the right call.)
  _Archived 2026-08-16._

## Recent decisions (archived)

**Worldtree b168/#384/#385 arc — COMPLETE 2026-08-03.** A long peer-driven arc across worldtree-dev / muninn-dev / mimir-dev / ratatoskr-dev, all on corviduo-dev's demo+personal instances. Sequence: providers.yaml boot-gate pre-sync → b168 deploy → DCC #384 reindex → round-2 full re-ingest → #381 restart → operator-approved production dedup sweep. Landed clean; three of MY foot-guns along the way, each caught + hardened into a fleet runbook rule (see Tried-and-abandoned: `mv -t`, `docker exec -u 1000`, shared-containerd race).

## providers.yaml pre-sync (boot-gating config)
b168 (commit `293f8f3`) added a `summarization` capability block that in-image `agents/muninn/config.yaml` references → boot-blocking if the host bind-mounted providers.yaml lacks it. Synced both hunks (summarization block + deep-reasoning desc) into demo+personal via `deploy-wt-config`; instance-configs commit `53349f8`.
- **deploy-wt-config runbook:** `~/development/worldtree-instance-configs/scripts/deploy-wt-config {diff|deploy|capture} <inst> --file providers.yaml` (per-instance dirs demo/personal/pinned; `deploy` = host write + api/matrix restart + 90s health-gate + auto-rollback; `diff`/`capture` safe). demo+personal providers.yaml are byte-identical.
- **GOTCHAS:** (1) an UNPUSHED source commit → `git show <sha>` 404s and a gitea `raw?ref=<sha>` silently falls back to the default branch; verify the commit exists (`/git/commits/<sha>`) before trusting a fetch, else ask the peer to paste hunks. (2) a peer's hunk paste may be mis-indented (8-space vs the block's 4-space) → invalid YAML; always YAML-validate after a paste-sourced edit.
- **Config-delta pre-sync rule (verified via `docker inspect`):** worldtree containers bind-mount ONLY `config/` host-side (`/opt/worldtree-*/config/` → providers/model_roles/matrix/policies/defaults/env.public = the pre-syncable set); `agents/` (schemas.yaml, prompts) + all code ship IN-IMAGE. So only a `config/*.yaml` change is boot-blocking-pre-syncable; an `agents/`-or-code delta needs NO host pre-sync (CI carries it). b169's schemas.yaml (#387) was correctly no-pre-sync.

## #384 reindex + #381 restart + verify
DCC job `mimir-6351554e8e8f`. Reindex: `sudo docker exec -u 1000 worldtree-personal-worldtree-muninn-1 python -m core.muninn --reindex <job>` (⚠️ MUST `-u 1000` — default-root writes contaminate the uid-1000 KB tree; see Tried-and-abandoned). Then **#381 restart** (stale-Chroma-client fix): `sudo docker restart worldtree-personal-worldtree-api-1` (plain bounce, NO compose up / no image repoint) → healthz/readyz 200 ~25s.
- **Chroma-verify runbook:** `sudo docker exec -i <muninn> python -` (MUST pass `-i` or stdin never reaches `python -`) → `chromadb.PersistentClient('/data/kb/.chroma').get_collection('fiction').get(where={'job_id':<job>}, include=['metadatas'])`. Chroma persists at container `/data/kb/.chroma` = host volume `worldtree-personal_worldtree-kb`.
- **Retrieval-visibility check (NOT grounding — that's ratatoskr's):** a Mimir session — admin token `~/.config/worldtree/personal-admin-token` (wildcard scope) → POST `/sessions` (agent_id=`mimir`, `record_tool_intermediates=true`) → POST `/sessions/{id}/messages` (STREAMS SSE, not JSON) → parse SSE `tool_result` for `search_library` wing hits → DELETE session.

## Production dedup sweep (operator-approved)
Deleted the 785 April-era DCC orphan rows (`job_id=b59c147c5ce0`, no wing/source_identity metadata → predate identity tracking) from the `main` collection. Supervised protocol: read-only verify count == 785, back up all rows (ids+docs+embeddings) to `corviduo-dev:/tmp/main-sweep-backup-b59c147c5ce0.json` (reversible), `main.delete(where={job_id})` (assert target==785 first), verify `main` 4009→3224, then **bounce the api** (a separate-process delete leaves the api's in-memory HNSW index holding the vectors until reload — the #381 pattern generalizes to deletes), confirm search now fiction-only. Backup left for /tmp natural cleanup (fiction wing is canonical; `~/archives` has the historical record).

Result: fiction wing 166 → 1,372 concepts; three consumer verify rounds 0/5 → 5/5 → saturated; #385 budget fix validated (705 vs April's 785 control, extraction AND indexing, zero truncations). worldtree-dev filed #388 for a deploy concurrency-lock (the shared-containerd race fix). See [[2026-08-02-mimir-inbox-arc]].
_Archived 2026-08-18._

`[2026-08-02]` **The mimir-inbox / #377-read-path arc — deploy, four bugs found+fixed+verified, a cloned voice, all in one long session (2026-08-01→02).**

The browser-facing half of the #377 Muninn ingestion arc, end to end: mimir-inbox stood up, the write path proven, the read path chased through four defects to a verified-working state, and a character voice cloned into the TTS zoo. Peers: mimir-dev (the app), muninn-dev (gate/watcher spec), worldtree-dev (Worldtree app layer + the #380/#381/#382/#383 fixes), ratatoskr-dev (a consumer + the rigorous verifier).

## mimir-inbox deployed (#377)
- **New infra-ops stack, canonical eshpfi `stacks/mimir-inbox/`; live corviduo-dev `10.250.50.152:8091`** (co-located w/ muninn-gate :8090 + the worldtree-personal muninn watcher). Full deploy detail + procedures → auto-memory `reference_mimir_inbox_deploy`.
- **Placement decision (operator, reversed):** 7-31 he ruled mimir-inbox stays OFF corviduo-dev (shared/NFS mount); 8-01 he REVERSED to CO-LOCATE. Trigger: muninn-dev's code-check showed staging is NOT same-fs-constrained (gate reads staging metadata + passes path strings; `os.replace` is inside `ingestion_root`) — staging's real constraint is **path-identity across writer/gate/watcher**, which co-location buys outright while dodging NFS failure modes. I HELD the reversal for the operator's direct word (data/hosting on a team-managed box, reversing his own ruling) even against 3 peer relays — vindicated as the right instinct; muninn-dev agreed.
- Build: **`uv sync --no-dev --frozen`, SINGLE-STAGE** (project installs editable-linked to `src/`, so src/ MUST stay beside .venv — a multi-stage "copy only .venv" dies at import/404s assets). uid 1000, host-net bind 10.250.50.152:8091, TCP-liveness healthcheck (deliberately NOT gate-coupled). Redeploy = refresh build context (**preserve the on-server `.env`!**) → `docker build -t mimir-inbox:0.0.1 -t mimir-inbox:<sha> .` → `compose up -d`. Version stays 0.0.1 across dev commits → tag the image w/ the source SHA too. Live commit progression `0478452`→`c8ab38f`→`2dcc77e`→**`8ece117`** (3 redeploys).
- mimir-inbox key on the gate bumped [read,submit]→**[read,submit,control]** (cancel/retry); brokered via a 0600 drop on nh3-dev (never on the althing bus).

## The read-path bug chain (worldtree-dev's, all found via this arc)
- **#380 wing-blind indexing:** the book-ingest path upserted concepts into a hardcoded `main` Chroma collection while wing search reads the `fiction` collection → P&P written to disk but `search_library` returned total 0. A silent-success defect ("complete/69 indexed" was right about the WRITE, wrong drawer). Root-caused off MY physical evidence (files on disk + search empty). Fixed b164 + a one-shot `--reindex <job_id>` (re-upsert into the right wing collection + delete stray `main` rows).
- **#381 stale Chroma client:** the personal api opens its Chroma client before the watcher's cross-process writes → **a freshly-ingested/re-indexed book is NOT queryable until the api is restarted.** Proven by my restart-diagnostic (pre-restart total 0 → post-restart hits, same index). Workaround until fixed: `docker restart worldtree-personal-worldtree-api-1` after any ingest/re-index. Filed as #381.
- **#382 unreliable Mimir grounding (the subtle one):** post-#380-fix the index was correct, but Mimir's grounding was INTERMITTENT — some sessions navigated the opaque job-hash dir (`mimir-f3887c9b97b7`) to the content, others distrusted the correct vector hits and **silently answered from training knowledge** (worst of the looks-fine-isn't family). ratatoskr-dev caught it; I'd been over-confident ("Mimir read Austen back to you") having verified the INDEX, not the GROUNDING. Fixed b166 with BOTH shapes: a self-describing `_index.md` per wing job-dir (resolves the hash dir to its title) + a Mimir prompt rule (wing-scoped hits ARE library content, never discard on a name mismatch, never substitute training). **Verified: ratatoskr-dev re-ran 3× fresh sessions → 3/3 grounded**, citations in note-extracted language not raw Austen. #382 CLOSED.
- **DCC (Dungeon Crawler Carl, job `b59c147c5ce0`) backfill:** `--reindex` FAILED ("job not found in any state dir" — predates state-tracking). SETTLED = **no re-file** (the b166 prompt rule already grounds it even without an `_index.md`; ratatoskr confirmed incidentally); an `_index.md` rides whenever DCC is next re-ingested.
- **#377 mimir-inbox banner bug (mimir-dev's, `8ece117`):** `/health-banner` misattributed an unwritable `ingestion_root` to the WORKER, rendering "The worker is not running." for a running worker — a false lead pointed at infra-ops's half of #377. Fixed (guard split into two banners); I confirmed from the DEPLOYED handler (not just the test) that `ingestion_root_writable:False` now renders "The ingestion root is not writable."

## muninn-gate → muninn-dispatch 0.1.5
Rebuilt `muninn-gate` off `vh/muninn-gate` main `bc04c4c` (dispatch 0.1.4→0.1.5) so the gate serves the new `concept_schema`/`concept_schema_source` row fields (computed gate-side). Gate version unchanged 0.0.14 (dual-tag the SHA). Build needs the vh gitea token as a BuildKit secret (`--secret id=gitea_pw`, UV_INDEX_GITEA_USERNAME=vh, drop+shred). Recreate with `compose up -d` (NOT bare restart — needs the new image). Verified: P&P job serves `concept_schema='fiction'`, `concept_schema_source=null` (null correct — pre-b164 job). Registry tags by commit SHA — `v1.0.0bNNN` docker tags don't resolve; use the deployed SHA (confirm `--reindex` present before using an image for a data-op).

## donut voice (65-frost → Zonos gateway)
Operator: "pick up 65-frost, use that bundle as a voice for a character named donut." 65-frost = a **Booth id** (`~/booth-data/65-frost/`) holding a curated yt-voice-clipper dataset (`dataset-…-curated.zip`: 4 clips + manifest, all SPEAKER_02 = Princess Donut). **Zonos gateway voice registry = a filesystem drop:** `<Name>.wav` in the voices dir (44.1kHz mono s16 PCM) auto-registers as `voice:"<name>"` on **startup** (needs a restart). The LIVE dir is the bind mount `/opt/docker/compose/zonos-gateway/voices/` (lkraven-writable), NOT the working tree. Built `Donut.wav` from seg000 (best clip), dropped it, restarted → `voice:"donut"` live in the gateway AND the Asset Engine's make form. Also copied to the build-source tree `~/zonos-gateway/voices/` for rebuild-durability (true canonical = the gitea repo, not yet CI-wired). Auditioned in booth `donut-voice`. **Expanded 2026-08-02 (onyx-58 bundle):** operator curated a 2nd Booth bundle `onyx-58` (`dataset-467d2cf8…curated.zip`, 3 Donut clips) as additions. Rebuilt the reference = **seg000 (65-frost) + seg101/seg110/seg148 (onyx-58)** ffmpeg-concat + resampled 24k→44.1k mono s16 = **52.0s**. `seg148` was diarized SPEAKER_03 but is Donut (operator-confirmed misdiarize → included). Assembly is NOT `assemble_voice.py` (that `-c copy` can't resample + caps ~15s); used a manual `aresample=44100,aformat=…,concat=n=4` filter. Backed up old ref → `irv-ml1:~/Donut.wav.pre-onyx58`; dropped to live bind-mount + build-source tree; `docker compose restart` (healthy 2s, `voice:"donut"` still 1 of 9). A/B booth `donut-onyx58` (A=old 16.3s ref, B=new 52s ref, same line). Longer ref is fine mechanically: gateway passes it as `speaker_audio_base64` → speaker *embedding*, not an audio prefix. **BUT auditioned → REVERTED same day:** pinned-seed neutral A/B (5 pairs, booth `donut-onyx58`) showed the single-clip seg000 (16.3s) beats the 52s 4-take concat on timbre — concatenating disparate takes muddied the embedding more than the range helped. Reverted both live + build-source to seg000-alone. Lessons (→ Tried-and-abandoned): more reference ≠ better when takes vary; and **emotion steering pulls output away from the clone fast** (operator craft rule) — keep clones emotion-neutral; bare `{input,voice}` calls send NO emotion (gateway only enables it on an explicit `emotion_*`/`preset` dial).

## Zonos streaming (no gateway change needed)
ratatoskr wanted play-as-it-arrives. `/v1/audio/speech` ALREADY streams — chunked `StreamingResponse`, opens native `/tts/generate` with `stream=True`, wraps as a streaming int16 WAV with `0xFFFFFFFF` placeholder sizes (meant for progressive `<audio>`). Verified TTFB 0.44s vs 6.84s total, `transfer-encoding: chunked`, dials preserved. ratatoskr's proxy was rewriting the placeholder header → forced buffering. Fix was theirs (pass chunks through); shipped + confirmed (TTFB 0.46s progressive). The Asset Engine (ana-docker:8200) IS the fleet "TTS zoo" (~20 audio svcs w/ irv-ml1 endpoints); zonos-gateway registered there, state=ready.

## Lessons (also in Tried-and-abandoned)
- **Verifying the INDEX (search returns hits) is NOT verifying GROUNDING** (does the agent trust+use them vs. silently answer from training). Check that citations are note-extracted, not model-knowledge. ratatoskr caught this after my over-confident "it works."
- **Reading the DEPLOYED artifact > trusting the test** for "is the fix live" — the test proves the source is right; reading the running code proves the artifact is, which is what an on-call actually meets.
- Held a boundary-box/data reversal for the operator's DIRECT word against 3 peer relays — the right call (peer relay ≠ operator consent; the placement guard was vindicated).

See also: [[2026-07-31-muninn-gate-deploy]]. auto-memory: `reference_mimir_inbox_deploy`, `reference_muninn_gate_deploy`, `reference_muninn_gate_staging_path`, `reference_zonos_tts_stack`, `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
_Archived 2026-08-18._

- `[2026-07-27]` **jackdaw-compose.service DECOMMISSIONED** (jackdaw-dev request; the JackDAW AI Composer was cut from v1 by operator decision 2026-07-27). Stopped + disabled the nh3-dev `:8787` user service (no client calls it — ai/server/AiChat deleted from main, `/compose` proxy removed); unit **archived not deleted** → `~/.config/systemd/user/jackdaw-compose.service.decommissioned-20260727` (revival = rename + `daemon-reload`). **No credential revoked** — the unit used the SHARED all-agents LiteLLM key (`sk-eA_XOd…`, model `gen`), not a dedicated one. Code preserved on jackdaw `origin/ai-composer-preserved`; treat as permanent. The `:4500` HTTPS audition bench is untouched. (Supersedes the 2026-07-23 stand-up line below.)
_Archived 2026-08-18._


## Tried and abandoned (archived)

- `[2026-08-02]` **donut voice multi-clip reference (onyx-58 expansion) — TRIED, REVERTED.** Folded the `onyx-58` bundle's 3 Donut clips (seg101/seg110/seg148) in alongside the original seg000 → a 52.0s 4-take concat reference, hoping a longer ref → more robust speaker embedding. A pinned-seed A/B (5 pairs, varied registers, booth `donut-onyx58`) showed the **original single-clip seg000 (16.3s) sounds better** — concatenating disparate takes muddied the timbre more than the extra range helped. Reverted to seg000-alone (live + build-source). **Two durable lessons:** (1) for a faithful clone, a single clean representative take can beat a longer multi-take concat — more reference audio is NOT automatically better when the takes vary. (2) **Emotion steering pulls the output AWAY from the cloned voice fast** (operator's craft rule) — keep donut (and clones) emotion-neutral for fidelity; the gateway only enables emotion when an `emotion_*`/`preset` dial is explicitly sent, so bare `{input,voice}` calls stay pure-clone. `seg148` was diarized SPEAKER_03 but IS Donut (operator-confirmed misdiarize). onyx-58 curated bundle lives in booth `onyx-58` (24h TTL — stash to `/mnt/smithy/voice_clones/` if a future middle-ref experiment is wanted).
_Archived 2026-08-18._

- `[2026-08-02]` **Verifying the INDEX is not verifying GROUNDING** (#382). A `search_library` returning wing=fiction hits proves the content is *retrievable*; it does NOT prove the agent (Mimir) *trusts and uses* those hits vs. silently answering from training. I reported "Mimir read Austen back to you" off a grounded-*looking* answer; ratatoskr-dev caught that grounding was intermittent (some sessions discarded the correct hits and substituted training knowledge). Test the harder claim — are the citations note-extracted or model-knowledge? — and reading the DEPLOYED artifact beats trusting the test for "is the fix live."
_Archived 2026-08-18._

- `[2026-07-30]` **brokkr's WebSearch "verification" CONFIRMED a hallucination — 3 phantom `microsoft/Mage-Flow-{Base,Turbo,Edit}` repo IDs.** brokkr-smithy-dev handed 3 gated-looking repo IDs for an operator-directed model pull; they don't exist (its own web-search fabricated an arXiv ID + project page, twice). Lesson: the HF **registry API is ground truth** — an unauth 401 ≠ exists (`{"error":"Invalid username or password"}` masks private/gated/nonexistent alike), an authed 404 = phantom, and `author=X&search=Y` refutes existence. API-verify every repo ID before a pull; LLM-summarized web fetches confabulate. auto-memory `reference_verify_hf_repo_ids_before_pull`.
_Archived 2026-08-18._

- `[2026-07-30]` **magpie TTS serving — evaluated, ABANDONED.** Pulled `magpie_tts_multilingual_357m` (the one real repo of brokkr's batch) to NFS, stood it up on irv-ml1 (ephemeral NeMo-Speech-`main` container — stock PyPI/NGC NeMo can't load v2607), A/B'd vs Zonos → Zonos wins expressive English decisively, multilingual not needed. Not served; `magpie-nemo` torn down. `.nemo` KEPT on NFS as brokkr's fine-tuning base. auto-memory `project_magpie_tts_eval_rejected`.
_Archived 2026-08-18._

## Recent decisions (archived 2026-08-19 batch)

`[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap; fix relayed to heid, NOT applied gateway-side.**

heid reported that `kimi-k3` (the primary route = Kimi Code coding endpoint `openai/k3` @ `api.kimi.com/coding/v1`) silently degraded its cross-frontier panel: on large/reasoning-heavy dispatches, `completion_tokens: 16381` **exactly**, `content` empty, `reasoning_content` ~64KB, `finish_reason: **stop**` (a truncation mislabeled as a clean stop). `max_tokens: 100000` in the request was not honored.

**Investigation arc (a clean cross-frontier-triage + verify-on-the-wire case):**
1. My first read: a flat ~16384 OUTPUT cap; fix = a LiteLLM `stop→length` relabel callback (heid's fallback ask). Confirmed the cap isn't in our LiteLLM config (no `max_tokens` clamp on the route).
2. Operator routed a fix-research pass to **dvalin-smithy-dev + bil-smithy-dev** (independent). Both CONVERGED (docs-based): `max_tokens` is a deprecated alias on Kimi/Moonshot; the canonical field is `max_completion_tokens` (default 131072, max 1M); the coding endpoint defaults output to 16384; fix = send `max_completion_tokens` + `reasoning_effort` via `extra_body` (drop_params-safe).
3. **heid's live data REFUTED the docs hypothesis:** a later dispatch hit `completion_tokens: 18455` (ABOVE 16384) cleanly, with `reasoning_tokens: 16198` (just under 16384) and content present. So COMPLETION is uncapped; the bound is on **REASONING at ~16384**. When a hard task's thinking exhausts that budget, nothing's left for content → empty answer under `stop`.
4. **I proved it on the wire** — ran heid's real 500KB failing bundle direct at both endpoints (bypassing LiteLLM so `reasoning_effort` isn't dropped): default effort → 504/timeout (the failure); **`reasoning_effort: low` → reasoning ~12–13.5k (under the ceiling), content returns (6–7.6k chars)**, on BOTH coding AND general endpoints. So re-routing to the general endpoint buys nothing — the fix is the effort param, and it works on the wire.

**THE FIX (caller-side, no shared-gateway change/restart):** send `reasoning_effort` via **`extra_body`** on kimi-k3 dispatches (`low` for large bundles). LiteLLM `drop_params: true` strips the top-level `reasoning_effort` — which is exactly why heid's earlier `reasoning_effort: low` was a no-op. `extra_body` survives drop_params (the house GLM-thinking pattern). Tradeoff: low effort = shallower reasoning, but a complete answer beats today's empty one.

**Relayed to heid to validate on a real round** (the one unconfirmed hop is whether `extra_body` survives OUR LiteLLM). **Backstop if it doesn't:** add `allowed_openai_params: ["reasoning_effort"]` to the `kimi-k3` route in the gateway config — that IS a shared-gateway change + a ~10s restart (blips all consumers), so it needs a heads-up.

Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/litellm/conf/config.yaml` (see Recent-decisions `[2026-07-25]` Kimi K3 wiring). No gateway change was made this session. Failing dispatch on record: `01KYTASKTY3T` (jackdaw-dev bug-hunt).
  _Archived 2026-08-19._

- `[2026-07-25]` **bil-smithy-dev wired as an althing zellij-window-ping (pane route).** She's a `driver: human` dwarf peer (pane `bil-smithy` already live alongside eitri/dvalin/regin-smithy in the `Claude` zellij session) but had no delivery route → smoke messages posted to the bus but never reached her window. **Mechanism (reusable for any pane-route handle):** `~/.althing/config.yaml` → `zellij_sessions.Claude.agents[]` maps `handle` → `target` (a zellij pane **TITLE**, matched via `list-panes -j` in `althing/zellij.py:resolve_pane_id`) → `command` (herald `write-chars` + CR into that pane). The **herald loads config ONCE at startup** (`herald.py main()`), so **`systemctl --user restart althing-herald.service`** after editing. Added bil (`target: bil-smithy`), restarted, verified: herald delivered the pending smoke `01KYD7W7CF…` (available→attempted→**delivered**). ⚠️ Noticed pre-existing pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range", empty msg_ids — likely `render_command messages[0]` on an empty list; NOT caused by this change, bil works) — worth a herald look.
  _Archived 2026-08-19._

## Tried and abandoned (archived 2026-08-19 batch)

- `[2026-08-02]` **`docker exec` into worldtree containers defaults to ROOT — root writes contaminate the uid-1000 (vh) KB tree.** My `sudo docker exec … --reindex` on personal ran as ROOT (muninn app = uid 1000); its wing git-commit + atomic note-swap left root-owned files in the `worldtree-personal_worldtree-kb` volume: a root-owned `.old-<job>` backup dir (blocked the uid-1000 retry's `rmtree` → Errno 13, because unlink needs write on the DIR and it was root:root 755) AND **60 root-owned loose git objects** in `.git/objects/`. Fix (host-side, corviduo-dev): `sudo rm -rf` the superseded `.old-` dir (tar'd aside to /tmp first) + `sudo find … -user 0 -exec chown 1000:1000` the objects (ownership-only, git-content-safe; the `.git/objects/XX/` dirs were vh-owned so these weren't a hard blocker, but violated "clean tree"). **RUNBOOK RULE (worldtree-dev, ADOPTED):** any `docker exec` into worldtree containers that WRITES pipeline state runs **`-u 1000`**, never default-root — same genus as the mv footgun (acting without matching the target's constraints; 3rd such slip in one session). **GOTCHA that hid the scope:** `find … -user 0 | head -20` TRUNCATED (the `.old-` dir alone had 153 files, so the first page was all `.old-`) → I "verified clean" off a partial list. Never `head` a scope-defining find; count first (`| wc -l`). **Related blind-spot (muninn-dev):** a root-owned job SUBDIR passes every requeue guard (job_row/dispatch/list_jobs render fine) AND `/health` (contract's `os.access(ingestion_root, W_OK)` tests only the ROOT dir, so a foreign-owned subdir under `pending/` still reports `ingestion_root_writable: true`) — then the uid-1000 gate can't write into it. "Clean board + green /health + failure at next mutation." muninn-dev added an OWNERSHIP column to the standing post-move check to catch it; two green signals both miss a foreign-owned subdir otherwise.
  _Archived 2026-08-19._

- `[2026-07-25]` **Chaining the althing wake-listener arm orphans it.** `reply && althing-wake-listener &` (or spawning `althing-wake-listener` with `&` *inside* a `run_in_background` task) → the `&`-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a *plain operator turn* (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn `althing-wake-listener` as its OWN `run_in_background` task, and re-arm ONLY after a real fire (`<task-notification> completed rc0`). Reclaim an orphan with `althing-cli stop-monitor` then re-arm.
  _Archived 2026-08-19._

## Recent decisions (archived 2026-08-20 batch)

- `[2026-08-05]` **Booth — 3 features shipped, live on `:8090` + tagged.** (1) verbatim-`index.html` booths get a floating top-right "‹ all booths" chip + inherited favicon, doctype/charset-safe byte-injection (`booth-v0.1.5`, `8577e7e`); (2) `.md` renders + `.txt`/`.log` view in-booth without downloading via the `/b/<n>/view` route + a `markdown` dep + `doc.html` (`booth-v0.1.6`, `315faac`); (3) prev/next arrows in the image zoom viewer — wrap-around + keyboard ←/→, hidden for single-image booths (`booth-v0.1.7`, `c37a425`). Canonical `services/booth/`; deploy = `systemctl --user restart booth.service` on nh3-dev (runs from the checkout's `.venv`; `uv pip install` new deps into it first); 47 tests. `uv.lock` gitignored (`348c5c1`).
  _Archived 2026-08-20._

- `[2026-07-31]` **worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred.** memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on `vh/worldtree-sdk` (source pulled via the **Gitea API archive** — git-HTTP 403s on that repo); publishing to the vh USER namespace **can't be delegated** (401 `reqPackageAccess` even with `write:package`) so it needs an owner token — operator saved a **FULL vh site-admin token at `~/.config/gitea/vh-token` (0600)** for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
  _Archived 2026-08-20._

## Tried and abandoned (archived 2026-08-20 batch)

- `[2026-08-03]` **corviduo-dev shared containerd: a concurrent-pull race fails ONE instance's deploy; DON'T "prune to fix" — the image is in-use by the instance that won the race.** b169 personal deploy failed at `docker compose pull` (`Lchown … no such file or directory` on the big torch layer → looked like disk pressure / corrupt snapshot). ACTUAL: NOT disk (56G free, inodes 7%). demo + personal + pinned share ONE `/var/lib/containerd` on corviduo-dev; demo (from main) and personal (from staging tag) extracted b169's shared torch layer simultaneously → personal's hit a partial snapshot mid-race and aborted while demo's completed. The image `6e34a87` was FULLY VALID — demo was RUNNING it healthy. Fix = just re-run the failed deploy (image already materialized; compose pull finds it present). **NEAR-MISS:** worldtree-dev's suggested "prune unused images/snapshots" would have rmi'd `6e34a87` = the image the running demo depends on → demo outage. **Lesson: before any prune/rmi "cleanup," `docker ps` the running images — an "unused" image may be a co-tenant's live one; and verify the failure's REAL cause (disk? inode? in-use? race?) before applying the suggested remedy.** (Pipeline fix, deferred: serialize demo-from-main + personal-from-staging, or a per-image pull lock, to avoid the shared-layer extraction race.)
  _Archived 2026-08-20._

- `[2026-08-02]` **`mv <job> complete/ → failed/` RENAMED the job to `failed` because failed/ didn't exist.** worldtree-dev's round-2 unblock command (`mv /data/state/ingestion/complete/<job> /data/state/ingestion/failed/`) assumed `failed/` existed; on PERSONAL muninn it did NOT (fresh instance — root was `active/ complete/ pending/ sources/`, no `failed/`). `mv src nonexistent/` **renames** src→nonexistent, so job1 became the `failed` dir and job2 nested inside it. Caught on post-move `ls` (failed/ held job *contents*, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt `failed/` (worldtree:worldtree 755) — NO data loss. **Lessons:** (1) before `mv X into-dir/`, verify the dir EXISTS (`[ -d dir ]`) — an empty `ls dir/ 2>/dev/null` is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is **`mv -t <targetdir> <src>`** (`--target-directory`): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a **trailing slash does NOT protect** — `mv src failed/` with `failed/` missing STILL silently renames to `failed` (rc=0); "just add the slash" is a false guard. (`mkdir -p failed/` first also works, but `mv -t` inverts the failure from silent-wrong to loud-safe in one flag.) Container `sh` is dash — no `(` in echo strings. **SILENT failure mode (muninn-dev carry-forward):** a misplaced ingestion-state move doesn't crash anything — `list_jobs()` stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (`job_row`→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (`job_row` found + guards pass), don't trust mv exit codes — and confirm `job.dispatch.json` survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. **DON'T TIDY (round-2 pending):** both DCC + P&P jobs currently REST in personal `failed/` with manifests reading `state: complete` until round-2 requeue runs — deliberate + load-bearing (`requeue` keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. **Round-2 sequencing:** the requeue is **mimir-dev's** browser flow (pending their operator's board-vs-API ruling); **muninn-dev** is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); **infra-ops** = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z. **ARC COMPLETE (2026-08-03 ~05:49):** both books terminal — DCC `mimir-6351554e8e8f` 705 concepts + P&P `mimir-f3887c9b97b7` 667, extracted AND indexed, 5/5 phases, 0 failures/truncations (validates the #385 budget fix vs April's 785 control); **#381 restart-after-ingest FIRED** (personal api, healthz/readyz 200 ~25s), retrieval-visibility confirmed (search_library returns DCC+P&P from fiction post-restart); handed ratatoskr-verify go to worldtree-dev. **Delete-sweep precondition NOW MET** — the stale DCC rows in `main` are genuine duplicates of live `fiction` rows, so worldtree-dev's supervised sweep of the ~785 April orphans is unblocked (still comes to me supervised: snapshot + operator-in-loop).
  _Archived 2026-08-20._

`[2026-08-10→12]` **secrets-broker — per-box Vaultwarden credential store, SHIPPED + consumer-confirmed.**

**What.** A per-dev-box credential store over the fleet Vaultwarden (`vaultwarden.phasefinal.com`,
on ana-docker, DB on pfi-postgres, in the pg_dump backup set). The `secret` CLI at eshpfi
`services/secrets-broker/secret` (also installed to `~/.local/bin/secret`, on PATH for all sessions):
`put / get / list / rm / backfill`. Stores into the **`infra-ops` org's Default collection** (org
shared to the operator's primary account, so he sees items too), folder = hostname, item name =
`<host>/<path>`, title-derived slug. Small text → item note; small binary → base64 hidden field;
**>6000 B → a bw attachment** (Vaultwarden caps notes at ~10000 encrypted chars); sha256 + source
metadata fields; idempotent upsert keyed by name.

**Auth.** Bootstraps from `~/.config/secrets-broker/bootstrap.env` (0600): apikey login
(`BW_CLIENTID`/`BW_CLIENTSECRET`) + master-password unlock (`--passwordenv`) → per-invocation
session. That file is **secrets-zero** (it unlocks the vault, can't live in it) and is excluded from
backfill.

**Client = `bw`, NOT `rbw`.** rbw was the operator's first choice but its `register` returned an
undebuggable 400 against this Vaultwarden despite valid creds (a direct `client_credentials` grant +
both prelogin paths return 200; rbw emits no HTTP logs). Switched to the official `bw` CLI
(user-prefix npm install) — clean unattended flow, full write support (org collections + attachments).

**Backfill.** Local-only (each box backs up itself; NOT a fleet daemon). Scanned nh3-dev's
`~/development/*/{env.sh,.env}` + `~/.config` credential files, **25 items stored + round-trip
verified** (2 large via attachment). Excludes bootstrap.env / `.example` / `~/AIPA-Data` archives /
cargo noise.

**Post-launch (jackdaw-dev feedback).** Added **`secret rm <name>`** (bw soft-delete to trash,
recoverable) — closes the "no delete path, append-only" gap; and a **new-top-level-namespace warning**
on `put` (stderr, non-blocking) — catches a typo'd/missing host prefix at store time. Chose
warn-not-auto-prefix because domain-scoped names (`gitea/…`, `certs/…`) would misfire on auto-prefix.
Deferred edge recorded in the contract: the warning is non-blocking, so a scripted put suppressing
stderr can still mis-namespace — add an opt-in `--strict` only if scripted callers appear.

**Standing directive (now GLOBAL in `~/.claude/CLAUDE.md`):** the vault is the credential source of
truth — **`secret put` durable secrets into it AND `secret get` the creds a task needs FROM it**
rather than reading on-disk copies. Dogfooded by pulling the gitea `vh` token from the vault to create
`vh/stonehenge-park`.

**Deploy shape.** Not a service / no daemon — per-box; a new dev box duplicates the stack
(`services/secrets-broker/README.md`): npm-install `bw` to `~/.local`, drop a per-box `bootstrap.env`,
`secret backfill`. Commits: `41359ea` (CLI + contract), `850a197` (backfill 25/25 + attachment +
resilient run), `a249073` (rm + namespace warning), `a1304b7` (deferred-edge contract note).
Consumer-confirmed end-to-end by jackdaw-dev.

Auto-memory: `reference_secrets_broker_cli`.
_Archived 2026-09-03._

- `[2026-08-19]` **`claude-bot` granted read on `vh/waterland`** (operator-empowered, verified `admin:false push:false pull:true`) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit `8189076`.
_Archived 2026-09-03._

# `[2026-08-19]` esh-pve hard-froze for 4.5h — and took the whole house's DNS with it

Reported by the operator as "routing or DNS issues on the PVC wifi." It was
neither: the internet was healthy the entire time (gateway reporting 3 ms and
209/26 Mbps; 1.1.1.1 and 8.8.8.8 answering at ~3 ms from inside ESH with zero
loss). **The house had no name resolution because one VM was down.**

## The SPOF: one resolver, cross-VLAN, no fallback

`esh-userland` (VLAN 10, `10.0.10.0/24` — the `PVC` SSID *and* the wired
userland LAN) handed out **exactly one DNS server, `10.0.50.45`** — AdGuard, on
`esh-docker-vm`, on the **server** VLAN. No secondary. That VM dies, every
client on the VLAN loses DNS, and it presents as "the wifi is broken."

It was the only network in the house exposed this way. `Default`, `esh-mgmt`,
`esh-server` and `esh-cameras` run DNS on auto (the gateway hands itself out);
`esh-iot` and `ESH-WG` point at 1.1.1.1 + 8.8.8.8.

**Fixed** (operator-approved): `esh-userland` now hands out `10.0.50.45`
primary, **`10.0.10.1` (the gateway) secondary** — the UDM's own resolver,
verified answering. Applied via the Classic API,
`PUT /proxy/network/api/s/default/rest/networkconf/687985eae5d15b673cef1a73`
with the full object (GET → modify one field → PUT), `rc: ok`. **This was also
the first confirmed WRITE on the ESH UDM key** — previously only the NH3 key
was write-tested. See [[reference_unifi_udm_integration_api_keys]].

⚠️ **A secondary is not clean failover.** macOS/iOS query resolvers in
parallel, so once AdGuard is back a real share of lookups go to the gateway and
**skip ad-blocking**. This converts a total outage into degraded-but-working.
The actual fix for blocking integrity is a second AdGuard instance NOT on
esh-pve.

## Root cause: hard freeze, no diagnostics, two suspects

`esh-pve` (Minisforum MS-01, i9-13900H, `productname: YajuuSenpai`) froze at
**03:34:39**. The journal stops mid-operation — **no panic, no OOM, no MCE, no
thermal event**. Powered on with its 10G link up, but not answering ARP.

Two changes landed the day before, and they are not exclusive:

1. **New kernel.** A large `apt` batch on **2026-08-18 07:00:21** installed
   `proxmox-kernel-6.8.12-42-pve`; clean reboot at 07:08:44. Before that the
   box had **4.5 months of uptime** (Mar 30 → Aug 18) on `6.8.12-16`. First
   boot on the new kernel lasted **20 hours**.
2. **GPU passthrough.** The last kernel messages of the dead boot are
   `vfio-pci 0000:01:00.0/.1: enabling device` at **02:55:17** — VM 102
   `esh-vm-workstation` starting with `hostpci0: 0000:01:00,pcie=1,x-vga=1`,
   **39 minutes before the freeze**.

A vfio/i915 regression in the newer kernel would produce exactly this
signature. `6.8.12-16` is still installed and is the held-in-reserve rollback.

**VM 102 is now pinned off** (`qm set 102 --onboot 0`, stopped) per the
operator — it is on-demand and there has been no demand. That removes the
suspect without a kernel rollback.

## Why nobody could recover it remotely — and the fix

Nothing on the box could reboot it:

- **`softdog` was the loaded watchdog.** A *software* watchdog cannot rescue a
  hard kernel freeze: the frozen kernel is the thing that would have to fire
  its timer. This is the trap — the machine *looked* watchdog-protected.
- **Proxmox's `watchdog-mux` held `/dev/watchdog` but never armed it.** It only
  pets the device while an HA client is connected, and this cluster has no HA
  resources.
- **vPro/AMT was unusable.** The MS-01 reaches the network only via **SFP+**
  (Intel X710, port 27 on the Garage switch) and presents exactly one MAC.
  **AMT cannot ride a discrete/SFP+ NIC** — it needs the chipset-integrated
  Intel PHY, i.e. one of the two i226 RJ45 ports, and both are unplugged.
  Cabling one and provisioning AMT in MEBx remains the open item for *control*;
  the watchdog below is the fix for *recovery*.

**Fixed:** `playbooks/esh-pve-hardware-watchdog.yaml` — systemd now owns the
PCH hardware watchdog (`iTCO_wdt`, `RuntimeWatchdogSec=60`), `softdog` is
blacklisted and unloaded, `watchdog-mux` is masked. Verified live:
`watchdog0: identity=iTCO_wdt state=active timeout=60s`, held by PID 1,
journal `Using hardware watchdog 'iTCO_wdt', version 6`. Playbook re-run proves
idempotency (6 skipped / 6 verify OK).

Firmware does **not** block the TCO timer here — checked for the
`unable to reset NO_REBOOT flag` line before committing to the approach; the
board reports `Found a Intel PCH TCO device (Version=6, TCOBASE=0x0400)`.

⚠️ **Masking `watchdog-mux` trades away HA fencing.** If Proxmox HA is ever
configured on esh-pve this must be reverted. Not a near-term concern:
`esh-pve-cluster` is **two nodes with no qdevice**, so a single node loss
already costs quorum and the survivor would fence itself — HA here would reduce
availability, not raise it.

⚠️ **The watchdog is configured and armed, but has NOT been proven to fire.**
Proving it means deliberately wedging the host. Untested-but-armed is still
strictly better than softdog; treat a real firing as unconfirmed until tested.

## Diagnostic corrections worth keeping

- **"No route to host" was the dead host, not a routing gap.** Two claims made
  mid-incident were wrong: that the mgmt VLAN (`10.0.250.0/24`) is not routed
  over the NH3↔ESH tunnel, and that a firewall isolates it from the server
  VLAN. Both were artifacts of esh-pve being dead. With it up, `root@esh-pve`
  SSHes fine from nh3-dev at 7.5 ms, and `10.0.250.1` answers from
  `esh-pve-nas` in 0.078 ms. **Control-test against a *different* host on the
  target subnet before concluding "the subnet is unreachable."**
- **UDM `uptime` on a client record is association time, not host uptime.** It
  read 2.2 days while the host had been up 20 hours. Use
  `journalctl --list-boots` on the host for real boot history.
- **`rest/user` `last_seen` is not maintained** (it read ~203 days for hosts
  that are demonstrably online). `stat/sta` is the live view.
_Archived 2026-09-03._

- `[2026-08-21]` **speaches ASR live on irv-ml1 for Eyra — and `no_speech_prob` alone is a weak hallucination gate.** Silence and room tone both hallucinated "Thank you." under 0.11; `avg_logprob` separates ~6× better. Consumers should gate on a composite. (`aa5863c`, `c7e2187`)

_Archived 2026-09-05._

# `[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight

The real work of the session: abliterate `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1`
using the MTP-aware, vision-preserving **Robinson formula** (documented in
`docs/pfi/abliteration-recipe-qwen38.md` from `RobinsonLabs/Qwen3.8-27B-abliterated`).
Harness: `services/coldfusion-abliteration/`. Runs on ana-ml2.

## Why this model, why abliterate it ourselves

Stock Cold-Fusion's refusal profile was **probed 2026-08-19** (Q6_K GGUF on
llama.cpp, 24-prompt battery, hand-verified after a keyword-classifier bug):
**~33% creative refusal**, concentrated on **explicit-sexual + graphic-torture**;
4/5 hard-harm technical refused; self-harm guardrails intact 3/3; benign
over-refusal 0. So there is a real creative-content refusal surface to remove.
This **supersedes** the earlier "watch for DavidAU's own heretic build" posture —
we abliterate it ourselves.

**It is additive over the current gen seat.** The live Heretic seat
(`qwen38-27b-heresy-bf16`) left its MTP head a **byte-identical base graft** —
the `Qwen3_5ForConditionalGeneration` wrapper never loads it, so Heretic could
not touch it. The Robinson formula abliterates the MTP head **in-band** (its 2
residual-write matrices), and the MTP head is what gates speculative acceptance.
That in-band MTP edit is the delta this experiment tests.

## Recipe maps 1:1 — dry-run PASSED

Against the staged bf16: 1199 tensors, 333 vision preserved,
`down_proj=64 o_proj=16 linear_out=48 mtp=2 embed=1`, coverage gate 6/6, exactly
**131** tensors to orthogonalize. Same architecture as RobinsonLabs' base, no
name drift. Two hard gates in the harness halt before any write: the coverage
identity `o_proj(16)+linear_out(48)==64`, and the attention-sink screen on
**dim 3994** (orthogonalizing a direction living there bricks the model).

## Capture SUCCEEDED — but only after a real environment fight (the durable lessons)

**The transformers Qwen3.5 DeltaNet linear-attention NaNs in bf16 on ana-ml2.**
The fast-path needs BOTH `flash-linear-attention` (`fla`, triton, installs fine)
AND `causal-conv1d` (**needs nvcc to build — absent, no prebuilt wheel**).
Without causal-conv1d the DeltaNet short-conv runs the torch fallback, which
produces **nondeterministic all-NaN** hidden states in bf16 (same 11-token input:
finite on one forward, NaN at layer 4 on the next). bf16 and fp32 share exponent
range, so this is **precision-driven catastrophic cancellation, not overflow** —
**fp32 resolves it.** Diagnosed via `diag_nan.py` / `diag2.py`: `sdpa` + plain
prompt = 65 layers all finite; chat-template input = NaN; the trigger is the
input path through the unstable recurrence.

Fixes, all in the committed harness (`7abd301`):
- **`--capture` loads fp32**; the write/surgery path stays bf16 (no forward, no NaN).
- **A finite-gate aborts on a non-finite direction** — the sink screen alone
  can't catch it (`nan > threshold` is False, so a NaN direction "passed" it and
  saved silently on the first run).
- `attn_implementation="sdpa"` pinned.

**fp32 (110 GB) needs the whole GPU.** device_map=auto packed it tight and the
forward OOM'd against the resident seats. Had to **stop three seats** for VRAM:
`vllm-meromero-rp`, `vllm-fablefusion-probe`, and production `vllm-gen`.
⚠ **Restart order matters:** gen restarted into an empty GPU0 and greedily
grabbed 64 GB (vLLM takes a fraction of *free* memory at startup), starving
meromero into a crash-loop. Fixed by bringing **meromero up first**, then gen
into the remainder. All three restored to healthy.

⚠ **fla lives in a side dir, not the venv.** The shared
`/tank/aimodels/quant-work/.venv` is not llmuser-writable; `fla` + `einops` are
`--target`-installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and
reached via `PYTHONPATH`. Prune deps that shadow the venv's torch/transformers.

## Result

Refusal direction: **finite, unit-normed, layer 22**, sink energy **0.0008%**
in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated.
Saved to `/tank/aimodels/qwen38-27b-coldfusion-bf16/refusal-direction.pt`.

⚠ **QUALITY CAVEAT — the reason the next step is calibration-set expansion.**
Two-template `|cos|` agreement at layer 22 is **0.59**, well below Robinson's
0.99. Almost certainly the small calibration set: **8 harmful / 8 harmless**
(HARMFUL/HARMLESS in `abliterate.py`) vs Robinson's **416 / 104**. The direction
is valid and sink-clean but noisier than ideal; abliterating on it risks
under-removing refusals or nicking capability. **Expand the sets to a few
hundred each and re-capture** before the `--out` write.

## Sequence from here

1. **Expand HARMFUL/HARMLESS calibration sets** → re-capture (fp32, seats down).
2. `--out` write (bf16 surgery, no forward) → `qwen38-27b-coldfusion-abliterated-bf16`.
3. Verify: vision byte-identical, refusal re-profile via `services/refusal-probe/`
   (the canonical harness, NOT the ad-hoc GGUF one), MTP acceptance on the quant
   (gate ≳40%, not KL — `reference_abliteration_mtp_lessons`), PPL/coherence.
4. NVFP4-quantize via `services/gen-seat-mixed-quant/` → gen-seat candidate.
   **Do NOT delete the incumbent** (`qwen38-27b-heresy-nvfp4-mixed`) until it
   holds through real multi-turn use.

bf16 staged at `/tank/aimodels/qwen38-27b-coldfusion-bf16` (pinned `9c44193`,
provenance recorded). All write paths re-stop the seats for fp32 VRAM — batch
re-capture + write in one window. Commits `ccb56a0`, `1857a8e`, `b56cb0d`,
`7abd301`.

_Archived 2026-09-05._

# `[2026-08-19]` Fleet `.internal` DNS — git-sourced, agent-managed, three resolvers

Operator: *"with ipv6 i can't memorize the IP addresses anymore. need a way to
keep track of local .internal dns names that can be agent managed and is
lightweight."* Built and live in one session; commit `b8003c7`.

## Shape

```
dns/internal.yaml     source of truth — 38 hosts + 4 service aliases
scripts/dns-sync.py   reconciles AdGuard resolvers against it
stacks/adguard-ana/   the colo's resolver, which did not exist
dns/README.md         workflow, naming, the IPv6 caveat
```

Deliberately the same posture as `deploy-stack.sh`: the file is intent, the
resolvers are derived state, you see a diff before anything changes.
`--dry-run` / `--yes` / `--site <s>`. Verified idempotent — a second run prints
`nothing to do`.

Naming is `<host>.<site>.internal` with sites **`ana` / `esh` / `nh3`**
(operator's call). `.internal` is ICANN-reserved for private use since 2024;
`.local` is reserved for mDNS, which is why the pre-existing
`searxng.pfi.local` was a standards collision that merely happened to work.

Every name is published to **every** resolver — the site label says where a
host *is*, not which resolver knows about it.

## The framing correction that mattered most

The ask reads as "I can't memorise v6 addresses", but the deeper problem is
that **v6 addresses are derived, not assigned**, so they cannot reliably be
*written down once* either. SLAAC gives EUI-64 (MAC-coupled) or
privacy-extension (rotating) addresses, and UniFi has **no v6 equivalent of a
DHCP reservation** — so a hand-maintained v6 table rots on its own.

⇒ The fix has two halves and only the second is DNS: (1) pin static v6 on
server-class hosts, (2) then the name table is just a file. Surfaced to the
operator before building.

**Verified 2026-08-19: no fleet host has a global v6 address at all yet** —
ESH's `/56` is live only on `esh-cameras`, NH3's LANs are back to
`ipv6_interface_type: none`, the colo has none. So the `v6:` column ships
EMPTY and correct, and the naming layer was built first rather than blocking
on v6. Names established now need no renaming when addresses land.

Suggested convention when they do (awaiting operator): each server static at
its site's `/64` with low-order bits echoing the v4 host octet —
`esh-docker-vm` at `…::45` — so addresses are declarable *and* semi-memorable.

## Two properties not to break

**Authority is scoped to the ZONE, not the resolver.** Only rewrites ending in
`.internal` are managed. ESH's resolver turned out to carry three hand-made
`esteban.net` rewrites (`eshnas`, `brotherprinter`, `eshhome`) — **my first
read of the config missed them**, because an `awk` range on `rewrites:` matched
an empty-looking block. A resolver-wide authoritative sync would have silently
deleted all three on first run. Verified intact after sync.

**Within `.internal` it IS authoritative** — names added by hand in the AdGuard
UI get deleted by the next sync. That is the point: one place to look.

## The colo had no resolver at all

ESH and NH3 each ran AdGuard; **ana-docker resolved straight against
`1.1.1.1`**, so the colo had no way to answer for internal names. Closed with
`stacks/adguard-ana/`.

⚠️ Its API is on **8053**, not 8080 — `:8080` and `:3000` were already taken on
that busy host. The port is therefore carried **per-site in the yaml**, not
assumed by the script, so the odd one out cannot be forgotten.

⚠️ It ships with **no blocklists**, deliberately. The other two filter ads for
human browsing; this one resolves for a rack of servers, where a blocklist
false-positive breaks service-to-service calls at 3am for no upside.

First boot uses a **seed config** (`conf/AdGuardHome.seed.yaml`) copied into
the conf volume before first start, so the container comes up configured
instead of sitting in the setup wizard.

## Credential — service account, not the operator's

Added a dedicated **`infra-ops`** AdGuard user to all three resolvers rather
than asking for the `lkraven` password (per the standing migrate-off-operator-
creds directive). Password vaulted at
`nh3-dev/adguard-infra-ops-password`; `lkraven` untouched; pre-change configs
backed up on each host as `AdGuardHome.yaml.bak-preinfraops-*`. Both existing
resolvers kept answering across the restart.

Two landmines worth keeping:

- **Go's bcrypt rejects `htpasswd`'s `$2y$` prefix.** Same algorithm, different
  marker; `golang.org/x/crypto/bcrypt` accepts only `$2a$`/`$2b$`. Normalise
  the prefix, and self-verify the hash with `htpasswd -vb` BEFORE installing it
  on a live resolver.
- **The vault appends a trailing newline on `get`.** A password carrying a
  stray `\n` fails auth in a way that looks exactly like a wrong password.
  `dns-sync.py` strips it.

## `pfi.local` migration — and the one that must NOT move

`searxng.pfi.local` → `searxng.ana.internal`, with the **old `Host()` kept
alongside** in the Traefik rule so nothing breaks mid-migration; both return
200. Drop the fallback once the access log shows the old name unused.

**`matrix.pfi.local` deliberately NOT migrated.** A Matrix `server_name` is
baked into every user ID, room ID and signing key, and federation identity
derives from it — renaming it is not a DNS change, it is rebuilding the
homeserver's identity and invalidating its history. The operator approved
"migrate pfi.local" generally; this was surfaced as a deliberate exclusion
rather than executed blindly.

## Still open

Colo hosts still point at `1.1.1.1`, so they do not yet *use* the new resolver
— it only answers what asks it directly. Repointing a whole site's DNS is a
bigger change than standing the service up, and is the operator's to schedule.

See also [[2026-08-17-fleet-ipv6-mesh]].

_Archived 2026-09-05._

# `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured

Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`):
a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare
`nohup` (PID 1283383) that would not survive a reboot. Now
`stacks/waterland-studio/`, `restart: unless-stopped`, healthy on
irv-ml1:8410. Commits `a2b5b58`, `8189076`.

Tracking `main` per operator: PR #4 merged and `main` HEAD was exactly the
pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway.

**Now deployed at `b72425b` (2026-08-19).** The container sat on `8025366` for
a few hours after PR #5 (`464dfc2`) landed — deliberately, since the image's
own guards already neutralised both landmines and the project was in
wind-down. PR #6 (the job-store rehydrate, operator-green-lit) was the rebuild
with a real reason behind it, and one `update.sh` run carried both. Verified
end to end after the update: healthy, `backend: cupy`, and a real 256² plate
render completes warm — the kernel-cache volume survived the image swap.

## Build context lives OUTSIDE the compose dir — on purpose

`/opt/waterland-studio/src` is the checkout; the Dockerfile is passed
out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh`
rsyncs `stacks/<stack>/` with `--delete`**, so a checkout kept beside
`compose.yaml` would be destroyed by the next deploy of this stack. `update.sh`
refreshes source → rebuild → recreate → health, and is verified end to end.

## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN

`gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer
silently drops to the numpy path at ~21x wall time — it does not error, it
just gets slow. waterland-dev warned about the build side.

The runtime side is worse and was not in the handover: **`studio/jobs.py`
shells the renderer out as a literal `uv run waterland ...` with no `--extra`
flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project
mid-job to its default extras and prunes cupy back out from under a correctly
built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the
pin ever stops holding the job fails **loudly** instead of quietly rebuilding a
slower environment.

**Fixed upstream in `464dfc2`:** the server now spawns
`sys.executable -m waterland.cli` directly — no resolver in the render path at
all. **The pins stay anyway.** They cost nothing and are now defence-in-depth:
if any future code path re-enters `uv` inside the container, the job fails
loudly instead of quietly dropping to the numpy backend. `uv` itself must stay
in the image regardless — it performs the build-time `uv sync` /
`uv pip install`, and this is a single-stage build.

## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare

Every render died 1.7s in with:

```
RuntimeError: Failed to find CUDA headers.
```

printed **through argparse's usage banner**, which makes it read like a CLI
argument bug rather than a missing toolkit. That misdirection is the reason
this is written down.

cupy compiles kernels at runtime through NVRTC, which needs the toolkit
**headers** — not just the driver and the runtime libs bundled in the
`cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the
bare `nohup` process found them **by accident**; a slim image has none.

Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few
hundred MB against ~6 GB for a `-devel` base image. It runs **after**
`uv sync`, because sync prunes what it does not know about.

Reported upstream: it is an undeclared runtime dependency of the `gpu` extra,
and anyone running this without a system toolkit hits it. **Declared upstream
in `464dfc2`** (`gpu` is now `cupy-cuda12x[ctk]>=13`). **The explicit install
stays in the Dockerfile**: the header requirement is a property of *this*
image — a slim base with no system CUDA toolkit — so it belongs in the file
that creates the problem, not inherited from an extra two repos away. It also
survives any future restructuring of the `gpu` extra. Cost of keeping it is
now measured, not assumed: since `uv sync` satisfies it first, the line
reports `Audited 1 package in 49ms` and adds **0.3s** to the build. A no-op
that documents a non-obvious requirement is worth 0.3s. (waterland-dev
independently agreed they would keep it too.)

## Landmine 3 — the GPU index inside the container is not the host's

The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects
`CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where
`nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU
(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the
container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying
the host's value selects a device that does not exist. Host device 0 is the
3090, which carries the TTS zoo and must not be touched.

## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume

| job | wall |
|---|---|
| 256² + anim, cold container | 23.3 s |
| 256² + anim, warm | **6.1 s** |
| 256² plate only (`--codec none`) | 3.9 s |
| 512² plate only | 6.4 s |

Warm beats the **7.4 s** recorded against the bare-metal process, so
containerising cost nothing. Verified the cache volume properly: recreate
(fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without
it every restart makes the next user wait 4x and the service merely *looks*
slow.

## Upstream finding — the on-disk job store grows without bound

`JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at
startup**. Consequences:

1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and
   how this was spotted: the API reported **1 job** while the volume held all
   **16 directories, 60.6 MB**. Not data loss.
2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict,
   so directories orphaned by a restart are **never reclaimed**. The
   handover's "bounded around 500 MB" holds within a single process lifetime;
   across restarts the store grows monotonically at ~12 MB per animated job.

Reported to waterland-dev with evidence; **not patched from the infra side** —
it is their code. Prune the volume by hand if it bites first.

**waterland-dev confirmed it (2026-08-19)** — their "bounded ~500 MB" handover
claim holds within one process lifetime and nowhere else, which on a
`restart: unless-stopped` service is the wrong lifetime to have bounded. They
have **surfaced a startup-rehydrate fix to the operator** rather than opening a
third PR during wind-down. **Operator green-lit it; PR #6 merged as `b72425b`
and is DEPLOYED (2026-08-19).**

Startup rehydrate, as recommended — and waterland-dev deliberately went
further than the framing I sent them. I had said a directory the scan cannot
parse "just does not enter the index"; they made the opposite call, because a
directory that never enters the index is exactly the one that never gets
reclaimed. **That is the sharper reading and it is the reason the fix works on
this volume at all** — the 16 pre-existing dirs have no sidecar. Their
adoption ladder: sidecar → restored verbatim; no sidecar → adopted with
dimensions recovered from the PNG IHDR (24-byte read, not a decode); corrupt
sidecar → degrades to inference, no startup crash; **neither source nor
sidecar → skipped on purpose**, since adopting it would turn eviction into a
delete-arbitrary-directories primitive pointed at this volume. Sidecar writes
go through `os.replace`, and `job.json` is excluded from `ARTIFACTS` so it is
unreachable via the artifact route.

They also closed a second leak I never saw, because it needs a restart
*mid-render* to surface: a job left `running`/`queued` in its sidecar is
non-terminal forever, and eviction skips non-terminal jobs — so it is a
phantom that is never reclaimed and `queue_depth` over-reports for the life of
the process. Adoption now marks those `failed`.

**Verified on this host after the update:** `/api/jobs` went **1 → 16** while
the volume stayed at 16 dirs / 61 MB — disk and API agree for the first time.
Nothing was reclaimed, correctly: 16 is under `RETAIN=40`, so adoption only
made them visible. A subsequent real render took both to 17. From here the
store is bounded **across** restarts, not merely within a process.

## Access

Repo is not anonymously readable (a bare clone 403s). Operator granted
**`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False,
pull: True`. Token on irv-ml1 at
`/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a
**repo-scoped** credential helper; `.git/config` carries no token (verified),
so the remote stays clean in any diff or backup. The operator's `vh`
site-admin token was used only for the initial clone and the grant itself and
was **never written to disk on that host** — a site-admin credential on a GPU
box is a blast radius nobody needs for a read-only fetch.

## Constraints honoured as stated (not inferred)

- **Serial by design — one replica, one card.** A render is 20–45s of near-full
  GPU with a single worker thread. Two on the same A6000 would OOM or thrash.
  Throughput is a hardware conversation, not a replica-count one.
- **No authentication, arbitrary file uploads** ⇒ stays inside the
  LAN/WireGuard boundary. Do **not** paper over it with a proxy password;
  waterland-dev offered to add a real auth layer if wider reach is ever needed.

_Archived 2026-09-05._

# `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background

Commits `9d92c4b`, `c3de7db`, `45c1995`, `f38cf69`, `df68dd2`.

## The cleanup (three real defects)

- **UltraSeedbox rendered on all four tabs.** The bookmark group had no entry in
  `settings.yaml`'s `layout:` block at all, and Homepage's documented behaviour
  is that a group with no `tab:` is shown on **every** tab. Pinned to Main.
  ⚠️ This will happen again to the next group added without a `tab:` — the rule
  is now written at the top of the layout block.
- **Uptime Kuma rendered twice** — a manual `services.yaml` entry under
  Monitoring *and* `homepage.group=Apps` on the container. Exactly the
  "never list a labelled container manually" failure the stack README warns
  about; it survived the previous day's audit because a duplicate reads as two
  plausible cards rather than as an error. Manual block deleted, label moved to
  `Monitoring`, `homepage.siteMonitor` added.
- **Column counts were fiction** — several groups declared more columns than
  they had members, so the last row of each was dead space (Notes: 1 card in a
  4-wide row). Columns now track member counts; `GET /api/services` prints the
  live per-group counts and is the check.

Later, on operator instruction, the **AI tab was reordered by clickability**:
Gateways & Chat → Image & Media → Audio Tools on top, then the vLLM `/docs`
seats and TTS endpoints. Reasoning written into the config so it survives:
order by "would I click this?", not by how central the service is.

## ⚠️ The expensive red herring — the tab bar after a recreate

After a recreate the client render comes up with **no tab bar, no wallpaper and
no i18n** (search box shows the raw key `search.search`), groups falling back to
side-by-side columns. **It restores itself with no intervention.**

Timing, measured rather than assumed: a fresh container was still tab-less at
**4m30s, twice**; it was healthy again after roughly an hour. `docker ps`
reporting `healthy` says nothing about it — the container is serving, the page
is just wrong.

An hour went into ruling out four causes that were never the cause:

1. **Not the config** — restoring `settings.yaml` *and* `services.yaml` to
   their committed versions reproduces it, as does the pre-adoption backup in
   `/opt/docker-bu/conf/homepage/`.
2. **Not the v2.0.0 release** — a throwaway container on `v1.13.2` shows
   identical symptoms, and the image never changed anyway (working and broken
   both report `v2.0.0` / rev `17456f2`).
3. **Not `PUID`/`PGID`**, and not Docker discovery — tested both, and with the
   socket unmounted entirely.
4. **Not server-side** — the server-rendered HTML still contains the tab
   markup, the background URL and `useEqualHeights`; `GET /api/validate`
   returns `[]`. The loss is client-side, with no page error, no failed chunk
   and no non-200.

Every throwaway container in that list was judged within ~30s of starting, so
they were all inside the same window — and that consistency **read as a
reproduction when it was the same measurement mistake five times over.**

**Operative rule: recreate, walk away, re-check later. Do not chase it.**

## ⚠️ The iteration loop that would have prevented the overcook

`custom.css` is served **per request** from `/api/config/custom.css`, so a CSS
change needs a **browser reload** — not a container recreate, and it never owed
the layout warm-up above. Conflating the two costs ~10 operator-visible minutes
per attempt (operator called this out directly).

Faster still, and how the final pass was done: **inject candidate CSS into the
running page and screenshot it** —
`await p.addStyleTag({content: css})` in Playwright against the live
dashboard. Seconds per iteration, no deploy. Build + deploy only once the
render looks right.

## The theme — Australis Skyfall

Operator supplied a Claude Design handoff bundle via the Booth (`26-copper`).
Skyfall is a dual-theme OKLCH system: one lightness law across every chromatic
family (deep 0.48 / base 0.66 / bright 0.80), all hues cooler than neutral, a
Sea neutral ramp drifting ice-blue→ocean-green as it brightens, and a
"calm depth" language of **hairline + two-layer shadow on every elevated
surface, never one without the other**.

```
theme/colors.css layout.css typography.css   vendored VERBATIM from the bundle
theme/fonts/Supreme-{400,500,700}.woff2      the body/UI face
theme/skyfall.css.in                         the Homepage bindings (ours)
theme/build.py            → conf/custom.css  (generated — do not hand-edit)
```

The build step exists for one reason: **Homepage serves only `custom.css` and
`custom.js` out of its config dir**, with no static route beside them, so a
`@font-face` pointing at a vendored `.woff2` would 404 — the face must arrive
as a data URI. The background image takes the other road, because
`/app/public/images` **is** a real static route (mounted read-only in
`compose.yaml`).

Only Supreme is embedded: a link dashboard has no display type, and Victor
Mono ships as 2.4 MB TTF statics per cut — 30x the whole stylesheet for a
handful of latency figures.

## The background is generated, not stock

**Arbo as an image-gen engine** (the operator's actual ask, which I first
misread as "use Arbo's palette" and had to redo). Arbo's `t2i-ui-background`
workflow is purpose-built: *"abstract full-bleed backgrounds, no subject"*.
Job `13f0891f4e42`, seed 26, flux2-klein-9b, 2048×1152, 1.6 MB PNG → **22 KB
WebP** (smooth gradients compress absurdly well).

⚠️ Arbo API gotcha: `prompt` is a **discriminated union, not a string** — a
bare string 422s. `{"kind":"raw","text":…,"negative":…}` is the shape.

## Two documented deviations from the design system

1. **Skyfall forbids this background.** Its rule is "flat semantic surfaces; no
   photography, no textures", with one permitted motif — a subtle aurora
   gradient on hero/empty-state areas only, *"never behind body text blocks"*.
   A dashboard is a body-text block. Present on the operator's explicit
   instruction, mitigated rather than excused: abstract, no subject, strictly
   cool temperature, held at **`opacity: 30`**. That number is load-bearing —
   at 14 the aurora was invisible, and turning it up makes the cards fight the
   ribbon.
2. **Service icons stay full-colour vendor logos.** Desaturating them from CSS
   only makes them illegible.

## Overcorrection, and the colour pass

First stat-well pass went from `font-thin` 13px straight to **bold 22px in
heading white** — operator: *"went from subtle to BASH YOU OVER THE HEAD."*
The principle missed: a stat only has to out-rank **its own label**, not the
service name above it. Now `--text-md` medium in cyan.

Colour was then lifted **from inside the system**: Skyfall names Aurora (blue,
cyan, green) the *primary* families, "used generously, in that order", while
Dawn (amber/red/violet) is semantic-only. So group markers cycle
blue→cyan→green down the page (icons full strength, names at 0.72), service
icons take a single cool wash, latency tags move to the info family so
"how fast" stops looking like "is it alive". **No Dawn colour is used
decoratively anywhere.**

Two DOM findings that made it possible:

- **Homepage renders mdi icons as a gradient behind an SVG mask** — recolour
  via `background`, not `color`.
- **Homepage emits `docker-status-<state>`, not `status-<state>`.** The
  original selectors matched nothing, so every green pill up to that point was
  stock colouring rather than the theme. Both forms are now matched.

_Archived 2026-09-05._

# `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken

Commits `42c594c`, `dc3e47b`, plus `uptimekuma` in `9d92c4b`.

## The pattern worth remembering

Chasing two bad-looking cards on the dashboard turned up **four stacks running
on fleet hosts with no canonical copy anywhere**: `uptimekuma` and (already
known) the two AdGuards on esh-docker-vm, `searxng` and `seafile` on
ana-docker, and `heretic2-charrp-reasoning` on ana-ml2 (untracked in git).

⇒ **A dashboard card is a cheap census of what is actually running.** When
something on it looks wrong, check whether the stack behind it is even in
`stacks/` before debugging the symptom — twice here the answer was "no", and
the fix belonged in version control as much as on the host.

Adopted: `stacks/uptimekuma/`, `stacks/searxng/`, `stacks/seafile/`,
`stacks/heretic2-charrp-reasoning/`. ESH/NH3 AdGuard compose files were
**deliberately left unmanaged** — adopting three live resolvers while also
introducing a new DNS naming system is two risky changes at once.

## SearXNG — the healthcheck was eating itself

Card flapped UNHEALTHY; the container was fine the whole time. The compose
passed `--tries` and `--spider` as **two separate argv entries**, so wget
consumed `--spider` as the *value* of `--tries`. Spider mode never engaged,
which means every probe since April **downloaded** the healthz response to a
file:

```
295,287 healthz.N files in the container's working directory
```

With that many files, wget's scan for the next free filename is what
intermittently blew the 10s timeout. **Self-worsening — every probe made the
next one slower.** Restored `--tries=1`; the junk lived in the writable layer
so the recreate cleared it. Now `healthy`, `fails=0`, 200 in 0.16s.

Lesson: an argv list in YAML has no shell to catch a missing `=`. A flag that
silently swallows the next argument turns a liveness probe into a workload.

## SeaFile — not broken, never restarted

Card showed EXITED for three months. **None of the three services declared a
restart policy**, so Docker defaulted them to `no`. On
**2026-05-06T21:27:45Z** the daemon stopped all three within 200ms of each
other — a daemon restart or host reboot — and nothing brought them back.

⚠️ **Exit code `255` is a red herring**: it is what a container that ignores
SIGTERM reports when the daemon stops it, **not** evidence of a crash. Reading
it as one sends you hunting a bug that does not exist. The tell was all three
services stopping within 200ms.

Added `restart: unless-stopped` to all three; brought up; mariadb gated on its
healthcheck exactly as the existing `depends_on` comments intended, seahub
started without the race, `302` → login page. Data was in local named volumes,
not on the ana-nas NFS, so nothing was at risk.

Three months of silent downtime whose only signal was a card nobody read as an
outage — the argument for semantic status colour on the dashboard (see
[[2026-08-19-homepage-skyfall-theme]], where amber EXITED pills made six
mis-grouped AI seats obvious at a glance).

## heretic2-charrp-reasoning — tracked, with its shim

The `char-rp-reasoning` seat (NEO-CODE Heretic2 27B, modelopt NVFP4 + grafted
BF16 MTP head, ~77 tok/s via `qwen3_5_mtp` spec-decode) had been running
untracked. Now in `stacks/`, including
`conf/mtp-workaround/sitecustomize.py`, which is **not optional**: vLLM 0.24.0
does not propagate modelopt `exclude_modules` to the spec-decode **draft**
model, so the BF16 MTP head gets quantized and the engine dies at load. Both
the mount and `PYTHONPATH` are load-bearing.

Added the two files house convention expects and the directory lacked — a
`.env.example` naming every knob (all values are compose defaults; the host
overrides only the three VRAM ones) and a README pointing at
`docs/runbooks/heretic2-nvfp4-mtp-seat.md` rather than duplicating it.

_Archived 2026-09-05._