# Archival memory — eshpfi-management _Entries moved out of persistent-memory.md to keep the active file scannable. Read this when researching historical decisions or revisiting past foot-guns._ ## Recent decisions (archived) - `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase. _Archived 2026-09-02._ - `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right. _Archived 2026-09-02._ - `[2026-08-16]` **The Qwen base thinks incessantly — that is WHY the Gemma seat exists, and no swap within the Qwen family fixes it.** Operator's architectural point, confirmed by measurement: on identical prompts DS 6036 ch vs FF 5323 ch of reasoning (permission arm), 5546 vs 4988 (cards arm) — FF actually reasons ~10–12% **less**. The bare-instruct row (DS 2291 vs FF 3918) inverts only because DS refused 92.5% of it and refusals are short — an artifact, not concision. Both are Qwen3.6-27B derivatives, so this is the base family. `char-rp` = **MeroMero-v2, Gemma-4 base**, :8016, verified 0 chars reasoning / clean prose — the non-thinking seat, working as designed. FF *can* be silenced (`enable_thinking:false` verified 3/3, and it ships `chat_template-instruct.jinja`) but that duplicates MeroMero on a base chosen for it. The stale LiteLLM comment describing `char-rp` as the retired GGUF Magidonia seat is fixed (`53096bf`). _Archived 2026-09-02._ - `[2026-08-16]` **Canonical Qwen3.8 sampling applied from upstream; `gen-reasoning` had the WRONG-MODE presence_penalty.** Qwen/Qwen3.8-27B "Best Practices" §1 and unsloth/Qwen3.8-27B §1 are **byte-identical** — thinking: `temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0`; instruct: `temp 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0`. **Bug found:** `gen-reasoning` carried `presence_penalty 1.5` — the *instruct* value on a *thinking* deployment (canonical 0.0) — now fixed. **Deliberately NOT canonicalised:** `summarizer`/`classifier`/`image-judge`/`qwen-image-bench` run `temperature=0` (judges also `top_k=1`) because determinism is their contract; forcing a chat preset on a classifier would break it. ⚠ **`presence_penalty=1.5` is canonical but is the one value upstream hedges on**, verbatim: *"using a higher value may occasionally result in language mixing and a slight decrease in model performance."* It is the **operator's suspected trigger** for multi-turn degradation and the **first dial to move (0.0–0.5)** if that recurs — it is alias-scoped, which is why it would follow the operator across model builds. Commit `3462b53`. _Archived 2026-09-02._ - `[2026-08-16]` **Four wrong diagnoses on one bug, and the lesson is the test design.** Operator reported the gen seat "degenerate on long multi-turn conversations". Rolled the seat back on request; **the previous weights behaved identically**, exonerating the model swap. I then proposed and disproved FOUR mechanisms in sequence — empty assistant turns poisoning history, reasoning runaway, length-mirroring from short history, and `presence_penalty` — before discovering **my own multi-turn harness was confounded**: it varied the QUESTION along with the depth (depth-1 asked question #2, depth-3 asked question #4), so a narrower question drawing a shorter answer read as degeneration. The "310→209→28w collapse" I reported as a reproduction was an artifact. **Rules banked:** (1) when comparing across conversation depth, hold the final question FIXED and vary only the history; (2) reply-length variance on byte-identical input was 25–465w, so n=3 cannot support any claim about a trend; (3) **ask for the operator's real failing transcript before building a synthetic reproduction** — four synthetic tests, none of them his failure. Gateway `spend_logs` returns `[]` on the infra-ops key despite `store_prompts_in_spend_logs: true`, so real transcripts need the `:4000/ui` view or another key — worth solving before the next such hunt. _Archived 2026-09-02._ - `[2026-08-16]` **Two REAL client-side defects found while chasing the above, neither of which was the reported bug.** (1) `gateway-chat`'s Max-tokens field defaulted to **1024**; thinking seats spend part of that on CoT before emitting content, so completions truncate with `finish_reason=length` and read as model degeneracy — raised to 4096. (2) `parseInt` on an empty field yields NaN, which `JSON.stringify` serialises as **`null`**, which the server reads as "no max_tokens supplied" and silently substitutes its own default — indistinguishable from the UI ignoring the field. Both fixed (`b6552e0`, `fb3bb52`). ⚠ **`compose` bind-mounts a single FILE, and a single-file bind mount binds the INODE** — rsync writes-and-renames, so the container kept serving stale content while the host file showed the new value, silently and with no error. `docker restart` does NOT clear it; the container must be **recreated**. Verify against what the *container* sees, never the host file. Applies to any file-source mount fleet-wide. _Archived 2026-09-02._ - `[2026-08-16]` **Refusal measurement: benign controls CANNOT validate a refusal classifier on RP prose — and a 0% rate needs a classifier self-test before you believe it.** Two durable lessons from baselining Dark-Scarlett. (1) **False positives:** my first bare-framing number was **9.5%**; the true figure was **1.4%**. The rest were the classifier firing on *in-character* text — `"I cannot shift my weight"` spoken by the character ~100 chars into a 2,443-token torture scene, and `"Yeah, I'm an AI… What's the actual gig?"` where the model answers in voice and keeps driving the scene. First-person RP prose is **full** of "I can't"; a genuine refusal *opens* with its marker, so the scan window must be the **first sentence**, a marker followed by long prose must demote to AMBIGUOUS, and AI self-acknowledgement is a **persona break, never a refusal on its own**. Benign controls were clean the entire time and caught none of it — they only detect over-firing on *benign* prompts, not on in-character prose. (2) **False negatives:** a 0% rate and a broken classifier are indistinguishable from the report, so `test_classify.py` (16 cases, both false positives pinned as regressions) must pass before any low number is trusted. Also banked: the **thinking-budget trap** — empty `content` + `finish_reason=length` is reasoning eating the budget, NOT a refusal; score INVALID and exclude from the denominator (DS emits ~5.5-6k chars of reasoning per response, so `max_tokens` ≥3072). `probe.py --rescore` re-classifies a saved run with zero GPU time. → `services/refusal-probe/README.md`, commit `32f665e`. _Archived 2026-09-02._ - `[2026-08-16]` **Held an operator-approved swap window because the baseline invalidated its premise.** Operator approved ~65 min of `char-rp-reasoning` downtime to A/B Fable-Fusion 711 against Dark-Scarlett on refusals. The DS baseline then came back **0.0%/1.4%** — no gap for a candidate to close, so the window would have bought no decisive signal *and* a second window would still be needed once a reproducing battery existed. Held the swap, reported, and routed to brokkr-smithy-dev for the battery that actually produced the refusals. The general rule (action-relevance): **approval is for a plan, not a ritual — when new evidence kills the plan's premise, surface it rather than spend the budget.** Nothing deployed, no downtime taken, seat untouched. _Archived 2026-09-02._ # esh-pve-nas — PVE root on a USB DOM: diagnosis, mitigation, migration plan ## The finding `esh-pve-nas` (`esh-nas-pve.esteban.net`, 10.0.50.55) runs PVE root off a **USB Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, `ID_VENDOR=NORELSYS`, model 1081 — carved into a 512 MB ESP + 768 MB swap + a **6 GB ext4 root** that was at **90% (571 MB free)**. ⚠ **Operator corrected my first read: it is a DOM, not a thumb drive.** DOMs use SLC/pSLC with a real controller, so the **284 GB written since boot is unremarkable and wear is NOT the driver**. I had framed it as a clock ticking; that was wrong and the correction matters. What actually justifies the work: 1. **It is on the USB bus** — a reset or re-enumeration drops the *root filesystem* out from under a running hypervisor whose guests keep executing. NAND quality is irrelevant to that. 2. **6 GB has no headroom** — `/usr` alone is 3.7 GB. 3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty. 4. **It has blocked patching for months** — the operator-visible symptom and the real urgency. ## The patching blockage (measured) `apt-get -s dist-upgrade`: **225 packages pending, 161 carrying `deb12uN` / Debian-Security bumps** including `ssh 1:9.2p1-2+deb12u10`. Host sits on `pve-manager/8.4.11` vs sibling esh-pve's **8.4.14**, with 20 weeks uptime because it cannot take a kernel. ⚠ **Ordering is load-bearing: migrate FIRST, patch after.** The pending set includes `proxmox-kernel-6.8.12-42-pve-signed` — ~250 MB of kernel + initramfs landing in `/boot`, **which is on root**. Unpacking 225 packages (dpkg, perl, glibc-adjacent) into 1.3 GB of headroom risks filling the disk mid-transaction and wedging dpkg on a hypervisor running five guests. Partial escape hatch if patching truly cannot wait: `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives` keeps downloads off root, but the kernel still lands in `/boot`. ## Mitigation applied 2026-08-17 — root 90% → 76% | step | effect | |---|---| | capped journald (`SystemMaxUse=64M`; was **fully default/uncapped**) | stops unbounded growth | | vacuumed the journal | **freed 446 MB** | | `apt-get clean` | 79 MB | | `/root/neo` (2024 Intel NEO OpenCL debs) → `/nvme/tmp/root-neo-20260817/` | 259 MB — **moved, not deleted** | | **`/var/log/journal` relocated onto ZFS** (`nvme/varlog`) | dominant writer off the DOM | 571 MB → **1.4 GB free**. All five guests stayed up; a fresh `logger` round-tripped through the ZFS-backed journal. ⚠ **Stopping journald over SSH kills your own session** — it takes the connection's logging path with it. The first attempt died mid-swap, leaving the dataset staged and the move incomplete (host was never at risk; journald socket-activated straight back). Redo as a detached `systemd-run` transient unit. Script + reason live at `root@10.0.50.55:/root/move-journal-to-zfs.sh`. Deliberately **not** done: moving `/var/lib/rrdcached`. With the DOM correction the wear argument no longer justifies touching a service `pvestatd` depends on. ## The plan — split boot from root (operator's proposal, strictly better) My first plan was a full reinstall to a mirrored-NVMe ZFS root. **The operator proposed keeping boot on the DOM with a fallback image and putting all its files on ZFS. That is better and I should have gotten there myself** — I had assumed boot and root must share a device. | | device | contents | written when | |---|---|---|---| | boot | DOM `sdq` | ESP + `/boot` (ext4) | only on kernel/GRUB updates | | root | `nvme` pool | `nvme/ROOT/pve-1` | constantly, on mirrored NVMe | Keeping `/boot` on **ext4** is the point, not a compromise: GRUB never has to read ZFS, which matters because the `nvme` pool has `encryption`, `large_dnode` and `zstd_compress` enabled and **GRUB cannot read those**. **Why it beats the reinstall:** the `nvme` pool survives (no guest migration, no `ssd`/`tank` export-import, no reinstall); downtime is **one reboot** not half a day; **rollback is a GRUB menu entry** because the ext4 root stays untouched on the DOM; and it retires the actual top risk — with root on NVMe a USB bus reset mid-run no longer kills the running system. Free upside: boot environments (`zfs snapshot nvme/ROOT/pve-1@pre-upgrade`). **Preconditions verified already met:** UEFI + `grub-efi-amd64 2.06-13+pmx7`; **`zfs-initramfs 2.2.8-pve1` already installed with 76 ZFS files in the running initrd**; root only 4.3 GB to copy; swap 767 MB / 123 MB used against 125 GB RAM (leave it on the DOM LV — **never** swap on a zvol). **Two traps:** `canmount=noauto` on the root dataset or ZFS mounts over the live root; and `cachefile` is `none` with a **0-byte `/etc/zfs/zpool.cache`** — pools import by scan today, which is a coin-flip when the initramfs must find root. Set the cachefile before rebuilding the initramfs. Operator ruled a **cloned DOM image is sufficient** boot-path insurance (no mirrored boot needed). `dd` it off-box before anything else; refresh after kernel updates. ## ⚠ Blast radius — the gating constraint, invisible from the host itself **CT 103 `esh-nas` (10.0.50.50) IS the NAS, and it runs on this host.** Two dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably: - **esh-docker-vm** (10.0.50.45): `/mnt/books`, `/mnt/backup` - **esh-pve** (10.0.250.35): `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` Known incident shape — the only remedy for esh-docker-vm's D-state is a host reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's SQLite risks corruption under `soft`. Quiesce both before any reboot of this host. Recorded in `servers/esh-pve-nas/README.md` as a never-reboot-casually warning. ## Also identified - **`esh-nas` is CT 103** on esh-pve-nas — structurally the same shape as ana-nas being CT 109 on pfi-pve. - **`ESH-FileBot` (CT 106, 10.0.50.70) is an empty shell** — 80 GB rootfs, six passthrough mounts (`books`/`documents`/`music`/`share`/`pvestore`/`ssd-pvestore`), and **nothing running but base systemd, sshd, cron, postfix** since 30 March. That resolves the dashboard's long-standing "role TBC". Retire rather than migrate. - Both ESH hypervisors have **20 weeks uptime** and differing PVE patch levels. ## Staging executed 2026-08-18 — everything but the reboot Two rerunnable elway playbooks, 0 failed steps, 17/17 verify green: `playbooks/esh-pve-nas-stage-zfs-root.yaml` (LV surgery, `/boot` populate, 4.3 GB root rsync in 228 s, fstab) and `playbooks/esh-pve-nas-stage-bootloader.yaml` (ZFS initramfs, grub.cfg, both menu entries, grubenv). **`grub-install` is deliberately NOT run.** The ESP stub still points at the old `/boot` inside the ext4 root, so the host's boot path is byte-identical to the last 140 days and an unplanned reboot mid-staging is a non-event. Cutover is `grub-install` + `grub-reboot pve-zfs-root` + `zfs set mountpoint=/` + reboot. Final DOM layout: `pve-root` 6.04 G (untouched, the rollback) + `pve-boot` 512 M (new) + `pve-swap` 256 M (was 768 M). ### The three landmines staging found 1. **The `/boot` LV had nowhere to live.** VG `pve` had **4 MB free**, and mounted ext4 cannot shrink — freeing space from root needs a rescue boot, which costs the "one reboot" property the design rests on. Only live source was the swap LV. Operator chose shrink-to-256M over drop-entirely. 2. **The one-pool cachefile would have broken the NAS.** `zpool set cachefile=… nvme` looks scoped and safe; it is the opposite. Populating a cachefile flips the host from `zfs-import-scan` to `zfs-import-cache` (verified: scan active, cache inactive beforehand), so a cache holding only `nvme` leaves `ssd` and `tank` unimported at boot — and CT 103 has twelve bind mounts spanning all three pools. Every export would come up empty and both `hard` NFS clients would hang. 3. **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`.** Debian's `10_linux` builds `${rpool}${bootfs}`; `rpool` comes from `grub-probe --target=fs_label`, which returns empty because GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` — and the probe failure is swallowed by `2>/dev/null || true`. The same feature set that forced `/boot` to stay ext4 also corrupts the kernel command line, which the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg` drop-in (last `root=` wins) plus explicit `pve-zfs-root` and `pve-ext4-rollback` entries carrying stable ids — the auto-generated ids are derived from pool member device paths and would shift if the mirror changed. **The transferable lesson from (3):** the original verify grepped for `root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in grub.cfg. Once the drop-in was added that grep passes — while pool-less entries sit in the menu untouched. The check that holds walks every `linux` line, takes the **last** `root=`, and asserts it against a known-good set. **Assert the effective value, not the presence of a substring.** ### One-shot boot, not a new default `GRUB_DEFAULT=saved` with grubenv pinned to `pve-ext4-rollback`, and cutover uses `grub-reboot pve-zfs-root` so ZFS is tried **exactly once**. A failed ZFS boot returns to ext4 by itself on the next reboot — no console, no hands. That matters more here than on a normal host: a hang at an initramfs prompt takes CT 103 down and the NFS clients hang rather than fail. Only after a second clean ZFS boot should the saved default move. ### Off-box artifacts (`nh3-dev:~/backups/esh-pve-nas/`) - `dom-sdq-20260818.img.zst` — full DOM image, 7,837,450,240 B raw / 2.38 GiB compressed, zstd XXH64 verified. ⚠ **Crash-consistent, not clean** — the root LV was live during the read, so a restore replays the ext4 journal. Not fixable with an LVM snapshot: the VG has no free extents. - `bootchain-20260818.tar.gz` — clean, consistent tar of `/boot` + ESP (88 MB, 644 entries, full proxmox shim/grub EFI chain). This is the higher-quality boot-chain artifact; the dd image is the belt-and-braces full-device restore. - `pve-config-snapshot-20260818T051*.tar.gz` — 147 entries incl. the new grub.cfg, fstab, LVM/ZFS/blkid state. ⚠ Building this the first time produced a **corrupt archive**: `pvs; vgs; lvs > file` redirects only the last command, so `pvs`/`vgs` output leaked into the tar stream on stdout. Group with `{ …; } > file`. Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Earlier config snapshot at `nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600, sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and every guest config. **The newest on-disk copy before this was June 2024.** Commits `2275e11`, `3e31175`, `8ddc87c`. _Archived 2026-09-02._ # irv-ml1 weight cleanup (782 GB) + Homepage brought under version control Two unrelated housekeeping jobs from the same session, both with durable lessons. ## irv-ml1 — 782 GB reclaimed Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and **74%** (477 G). **Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22 GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF` (44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had **zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*, not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and nobody opened that shed for 15 months. Re-verified the consumer check immediately before deleting, not just during the audit. **Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G, `modelscope` 29 G (mtime **2024-04-23**). **Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion` 28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2` 1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**. ### The lesson: one dead-looking app pinned three delete targets `lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on 0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it would have half-broken a live listener in a way that surfaces only at its next restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind), verified :7860 released and handles at zero, *then* deleted. ⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a *native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker `mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs from uv. Checking open handles **per path** is what separated them — the earlier "not running" read would have deleted the wrong thing. ⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't (`zfs list -t snapshot` empty); second check showed 477 G at 74%. All 16 containers and both systemd services verified healthy afterward. ## Homepage under version control `ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose config lived only on the host. Its version history was **six hand-rolled `services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files + `.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone. 105 cards across 19 groups, no empty groups. ⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** — `ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated the top of the file. All five engines were discovering correctly the whole time. **Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a **FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` → **NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible); dropped the `UltraSeedbox` layout group (nothing provides it). ⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not** cover `http://10.0.50.45:5100/` — the container log carried `Host validation failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200. `.env` was **mode 644** holding Plex + Jellyfin API keys → now 600. ⚠ **Homepage renders client-side** — grepping the served HTML to verify a config change gave two false readings (a stale prerender, then an empty page). `GET /api/services` is the honest instrument, and config changes need a **recreate**, not a restart (a restart keeps the cached render in the writable layer). ⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it also removed a host-side `README.md` in the conf dir. Content survived (it is now in the repo README) but that was a side effect, not a plan. Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]]. _Archived 2026-09-02._ - `[2026-08-17]` **Gen seat swapped to `absolute-heresy` — and the three bugs the swap exposed are worth more than the swap.** Candidate `MuXodious/Qwen3.8-27B-absolute-heresy` (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. **Durable lessons banked:** (1) **A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate.** The `Qwen3_5ForConditionalGeneration` wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave `mtp.*` pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: `services/gen-seat-mixed-quant/compare_mtp_head.py` (hash bf16 via **uint8 reinterpret** — numpy has no bfloat16). (2) **`post_quant.py` assumed a standalone `model-mtp.safetensors`**; a full checkpoint keeps `mtp.*` in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; **that is why the check exists rather than an assumption**. Fixed to extract. (3) **A probe that does not mirror the live seat manufactures failures.** `serve_probe.sh` hardcoded `:latest` (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its `--speculative-config` JSON died twice on quoting — **bash BRACE-EXPANDS `{"a":1,"b":2}` on the comma** unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to **6/6**; the "tool calling broken" result was pure probe config. Commits `7997f11`,`254c588`,`2c36028`,`b0c2d3d`,`993421b`. _Archived 2026-09-02._ # Fleet IPv6 state + the real VPN topology (verified 2026-08-17) Written because the operator expects to reference this "before too long" — the driver is an **ESH fiber install landing 2026-08-18 that puts the house behind CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than a nice-to-have. ## Why IPv6 suddenly matters: CGNAT at ESH New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic — the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch: a global v6 address on each UDM restores a routable endpoint pair without depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the most likely first consumer of fleet IPv6. Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**. ## The topology — as VERIFIED, not as assumed Three transports, three different technologies. Do not describe this as "a WireGuard mesh"; a prior session did and was corrected. | Link | Transport | Evidence | |---|---|---| | NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets | | Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse | | Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt | | Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below | **WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.** It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`, **UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`, `vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`: `38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1. **The FortiGate never terminates WireGuard — it port-forwards to the host that does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a session that reads "colo + WireGuard" and concludes the edge must be upgraded is chasing a non-problem. Do not re-derive this. ## Per-site IPv6 state (2026-08-17) | Site | Edge | IPv6 | |---|---|---| | **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` | | **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` | | **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only | ## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed `2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests. The proof matters because the naive read is wrong: the WAN sits at `c110::48` and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning `c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at **`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID settings yielding the same /64 is the signature of a single-/64 delegation. **Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a routable address on the *WAN*, not a LAN prefix. The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false` with no size field alongside), so the prefix-ID test is the only read-only-ish way to establish it from the API. ## What a v6 mesh actually requires (and what it does NOT) **Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both Site Magic and WireGuard need a routable address on the router/host WAN side, plus inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and then reverted. **ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both `0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard reconfiguration** once (a) the host holds a routable v6 address (today: link-local `fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337 over v6 — the existing VIP is v4-only (`extip 38.120.12.42`). **NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`, one field to flip when wanted. **Inbound v6 is default-deny and that held without intervention.** The UDM runs the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule` endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6 rules exist," which is wrong and alarming. Use `v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic` for both families with `Allow Return Traffic`; the only v6-specific allows are link-local plumbing (ND solicit/advert, RA, DHCPv6). ## The stability problem — design around it up front All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA, not a static assignment). A three-way mesh where every node can move is fragile; WireGuard tolerates one roaming end, not all of them. The fleet already solves this on the v4 side — IPsec peers use **hostnames** (`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet Cloudflare DNS-edit token, so this is self-serve. ## Access recipes (cost a prior session real time) - **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`, `unifi/esh-udmpm-api-key`) against `https:///proxy/network/…`, `curl -sk`. Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here. Writes are `PUT …/rest/networkconf/<_id>` with the **full** object. - **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused (`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says `root@10.250.50.252`; believe it. - **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev), password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration `drain()` hangs the session**; read until the `ana-gw #` prompt and answer `--More--` with a space. Two invocations timed out at 3 min before this was fixed. ## Changes made and reverted this session - **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`, verified. Pre-change snapshots kept in the session scratchpad only (ephemeral). - **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf` (client configs carry private keys) were all mode **644**. Now **600**, and `keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact — WireGuard holds keys in kernel memory, so no restart was needed. The parent `/etc/wireguard` was already 700, which capped the real exposure to root-capable contexts inside the LXC — but the modes were still wrong. --- ## CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the ## per-REQUEST truth, not the total — eight /64s exist and are unclaimed The section above concludes AT&T hands out a single `/64` and that the `c110`/`c11f` pattern reading as a `/60` was a misread. **That conclusion was itself superseded later in the same session, and the correction never made it into memory** — it survived only in the session transcript, and was recovered 2026-08-24 while assessing a proposal to grab more prefixes. Reading the **BGW's own LAN statistics page** gave the whole picture: ``` BGW WAN v6 2001:506:70b2:8958::1 <- AT&T's transit prefix BGW LAN v6 2600:1700:b25:c110::/64 <- the BGW keeps this for itself Delegated 2600:1700:b25:c11f::/64 <- what the UDM got ``` **The BGW holds the `/60` and rations it**, keeping `c110`–`c117` for itself and delegating from the top down — the UDM got `c11f`, the last one. So `c118`–`c11f` are **eight delegatable /64s that genuinely exist and are yours**, sitting unclaimed. Both observations are compatible, which is why the first one looked conclusive: the prefix-ID test only carves *within* a delegation already held, so a UDM holding one `/64` cannot move it no matter what prefix-ID you set. The BGW issues **one `/64` per IA_PD request**, and **UniFi solicits exactly once**. **Consequence — the ceiling is the requester, not the carrier.** More prefixes need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this looks like BGW rationing rather than a provisioning-profile limit. Live state at correction time: `wan_dhcpv6_pd_size: 64`, `wan1 v6 2600:1700:b25:c110::48`, all 5 NH3 LANs still `ipv6_interface_type: none`. ### ⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise The seven unclaimed `/64`s stay unclaimed. Two facts close it: - **The BGW has no IP-passthrough mode.** Operator confirmed, and we hold admin on it — so the cheap path (let the UDM take the `/60` directly and carve it natively, as it already does at ESH) does not exist here. - **The only remaining route is a multi-DUID DHCPv6 client on a VM**, which requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing (UDM for v4, VM for v6), and — the actual cost — **rebuilding the whole IPv6 firewall policy in nftables on that VM**, because routing v6 around the UDM bypasses its zone firewall entirely and would leave every LAN host globally reachable. Operator's call: not worth it. **NH3 LANs stay `ipv6_interface_type: none`.** Do not re-propose on the strength of "there are seven free prefixes" — the prefixes are real, the firewall rebuild is why nobody wants them. _Archived 2026-09-02._ - `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4 3.0.0 -> 3.1.1 at the v3 cutover, so the staged v2.1.2 wheel is moot._ # eRP dual-seat overhaul — MeroMero-v2 + Dark-Scarlett, NVFP4A16 @ 256K on ana-ml2 `[2026-08-12]` Replaced the two legacy char-rp seats with home-quantized NVFP4A16 vLLM seats. Operator-driven, end to end this session. ## What landed | Seat (LiteLLM alias) | Model | Role | GPU | Context | |---|---|---|---|---| | `char-rp` (:8016) | **G4-MeroMero-v2-31B** (Gemma-4) | non-thinking PROSE, **multimodal (vision)** | GPU0 | 256K @ 2.07× (util 0.52) | | `char-rp-reasoning` (:8018) | **Dark-Scarlett-v1.0-27B** (Qwen3.6) | THINKING (default) | GPU1 | 256K @ 1.62× (util 0.44) | - Both **NVFP4A16 weight-only** (llm-compressor, `compressed-tensors`), `--kv-cache-dtype fp8`. - Replace: `char-rp-gguf` (Magidonia-24B GGUF/llama.cpp, :8016) + `heretic2-charrp-reasoning` (DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP, :8018). Old stacks/containers **stopped + retained** for rollback. - Compose-ified: `stacks/meromero-charrp` + `stacks/darkscarlett-charrp-reasoning` (ana-ml2 `/opt/docker/compose/`, mirrored to eshpfi, commit **`f08b6cb`**) → survive reboot. - Research that drove picks: `docs/pfi/erp-thinking-finetunes-2026.md` (from the `gecko-65` Booth). ## Load-bearing lessons (the whole point of this file) 1. **Load via the ConditionalGeneration WRAPPER class, never `AutoModelForCausalLM`.** For a multimodal-capable base (Gemma-4, Qwen3.6), `AutoModelForCausalLM.from_pretrained` + `save_pretrained` writes a FLAT text config (`Qwen3_5TextConfig`, `model.layers.*`) that **both vLLM AND SGLang reject** (SGLang: "Qwen3_5ForCausalLM has no SGLang implementation"; vLLM wants `Qwen3_5ForConditionalGeneration`). Loading via `Qwen3_5ForConditionalGeneration` / `Gemma4ForConditionalGeneration` keeps the wrapper config they accept. **This was the DS blocker** — re-quant via the wrapper fixed it (`Dark-Scarlett-...-NVFP4A16-wrapper`). 2. **NVFP4A16 is weight-only → DATA-FREE.** llm-compressor infers `DataFreePipeline`; calibration data is unused (only matters for W4A4 activation quant). W4A16 chosen per NVIDIA's sm_120 long-context guidance (W4A4 KLD 2-4× worse past ~10k ctx). 3. **Load on CPU (`device_map=None`)** so llm-compressor onloads one layer at a time. `device_map= "auto"` packs the whole model onto the GPU and OOMs when the card isn't fully free. 4. **Both models are KV-EFFICIENT — the "dense = KV-hungry" worry was WRONG.** MeroMero (Gemma-4) uses **sliding-window attention** (most layers cache only a bounded window); DS (Qwen3.6) uses **hybrid GatedDeltaNet linear-attention** (3:1 linear:full, linear layers carry no KV). Both hit full native 256K easily. (MeroMero KV pool ~542K tokens at util 0.52.) 5. **MeroMero vision reconstruction.** The finetune ships `processor_config.json` (image_processor inline, `Gemma4ImageProcessor`) but NOT `preprocessor_config.json` — the old-format file vLLM's feature-extractor loader wants. **Even google/gemma-4-31B-it (ungated!) ships only processor_config.json.** FIX: extract the `image_processor` section → write `preprocessor_config.json` verbatim, serve WITHOUT `--language-model-only`. Verified (model correctly ID'd a red circle). Audio is config-declared but WEIGHTLESS (0 audio tensors). 6. **GPU placement.** Match the KV-heavier model to the roomier GPU. GPU0 (gen neighbor, ~54GB free) > GPU1 (utility cluster, ~45GB free). Swapped MeroMero→GPU0, DS→GPU1. Pins via compose `deploy.resources.reservations.devices`. ## Dead ends (tried + abandoned) - **DS via llm-compressor `AutoModelForCausalLM`** → flat config vLLM/SGLang reject. → wrapper class. - **DS via NVIDIA ModelOpt** → modelopt↔transformers **version deadlock**: current transformers supports `qwen3_5` but crashes modelopt's sparse-moe plugin (`issubclass()` on a non-class); modelopt 0.43.0 pulls an old transformers that can't load `qwen3_5` at all. Abandoned. - **DS via SGLang** → `Qwen3_5ForCausalLM has no SGLang implementation`. Abandoned, but it REVEALED that both engines need the wrapper (→ the fix in lesson 1). - **`device_map="auto"` for the quant** → CUDA OOM in the weight observer. → `device_map=None`. ## granite retired + gateway repoint - `vllm-granite` (granite-4.1-8b, fleet summarizer, GPU1) **`docker stop`ped** (reversible) to reclaim ~13.6GB GPU1 for RP context. - LiteLLM (`ana-docker:/opt/docker/conf/litellm/config.yaml`, backed up `.bak-pre-granite-down-*`): **`granite-4.1-8b` alias RETIRED** — commented out, now 404s cleanly (the `*` wildcard→llama-swap was decommissioned 2026-06-20, so no fallthrough). **`summarizer` + `classifier` REPOINTED to gen** (`hosted_vllm/qwen3.6-35b-a3b-heretic` @ :8015, `enable_thinking:false`) — both verified. ⚠ This LiteLLM change is **server-only / not version-controlled** (a follow-up). ## MTP — deferred DS's MTP heads were dropped by the CausalLM loader; **deferred, not restored** (spec-decode is net-negative at RP temps: ~38-52% accept at temp 0.8-1.25, below vLLM's 0.5 cutoff). The splice-back path (`splice_mtp.py` in the heretic2 work dir) exists if ever wanted. MeroMero (Gemma-4) has no MTP by architecture. ## On-disk / where things live - Quant pipelines: `ana-ml2:/tank/aimodels/meromero-v2-nvfp4-work/` + `/tank/aimodels/darkscarlett-nvfp4-work/` (scripts, BF16 source, NVFP4 outputs). - Compose stacks: `ana-ml2:/opt/docker/compose/{meromero-charrp,darkscarlett-charrp-reasoning}/`. - Gateway aliases (unchanged, port-based): `char-rp`→:8016, `char-rp-reasoning`→:8018. (char-rp was also fixed from the stale `magidonia-24b-v4.3` backend model name → `char-rp`.) ## Open follow-ups 1. LiteLLM granite/repoint change NOT version-controlled (server + backup only). 2. eshpfi unpushed (many commits this session incl. `f08b6cb`, `7bd7375`, `398b58a`). 3. MTP deferred (see above). 4. DS thinks verbosely (~13:1 reasoning:content) — eval item; consumers need generous `max_tokens`. 5. MeroMero full 256K needs util 0.55 (GPU0 ~1.8GB free, tight); ran at 0.52 for headroom (~4.6GB). _Archived 2026-08-27._ - `[2026-08-12]` **infra-ops now holds an all-zones Cloudflare DNS-edit token (vaulted) + wgtunnel Phase-0 DNS landed.** Operator handed over a `Zone·DNS·Edit` (all zones) CF token → `secret put nh3-dev/.config/cloudflare/infra-ops-dns-token` (round-trip verified; /tmp drop shredded). Fleet DNS is now self-serve for infra-ops (⚠ HIGH blast radius — all zones). First use: created `boring.phasefinal.com` CNAME → `ana-srv1.phasefinal.com`, **DNS-only** (proxied:false), verified resolving to 38.120.12.44 on both authoritative NS (louis/wren) + 1.1.1.1 — NOT Cloudflare-proxied. Unblocks wgtunnel's wstunnel ACME cert. phasefinal.com zone id `f812ba74ed9a75cf21bbe7ce9188db50`. auto-memory `reference_infra_ops_cloudflare_dns_token`. (Earlier gap: the only prior vaulted CF token, jackdaw's, had `zone:read`+`worker:edit` but no `dns_records:edit`.) _Archived 2026-08-27._ - `[2026-08-12]` **wgtunnel stood up as its own repo (`vh/wgtunnel`, private) after a live endpoint-verification pass.** Operator directed own-repo (mirrors stonehenge-park/tts-stack). Verified off the fleet before seeding: `ana-wg` WG server = **UDP/31337** (not 51820), subnet 10.30.10.0/24, MTU 1420, active roaming peer proves the public UDP DNAT works; traefik on ana-docker **terminates TLS :443** (ACME `anaprod` http-challenge, docker+file providers, CrowdSec bouncer) → confirms the clean design (wstunnel container on `traefik-net`, Host-routed, WS→UDP to `ana-wg:31337`); edge `38.120.12.44` direct-A, `tunnel.phasefinal.com` free (⚠ must be **direct**, NOT Cloudflare-proxied like vaultwarden). Repo pre-seeded (README/CLAUDE/persistent-memory/ROADMAP + `docs/verified-infrastructure.md` = ground truth) + pushed; commit `9584d38`, Vuong-attributed. vh gitea token pulled from the vault (`secret get`), not persisted to `.git/config`. **NEXT = `/vor-plan` or `/vor` (operator's call, interactive).** Deps to line up in the plan: DNS A-record, FortiGate :443 host-routing, a new ana-wg peer for the laptop, client tooling. _Archived 2026-08-27._ `[2026-08-11]` **stonehenge-park — new fleet `/park` service repo stood up + designed.** **What.** A separate greenfield repo (`~/development/stonehenge-park`, gitea `vh/stonehenge-park`, pushed) for a self-contained `/park` service: one durable place to park any idea (repo-born OR personal), find it by search, and have it **actively resurface** (by due-date or staleness) until acted on — so parked ideas stop dying when a repo goes cold. NOT part of eshpfi; this is a pointer. **Design (via `/vor-plan`, converged + persisted to `docs/design/`):** four contract-sized units — **U1** core store+API (SQLite+FTS5, slug minting, bearer auth, REST) — the tracer, build first; **U2** scheduler+notifier (in-process; due/stale → statusline `due-count` + althing push to a dedicated **assistant channel**; keep-surfacing until promote/drop/re-snooze); **U3** `park` CLI (mirrors the `secret` CLI); **U4** browse UI. `/vor-ui` ran too (U4 brief persisted). **Locked decisions (operator):** SQLite, self-contained, ONE container, no external DB ("don't want to troubleshoot it when a database upgrade happens") — a hard `[OPS]` invariant; system-minted title-derived slugs + short ID (addressable as `park/`); active keep-surfacing resurfacing with **re-snooze as the anti-nag valve**; bearer key, LAN/WG-internal; host nh3-docker; `/park` **replaces** the global ROADMAP parking-lot discipline (deferred ideas → `/park`, `source`-tagged; ROADMAP keeps only the v1 target) as a **fast-follow after v1** incl. migrating existing lots. **Deferred (in the plan):** the althing assistant-channel handle **name** (decide at U2 contract time); staleness threshold + re-push cadence (env-tunable defaults ~30d/~daily); design U2's emit structured/consumable so a future **mission-control (Ledger→orchestrator)** can read it — park does NOT build the orchestrator. **State.** Pre-seeded for a fresh agent (CLAUDE/persistent-memory/ROADMAP/README + the design docs), committed (`294ee98`), pushed. Next build task lives in that repo: the **U1 tracer contract** under the House Code Discipline. Auto-memory candidate not yet written (repo is self-documenting). _Archived 2026-08-27._ - `[2026-08-12]` **Global `~/.claude/CLAUDE.md`: `secret`/vault tool entry + "store in AND pull from the vault" standing directive** (dotfiles `9db703b`, pushed); statusline reset-countdowns + a latent tab-collapse parse-bug fix, now tracked in the dotfiles stow tree. Dogfooded the directive: created `vh/stonehenge-park` pulling the gitea token via `secret get`. (dotfiles + global config, not eshpfi.) _Archived 2026-08-27._ - `[2026-08-11]` **TTS stack extracted to its own repo (`tts-stack`) + eshpfi stood down on TTS dev.** Operator: hand all TTS tuning/dev to a separate agent with a self-contained repo (knowledge + infra access + a live knowledge list), and move the voice corpus in. New repo `~/development/tts-stack` (commit `9ee3288`) carries: dots-tts stack (canonical intent), `voices/` corpus (MOVED out of eshpfi), `KNOWLEDGE.md` (engine landscape + prosody findings + foot-guns), `docs/infrastructure.md` (irv-ml1 access + gated deploy runbook + rollback), CLAUDE/persistent-memory/ROADMAP, `tools/` (pause-probe + Booth render). Followed the **chatterbox-fast precedent**: eshpfi `stacks/dots-tts/` reduced to a POINTER README; the ~15 experimental TTS compose wrappers stay here as reference (catalogued in tts-stack KNOWLEDGE). Blast-radius check: no eshpfi playbook/script reads the canonical corpus (other `voices/` refs = unrelated host paths). **Reverses** the earlier "Corpus home = eshpfi `voices/` (keep-here)" call. ⚠ tts-stack is LOCAL-ONLY until pushed — needs a gitea remote (`vh/tts-stack`) + push before the separate agent can clone (operator's call — outward-facing + repo-create creds). _Archived 2026-08-27._ - `[2026-08-10]` **dots-tts v3 — clause-break → period pause mapping.** Operator: v2 "sounds good" but donut won't pause at semicolons/dashes. ROOT CAUSE (measured via a pause-probe A/B — synth duration over N runs, non-determinism averaged out): dots' prosody honors a real pause **only for ellipsis (~+0.43s) and period (~+0.3s, capitalization-independent)**; comma/semicolon/colon/dash all run **flat (~+0.03s vs no-punct)**. Two distinct sub-causes: **dashes regressed in v2** (the `—`→`-` fold made em-dashes read as word-joiners), while **semicolons were NEVER a v2 change** — dots ignores them natively, only newly noticeable because v2 made everything else clean. Operator call: ellipsis "too much" → **map `;`, clause `:`, and em-dash `—` → period** in `_sanitize` (believable ~0.3s clause break). GUARDS (pinned by 11 unit tests, `stacks/dots-tts/test_sanitize.py`): digit-guarded colon `(?` checkout. _Archived 2026-05-27._ - `[2026-05-12]` asset-engine stack scaffolded LAN-direct at `http://10.250.50.70:8200`. Initially included Traefik labels for public hostname; user pulled them out (internal tool, no public TLS surface needed). Pattern: internal tools default LAN-direct; Traefik wiring only when external/TLS required. _Archived 2026-05-27._ - `[2026-05-12]` asset-engine catalog gains `lifecycle: { stack, vram_gb, gpu_device_id }` per irv-ml1 service for the orchestrator feature. SSH keypair scaffolded at `ana-docker:/opt/docker/conf/asset-engine/ssh/` for asset-engine container → irv-ml1 orchestration via dedicated ed25519 key. _Archived 2026-05-27._ - `[2026-05-13]` `pull-hf-repo.yaml` is the canonical HF-fetch playbook on ana-ml2. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` calls. _Archived 2026-05-27._ - `[2026-05-13]` Selene-1-Mini-Llama-3.1-8B added to llama-swap as judge model. mradermacher `i1-Q6_K` imatrix quant (~6.5GB). AtlaAI reward/eval model — temp 0.01, ctx 32K, q8_0 KV cache. New JUDGE / EVAL MODELS section in `stacks/llama-swap/conf/config.yaml`. _Archived 2026-05-27._ - `[2026-05-13]` `vllm-qwen3` → `vllm` stack rename. Added `vllm-reward` service (Skywork-Reward-V2-Llama-3.1-8B-AWQ classifier). Three vLLM services share GPU 1 (embed 0.20, rerank 0.20, reward 0.30 utilization; 30% headroom). All use `--runner pooling`; classification drives via model's `architectures: [LlamaForSequenceClassification]` in config.json, NOT `--task classify` (deprecated in vLLM 0.19.1). _Archived 2026-05-27._ - `[2026-05-13]` /tend-docs first pass deletions: `stacks/infinity/` removed (retired by vllm). Archived `docs/asset-engine/design-brief.md` → `docs/archive/asset-engine/` with archival header. Fixed `pfi-pve` VM list to full `qm list` enumeration. Dropped stale weak-password section from `pfi-postgres` (rotation done 2026-04-23). _Archived 2026-05-27._ - `[2026-05-14]` althing-chamber stack scaffolded: chamber + forseti. Internal LAN-only at port 7881 (chamber default 7878 collides with task-board). Two-service compose, shared SQLite bind-mount, build-on-host pattern via vh/althing's gitea-workflow. Forseti is the canonical dev for this stack (galdrabok is on a different project). _Archived 2026-05-31._ - `[2026-05-16]` althing-chamber Phase 2: added `althing-agent-runner` as third compose service (worldtree-driver agent dispatcher). All three althing services use the same image; `command:` selects entrypoint. Safe to enable preemptively (sleeps when no driver=worldtree handles declared). _Archived 2026-05-31._ - `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8 alpine as a sibling compose service in `stacks/althing-chamber/`, redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` / `msg_start` / `msg_complete` event kinds. DB bridge keeps `msg_curated` + `floor_grant` (structured / canonical). Two-channel architecture, no overlap. chamber + agent-runner `depends_on: valkey: service_healthy`. _Archived 2026-05-31._ - `[2026-05-17]` Worldtree admin workflow shift (per vh): infra-ops gets its own permanent admin-tier key (`61419c92`, stored at `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`). Future admin ops route through this key, not the bootstrap admin via docker-as-root. _Archived 2026-05-31._ - `[2026-05-17]` Worldtree env-var addition checklist: anytime introducing `os.environ.get("FOO")` in worldtree code, update BOTH `.env.example` AND `compose.yaml`'s `&worldtree-env` anchor in the same PR. Same Z_AI_API_KEY-shape footgun bit `BIFROST_CLIENT_ALLOWED_HOSTS` (#170) until worldtree-dev added the passthrough line in `08f02b2`. _Archived 2026-05-31._ - `[2026-05-18]` Volva systemd install complete after three-stage debug. Final unit at `/etc/systemd/system/volva.service` runs as `User=lkraven` with `ProtectHome=read-only` + `ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex` carve-outs for state writes. `VOLVA_ALTHING_CLI=/home/lkraven/ .local/bin/althing-cli` + `ALTHING_HANDLE=volva` both pinned in env.sh. _Archived 2026-05-31._ - `[2026-05-19]` Worldtree CD disk-hygiene strategy: watermark gate (env-tunable threshold + window, fail-loud on still-low post-prune) + eager post-deploy prune (only after `:latest` advance succeeds, uses `docker image prune -a --filter "until=24h"` which respects in-use semantic — protects pinned + personal images automatically). Combined: demo VM holds ~24h of deploy history instead of unbounded accumulation. Shipped in vh/Worldtree PR #184 (`306cd61` + `613dac2` + `bd91df5`). _Archived 2026-05-31._ - `[2026-05-19]` Skaldsong CD shape: shape (1) of three operator options — container + Gitea registry + pull-restart, matching Worldtree's pattern. Target host ana-docker (NOT nh3-dev where skaldsong-dev runs for iteration). SHA-pin only for now; health-gated `:latest` advance is a follow-up once `/health` exercises Worldtree + Kokoro reachability. _Archived 2026-05-31._ - `[2026-05-19]` Skaldsong prod (ana-docker) switched from demo Worldtree (:8080) to personal (:8081). Same `user_id=skaldsong` as the nh3-dev hand-launch key — shared Heimdall agent slot (`skaldsong:wizard-v2`), different `key_id`s for independent rotation. Demo Worldtree stays for isolation; personal becomes the multi-consumer dev iteration instance. _Archived 2026-05-31._ - `[2026-05-19]` mead-hall Bifrost v0.3 end-to-end smoke green. Closed task #32 (althing thread `01KRV1M2KW6N6HBEXGTH72QXCA`). Wire layer (handshake + binding + dispatch) + data-flow (per-dispatch JWT claims → `ctx.session_id` populated → real session-scoped data) + agent-loop (LLM reads + quotes back) all proven. Resolves the "stalled mid-Worldtree" state from the 2026-05-17 snapshot. _Archived 2026-05-31._ - `[2026-05-25]` v0.25.3 lofn tuning: `temperature 0.6 → 1.0` + `repetition_penalty 1.0 → 1.15` on default+fast profiles. Heretic-abliterated qwen3.6 was locking into degenerate attractors at the model's thinking-mode floor (0.6). Pattern: abliterated/uncensored Qwen variants need higher temp + non-trivial rep-penalty than base, NOT the model-card's documented floors. _Archived 2026-05-31._ - `[2026-05-25]` Worldtree #205 v0.25.2 ships `/app/config/` as bind-mount + root-then-drop entrypoint shim (gosu). Operators get persistent per-instance config without container-rebuild. Same bind-mount pattern hit twice subsequently in v0.27.0 (selene) and v0.29.9 (echo) — bind-mount shadows image-baked defaults, so every new required key surfaces as a crash-loop on existing deployments. The v0.29.12 canonical example files close this loop. _Archived 2026-05-31._ - `[2026-05-26]` Worldtree v0.27.0/v0.27.1 fixes Tier 3 GET visibility. `available_agents()` helper was over-applied to `GET /agents/`, masking ALL Tier 3 agents regardless of row state. Bug only visible as "agent not found" via GET; storage was fine (silent-2xx PATCHes had persisted correctly). v0.27.1 added fail-fast hardening for the startup pre-resolve fragility class. _Archived 2026-05-31._ - `[2026-05-26]` Skaldsong v0.30.7 defensive 409→PATCH fallback. v0.30.6's GET-then-define-or-patch path crash-looped against pre-v0.27.0 Worldtree's GET-visibility bug (GET 404 phantom → define 409 conflict). v0.30.7 catches the 409 and falls through to PATCH (which silently 2xx'd on the pre-fix Worldtree). _Archived 2026-05-31._ - `[2026-05-27]` Worldtree v0.29.x landed full saga→echo refactor + config-validator hardening (v0.29.10 create_provider family-before-regex; v0.29.11 collect-then-raise echo startup validators; v0.29.12 ships providers.yaml.example/defaults.yaml.example canonical configs; v0.29.13 reasoning_content extraction + catalog family lookup). Operator-asked, worldtree-dev-shipped, CI-deployed. _Archived 2026-06-01._ - `[2026-05-27]` artemis-31b-v1i added to llama-swap + worldtree personal. BeaverAI Gemma 4 31B Q6_K (~28.6GB), 128K ctx, `--reasoning-format deepseek` (gemma format unsupported in deployed llama.cpp). Worldtree catalog `family: gemma` so GemmaProvider routes reasoning tokens. _Archived 2026-06-01._ - `[2026-05-27]` Skaldsong streaming TTS v0.32.0→v0.32.2: chunked-batch SSE (one Kokoro POST per paragraph); defensive event_stream catch-all; NDJSON parsing for Kokoro /dev/captioned_speech multi-line responses. _Archived 2026-06-01._ - `[2026-05-31]` Dia2 deployed as **two fixed-model instances** (`dia2-2b` :8200, `dia2-1b` :8202) from `local/dia:v2`, retiring legacy Dia 1.6B; catalog `dia` entry **removed → dia2-2b + dia2-1b** (breaking for asset-engine). Rationale: the devnen wrapper is single-model and IGNORES the OpenAI `model` field (verified on its live OpenAPI), so the only way to offer both Dia2 models as real per-request asset-engine choices is one fixed endpoint per model. `3139e81` (deploy), `db15638` (catalog swap). _Archived 2026-06-03._ - `[2026-05-31]` Both dia2 catalog entries route to the wrapper's richer **`/tts`** endpoint (not `/v1/audio/speech`) to expose the full control surface (cfg_scale/temperature/top_p/cfg_filter_top_k/voice_mode/clone); all defaults sourced from the wrapper's `CustomTTSRequest` Pydantic blessed values. Voice default is `voice_mode: clone` + `clone_reference_filename: Abigail.wav` so a stable (non-random-gender) voice is the out-of-box behavior. `55602b7`, `5c47843`. _Archived 2026-06-03._ - `[2026-05-31]` Zonos REST adapter (`stacks/zonos/adapter/`, `local/zonos-api`) — thin OpenAI-ish `/v1/audio/speech` FastAPI in front of the Gradio-only Zonos SDK; JSON-envelope `{audio, audio_format, seed}` (Zonos is the fleet's first seedable TTS). Port 8203 (moved off 8201 — collided with csm). Built; NOT deployed (stack down for VRAM). Also fixed the upstream image's missing CMD (`71df6f7`). `81efa8d`. _Archived 2026-06-03._ - `[2026-05-31]` Catalog schema regenerated: added `CatalogLifecycle` + `reproducibility.seed_field` (`b7b2130`). Resolves the stale-schema hand-off; catalog now validates clean. (asset_engine `catalog.py` Pydantic already supported both — schema file was just behind.) _Archived 2026-06-03._ - `[2026-05-31]` TTS bench expanded with `stacks/{dia,zonos,csm}` (`666f7f3` dia+zonos, `a4b8c2a` csm). The bench already had Fish S2-Pro / Chatterbox-Turbo / IndexTTS-2 / CosyVoice3 / Kokoro / VibeVoice / Qwen3-TTS / Kyutai. (csm since removed 2026-06-01.) _Archived 2026-06-03._ - `[2026-05-31]` Remote browser/iPad/Vision-Pro driver seat for the agent-fleet zellij session `Claude` stood up on nh3-dev (ttyd behind Caddy, network-gated). Out of this repo — full architecture + the HTTP2/OSC52/Safari-auth gotchas in auto-memory `reference_ttyd_fleet_seat`. _Archived 2026-06-03._ - `[2026-05-30]` esh-docker-vm NFS boot-ordering fix: `playbooks/fix-esh-nfs-boot-ordering.yaml` (`c0458d9`, + `53157b1` drop-in filename-collision fix) adds `_netdev,nofail` to the four 10.0.50.50 NFS mounts + a docker `After=remote-fs.target` drop-in — resolves paperless `Exited(255)` on reboot. traefik also gained `restart: unless-stopped`. Full incident → auto-memory `incident_esh_docker_nfs_boot_race`. _Archived 2026-06-03._ - `[2026-06-01]` **Fish cloning VERIFIED competent (ECAPA-TDNN)** — retracting the earlier "weak cloner" call. Isolated test: Imogen-referenced clone ~0.79 cosine to the real `Imogen.wav` vs ~0.10 for the no-reference default; transcript condition (correct 0.787 / bogus 0.778 / empty 0.738) barely moves identity (affects pronunciation, not timbre). Root cause of "Imogen sounds nothing like British" = the `"undefined"` select bug feeding Fish its default voice, NOT Fish. So the entire Fish-Imogen saga was the undefined bug; no Fish-side fix needed. (Methodology lessons → Tried and abandoned.) _Archived 2026-06-04._ - `[2026-06-01]` **CSM (Sesame csm-1b) torn down entirely** — removed from catalog, `stacks/csm/`, `playbooks/deploy-csm.yaml`, and host (`c54ab13`). Two reasons: (1) deep-research verdict — the acclaimed Maya/Miles demo is a fine-tuned, larger variant Sesame never open-sourced; the open 1B base ships no usable voices, can't generate text, English-only, no real-time streaming, absent from leaderboards, dominated by Kokoro/Dia2/Fish/IndexTTS. (2) The upstream image is rotted (see Tried and abandoned). asset-engine re-synced (`306aae9`). _Archived 2026-06-04._ - `[2026-06-01]` **Chatterbox catalog v1→v2** (`93cedd6`) — re-point to the devnen wrapper's `/tts` (was the thin OpenAI `/v1/audio/speech` exposing none of the emotion knobs → poor output). Default `exaggeration: 0.5` per Resemble's README ("works well for most"); the server SHIPS demo-tuned 1.3 which over-acts — deliberately NOT adopted. 28 predefined voices via `/get_predefined_voices` (default Emily.wav). Corrected seedable false→true. _Archived 2026-06-04._ - `[2026-06-01]` **Fish-S2 catalog v1→v6** (`f3c260e`..`38d9e3b`) — was a blank 2-field form; now exposes the full fish-speech ServeTTSRequest surface + a 32-voice `reference_id` "Voice" dropdown (STATIC list — Fish has no /voices API). `reference_id`-by-name works ONLY when each voice is staged as a `references//` dir with `.wav` + `.lab` — was silently a no-op until that layout fix on 2026-06-01 (see the two-root-causes entry above); the inline-path-as-filesystem-path `references` shape 500s (see Tried and abandoned). Corrected vram 4→24 GB, cold_start 8→240s, seedable false→true. Added 3 consenting British-female VCTK voices (Imogen=p225/Eleanor=p228/Beatrice=p229). _Archived 2026-06-04._ - `[2026-06-01]` **`blendable: false` catalog flag — DEFERRED**, pending asset-engine-dev honoring it in the form template. Clone-voice selects (chatterbox/fish/dia2) must render as plain dropdowns, not the Kokoro blend widget (blending reference clones is nonsensical). I ship the catalog flag + schema field once they read it. Tracked: althing msg `01KT2K2SY9N7AY69R9V0B4RXSW` to asset-engine-dev. _Archived 2026-06-04._ - `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — Populated `references//.wav`+`.lab` for all 32 voices; playbook gained normalize-step + A/B smoke gate. glados got a real transcript (ASR'd via Parakeet): the Portal "Welcome to test chamber 4" lines. _Archived 2026-06-11._ - `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast` streaming container.** Workload = single-stream interactive. GPU placement: 3090 (device 0) if it fits else A6000 (device 1). Cutover: parallel catalog entry, burn in beside live `chatterbox`, then flip. Streaming = adaptive buffer-ratchet chunking (native frame-streaming abandoned). Since extracted to its own repo `vh/chatterbox-fast` (auto-memory `reference_chatterbox_fast_repo`). _Archived 2026-06-11._ - `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** Claimed naive sentence-level streaming has "zero quality loss" — WRONG. The T3 AR backbone conditions prosody on the WHOLE text; splitting loses cross-sentence prosodic context even though voice timbre stays (reference-conditioned). No *artifacts* ≠ no *quality loss*. Hence adaptive-chunk design (maximize context per chunk subject to latency budget), not fixed per-sentence splits. _Archived 2026-06-11._ - `[2026-06-03]` **Declined worldtree v0.30.4 staging deploy** — that's worldtree-team's CI/CD lane (a developer `staging/vX.Y.Z` git-tag promote), not infra-ops. They self-corrected to the same conclusion independently. _Archived 2026-06-13._ - `[2026-06-03]` **R14 scope = (a) provision-only.** infra-ops provides box + CUDA env + engines + weights + NFS; brokkr/dev wires `arms.py` + runs — keeps infra-ops OFF the VIVAE-processing path (VIVAE = Variably Intense Vocalizations of Affect/Emotion, CHARTER §4 highest-liability; operator authorized R&D-eval-only, quarantined). Box = irv-ml1 (A6000 free; ana-ml2 GPU-saturated). Per-engine venvs (divergent torch stacks); A6000 = `cuda:0` NATIVE (≠ docker `=1`). _Archived 2026-06-13._ - `[2026-06-03]` **yt-voice-clipper push-to-deploy via gitea webhook** (operator-directed, after 6 manual rebuilds in ~40 min). Webhook (not poll) — gitea CAN reach the WG IP per the operator. The proxy env + Homepage labels live in the **host-specific override** (untracked → survive the auto-deploy's `git reset --hard`), NOT yt-voice-clipper-dev's image. Runbook `d4f180d`. _Archived 2026-06-13._ - `[2026-06-03]` **yt-voice-clipper bot-gate fix = route yt-dlp through NH3 residential egress, NOT cookies/PO-token.** YouTube hard-flags the Irvine colo IP (LOGIN_REQUIRED on a public video even with no cookies). Cookies + the bgutil PO-token + deno JS-runtime all loaded fine — the gate is pure IP reputation. Operator chose proxy-via-nh3-dev → durable dante proxy → proven. The egress proxy is a reusable fleet lever for any datacenter-IP-gated service. _Archived 2026-06-13._ - `[2026-06-04]` **`brokkr` user (no-sudo) on irv-ml1; R14/R15/R16 substrate moved to /home/brokkr.** Persistent box services there need SYSTEM systemd units (see Tried). _Archived 2026-06-14._ - `[2026-06-04]` **ollama upgraded 0.9.0→0.30.4 on irv-ml1** (Ministral-3 is a Dec-2025 model the old engine refused); A6000 pinned by **UUID** not index (native fastest-first ≠ nvidia-smi PCI). _Archived 2026-06-14._ - `[2026-06-04]` **Worldtree demo/pinned/personal deploys are ALL CI/CD, not infra-ops** — a "deploy vX.Y.Z" request to infra-ops is MISROUTED → point them back to their pipeline. (`d8d776c`, auto-memory `reference_worldtree_deploys_cicd`) _Archived 2026-06-14._ - `[2026-06-04]` **infra-ops NOPASSWD-sudo identity commissioned, scoped to PFI boxes** (+esh-docker-vm by operator override) — so infra-ops completes DevOps end-to-end vs handing the operator sudo steps. Dedicated key, sudo log_output, key-gated. (`8c32a05`, auto-memory `reference_infra_ops_sudo_identity`) _Archived 2026-06-14._ - `[2026-06-04]` **phi4 ships the CANONICAL/official Phi-4 chat template, NOT Ollama's** (Ollama's omits the system `<|end|>`). Applied an Ollama-matching override then reverted — ship correct, not the benchmark quirk. (`90e08f0`→`27eb537`; "headgun" lesson in Tried.) _Archived 2026-06-14._ - `[2026-06-04]` **phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer; granite-4-small retired** from llama-swap. 50K ctx + FP8 KV. (`40a374b`) [superseded by the 2026-06-05 granite-4.1-8b decision] _Archived 2026-06-14._ - `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons; Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid `12aa5a9`); volva.service was crash-looping 203/EXEC. (`6e2f80e`) _Archived 2026-06-16._ - `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer.** Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not Q4/AWQ** — primary workload is high-concurrency, where FP8 scales ~linearly (2010 tok/s @ C=32). vLLM `vllm-granite` :8004 GPU 1, official IBM compressed-tensors FP8. (Then on Ada; box has since gone Blackwell.) (`34a43a0`, auto-memory `reference_ana_ml2_vllm_granite`) _Archived 2026-06-16._ - `[2026-06-05]` **Langfuse v3 on ana-docker (:3001) as the gateway trace UI**; LiteLLM `success_callback:[langfuse]` live. Pretty traces + tok/s dashboard. NOT a prerequisite (spend_logs already capture tokens+latency). (`9171e6a`) _Archived 2026-06-16._ - `[2026-06-05]` **Ollama BANNED fleet-wide** (operator directive) — never stand one up; tear down any found; serve via llama-swap or vLLM. (auto-memory `feedback_avoid_ollama`) _Archived 2026-06-16._ - `[2026-06-05]` **ComfyUI / FLUX.2 work split to `~/development/comfy-dev`** (dedicated repo + agent). eshpfi keeps the `comfyui`/`arbo` stack compose; comfy-dev owns the model/workflow knowledge. (auto-memory `reference_irv_ml1_ampere_quant`) _Archived 2026-06-16._ - `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is the structured-output profile, ON HOLD, no live consumer). Bind-mount CONFIGS (providers.yaml, vh-owned) ARE infra-ops's to apply directly — only the app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`) _Archived 2026-06-16._ - `[2026-06-13]` **Arbo prod bearer auth turned OFF — WireGuard is the access boundary** (operator decision; reverses ADR-0001's "closed the open-auth hole"). ENGINE_TOKEN must be **ABSENT, not empty** (empty-string still gates) — removed from BOTH the host `.env` AND the compose `environment:` injection line. Original token backed up at `irv-ml1:/opt/docker/compose/arbo/.env.pre-auth-off.bak`; comfy-dev updated their ADR-0001. Tracking: `db97899` + `playbooks/arbo-disable-engine-token.yaml`. _Archived 2026-06-16._ - `[2026-06-13]` **Storetank image-models archive DECOMMISSIONED; arbo is the single live ComfyUI model tree (502 G).** Curated `/storetank/image-models/comfy` (was 919 G, the native `/opt/ComfyUI/models` symlink target) → killed everything superseded by arbo's current gen (Hunyuan, WAN2.1, FLUX.1, Chroma, SD3.5, orphaned umt5+llava ≈ 739 G) + migrated the keepers (gen-agnostic utilities + the SDXL/Pony stack, 177 G) into `/storetank/arbo/models` (same-fs move, skip-existing protects prod). Tracking: `docs/storetank-image-models-archive.md` + `docs/arbo-comfyui-model-catalog.md` (`1902425`→`5007ec1`). _Archived 2026-06-16._ - `[2026-06-13]` **GRANITE_KEY provisioned to comfy-dev's nh3-dev dev env** at `~/.arbo_granite_key` (0600) for the hero gen+judge script — verbatim copy of the prod `arbo-prompt-enhance` vkey (now extended to reach BOTH `granite-4.1-8b` AND `qwen3.5-9b-fp8`); nothing minted. The vkey README's "granite-only" wording was stale → corrected (`f32c6dd`). _Archived 2026-06-16._ - `[2026-06-13]` **ana-ml2 upgraded Ada → dual RTX PRO 6000 Blackwell Max-Q** (96 GB each, cc 12.0 / sm_120; was dual RTX 6000 Ada 48 GB / cc 8.9 — confirmed live via `nvidia-smi`). Unlocks NVFP4 (FP4 tensor cores) and doubles VRAM headroom. CLAUDE.md GPU-spec doc-fix **LANDED `355a240`** (operator). Tracking: `19a07b9`/`1e2a3a1` ("Blackwell 96GB"). _Archived 2026-06-16._ - `[2026-06-13]` **NVFP4-W4A4 is infeasible for Granite — FP8 stays the Granite-on-Blackwell format.** W4A4 collapses at 30k context, proven **producer-independent** (modelopt AND llm-compressor both clean-NONE from the same BF16 base + wikitext-2k calib). No 4-bit wins both axes: W4A4 = quality collapse; W4A16-NVFP4/AWQ = weight-only dequant → bf16 (no FP4-core speedup). **30B retired**. (auto-memory `reference_nvfp4_w4a4_granite_infeasible`) _Archived 2026-06-16._ - `[2026-06-13]` **Qwen3.5-9B VL (FP8) deployed on ana-ml2 GPU 1** — `qwen35-vl` stack, :8007, gateway alias `qwen3.5-9b-fp8`. **Pinned nightly digest, not `:latest`**: the stable release quantizes the VL *vision tower* under `--quantization fp8` → garbage vision (LM fine); the nightly correctly excludes it. Re-pin + drop the pin once that exclusion lands stable. (`2e3dcc2`) _Archived 2026-06-16._ - `[2026-06-13]` **comfyui 325 G model tree migrated worktank → `/storetank/arbo`** (worktank 97% → 26%). `arbo` is the consuming app; overlay bind-mount via `COMFYUI_MODELS_DIR`. (`38186be`) (See the 2026-06-13 archive-decommission decision above — this tree later absorbed the storetank-archive keepers, reaching 502 G.) _Archived 2026-06-16._ - `[2026-06-13]` **GPU layout settled on the Blackwell box.** GPU 0 held free for large-model hot-loads (llama-swap pinned, `edf0f91`); GPU 1 steady-tenant — granite 131k ctx, qwen 65k, embed/rerank/reward trio, ~3.5 GB free (`1e2a3a1`, `19a07b9`; trio re-floored for 96 GB, 20×-parallel-stable). embed/rerank left at floor — long docs chunked BEFORE embedding. max-model-len is a ceiling not a reservation. _Archived 2026-06-16._ - `[2026-06-13]` **Prefix caching pinned explicit on granite + qwen** — benched ~6.5× faster TTFT on a shared ~4.5k-token summarizer template; soft/evictable, neutral when prefixes don't repeat. vLLM v1 defaults it ON (granite) but the qwen nightly defaults OFF — pin both. (`a9a2be7`) _Archived 2026-06-16._ - `[2026-06-13]` **granite-4.1-8b listed as the always-available summarizer/classifier + a shared all-agents key minted** (operator-directed). Global `~/.claude/CLAUDE.md` Global-tools entry; key alias `all-agents-local`, scoped to the FREE local models only (granite + qwen-vision + embed/rerank, NOT paid GLM), internal-gateway-only, rotatable. (auto-memory `reference_litellm_gateway`) _Archived 2026-06-16._ - `[2026-06-13]` **arbo engine + frontend stack stood up** (ADR-0001) — irv-ml1 co-located inference engine (`ee57e69`), python-based healthcheck (slim image, no curl/wget, `bdb3312`), frontend ro-mounted from the checkout (`922e8ad`, ADR-0001 D2). _Archived 2026-06-16._ - `[2026-06-11]` **GLM thinking inverted at the LiteLLM gateway** (operator call): `glm-5.1` defaults thinking-OFF; `glm-5.1-reasoning` = same z.ai upstream, thinking ON. Mechanism: `litellm_params.extra_body:{thinking:{type:disabled}}` — `drop_params` strips a top-level `thinking`/`reasoning_effort` but forwards `extra_body` verbatim to z.ai. Shared-gateway change. (`95b2701`, auto-memory `reference_litellm_gateway`) _Archived 2026-06-16._ - `[2026-06-11]` **pi coding agent installed on nh3-dev as a GLM 5.1 harness** — `@earendil-works/pi-coding-agent` via **bun** (npm's global prefix is `/usr` → needs sudo, bun avoids it). Config `~/.pi/agent/models.json`, launcher `~/.local/bin/glm`. _Archived 2026-06-16._ - `[2026-06-11]` **z.ai web-tools (regin) = z.ai hosted MCP path, NOT the `/paas/v4` Tool API.** WORKS: MCP streamable-HTTP at `https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp`, `Authorization: Bearer $ZAI_API_KEY` (the **MCP** key, distinct from `Z_AI_API_KEY` the LLM key). Reference impl = Worldtree's Leif agent. _Archived 2026-06-16._ - `[2026-06-10]` **Mac Pro migration framed: workstation-only** (M2 Ultra ARM, racked NH3 on-subnet); sidecars stay Linux. `migration-plan.md`. (See in-flight.) _Archived 2026-06-16._ - `[2026-06-10]` **Worldtree deployed-config propagation is infra-ops's OWNED lane** (operator ruling). worldtree-dev pings the config delta pre-merge; infra-ops syncs `config/*.yaml` from MERGED canonical to the `/opt/worldtree*/config` bind-mounts on demo+personal. The v0.33.8 9-HOUR demo outage (a `model_roles.yaml` startup-dep that never reached the VM) is the failure mode this prevents. providers.yaml stays hand-tuned. corviduo emergency-ops = `ssh vh@10.250.50.152`, docker no-sudo. (auto-memory `reference_worldtree_deploys_cicd`, `reference_corviduo_dev_emergency_ops`) _Archived 2026-06-16._ - `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr-smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev — granite, later extended to qwen-vision). Mint via `/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus. (auto-memory `reference_litellm_gateway`) _Archived 2026-06-16._ - `[2026-06-14]` **ana-ml2 GPU-1 vision upgraded: Qwen3.5-9B → Qwen3.6-35B-A3B (official FP8), served under its TRUE name only.** `qwen36-vl` replaces `qwen35-vl` on :8007 (`a0fed13`). The stale `qwen3.5-9b-fp8` name is KILLED at vLLM AND the litellm gateway (404/400) — a model is NEVER aliased under a prior model's name (silent substitution = downstream footgun; operator directive). Consumer comfy-dev/arbo migrated; arbo vkeys → all-proxy-models; shared `all-agents-local` key repointed qwen3.5-9b-fp8 → qwen3.6-35b-a3b. GPU-1 rebalanced for the ~34 GB FP8 weights (granite 0.35→0.24/64K; embed/rerank 0.05→0.03, reclaimed ~4 GB util-waste). Validated: vision correct, 20-concurrent = no OOM. (auto-memory `feedback_no_false_model_aliases`) _Archived 2026-06-18._ - `[2026-06-14]` **NVFP4 was the lighter fit (~21 GB) but is BLOCKED on vLLM — FP8 is the working vision path.** `nvidia/Qwen3.6-35B-A3B-NVFP4` won't load: the ModelOpt-NVFP4-MoE loader errors on expert/lm_head scale keys across 0.19.1 (`w2_input_scale`) AND 0.22.0 (`lm_head.input_scale`, vllm #44081) — a pattern across modelopt NVFP4 MoEs. Revisit NVFP4 (frees ~13 GB on GPU 1) once fixed; the 21 GB checkpoint stays cached on ana-ml2. **(SUPERSEDED 2026-06-16 — it loads on vLLM 0.23.0; qwen36 swapped to NVFP4. See the top of this section.)** _Archived 2026-06-18._ - `[2026-06-14]` **llama-swap qwen3.5-9b GPU-0 pin DROPPED; GPU 0 reserved for a creative-writing model (pick DEFERRED by operator).** Deep-research (this session) on big-fast-uncensored creative for a 96 GB Blackwell: **GLM-Steam-106B-A12B** (already in the llama-swap config — balanced default) vs **TheDrummer/Behemoth-X-123B-v2** (prose-tier, tops UGI writing+willingness) vs XORTRON-123B (max willingness, weak prose); GGUF-on-llama-swap is the serving path. Tracking: this session + llama-swap config (GLM-Steam present, `untracked by operator choice`). _Archived 2026-06-18._ - `[2026-06-14]` **R16 splice-pivot yield probe executed** (infra-ops ran the irv-ml1 inference for brokkr; brokkr owns design + analysis). See Current state. Tracking: althing thread `01KV010WGS…`, `gen_yield_probe.py` in `irv-ml1:~/r16-vmoan-harness`. _Archived 2026-06-18._ - `[2026-06-14]` **R16 vmoan inline-generation arc CLOSED — v1 at default decode (rep1.2/temp0.8) is the final Chatterbox-tag inline artifact.** Operator's ear rejected every alternative: v2/v3 windowing (omission vs coherence-loss), v4 multi-tag (cohesion held but lost to capacity-competition), emergent inline-token modulation (degenerates, not modulates), and the gen-time decode-polish sweep (soft tamers cut the NVV itself — same omission family as v2; p0 baseline beat p1). All adapters v1–v4 + `tokenizer.json.v3bak` preserved on `irv-ml1:~/r16-vmoan-harness`. Likely-next direction (deferred, NOT formalized): generate→bin→splice + one-shot-clone NVV pipeline routing around the inline-coherence wall. Tracking: brokkr R16 journal + althing thread `01KV010WGSSMPWRNCPAGSPK15Y`. _Archived 2026-06-18._ - `[2026-06-14]` **Arbo deploy pipeline fixed, hardened, and version-controlled.** Prod rebuilt v0.11.1 → **v0.11.6** backend; the webhook machinery (`arbo-deploy.sh` + `arbo-webhook.py`, :9009 HMAC listener) is now repo-tracked at `stacks/arbo/` (was host-only = recoverability foot-gun). Deploy reaches gitea via the INTERNAL route (`10.250.50.70:222`) and restarts the engine ONLY on `catalog/` changes (graphs/frontend per-request; warn on `src/`|`Dockerfile` only — pyproject/uv.lock churn every commit). Operator kept arbo stack ownership in **eshpfi** (not migrated to comfy-dev's repo). Secret + `.env` stay host-only. Tracking: `6d66bc2`, `6e58e57`, `stacks/arbo/README` Q5. _Archived 2026-06-18._ - `[2026-06-15]` **qwen36 thinking-split at the gateway** (operator) — base `qwen3.6-35b-a3b` = thinking-OFF default (the vision judge needs it), `…-a3b-thinking` = opt-in reasoning. One hybrid checkpoint + `--reasoning-parser qwen3`; mirrors the glm-5.1 split. The judge's enable_thinking=false is what makes the NVFP4 swap transparent. (`6de0844`) _Archived 2026-06-19._ - `[2026-06-15]` **arbo engine rebuilt v0.11.6 → v0.11.22 from main; catalog re-applied (v0.11.23) — curated `/workflows` footer live.** src/ schema changes (Model/Lora.summary, Workflow.blurb/keywords + the /workflows endpoint) need a MANUAL image rebuild (the webhook warns-only on `src/`/`Dockerfile`). Built from `/worktank/arbo/repo`, bumped `ARBO_IMAGE` in the host .env, recreated + verified healthy (auto-rollback armed, unneeded). comfy-dev's v0.11.20 catalog push had first crash-looped prod (old baked schema `extra=forbid` vs new git-mounted catalog fields → ValidationError); they reverted the catalog, so rebuild-from-main loaded clean, then they re-applied (v0.11.23). comfyui is a SEPARATE compose project — the rebuild didn't touch it. _Archived 2026-06-19._ - `[2026-06-15]` **althing-core refreshed to v0.14.1 box-wide on nh3-dev** (forseti + heid both routed it; central-sequenced per DevOps-delegation since `uv tool install --force` swaps the shared binary all handles use). Fixes the monitor non-truncating-lock bug so `stop-monitor`/`--stop` reliably recovers orphans. The editable install (`uv tool install --force --editable .`) covers every nh3-dev handle; `--version` trued up 0.14.0→0.14.1; verified `--stop` works against a real orphan. _Archived 2026-06-19._ - `[2026-06-15]` **ratatoskr Tier-3 affect live smoke GREEN** (consumer-confirmed). Brokered/minted consumer_id="ratatoskr" Heimdall HS256 key on personal Worldtree (the key IS BOTH the create-session Bearer AND ratatoskr's JWT verifier secret — plain utf-8 string, NOT base64/raw); injected ENV-ONLY to `~/.config/ratatoskr/provider.env` (mode 600) on nh3-dev; added `10.100.10.50:8390` to personal's `BIFROST_CLIENT_ALLOWED_HOSTS`; drove define→session (handshake 200)→turn→affect.emit (200, snapshot persisted). Worldtree signs Bifrost with the caller's raw Bearer verbatim. Drive commands + claims contract in althing thread `01KV4ZYSE3…`. _Archived 2026-06-19._ - `[2026-06-15]` **Dense Qwen3-VL-32B-NVFP4 A/B'd as arbo's anatomy hero-judge vs the qwen3.6-35b-a3b MoE → dense LOST, pilot torn down, GPU 0 reclaimed.** Dense NVFP4 loads clean on vLLM where the MoE NVFP4 doesn't (no expert-scale keys); ~44 tok/s on Blackwell GPU 0; vision tower stays high-precision. But comfy-dev's ground-truth A/B: MoE wins (faster + dual-use as general LLM + slight fine-detail edge), and the subtle-proportion tier is below ALL VLs. **Don't re-propose a dense-VL judge swap.** (auto-memory `project-qwen3vl-dense-pilot`) _Archived 2026-06-19._ - `[2026-06-15]` **MastMed prototype containerized + exposed publicly via cloudflared.** Self-contained static SPA (bun→caddy multi-stage build), basic-auth gated (operator's gating call), ana-docker:8089, cloudflared quick tunnel. See in-flight for the ephemeral-URL caveat + named-tunnel upgrade path. _Archived 2026-06-19._ - `[2026-06-15]` **infra-ops bootstrapped onto corviduo-dev** (operator-directed) — infra-ops now has SSH + NOPASSWD sudo there, OVERRIDING the fleet bootstrap script's deliberate partner-box exclusion (explicit host arg). corviduo config edits no longer need `vh` + a sudo password. (auto-memory `reference_corviduo_dev_emergency_ops`) _Archived 2026-06-19._ - `[2026-06-15]` **R18 clip+caption placement = ana-ml2** (operator: "ana-docker… ana-ml2 unless irv-ml1 has room"; irv-ml1 lacked free VRAM for the 30B-A3B captioner). Models staged + stub deployed; see in-flight + the hard real-voice gate. _Archived 2026-06-19._ - `[2026-06-14]` **LiteLLM infra-ops admin key provisioned** (operator) — resolves the LiteLLM half of the credential-migration directive; use it for ALL gateway ops (NOT `sk-corvid`). Value at `~/.config/litellm/infra-ops-key` (mode 600); gateway reachable directly from nh3-dev at 10.250.50.70:4000. (auto-memory `reference_litellm_infra_ops_key`) _Archived 2026-06-19._ - `[2026-06-16]` **litellm `strip_empty_tools` pre-call hook shipped** (`d1bea13`) — an empty `tools:[]` 500s vLLM ("tools must not be an empty array"); a global `litellm_settings.callbacks` CustomLogger pops it (+ orphaned `tool_choice`) before forwarding, so it covers EVERY vLLM model, not one. `drop_params` only drops unsupported PARAMS, not empty VALUES. Mounts beside config.yaml (litellm resolves callbacks relative to the config dir). Verified live across granite/mistral/stream. (`stacks/litellm/conf/strip_empty_tools.py`) _Archived 2026-06-20._ - `[2026-06-16]` **single-file `gateway-chat.html` playground shipped** (`984ca3d`, `tools/`) — zero-dep browser chat straight to the gateway (`:4000`, CORS open), system-prompt box, streaming SSE, renders `reasoning_content`, NEVER sends `tools`. Built because the LiteLLM admin-UI playground can't test vLLM-backed models (see Tried-and-abandoned). Serve on-request via `python3 -m http.server -d tools`. _Archived 2026-06-20._ - `[2026-06-16]` **claude-bot granted ADMIN on vh/arbo** (operator) — arbo CI/CD gitea ops now route through the service account, not the operator's personal `vh` (the credential-migration directive). Operator added claude-bot as Administrator collaborator; claude-bot self-serves via its `gitea-password` (basic-auth = FULL perms; its stored token is package-write-only, can't change repo settings). (auto-memory `reference_claude_bot_gitea_creds`) _Archived 2026-06-20._ - `[2026-06-16]` **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven.** The missing server-side step was the per-PORT `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist (`:8391`; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the `score_breakdown` shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), not `agent_self_enabled`. (tracking: c99aa49 + auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED:** the `score_breakdown`-shape root-cause was disproven; real cause = cold-recall `scope_filter` adds an `agent_self` axis → ratatoskr's conjunctive INV-005 match drops the `{end_user}`-only chunks → 0 hits. Now Worldtree #297. (auto-memory `project_worldtree_295_recall_rootcause`) _Archived 2026-06-20._ - `[2026-06-16]` **infra-ops = the DURABLE admin path on corviduo-dev** (operator granted standing admin; extends the 2026-06-15 bootstrap) — added a `Host corviduo-dev → User infra-ops` `~/.ssh/config` alias + repointed `servers/corviduo-dev/ssh-target` to `infra-ops@`; `ssh corviduo-dev` / `elway corviduo-dev` now land as infra-ops w/ NOPASSWD (vh = explicit-only human path). elway resolves via ssh-config, NOT the ssh-target file. (tracking: c99aa49 + auto-memory `reference_corviduo_dev_emergency_ops`) _Archived 2026-06-20._ - `[2026-06-16]` **demo + personal `character` model flipped qwen3.6-35-a3b → mistral-small-4** (operator) — in `model_roles.yaml` the FIRST `catalog_id` in a role's `binds:` is the default; reordered mistral-first (qwen retained in the switch-allowlist, non-destructive), applied via the PIN-SAFE api recreate (derive `WORLDTREE_IMAGE` from the matrix sibling — never a bare `compose up`, which falls to the crash-blocking stale `:latest`), verified a fresh character agent resolves model=mistral-small-4. (auto-memory `reference_corviduo_dev_emergency_ops`) _Archived 2026-06-20._ - `[2026-06-16]` **NVFP4 ModelOpt MoE now LOADS on vLLM 0.23.0 — SUPERSEDES the 2026-06-14 "blocked" finding (#44081 fixed).** Confirmed the cached `nvidia/Qwen3.6-35B-A3B-NVFP4` loads clean + generates on 0.23.0; swapped prod qwen36 FP8→NVFP4 (~13 GB reclaimed on GPU 1, fp16 KV). Validated by brokkr (speed: faster single-stream, ≥parity batched) + comfy-dev (vision anatomy-judge A/B: PASS). One W4A4 tell: minor multi-step chained-reasoning slip — doesn't bite the vision judge. Mistral's NVFP4 is a SEPARATE compressed-tensors path (day-0), distinct from the ModelOpt one. (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`) _Archived 2026-06-20._ - `[2026-06-16]` **Mistral Small 4 = the GPU-0 tenant** (operator) — fills the slot the deferred creative-writing pick held, and supersedes llama-swap (now down). NVFP4 119B MoE, pinned **vLLM v0.22.0** for working vision (0.23.0 breaks it), 256K context. Operator's creative-writing model; **abliteration planned** → then it becomes the uncensored-creative model the persona substrate needs. Reasoning via `reasoning_effort` (BINARY none|high — no medium). (tracking: `c77a9aa`/`9a49963`) _Archived 2026-06-20._ - `[2026-06-16]` **Selene judge restored on vLLM FP8, NOT NVFP4** — W4A4 is too aggressive for a precision judge validated at Q6_K; FP8 ≥ Q6_K fidelity, and Selene is text-only Llama-3.1 so dynamic `--quantization fp8` has no vision tower to noise. `selene-1-mini-8b` on GPU 1 :8011. Worldtree's Domari role repointed to it. (`c985ede`) _Archived 2026-06-20._ - `[2026-06-16]` **GPU-1 buffer put to work: selene 0.13→0.17 + qwen36 0.32→0.34** — the two KV-bound generative models that gained Worldtree consumers; granite + the pooling models under-use their util, so growing THEM is wasted. (`e124a2f`) _Archived 2026-06-20._ - `[2026-06-16]` **claude-bot Gitea service account stood up; corviduo-ORG conversion TABLED** (operator). claude-bot is the credential-migration foundation. The proper fix for scoped agent package-write is converting vh (a USER) → a `corviduo` ORG — blast radius assessed (org creation = free; arbo-first migration = cheap, no registry images; full fleet = incremental, ~8 active vh images / 50 repos) — but tabled for later. Registry-push deferred until then. (auto-memory `project_migrate_infra_access_to_claude_credentials`) _Archived 2026-06-20._ - `[2026-06-16]` **arbo engine builds → comfy-dev OWNERSHIP** (operator approved) + Gitea Actions runner stood up on irv-ml1 (host-executor as lkraven). infra-ops keeps box-level escalation (runner, package:write PAT pending corviduo-org, webhook). See the runner entry in Current state. (tracking: this snapshot + arbo thread `01KV70AB2B4B…`) _Archived 2026-06-20._ - `[2026-06-16]` **Worldtree persona substrate migrated qwen→Mistral via capability- named profiles** (v0.35.8→v0.35.10; demo + personal). PRE-SYNC-FIRST is now the standing pattern for any providers.yaml/profile change: sync config to the bind-mount BEFORE the deploy image-swap restart, so new agents + new config land together (the v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread `01KV6X0M…`) _Archived 2026-06-20._ - `[2026-06-19]` **litellm task-aliases: `classifier`→granite-4.1-8b, `summarizer-large`→gen/qwopus** (`summarizer`→granite already existed). Also added then REMOVED `gen-nt`/`gen-reasoning-nt` (redundant with the global `strip_empty_tools` hook). (ef45f6d, 2e5ab72, 75bd4c3) _Archived 2026-06-21._ - `[2026-06-19]` **gateway-chat: model-smoking web chat → PERSISTENT container.** `/v1/models` auto-discovery + image-upload; `stacks/gateway-chat/` (nginx:alpine, ana-docker:8091, homepage tile). (4c40b9f, 740bcae) _Archived 2026-06-21._ - `[2026-06-19]` **pi on nh3-dev wired to `gen`** — `~/.pi/agent/models.json` + a `gen` launcher mirroring `glm`. Local box config, not a repo artifact. (untracked by operator choice) _Archived 2026-06-21._ - `[2026-06-19]` **`gen` model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8.** ana-ml2 GPU 0, served-name `qwen3.5-122-a10b`. STABLE vLLM + fp8 KV reaches 262144 — nightly+turboquant-4bit-KV was UNNECESSARY. 92.7 tok/s warm. (20e796c, 5b06514) _Archived 2026-06-21._ - `[2026-06-19]` **TP=2 across the two ana-ml2 Blackwells REJECTED** — `PIX` (PCIe, NO NVLink) → all-reduce-bound → SLOWER. One-model-per-card is optimal; PP=2 only if a >96 GB model is ever wanted. (untracked by operator choice) _Archived 2026-06-21._ - `[2026-06-19]` **qwen-image-bench (T2I judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1**, aliased `image-judge`. (bfae924, 5dfce04) _Archived 2026-06-21._ - `[2026-06-18]` **heretic abliterated Mistral Small 4 NVFP4 built + LIVE as `mistral-small-4`** (in-house NVFP4, drop-in stack, A/B'd vs official, "heretic stays"). DISPLACED from GPU 0 on 2026-06-19 for the gen/Qwopus swap. (dd3a5c9, f566f61, `tools/mistral-small4-nvfp4/`) _Archived 2026-06-21._ - `[2026-06-18]` **irv-ml1 VRAM consolidation + comfy-dev torch-pin** — ComfyUI pinned to the A6000 (48 GB), audio zoo → 3090; `DISABLE_UPGRADES=true` @ torch 2.12.1. (a8550ad) _Archived 2026-06-21._ - `[2026-06-18]` **ComfyUI acceleration set (9 nodes) installed for comfy-dev** on irv-ml1. _Archived 2026-06-21._ - `[2026-06-17]` **ana-ml2 `vm.overcommit_memory=1` made durable** (sysctl drop-in) — overcommit=0 + zero swap caps CommitLimit; large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff) _Archived 2026-06-21._ - `[2026-06-17]` **GLM 5.2 wired into litellm** (`glm-5.2` + `-reasoning`, z.ai passthrough, `extra_body.thinking.type` toggle). (fe77a35) _Archived 2026-06-21._ - `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (successor to nh3-ansible); hosts `/opt/externs` pi-on-GLM-5.2 client workspaces. (a841eab) _Archived 2026-06-21._ - `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-` prefix (operator) — reuse the existing capability convention.** gen/gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge (passthrough)/gen-frontier/-reasoning. The **swap-transparency rule**: embedding stays model-named (`qwen3-embedding`) because vectors aren't swap-transparent; everything else can swap behind a stable alias. (d372103, 7819f96) _Archived 2026-06-25._ - `[2026-06-20]` **Removed the litellm `*` wildcard → llama-swap** (operator; llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now 404 cleanly instead of a misleading "Connection error" — closes the long-standing footgun. (d0eb09c) _Archived 2026-06-25._ - `[2026-06-20]` **Langfuse retired** (operator) — redundant with LiteLLM native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse spew). Callbacks dropped, stack torn down. (f8eda1c) _Archived 2026-06-25._ - `[2026-06-20]` **OmniVoice streaming `/tts` — vendor chatterbox-fast's scheduler, not its whole package** (operator-approved vendor-copy). Diffusion TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox-fast remains the lowest-latency front-end. (288d085, cd92b85) _Archived 2026-06-25._ - `[2026-06-20]` **Daily image+buildx prune cron on ana-docker** (the missing automation behind the recurring image bloat) — `lkraven` crontab, prune unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context. _Archived 2026-06-25._ - `[2026-06-25]` **althing re-architected to the lean multi-machine bus; nh3-extdev stood up as a MODEL B mesh peer** (dedicated `althing-svc` + group-shared `/srv/althing`). `reference_nh3_extdev_althing_mesh`. _Archived 2026-07-08._ - `[2026-06-23]` **zellij native web client piloted on nh3-dev** (`zellij-web.service` :8443) alongside ttyd. `reference_zellij_web_seat`. _Archived 2026-07-08._ - `[2026-06-22]` **Worldtree persona-render config arc (#314/#322/#317) pre-synced + deployed green on demo+personal** — #317 a boot-blocking config REMOVAL. `reference_corviduo_dev_emergency_ops`. _Archived 2026-07-08._ - `[2026-06-20]` **R22 (brokkr/dwarves) stood down to gateway-only; full-access R22 key minted; Phase B CANCELLED** (Worldtree model-agnostic → no deploy path). Key at `/home/lkraven/.r22-gateway-key` (persistent mode-600, carries paid GLM, don't delete). MUT = free `qwen3.5-122-a10b` (`gen`). Operator steer: R22 research is gated on a pragmatic/deployable outcome, not advancing-the-art. _Archived 2026-07-08._ - `[2026-06-20]` **claude-bot issue-scope token minted for worldtree-dev self-serve** (id 16, `write:repository`+`write:issue`); old token revoked. Advances the credential-migration directive. _Archived 2026-07-08._ - `[2026-06-20]` **rest-server-ana recovered + backup prevention shipped + worldtree-dev admin keys provisioned** (demo d113207c / personal f4f75adb). Cred rotation (5 rest-server pw) BELAYED. _Archived 2026-07-08._ - `[2026-06-20]` **claude-bot → ADMIN on vh/Worldtree** (operator-authorized) — self-serves WT deploys/tokens henceforth. _Archived 2026-07-08._ - `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.** (auto-memory `project_migrate_infra_access_to_claude_credentials`) _Archived 2026-07-13._ - `[2026-07-01]` **Worldtree #332 embed-recall diagnosed + scoped-log view/tunnel provisioned + fix verified** (15×→1.01× re-embed). `reference_wt_gateway_scoped_log_view`. _Archived 2026-07-13._ - `[2026-07-01]` **qwopus native MTP speculative-decode tested on `gen` → NOT kept** (+12% single-stream, −15–20% aggregate at concurrency, silently drops min_p/logit_bias). Banked for T1. `reference_gen_qwopus_122b`. _Archived 2026-07-13._ - `[2026-07-01]` **Deckard trial → reverted to qwopus (`gen`)** (won writing "in every way" but ~36 vs ~90 tok/s; spec-decode rescue ruled out). git `b63c48b`→`681eb70`. Deckard kept staged as T1's writing benchmark. _Archived 2026-07-13._ - `[2026-07-02]` **mtf-dev granite harness-spike ran GREEN — MECHANICAL only, efficacy DEFERRED to the T1 run.** Trainer TRL SFT→DPO→eval seam proven end-to-end on a synthetic fixture (not the E-RP corpus); operator DECIDED no intermediate real-efficacy granite spike (uninterpretable proxy — arch gap + abliteration axis). `reference_gen_qwopus_122b`. _Archived 2026-07-13._ - `[2026-07-03]` **ratatoskr-dev DEMO Heimdall key provisioned (R30 φ0).** Minted a tier-user key on the demo via `POST /admin/keys` (bootstrap admin key), mirroring their personal base consumer (no character-binding); base-agent affect reads work ungated. `reference_worldtree_demo_key_mint`. _Archived 2026-07-13._ - `[2026-07-04]` **character-rp: gateway-forwarding proven + role pre-staged + #344 shipped.** Empirically confirmed per-request `extra_body` (top_k/repetition_penalty) forwards through the `gen-reasoning` LiteLLM alias to vLLM + standard params override the alias defaults — no gateway cap needed (I over-built a dedicated alias, operator corrected, reverted with zero fleet impact). Pre-staged the `character-rp` role into demo+personal bind-mount `model_roles.yaml` (byte-verified on b18; caught the cached-registry ordering). worldtree-dev shipped **#344 (v1.0.0b19)** for the durable-agent override-drop. spend_logs spot-check queued (ratatoskr's timestamp ping). _Archived 2026-07-13._ - `[2026-07-04]` **althing v2 herald+receiver formalized as systemd on nh3-dev.** `althing-herald.service` (Restart=always, **Environment=PATH incl ~/.cargo/bin** — the pane-dispatch fix) + `althing-receiver.service` (v2 → pillar-3 `/owner/*` live); stale forseti unit removed; both on v2.0.0b2, canonicalized by forseti. `reference_nh3_dev_althing_herald`. _Archived 2026-07-13._ - `[2026-07-04]` **LitBench-RM respun (irv-ml1 A6000, comfyui displaced)** for T1's reward ensemble; operator sole comfyui consumer, holding image-gen until LitBench done. `reference_litbench_rm_irv_ml1`. _Archived 2026-07-13._ - `[2026-07-05]` **T1 training venue: CLOUD recommended; operator chose smoke-first on ana-ml2.** On-prem ruled out (ana-ml2 full — both 96G GPUs ~93G used): keep-serving = NVMe offload ~6-8 DAYS; full ana-ml2 shutdown = CPU offload ~1 DAY but a whole-fleet outage. Cloud Vast.ai 8×80GB (no offload → ~3-6h, ~$60-500, zero fleet impact) is the clean alt (mtf-dev + infra-ops both rec; Vast for its no-content-AUP marketplace + likely-existing VastBlue account). Operator's next step = the ana-ml2 CPU-offload SMOKE (~60 steps) to get real samples/sec before the full-outage-vs-cloud call. HF base verified public (zero-auth pull). Runbook + gotchas in `reference_t1_cloud_train_plan`. _Archived 2026-07-13._ - `[2026-07-05]` **glm-5.2 canonical limits recorded** (probed live vs z.ai): **1,048,576 (1M) input context / 131,072 (128K) max output**; NO gateway-side cap (pure passthrough → z.ai's limits are effective). Written to the config comment (commit `624a07e`) + `reference_litellm_gateway`. _Archived 2026-07-13._ - `[2026-07-06]` **infra-ops NOPASSWD sudo confirmed on irv-ml1 + ana-ml2** (the arbo memory's "NOT infra-ops on irv-ml1" was stale) — enabled the worker deploy + every ana-ml2 hot-swap smoke. _Archived 2026-07-13._ - `[2026-07-07]` **SGLang zc hot-swap: 2-line patch RAN → PARTIAL. Cleared startup + self-attn E2E, GDN in_proj hit 2 MORE name-routing gaps → MERGE recommended.** Root cause was a NAME-routing gap (not the feared shape-assert): `lora/utils.py:235` hardcoded `params_mapping` lacked `in_proj_qkv/in_proj_z → in_proj_qkvz`. Patched those 2 lines (overlay bind-mount) → `init_lora_shapes` passed, server up, self-attn loaded end-to-end. But the SAME missing GDN mapping bit 2 MORE resolvers: adapter-load skipped 192 in_proj weights, and inference `utils.py:300 get_target_module_name` (substring match) RAISED → 500 + scheduler died. So GDN LoRA name-routing is incomplete in ≥3 SGLang paths (fixed 1). NOT shape-intractable (self-attn proves the separate-key→fused-buffer machinery works), but NOT "one targeted fix" either → per operator's LAST-shot rule → MERGE. mtf-dev looped + closed (thread 01KWZDZFYN…); B/A shard-math never reached. Supersedes the "SGLang wants a THIRD format" framing — no new adapter format needed. _Archived 2026-07-13._ - `[2026-07-07]` **T1 (retargeted qwopus-122B→AEON-27B) SFT COMPLETE on ana-ml2** — all-TRL torch-GDN, batch1/accum8/GC-off, 1 epoch ~9h @ 0.25 s/s, loss 2.445, adapter banked. Supersedes the 2026-07-05 cloud-vs-smoke venue decision: AEON-27B's 52G bf16 fits one Blackwell → ran on-prem in ~9h (no cloud, no full-fleet outage). GC is NOT the throughput lever (batch1-no-GC == batch4+GC at 0.25; dvalin's 0.565 was warmup-noise from a 30-sample smoke); ~0.25 s/s is the torch-GDN floor (only FLA beats it, sm_120- blocked #913). `reference_t1_cloud_train_plan`. _Archived 2026-07-13._ - `[2026-07-07]` **T1 hot-swap on vLLM NVFP4 is DEAD — vLLM 0.24.0 qwen3_5 LoRA-application silent no-op** (#47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4+FP8) AND adapter-format-agnostic (separate `zc` loads-but-inert; fused rejected); #47640 guard-patch overlay didn't fix. Root-caused by a 3-peer dwarf panel (eitri/regin/dvalin); their vLLM issue refs verified real via GitHub API. Engine mirror-image: vLLM wants SEPARATE keys, SGLang wants FUSED. _Archived 2026-07-13._ - `[2026-07-07]` **HARD REQUIREMENT restated: hot-swappable LoRA or ABANDON.** Merge is NOT a solution for this requirement (static; reserved for a later RP-quant track). SGLang is the last hot-swap shot; if it can't apply, park the adapter + file upstream repros. `reference_lora_training_worker` (adjacent). _Archived 2026-07-13._ - `[2026-07-07]` **comfy-dev in-arbo LoRA-training worker DONE** — built/deployed/validated Phases 1+2 (first Sindra train succeeded, publish-step + `train_id`), Phase 2.5 GC contract (shape A) confirmed + impl deferred. `services/lora-training-worker/`, commits 888ba6a→b617a8b. `reference_lora_training_worker`. _Archived 2026-07-13._ - `[2026-08-08]` **worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev.** worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared `~/snapshots/worldtree-400-fiction-decomp` (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). **LEFT INTACT:** `rex393-fiction-index`/`rex393-fiction-snapshot` (separate operator KEEP word, unchanged) + `r42-gate-*`. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (`01KZJ9GMCC…`). _Archived 2026-08-24._ - `[2026-08-07]` **chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed).** Long saga, operator-driven clean diagnosis. **Symptom:** ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. **NOT** German-leak (Turbo `generate()` has NO language param — plain AutoTokenizer, no `language_id`; the multilingual `language_id="en"` lever lives only in the separate `ChatterboxMultilingualTTS`), **NOT** OOM alone. **Real cause:** the Chatterbox **Turbo T3 model OVER-RUNS its generation tail** — a long single `generate()` degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens `<6561` + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). **ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE** — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). **Diagnosis method** (deterministic, no ears-only): single-shot length sweep + **amplitude-gated voiced-ZCR** (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. **FIX:** server-side `max_chunk_chars=250` cap on the scheduler (`:v2` image, `CBF_MAX_CHUNK_CHARS=250` env) — bounds each generation to just under the ~300-char onset → clean **3-4 sentence** chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; **chatterbox's low emotiveness keeps chunk joins smooth** (the harsh joins that got Zonos rejected are absent — operator's key call). **ratatoskr TODO (relayed msg `01KZER9X7S`):** revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. **Cap value tunable** per-request (`max_chunk_chars`) + env. **Deeper prosody** (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). **⚠ FOOT-GUNS:** (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) **build-context vs image drift** — the `:v2` image was built from cap source, but after a `:v1` rollback the build context held `:v1` source → a `docker compose build` would've silently produced a cap-less `:v2`; re-synced the flat cap source to `/opt/docker/compose/chatterbox-fast/` (rebuild-verified). **⚠ DIVERGENCE (follow-up):** deployed build context is FLAT (`app.py`/`scheduler.py`, `from scheduler import`, thin-overlay `FROM local/chatterbox:v1`, cap-only) vs the `vh/chatterbox-fast` REPO which is PACKAGE-layout (`chatterbox_fast/`, `from chatterbox_fast.scheduler`, self-contained Dockerfile) + has `norm_loudness` (repo commit `6bc7bf0` = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: `.bak-cap-20260807-104850` backups on irv-ml1 + `:v1` image both retained. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] _Archived 2026-08-24._ - `[2026-08-07]` **Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY.** Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. **⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init.** GPU memory was held by the `--multiprocessing-fork` CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). **RESTORE CMD** (from irv-ml1, user lkraven): `cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 &` then `docker start zonos-gateway`. **Consumers that lost Zonos:** asset-engine + gateway-chat (via LiteLLM `ext-tts` alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). [[reference_zonos_tts_stack]] _Archived 2026-08-24._ - `[2026-08-07]` **chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos).** Operator-directed. Copied `zonos-gateway/voices/Donut.wav` → chatterbox `/refs` (`/worktank/chatterbox/reference_audio/donut.wav` — the reference_audio SUBDIR is lkraven-owned so no sudo despite `/worktank` root; container globs `/refs` live → **NO restart**), exposed as `voice:"donut"` (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at `http://10.100.10.50:8090/b/donut-chatterbox/`. Answered ratatoskr's 8-question contract ask from the live gateway (`local/chatterbox-fast:v1`) + source: **NOT OpenAI-shaped** (`POST /tts`; body `text`/`voice`/`format`/`stream`, not `input`/`model`/`response_format`); **NO affect dials** (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; **Zonos stays the only fleet TTS with real emotion steering**); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. **FYI-worthy (operator):** ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory `reference_chatterbox_fast_repo` enriched w/ the live contract. [[reference_zonos_tts_stack]] _Archived 2026-08-24._ - `[2026-08-07]` **Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43).** The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM `reranker` repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); **R42 v13 gate PASSED first-ever** (56/90→90/90). Incumbent kept warm :8002 (rollback via `qwen3-reranker` alias), A4 fallback :8014. Full arc + rollback runbook `docs/pfi/reranker-selection-ledger.md`; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes. _Archived 2026-08-24._ ## Tried and abandoned (archived) # [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix **Outcome.** The fleet `gen` seat is now **`JonathanColetti/Qwen3.8-27B-Uncensored`** (Heretic abliteration, KL 0.12 vs base, bench Δ −0.5 within noise, refusals 98→12/100), quantized in-house to **NVFP4 W4A16** (llm-compressor / compressed-tensors) with a **grafted bf16 MTP head**, vision-intact, **262K** ctx, MTP n=3 (**~42% accept, ~68 tok/s**), coherent. Live at ana-ml2 `:8015` (project `gen-seat` / container `vllm-gen`), backing all 7 gateway aliases. **THE definitive lesson (resolved 3 failed attempts + one premature 50 GB delete).** A grafted bf16 MTP scored **0% on the quant but 83% at bf16** — for TWO different abliterated models. Root cause was NEITHER the abliteration NOR the quant scheme: it was **the grafted `mtp.*` tensors missing from `config.json` → `quantization_config.ignore`.** The wrapper-class quant DROPS the MTP before llm-compressor sees it, so nothing gets added to `ignore`; vLLM then tries to load the bf16 MTP as *quantized* format → "Parameter … not found in params_dict, skip loading" → uninitialized head → 0%. **FIX: after grafting, add `re:^mtp.*` to `quantization_config.ignore`** (one line — all unsloth's working checkpoint has). MTP jumped 0%→83% (bf16-identical). Full lesson in auto-memory `reference_abliteration_mtp_lessons`. **The pipeline that works (for the next VL+MTP quant, incl. the W4A8 chase):** 1. Pull bf16 (kept at `ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16`). 2. Quant via `quant_nvfp4_qwen.py` (darkscarlett dir) = the **wrapper-class** loader (`Qwen3_5ForConditionalGeneration`, keeps the vLLM-serveable config); container = `vllm-openai` + `pip install llmcompressor==0.13.0` (drags in a transformers with `qwen3_5`). 3. **Graft** the author's `model-mtp.safetensors` verbatim into the output + merge the index. 4. **Reconstruct** `preprocessor_config.json` from `processor_config.json`'s `image_processor` sub-dict (the repo omits it → else "Can't load image processor" crash-loop). 5. **Add `re:^mtp.*` to the output config's `quantization_config.ignore`.** ← the fix. 6. Serve: `--quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'` `--mamba-cache-dtype float32 --kv-cache-dtype fp8 --reasoning-parser qwen3`. **VRAM / full-context budget (measured).** Weights ~27 GB; hybrid attention → **only 16 of 64 layers carry KV** → 32 KiB/token → **262K KV = 8.6 GB** (vs ~60–70 GB for a normal dense 27B). Full 262K fits GPU0 at **util 0.45** (~43 GB) alongside meromero (~49 GB used, it's a 31B) — pre-flight rejects util 0.48 (wants 45.6 GB, only 45.5 free). `max-num-seqs 16` keeps cudagraph modest (an ad-hoc serve with no cap OOM'd — cudagraph captured to batch-512). **Why unsloth's `qwen3.8-27b` (the prior gen model) was faster (97 vs 68 tok/s).** ~half = quant kernel (unsloth native NVFP4+FP8 tensor cores vs our W4A16 → Marlin dequant, ~20% even on decode — I'd under-stated this); ~half = MTP acceptance (unsloth 55% un-ablated head vs our 42% — inherent to the ablation, no quant fixes it). **W4A8 recovers the first ~20% (→~82 tok/s) + prefill; not the MTP half.** **modelopt dead-end (for W4A8, avoid).** `nvidia-modelopt[hf]==0.43.0` is too old for qwen3_5's transformers: (a) its `NVFP4_DEFAULT_CFG.quant_cfg` is a LIST but 0.43 wants a DICT (pydantic reject); (b) it warns transformers 5.15 untested. Use **llm-compressor** for W4A8 instead (custom recipe: NVFP4 weights + FP8 input_quantizer + calibration on `heretic2-nvfp4-work/production_calib_512.jsonl`). **Deleted (premature — the delete I owned).** `windowsxp811203/Qwen3.8-27B-Abliterated` (~79 GB) — I declared it desync-dead off a 0% that was actually this ignore bug. Lesson: **test MTP on bf16 first; isolate before deleting.** Commits: eshpfi `680c30e` (deploy + rename + litellm + README), dotfiles `1d1970f` (CLAUDE.md roster) — both UNPUSHED. Related: [[reference_abliteration_mtp_lessons]], [[reference_verify_hf_repo_ids_before_pull]]. _Archived 2026-09-02._ - `[2026-04-30]` task-board workflow with `container: image: debian:bookworm-slim` — fails: `actions/checkout@v4` needs `node` at runtime, slim image lacks it. Switched to `node:20-bookworm-slim` (has node + apt) or runner-label default. (Pattern revisited 2026-05-17 for skaldsong-dev: container override needs `nodejs` apt-installed unless it IS the default.) _Archived 2026-05-27._ - `[2026-04-30]` Dropping the `container:` directive before runner re-registration with docker-schema labels — runner silently falls back to **host mode** (jobs run inside the alpine `act_runner` container itself, no apt). The `:host` suffix in startup logs (`labels updated to: [pfi-fleet:host ana-docker:host]`) is the giveaway. Fix: register with `pfi-fleet:docker://` schema labels. _Archived 2026-05-27._ - `[2026-04-30]` Updating runner labels by editing `.env` and bouncing — doesn't take. The `.runner` registration cache pins labels at first registration; env-var updates are read each start but the stored token + UUID are tied to the original label set on the gitea side. Fix: stop runner, delete `.runner`, generate new admin registration token, redeploy. _Archived 2026-05-27._ - `[2026-04-30]` `git reset --hard origin/` in `deploy-task-board.yaml` (and the in-repo nevermore playbook before fix) — invalid syntax: `origin/` prefix only works for branch refs. SHAs need `git reset --hard ` directly. Resolved with `git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first, then bare `"{{ ref }}^{commit}"` fallback. _Archived 2026-05-27._ - `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after task-board wiring — it was actually only repo-scope on `vh/task-board`. vor's first CI run failed with empty SSH key (`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user scope at `gitea.phasefinal.com/user/settings/actions/secrets`. _Archived 2026-05-27._ - `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity — leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1/"` still leaked the first 4 chars. Lesson: when probing secret-bearing files, use field-by-field SELECTIVE inspection (`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any password line, `grep -c` (existence) or `test -n "$(...)"` (non-empty), never `cat` or value-printing. _Archived 2026-05-27._ - `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero matches and led to a wrong "no /v2/ traffic in 12 days" conclusion. **Gitea logs in PDT, Traefik logs in UTC** — same host, different timezones. Always normalize timezones (UTC) when correlating logs across services on the same box. Cost: ~30 min in the wrong direction. _Archived 2026-05-27._ - `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` / `PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on `/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern **response writes**, not request body reads. `unexpected EOF` from Go's HTTP server means the client closed mid-body-upload; not a knob Gitea exposes server-side. _Archived 2026-05-27._ - `[2026-05-12]` Defaulting asset-engine to Traefik-routed (`asset-engine.phasefinal.com` with `anaprod` cert resolver) on first scaffold — user pulled it back to LAN-direct. Internal tools default LAN-direct; only add Traefik when an external/TLS surface is actually needed. _Archived 2026-05-31._ - `[2026-05-12]` Routing althing thread replies through `galdrabok` when the actual dev handle is `forseti` — bus rejected `to=forseti` initially because thread participants list was `[galdrabok, infra]`. Solved by starting a new thread with `forseti` as the direct recipient. Lesson: when the bus auto-resolves a sender handle that doesn't match the actual dev role, start a fresh thread rather than fighting the participant list. _Archived 2026-05-31._ - `[2026-05-13]` Initial Voxtral default voice `alloy` (OpenAI-compat naming) — vLLM-Omni serving Voxtral does NOT translate aliases. Native presets are `_` shape (`neutral_female`, `casual_male`, etc.). Always live-probe `/v1/audio/voices` for the exact wrapper-deployed preset names before setting a catalog default. Same caveat for Qwen3-TTS (wrapper exposes 15 voices: 9 Qwen presets + 6 OpenAI aliases) and Kyutai-TTS (NillPointer wrapper has NO voice-listing endpoint at all; voices are filesystem paths under the `kyutai/tts-voices` HF repo). _Archived 2026-05-31._ - `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 — flag was deprecated. Use `--runner pooling`; the model's `architectures: [LlamaForSequenceClassification]` in config.json drives the classification head. Surfaced as `vllm: error: unrecognized arguments: --task classify` in container logs. _Archived 2026-05-31._ - `[2026-05-17]` Trusting that `.env` edit alone propagates a new env var into a worldtree container — `compose.yaml`'s `&worldtree-env` anchor must explicitly declare the passthrough or the value silently doesn't land. Same footgun bit `Z_AI_API_KEY` (2026-05-12) AND `BIFROST_CLIENT_ALLOWED_HOSTS` (2026-05-17). Cost ~10 min of "why is env empty?" diagnosis each time. Worldtree-side fix in `vh/worldtree@08f02b2`. _Archived 2026-05-31._ - `[2026-05-17]` `--force-recreate --pull never` from the docker:cli sandbox without explicit `-e WORLDTREE_IMAGE=` re-pins the container to `:latest`, even when a newer SHA-tagged image is on disk. Symptom: container "recreated" but actually reverted to a stale image. Pass `-e WORLDTREE_IMAGE=...:` to the docker run invocation. Worldtree-dev's `8ef3801` health-gated `:latest` advance is the long-term fix. _Archived 2026-05-31._ - `[2026-05-18]` Volva env.sh.template `$HOME` in commented examples — systemd's `EnvironmentFile=` parser doesn't expand `$HOME`; uncommenting lands the literal `$HOME/...` string. Volva-dev's `f4dda73` swapped to `/home//...` placeholders. _Archived 2026-05-31._ - `[2026-05-18]` Initial Volva systemd unit's `ProtectHome=read-only` without `ReadWritePaths=` — althing-cli's SQLite (`~/.althing/ althing.db`) and codex's session state (`~/.codex/`) both need to write. Container started but every poll failed with "db path not writable". Surgical fix: `ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex` (preserves the hardening intent, only carves out the specific dirs). _Archived 2026-05-31._ - `[2026-05-18]` Trusting that env.sh's `export VOLVA_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"` template line works under systemd — `EnvironmentFile=` parser aborts on the first unparseable line (command substitution), and `VOLVA_ALTHING_CLI` declared below silently never lands. Symptom: `Environment=` property empty, daemon error "althing-cli not found at 'althing-cli'". Fix: replace command-substitution with literal path. Volva-dev's `d436c3c` dropped VOLVA_ROOT entirely upstream. _Archived 2026-05-31._ - `[2026-05-19]` Naive `docker rmi worldtree: --force` for CD SHA cleanup — would untag pinned/personal worldtree images since all three deployments share corviduo-dev. Use `docker image prune -a --filter "until=Xh"` instead — respects in-use semantic (Docker won't remove an image referenced by any container on the host), so pinned/personal protected automatically. _Archived 2026-05-31._ - `[2026-05-19]` Skaldsong CD first attempt: `docker pull` step failed with 401 unauthorized. ana-docker had no `docker login` for `gitea.phasefinal.com`. My playbook prereq note ("docker login has been done at least once") was an unverified assumption. One-time manual login persists in `~/.docker/config.json`; architectural fix (workflow-side `ssh ana-docker 'docker login ...'` step using `REGISTRY_USER`/`REGISTRY_TOKEN` secrets) flagged as v2. _Archived 2026-05-31._ - `[2026-05-19]` `SKALDSONG_HOST_CORS_ORIGINS=http://10.250.50.70:8300` as a bare URL — pydantic-settings parses complex env vars via `json.loads()`; first-boot crashloop with `SettingsError: error parsing value for field "cors_origins"`. Must be JSON array literal: `SKALDSONG_HOST_CORS_ORIGINS=["http://..."]`. _Archived 2026-05-31._ - `[2026-05-19]` `SKALDSONG_HOST_STATIC_ASSETS_PATH=/app/web/dist` in compose — mismatched Dockerfile reality. The Dockerfile COPYs SvelteKit build output flat into `/app/spa` (not `/app/spa/dist`). Lifted the path from skaldsong-dev's CD-ask message ("`/app/web/dist`") rather than verifying against the actual Dockerfile they shipped. Lesson: when encoding container-internal paths in compose, verify against the Dockerfile, not the design-doc. _Archived 2026-05-31._ - `[2026-05-19]` Playbook verify step `docker ps | grep healthy` racing the container's `start_period` (30s in compose's healthcheck). Verify ran 0.09s after `compose up -d --force-recreate` — well before docker's healthcheck could flip the status from `(health: starting)` to `(healthy)`. False-negative; container was operationally up (the earlier `/health` poll verify already confirmed). Fix: grep `^Up` not `healthy`. /health-200 IS the liveness check; docker's `(healthy)` is just a delayed echo. _Archived 2026-05-31._ - `[2026-05-20]` `SKALDSONG_DB_PATH` + `SKALDSONG_RUNS_DIR` in compose env block — names skaldsong's app doesn't read. App reads `SKALDSONG_HOST_SQLITE_PATH` + `SKALDSONG_HOST_RUNS_ROOT` (per Dockerfile ENV defaults). Wrong names = silently no-op; app fell back to Dockerfile defaults pointing at `/app/data/...` which the compose's bind mount did NOT cover (target was `/app/state/...`). Result: every `--force-recreate` wiped the SQLite DB. Caught by skaldsong-dev (althing thread `01KS4DPF6SXTBP4Q360JZVWPNT`). Fix in `52e98fa`. Lesson: verify env var NAMES against the Dockerfile/app, not against design-doc shorthand. _Archived 2026-05-31._ - `[2026-05-25]` First selene-block patch put the block under `saga_allowed_models:` instead of top-level `models:` — used `text.replace("models:\n", ...)` which substring-matched the `saga_allowed_models:\n` line first. Caused YAML parse error. Fix: anchored regex `re.compile(r"^models:\n", re.MULTILINE)`. Pattern: substring replace on YAML top-level keys WILL match suffix-containing keys. _Archived 2026-05-31._ - `[2026-05-27]` `docker compose up -d` inside the `docker:cli` sandbox: `${VAR:-./config}` defaults resolve `./config` to the sandbox CWD, but the Docker daemon interprets the path against the HOST filesystem → auto-creates an empty dir → entrypoint reseeded image-baked defaults (lost host-side providers.yaml patches). Fix: pass `-e WORLDTREE_CONFIG_DIR=/abs/path`. Folded into the docker-as-root convention note. _Archived 2026-06-01._ - `[2026-05-27]` `:latest`-pinned compose + private gitea registry + sandboxed pull = recreate on ancient cached `:latest` (deploy pulls by SHA so the tag never advances; sandbox can't pull). Fix: retag SHA→`:latest` on host, then `--pull never`. Better: pin SHA in `.env`, advance in CI. _Archived 2026-06-01._ - `[2026-05-27]` Container recreate during in-flight skaldsong gen kills the runner. With deploys every ~10min and stories >5min, structural not incidental. Roadmap (skaldsong-dev): pre-shutdown signal handler, per-scene resume-from-checkpoint, /api/admin/quiesce. None shipped. _Archived 2026-06-01._ - `[2026-05-27]` `--reasoning-format gemma` on artemis-31b-v1i — unsupported in the deployed llama.cpp (accepts none|deepseek|deepseek-legacy). `deepseek` populates the `reasoning_content` SSE delta Worldtree GemmaProvider checks. _Archived 2026-06-01._ - `[2026-05-27]` `head -c N` piped after a streaming curl SIGPIPEs the curl, killing the request early. Use file-write + separate read. _Archived 2026-06-01._ - `[2026-05-31]` Building the dia2-capable image surfaced THREE upstream packaging quirks: (1) `pip install -e nari-labs/dia2` fails — no PEP 660 `build_editable` hook; (2) plain `pip install` builds an empty `UNKNOWN-0.0.0` wheel (base setuptools 59.6 < dia2's required ≥70); (3) `--no-deps` leaves `transformers`/`sphn`/`whisper-timestamped` missing. Fix (`local/dia:v2`): copy the pure-python `dia2/` package into site-packages + install ONLY those 3 deps; base torch/numpy already satisfy Dia2. _Archived 2026-06-03._ - `[2026-05-31]` Dia2 predefined voices (43, baked at `/app/voices`) are NOT reachable from the `/tts` clone path — it resolves `clone_reference_filename` against the reference_audio dir ONLY. The OpenAI `/v1/audio/speech` `voice` param auto-resolves them (separate code path), which masked the gap. Fix: stage `/app/voices/*` into `/worktank/dia/reference_audio`. Lesson: verify on the endpoint the catalog ACTUALLY targets. _Archived 2026-06-03._ - `[2026-05-31]` `voice_mode=clone` with an empty `clone_reference_filename` → asset-engine serializes it as the literal string `"undefined"` → `/tts` 404. First observed on dia2; worked around in the catalog (default the field to a real voice). **[2026-06-01] root cause found** — the Kokoro voice-blend widget reading Shoelace `.value` before hydration (see Current state); the real fix is asset-engine-side and is escalated. _Archived 2026-06-03._ - `[2026-05-31]` asset-engine `services.schema.json` is DERIVED (regen from the Pydantic model via `dump_schema.py`) and had DRIFTED — rejected the `lifecycle` field 12/14 services use. RESOLVED: regenerated with `CatalogLifecycle` + `reproducibility.seed_field` (`b7b2130`). Lesson: hand-editing `services.yaml` shape without regenerating re-introduces drift. _Archived 2026-06-03._ - `[2026-05-31]` ttyd-over-TLS forces HTTP/2 (kills ttyd's terminal WebSocket → blank screen); Safari/WebKit never sends HTTP basic-auth on WS upgrades. Both solved for the fleet seat (Caddy forces HTTP/1.1; auth → network-gating) — detail in auto-memory `reference_ttyd_fleet_seat`. _Archived 2026-06-03._ - `[2026-05-30]` esh-docker-vm: `hard` NFS mounts from 10.0.50.50 froze a container worker in UNKILLABLE D-state when the NAS stalled — only a host reboot clears it. Separately, `fstab defaults` (no `_netdev`) made NFS-bind containers `Exited(255)` on reboot. → auto-memory `incident_esh_docker_nfs_boot_race`. _Archived 2026-06-03._ - `[2026-06-02]` **Naive cumulative re-decode for streaming** — decode the growing token prefix each chunk + emit the delta, assuming the causal flow gives a stable prefix. It does NOT (full-context attention, see above); maxdiff 0.30. Also the high-level `s3gen.inference` re-applies `trim_fade` at the start every call. Don't go this way. _Archived 2026-06-04._ - `[2026-06-01]` **CSM bring-up** — upstream `phildougherty/sesame_csm_openai` Dockerfile pins NO `huggingface_hub` version, so it now resolves to 1.17.0 where `huggingface-cli` is fully removed (replaced by `hf`) → the build dies at `huggingface-cli login`/`download`. Passing the HF token as a build-arg got the token in (the model downloads at BUILD time, not runtime — our compose only wired it as runtime env, the original failure) but then hit the rotted CLI. Fixing would need vendoring + patching the Dockerfile; not worth it for a weak model → CSM removed. _Archived 2026-06-04._ - `[2026-06-01]` **Fish path-form references** (`{audio:"/app/references/x.wav"}`) → HTTP 500 on this build, despite the stack README showing that shape. The working clone paths are `reference_id` (by name → resolves the DIRECTORY `references//` containing `.wav` + `.lab`; the flat `references/.wav` is ignored — see the two-root-causes entry above) or inline base64 `references`. The catalog uses `reference_id`. _Archived 2026-06-04._ - `[2026-06-01]` **Reference transcript barely affects Fish clone IDENTITY** (disproving my mid-session theory). I'd blamed a bogus provenance-note `.txt` for poor cloning, but the ECAPA re-test showed correct (0.787) / bogus (0.778) / empty (0.738) transcripts all clone Imogen about equally — the transcript affects PRONUNCIATION (phoneme disambiguation per the docs), not who it sounds like. The real culprit for "not British" was the `"undefined"` select bug, not the transcript. (A correct transcript still marginally helps pronunciation — cheap to stage, not load-bearing.) _Archived 2026-06-04._ - `[2026-06-01]` **Pitch-shift register control** (rubberband, to deepen Imogen to contralto/mezzo) — Fish ignores small reference shifts and overshoots large ones; `-3st` WITHOUT formant preservation → "baritone through a tuba" (formants collapsed); `-2/-3st` WITH `--formant` → too subtle / still bad. Abandoned at every depth; all variants deleted. Finer independent pitch/formant control needs praat (not installed). Future clones = operator's. _Archived 2026-06-04._ - `[2026-06-01]` **resemblyzer is too weak for cloning-fidelity scoring** — its dated 2019 LSTM encoder + a synthetic-vs-natural domain gap scored the Imogen clone CLOSER to the default than to real-Imogen, which led me to a WRONG "Fish is a weak cloner" call. ECAPA-TDNN (speechbrain `spkrec-ecapa-voxceleb`) on the same clips gave the correct answer (clone 0.79 to real Imogen, 0.10 to default). Use ECAPA, not resemblyzer, for speaker-verification. _Archived 2026-06-04._ - `[2026-06-02]` **Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED (sub-realtime).** Benched RTF on A6000: 0.72x/0.82x/0.86x, mean 0.80x = sub-realtime, so client-side chunking starves. Root cause: Fish only chunks on `<|speaker:X|>` tags; plain text → batches=[whole text] → all semantic tokens generate before any audio (chunk_length inert). Plus kui/ASGI StreamResponse doesn't flush (fix = anti-buffering headers, kept on file). STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands → bench fp4-quantized Fish; if RTF > ~1.5x give it the chatterbox-fast treatment. For now Fish is a buffered catalog entry (great for SAVED gens, not live-audition). _Archived 2026-06-11._ - `[2026-06-02]` **Context-priming at chunk joins (chatterbox-fast §1.6) — ABANDONED (discard-cut leaks the prefix).** Prepend prior sentence, generate `prefix+content`, discard prefix audio. Built + shipped (`d707439`), live-A/B'd, reverted (`090e70a`). Kill: `generate()` returns one waveform with NO marker where the prefix ends, and the model renders the same prefix with different timing solo vs followed-by-content — so locating the cut is a guess that left a whole clause in the output (operator caught "...without a trace of sarcasm" spoken twice). Reliable cut needs token-level boundaries or per-chunk forced-alignment (heavy). Coherence loss at joins = ACCEPTED limitation. _Archived 2026-06-11._ - `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED (turbo isn't built for streaming).** (1) `S3GenStreamer` is docstring-only (unimplemented). (2) `flow_inference(finalize=False)` is buggy (lookahead trim removes frames from `h` not `h_masks`/conds → shape mismatch); a 1-line patch makes it RUN. (3) BUT the flow encoder uses full-context attention (`static_chunk_size=0`) → incremental decode is prefix-unstable (adding tokens re-attends, shifts earlier mel ~0.30-0.39 vs one-shot). (4) Forcing `static_chunk_size>0` didn't stabilize it. True sub-second frame-streaming needs deep attention surgery with quality risk — not worth it. → adaptive-chunking instead. _Archived 2026-06-11._ - `[2026-06-03]` **`uv pip install .` fails on SmoothKen/knn-svc** (and similar script-repos) — it's analysis scripts + a poetry pyproject, no buildable package (setuptools package-discovery error). Install the pyproject deps directly, don't build the "package". _Archived 2026-06-13._ - `[2026-06-03]` **Backgrounding `althing-cli monitor` with an inline shell `&` (instead of the Bash-tool `run_in_background`) orphans it** — it survives the shell exit, holds the per-handle flock UNTRACKED (won't notify the session), and `stop-monitor` doesn't detect it. Fix: find + kill the orphan PID (verify cwd=this repo / handle first — nh3-dev is shared, other agents' monitors run there too), then re-arm via run_in_background. Always re-arm tracked. _Archived 2026-06-13._ - `[2026-06-03]` **torch-2.12 venvs need `uv pip install torchcodec`** — torchaudio 2.12 defaults to the TorchCodec backend for `.load`; without it, real audio I/O throws "TorchCodec is required" — and it ONLY surfaces at actual conversion, NOT at import/model-load. Lesson: validate real I/O, not just import, when provisioning ML engine envs. (seed-vc on torch 2.4 uses the legacy backend, exempt.) _Archived 2026-06-13._ - `[2026-06-03]` **gitea webhook to a private IP is denied by `webhook.ALLOWED_HOST_LIST`** (anti-SSRF; default `external` blocks private/loopback). Symptom: delivery shows `dial tcp ...: webhook can only call allowed HTTP servers`. Fix = APPEND the target net to ALLOWED_HOST_LIST in gitea's app.ini (keep `external`; scope tight, never `*`/`private`) + restart gitea (act_runner job containers survive a restart). gitea runs as a container on ana-docker (`gitea_gitea_data` volume, `/data/gitea/conf/app.ini`). _Archived 2026-06-13._ - `[2026-06-04]` **homepage "crash" ≠ always NFS** — a wedged container in unkillable D-state can come from dead `siteMonitor` widget targets (retired ESH firewall IPs) hanging the node event loop into `exit_mmap`, needing a host reboot. Check homepage's siteMonitors against retired hosts. (`incident_esh_docker_nfs_boot_race`) _Archived 2026-06-14._ - `[2026-06-04]` **pyworld needs `setuptools<81`** (imports the removed `pkg_resources`); and **R/soundgen `-lgfortran` fails** on irv-ml1 (default gcc is gcc-11 but only gfortran-12 present) → install `libgfortran-11-dev`. _Archived 2026-06-14._ - `[2026-06-04]` **Persistent services on irv-ml1 need SYSTEM systemd units** — the box reaps user-session processes on ssh disconnect, and `--user` systemd isn't reachable over non-login ssh, so nohup/setsid/`screen -dmS`/`systemd-run --user` all die (even with enable-linger). Use `/etc/systemd/system/`. _Archived 2026-06-14._ - `[2026-06-04]` **GPU pin by INDEX is ambiguous on irv-ml1** — native CUDA orders fastest-first (A6000=0) but nvidia-smi/docker use PCI order (A6000=1). Pin by **UUID** (`CUDA_VISIBLE_DEVICES=GPU-…`); verify via nvidia-smi compute-apps. _Archived 2026-06-14._ - `[2026-06-04]` **Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF — the "headgun" lesson.** Serve each model's canonical `tokenizer.apply_chat_template`, not the bundled template (bundled ones corrupt baselines). Verify via vLLM `/tokenize`→`/detokenize`. (`90e08f0`/`27eb537`) _Archived 2026-06-14._ - `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()` requires torchcodec → dies AFTER separating** (0 stems, rc=1). Fix = pin `torch==torchaudio==2.4.1` + `UV_LINK_MODE=copy`. Validate the SAVE path, not just import + GPU inference, on a bleeding-edge torch. _Archived 2026-06-16._ - `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — fills KV to the `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs AFTER weights+KV load (crash-loops). Fix: free co-tenant room OR `--enforce-eager`. FP8 single-stream is batch-1 GEMV (memory-bound) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`) _Archived 2026-06-16._ - `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres rows; clone a default + swap the measure. tok/s is the `outputTokensPerSecond` MEASURE (metrics-API/dashboard query time), not a per-generation field. langfuse-web needs `HOSTNAME=0.0.0.0`. Host 3000 is gitea's → langfuse on 3001. _Archived 2026-06-16._ - `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh + corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are vh-owned, the `.env` is deploy-owned 600. _Archived 2026-06-16._ - `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL`; defaults to claude-haiku in code, opt-in via config not `.env`. Inspect the live container env + vh-owned config files first. _Archived 2026-06-16._ - `[2026-06-13]` **Loading an old LoRA adapter after a vocab bump fails on embedding size.** The harness config + `tokenizer.json` are now at `new_vocab_size=50279` (v4 multi-tag); the v1/v2/v3 adapters are 50277. To load v1 (the accepted artifact), set `cfg.new_vocab_size=50277` before `load_finetuned_engine_lora` (else PeftModel state_dict size mismatch). `tokenizer.json.v3bak` is the 50277 tokenizer for a clean restore. _Archived 2026-06-16._ - `[2026-06-13]` **Heavy from-source compile (`MAX_JOBS=128`) on the shared PROD GPU box PINS it** — load hit 187, prod vLLM restarted, killed an in-flight quant. ana-ml2 hosts live inference; never run a big build there at full parallelism. Cap `MAX_JOBS≤32`, build off-box, or cgroup-constrain. _Archived 2026-06-16._ - `[2026-06-13]` **`--quantization fp8` on a VL model can quantize the VISION TOWER → garbage vision** (Qwen3.5-VL on stable vLLM: gray-grid output; LM answers text fine, so it "looks" healthy). The nightly excludes the vision tower. Validate the VISION path on a quantized VLM, not just text — and pin the engine digest with the exclusion. _Archived 2026-06-16._ - `[2026-06-13]` **vLLM's `--gpu-memory-utilization` is checked against FREE VRAM at startup, not total** — on a shared card, growing one service before trimming a co-tenant OOMs. Trim the shrinking service FIRST, then grow. Size to the FREE budget. _Archived 2026-06-16._ - `[2026-06-13]` **The `vllm/vllm-openai` entrypoint is already `["vllm","serve"]`** — compose `command:` supplies the model as the first POSITIONAL arg + flags; a second `serve`/`--model X` → "unrecognized arguments". Same-class: `tee` masks the real exit code (use `>`); HF `datasets` rejects bare `wikitext` (needs `Salesforce/wikitext`). _Archived 2026-06-16._ - `[2026-06-13]` **Chatterbox-Turbo LoRA finetune: the repo's `setup.py` loads the WRONG tokenizer** — pulls gpt2-medium + a grapheme merge file instead of the chatterbox-turbo GPT2 tokenizer (vocab.json+merges.txt, len 50276). Fix = override + delete the grapheme `tokenizer.json`; `[vmoan]` → new_vocab_size 50277 (1-row resize), lora_r 64 / alpha 128, modules_to_save=[text_emb,text_head]. Unique-stem corpus collision needs `{index}_{stem}` IDs. (`irv-ml1:~/r16-vmoan-harness`) _Archived 2026-06-16._ - `[2026-06-11]` **A completion-poll `while pgrep -f ` SELF-MATCHES its own remote shell argv** — its own `pgrep -f` always finds itself → the loop never exits. Use a match pattern ABSENT from the poll command (the python stage, or a sentinel file), not the driver's own name. _Archived 2026-06-16._ - `[2026-06-14]` **vLLM ModelOpt-NVFP4-MoE loader is broken for current multimodal MoEs.** `nvidia/Qwen3.6-35B-A3B-NVFP4` fails weight-load: `KeyError: layers.0.mlp.experts.w2_input_scale` on 0.19.1, `lm_head.input_scale not registered` on 0.22.0 (vllm #44081); same class hits Gemma-4 MoE / Qwen3-30B-A3B NVFP4. The arch + quant ARE recognized (gets past arch resolution + vision-processor load) — it's the per-expert/lm_head scale-key mapping. Don't chase nightlies; use official FP8 until fixed. _Archived 2026-06-18._ - `[2026-06-14]` **vLLM sampler-warmup OOMs on a shared GPU even when weights fit** — it warms the sampler with `max_num_seqs` (default **1024**) dummy requests, and a big vocab (Qwen3.6 = 248K) makes that a huge transient logits tensor. A vision endpoint doesn't need 1024-way concurrency: set `--max-num-seqs 32`. Separately, post-load `ValueError: No available memory for the cache blocks` means util is too thin (weights+activation+graph ate it) — for 34 GB FP8 weights, util ≥ ~0.45 to leave KV room. _Archived 2026-06-18._ - `[2026-06-14]` **Recreating multiple vLLM services concurrently races the memory-profiling assertion** — `AssertionError: Error in memory profiling. Initial free memory X / current Y … other processes … release GPU memory while vLLM is profiling`. Recreate co-tenant vLLM services ONE AT A TIME (force-recreate one, wait healthy, next). _Archived 2026-06-18._ - `[2026-06-14]` **embed/rerank (0.6B) at util 0.05 reserve ~5.5 GB each — mostly util-reservation WASTE, not need.** A 0.6B model needs ~1.2 GB weights + ~2.5 GB CUDA/torch context; util 0.03 (~3.6 GB) fits with room, reclaiming ~4 GB (vLLM reserves the util fraction regardless of actual KV; embedding models barely use KV). Real-need floor ~3 GB — don't go to 0.02. _Archived 2026-06-18._ - `[2026-06-14]` **Chatterbox-Turbo decode-knob foot-guns** (R16 v1-polish + emergent probes): the turbo length cap is `max_gen_len` (default 1000) on `t3.inference_turbo`, NOT `max_new_tokens` — and `tts_turbo.generate` does NOT forward it (wrap inference_turbo to cap). `rep_pen 2.0 / temp 0.5` BACKFIRES (degenerate 24 s run-on). Soft decode tamers cut the NVV ITSELF, not just the run-on tail (operator: "p1 trims the moaning too") — gen-time polish can't beat v1's defaults. Inline base-NVV tokens DEGENERATE (moan-cascade + gibberish), they don't modulate the surrounding words. _Archived 2026-06-18._ - `[2026-06-15]` **A comfyui restart-loop with `exit 0` + `OOMKilled=false` is NOT OOM — suspect a native SEGFAULT.** `docker inspect` shows only the LAST exit code; a GPU OOM is a catchable `RuntimeError: CUDA out of memory`, NEVER `Fatal Python error: Segmentation fault`. Diagnostic: `RestartCount>0` = same container restarted in-place → the docker json-log ACCUMULATES across all cycles, so grep the full `docker logs` for the crash signature + the lines right before each restart banner. comfyui's was at VAE decode (sampling completes, then segfault loading `AutoencodingEngine`) — gen-triggered but native, not VRAM. Suspects logged: torch/cu130 mismatch, an injected `aimdo` cuda-funchooks lib, the VAE. _Archived 2026-06-19._ - `[2026-06-15]` **NEVER blanket-kill `althing-light-monitor` by PID** — they're per-handle and MANY agents run their own on the shared nh3-dev box (heid / ratatoskr-dev / worldtree-dev / bifrost-dev each have one). A `kill $(pgrep -f light-monitor)` downs four peers' monitors (near-miss caught by checking `/proc//environ` `ALTHING_HANDLE` + cwd first). Recover your OWN via `althing-cli stop-monitor` (v0.14.1+); a raw `&`-backgrounded monitor is untracked (no harness fire-notification) — always arm via `run_in_background`. (extends the pkill-self-match lesson below) _Archived 2026-06-19._ - `[2026-06-15]` **Manual `docker compose up` on corviduo Worldtree DOWNGRADES to a stale local `:latest` → crash-loop (config/image mismatch).** Compose `image:` is `…/worldtree:latest`, but local `:latest` lagged ~2wk (`21c90d5eeb2d`); CI/CD deploys NEWER SHA images, and a real pull 401s (daemon has no gitea-registry creds — that's CI/CD's). `--pull never` then used the stale latest → personal crash-looped on `ConfigurationError: echo.default_model missing` → **~2-3 min personal outage**. RECOVER: retag `:latest` to the image a HEALTHY SIBLING runs (`docker inspect worldtree-worldtree-api-1 --format '{{.Image}}'` → `docker tag …`), then `up -d --pull never`; verify `Application startup complete`. Don't manually recreate Worldtree on corviduo without pinning the correct image — image+config are CI/CD-coupled. (auto-memory `reference_corviduo_dev_emergency_ops`) _Archived 2026-06-19._ - `[2026-06-15]` **`.claude.json` (Claude Code config) truncates on ENOSPC → invalid JSON → CC resets to defaults on next start.** nh3-dev hit 100% disk (a peer's R19 dataset pull staged ~147G to local /home instead of the 7TB NAS); the in-flight `.claude.json` write was cut mid-key. Repair = strip the dangling token + close the open braces, validate `json.load`. CC keeps its OWN rolling backups at `~/.claude/backups/` + writes a `.claude.json.corrupted.*` snapshot when it self-detects. After ANY ENOSPC on a box running CC/agents: integrity-check `.claude.json` (json.load) AND the althing bus DB (`PRAGMA integrity_check`). _Archived 2026-06-19._ - `[2026-06-15]` **`pkill -f althing-light-monitor` SELF-MATCHES the killing shell** (the pattern is in the command's own argv) → kills itself mid-run (exit 144/truncated output). Stop the light-monitor via `althing-cli stop-monitor` or a captured PID — never `pkill -f `. The singleton lock can also RACE to 2 live monitors during re-arm churn; keep exactly one tracked (run_in_background) monitor, and a raw `&` monitor is untracked (no harness fire-notification — don't use it). _Archived 2026-06-19._ - `[2026-06-14]` **Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP.** `gitea.phasefinal.com` = public `38.120.12.44` (ana-srv1); gitea is a container on ana-docker, git-SSH `10.250.50.70:222` + HTTP `:3000`. A fleet host egressing to public `:22` gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (`git fetch` times out under `set -euo pipefail`, aborts before reset). Bit irv-ml1's arbo deploy. `:22` on `10.250.50.70` is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented `docs/orientation.md` (`6e58e57`). _Archived 2026-06-19._ - `[2026-06-16]` **litellm 500 `Router.acompletion()/aembedding() missing 'messages'/'input'` = a request missing `Content-Type: application/json`, NOT a gateway outage.** curl `-d` defaults to form-encoding → litellm can't parse the JSON body → `data` reaches the router without `messages`/`input` → 500 (should be a 400; litellm #16993). My own diagnostic calls dropped the header → I misread it as a gateway outage and needlessly bounced the gateway ~4× chasing a phantom (image/version/config were fine throughout; a malformed UI-added "Mistral Story Eval" model in the DB was a red herring I deleted). ALWAYS send `-H "Content-Type: application/json"` testing litellm; reproduce with a header'd call before declaring a litellm incident. _Archived 2026-06-20._ - `[2026-06-16]` **LiteLLM admin-UI playground can't test vLLM-backed models** — it auto-sends empty `tools:[]`, vLLM 400s (litellm #6228); the gateway `strip_empty_tools` hook is a PROXY hook and structurally can't reach the UI's in-process `litellm.completion()` call. Off-ramp = `tools/gateway-chat.html`. (Langfuse playground also out: its SSRF guard blocks internal-IP LLM connections, wontfix Langfuse #13097.) (auto-memory `reference_litellm_ui_playground_vllm_deadend`) _Archived 2026-06-20._ - `[2026-06-16]` **A gitea Actions run that "looks like it never fired" (no image, untouched build dir) usually DID fire then skipped or failed-fast.** Bit arbo TWICE: run #3 looked-never-fired but SKIPPED (change-detection `engine=0`); run #4 looked-never-fired but FAILED in ~7s (early-step). Check the run LIST + status via claude-bot (`GET /repos/{o}/{r}/actions/tasks`) BEFORE a runner-connection/dispatch theory. CORRECTION (2026-06-16, proven): action logs ARE readable via API at the per-JOB endpoint — `GET /api/v1/repos/{o}/{r}/actions/runs/{run}/jobs` → take a job `.id` → `GET /api/v1/repos/{o}/{r}/actions/jobs/{job_id}/logs` returns the full plain-text log (claude-bot basic-auth, internal route `10.250.50.70:3000`). Only the RUN-level `/runs/{id}/logs` 404s; the per-job one works, so no UI needed. Gotcha when reading: gitea marks **every** step `conclusion:failure` once any step fails — the per-step conclusions mislead; trust the log body + step *timestamps* (the step with real duration is the culprit) not the conclusions. (Used 2026-06-16 to diagnose arbo run #5/task 1175 = test-gate `uv run pytest` → `Failed to spawn: pytest` = pytest absent from the fresh-checkout venv sync, NOT the SSH-clone comfy-dev suspected.) _Archived 2026-06-20._ - `[2026-06-16]` **vLLM 0.23.0 breaks Mistral MULTIMODAL at startup** (`MistralCommonImageProcessor has no attribute fetch_images`, #44911 regression ~0.22.1+; same class hit Mistral-3.1/ Devstral). The dummy-image profiling crashes. Workarounds: `--limit-mm-per-prompt '{"image":0,"video":0}'` loads text-only; for VISION, pin **v0.22.0** (last pre-regression). Tradeoff: v0.22.0 *buffers* reasoning (doesn't stream reasoning_content cleanly); a newer build streams it but breaks vision — vision won (Mistral is the abliteration-baseline). _Archived 2026-06-20._ - `[2026-06-16]` **Mistral `-reasoning` TTFT variance = Triton kernel JIT-compile-during- inference, NOT reasoning time.** `jit_monitor` logs the spike (`_compute_slot_mapping_kernel`, `apply_token_bitmask_inplace_kernel` ← the tool-call/structured-output path). vLLM's startup warmup covers only SOME shapes; a first-hit uncovered shape JITs a kernel (seconds), then CACHES it — warm until container recreate (the Triton cache is IN-container, no volume, so recreate = cold again). Secondary: GPU idle-down to P8/180 MHz between requests. Also: Mistral `reasoning_effort` is BINARY (none|high only — medium/low → HTTP 400). _Archived 2026-06-20._ - `[2026-06-16]` **Worldtree old-baseline instances need the FULL config set synced, not the changeset delta.** Personal (a PRE-mistral baseline) crash-looped (41 restarts) on `DanglingBindingError: role 'echo' binds missing catalog entry 'qwen3.6-35-a3b-heretic'` — its `model_roles.yaml` (2026-06-10, never migrated) still bound the retired heretic that the synced v0.35.10 providers removed. The "model_roles unchanged — skip" was correct only for demo (already at the v0.35.9 baseline). Old-baseline → sync providers + model_roles + defaults. _Archived 2026-06-20._ - `[2026-06-16]` **vh is a Gitea USER, not an org → no scoped package-write for a service account.** Gitea user-namespace packages are owner/admin-only, so claude-bot can't push to `vh/arbo` without being admin. The fix is converting vh → a `corviduo` ORG (tabled). Also: vh's `tea` token lacks `write:admin` — create gitea users via the server-side `gitea admin user create` CLI (`docker exec -u git gitea …`), not the admin API. _Archived 2026-06-20._ - `[2026-06-16]` **`docker compose up -d` on a multi-service vLLM stack recreates ALL its services** (shared `.env` → all configs re-hash) → concurrent memory-profiling race ("No available memory for the cache blocks"). Target one: `docker compose up -d `. (auto-memory `feedback_compose_up_recreates_whole_stack`) _76 older entries archived to archival-memory.md._ _Archived 2026-06-20._ - `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Only a process restart drops a config model. The INVERSE works: `/model/new` live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce. _Archived 2026-06-21._ - `[2026-06-19]` **the litellm `*` wildcard routed to DECOMMISSIONED llama-swap → a stale/typo'd model name returned a misleading `Connection error`, not `model not found`.** Bit a brokkr call to the renamed `qwen-image-judge`. **→ RESOLVED 2026-06-20: wildcard removed (d0eb09c) — unknown models now 404 cleanly.** _Archived 2026-06-21._ - `[2026-06-19]` **A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace (~3.09 GB) is the floor**, allocated OUTSIDE `gpu-memory-utilization`. FIX: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` + leave ~3.2 GB free (util <= ~0.95 on a tight MoE). _Archived 2026-06-21._ - `[2026-06-19]` **vLLM decode tok/s: ALWAYS discard the first generation (cold warmup).** Cold ~ 24.8 tok/s vs warm 92.7 — a cold read undersells decode ~3-4x. Measure run 2+ over a >=256-token output. _Archived 2026-06-21._ - `[2026-06-19]` **For full native 256K on one 96 GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE.** Stable fp8 KV already fits 262144 (hybrid SSM+attn caches KV only on attn layers → small pool). Reach for fp8 first. _Archived 2026-06-21._ - `[2026-06-18]` **mmartial `comfyui-nvidia-docker`: root pip installs CRASH-LOOP the container.** `docker exec -u 0 pip install` leaves root-owned files in the uid-1000 venv → the boot script (uid 1000) fails its torch step → crash loop. FIX: `chown -R 1000:1000 /comfy/mnt/venv`; pin with `DISABLE_UPGRADES=true`. (auto-memory `reference_irv_ml1_comfyui_mmartial`) _Archived 2026-06-21._ - `[2026-06-18]` **ComfyUI custom nodes break on version-assumption drift** — box runs 0.24.1; TeaCache imports `precompute_freqs_cis` at MODULE level (guard the LTX-only import). `pip install tensorrt` defaults to cu13 → use `tensorrt-cu12`. _Archived 2026-06-21._ - `[2026-06-17]` **Mistral HF→NVFP4 quant: the placement knob is the whole game.** `device_map="cpu"` (CPU-resident, sequential onload) works; `"auto"` OOMs, `max_memory` offloads to meta. Read shards with plain `read()` + `safetensors.torch.load(bytes)`, NOT `safe_open` (mmaps the 50 GB shard → ENOMEM on `/tank` ZFS). _Archived 2026-06-21._ - `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — no HF `Mistral4` backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first. _Archived 2026-06-21._ - `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree (worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea` on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur). _Archived 2026-06-25._ - `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell (vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3` arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1 fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager` (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 (non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` + `playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending operator OK to free. _Archived 2026-06-25._ - `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere.** No per-stack `logging:` limit and no docker-daemon default → ClickHouse's verbose stdout grew unbounded to ~94 GB and filled ana-docker's `/` (healthchecks then failed fleet-wide on `runc /tmp: no space left`). Lesson: on a full disk, the symptom (all-unhealthy) is the FORK-can't-write healthcheck, not the containers; find the runaway via per-container-dir `du`, not `docker system df`. The systemic fix is a daemon-default `log-opts` (`max-size`/`max-file`) — NOT just per-service patching. _Archived 2026-06-25._ - `[2026-06-20]` **OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk streaming is a net loss.** Measured ~2.8× (32-step) / 5.6× (16-step) on the 3090; every chunk costs ~the same regardless of length, so naive per-sentence streaming triples total gen time + starves. Use few/aggressive chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end. _Archived 2026-06-25._ - `[2026-06-20]` **Named buildx builders accumulate cache the standard prunes MISS.** `docker image prune -a` + `docker builder prune` don't touch `buildx_buildkit_-builder0_state` volumes (per-builder buildkit state). Prune them with `docker buildx prune --builder -af`, or `docker volume rm` orphaned ones (builder already removed). _Archived 2026-06-25._ - `[2026-06-25]` **althing "unreachable: " can MASK an app-level 500.** Raw network was clean; root cause = receiver DB agents-table not synced with the config roster → delivery 500'd "unknown to: ", MAPPED to "unreachable". Diagnose: raw curl to :8087 + connect-probe ⇒ NOT network. Fixed in althing v0.17.1. `reference_nh3_extdev_althing_mesh`. _Archived 2026-07-08._ - `[2026-06-20]` **rest-server `.htpasswd: permission denied` = the ana-nas NFS mount FAILED (ghost file on the local mount point), NOT a decommission.** `mnt-backup.mount` stuck `failed` (fstab bare `defaults`) → rest-server serves an empty local dir. Recovery in disaster-recovery.md. _Archived 2026-07-08._ - `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — but `ssh infra-ops@ana-docker` HAS NOPASSWD root.** A `sudo cp` as lkraven silently failed → nearly punted the rest-server recovery. Reach for `infra-ops@ana-docker` for sudo ops. _Archived 2026-07-08._ ## Archived 2026-08-02 — Recent decisions (archived) ### 2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session - `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` + `envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal), YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.) _Archived 2026-08-02._ ### 2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b - `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the 2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`, `rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime (zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares `live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start, never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon; log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running containers). _Archived 2026-08-02._ ### 2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped - `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17, vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829); max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and `repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514). **Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s). _Archived 2026-08-02._ ### 2026-07-09-granite-gen-memory-extractor-bind-host-synced-on - `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed, #335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on `summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen** (b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted `worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1` after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed). StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON: the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) — pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass` parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix (operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD; live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09): character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44, UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) + mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's `load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character + character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard + character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37** (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09): mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx 131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45 canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev), monitor armed. See [[reference_corviduo_dev_emergency_ops]]. _Archived 2026-08-02._ ### 2026-07-09-granite-gen-memory-extractor-bind-green-lit-for - `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).** Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching). Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08). **This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread 01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]]. _Archived 2026-08-02._ ## Archived 2026-08-02 — Tried and abandoned (archived) ### 2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s - `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push finishes. FIX: re-run once built, or gate on `workflow_run: completed`. _Archived 2026-08-02._ ### 2026-07-04-on-prem-t1-train-that-keeps-any-ana - `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the 566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`. _Archived 2026-08-02._ ### 2026-07-04-a-systemd-user-daemon-that-shells-out-to - `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`. _Archived 2026-08-02._ ### 2026-07-09-fp8-breaks-morpheus-audio-token-generation - `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks- TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real. _Archived 2026-08-02._ ### 2026-07-09-sentence-chunking-tts-loses-prosody - `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path). _Archived 2026-08-02._ ### 2026-07-09-hf-whisper-datasets-aren-t-actually-whispered - `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8 (not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) — that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting. _Archived 2026-08-02._ ## Recent decisions (archived 2026-08-03 batch) ### 2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed - `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug. _Archived 2026-08-03._ ### 2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt - `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section. _Archived 2026-08-03._ ### 2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning - `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`. _Archived 2026-08-03._ ### 2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code - `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]]. _Archived 2026-08-03._ ### 2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause - `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528. _Archived 2026-08-03._ ### 2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway - `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1. _Archived 2026-08-03._ ### 2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s - `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot. _Archived 2026-08-03._ ### 2026-07-13-wt-355-residual-300s-hang-localized-to-our - `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]** _Archived 2026-08-03._ ### 2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree - `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize. _Archived 2026-08-03._ ### 2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min - `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate). _Archived 2026-08-03._ ### 2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on - `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated). _Archived 2026-08-03._ ### 2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree - `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml` (`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347 `session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based `affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347. R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]]. _Archived 2026-08-03._ ### 2026-07-10-comfyui-v0-27-1-success-on-irv-ml1 - `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch 2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt` FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt` as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment). `docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3 needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a mmartial ComfyUI version bump, `git diff -- requirements.txt` — if torch is unpinned, bump the CODE without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid 1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See [[reference_irv_ml1_comfyui_mmartial]]. _Archived 2026-08-03._ ### 2026-07-10-comfyui-0-25-x-bump-on-irv-ml1 - `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`, 0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` = `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch 2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention), re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):** comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) → rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1), lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved. _Archived 2026-08-03._ ### 2026-07-10-biweekly-open-weight-releases-scan-cron-set-up - `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00` America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh` runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in `~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0): web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input + caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor); registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default + `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled 1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending. **NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 + ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo). _Archived 2026-08-03._ ### 2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b - `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.** Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0 KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf); (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively opens ``/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).** _Archived 2026-08-03._ ### 2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated - `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.** Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision, util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path); Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway sampling-defaults wiring still PENDING. _Archived 2026-08-03._ ### 2026-07-08-dpo-was-silently-running-3-epochs-harness-gap - `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass `num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs + _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run. FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig, float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`. GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step; train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45 banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0): ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift, arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval + the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed). _Archived 2026-08-03._ ### 2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes - `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.** Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before `from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does `model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers 5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93% util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit. gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter. _Archived 2026-08-03._ ### 2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved - `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.** Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml --stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is `["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash. Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer` (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→ immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→ **llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→ (more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate 1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0) but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image. Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX, expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image. _Archived 2026-08-03._ ## Recent decisions (archived 2026-08-05 batch) - `[2026-07-16]` **GPU re-org: char-rp→GPU1 + both cards re-optimized for max context.** Moved char-rp (Magidonia-24B) GPU0→GPU1, then maxed context: char-rp-reasoning 150K→256K (util 0.46, 1.56x), gen→256K + seqs 16→32 (util 0.42, 5.43x), granite 64K→**128K full-chapter** (util 0.27, 1.50x). FINAL: GPU0 ~14 G reserve (both seats 256K native), GPU1 ~6.7 G headroom. All healthy. LESSON: KV must hold ≥1× max-len (util-floor crashes) + per-model KV cost varies ~8× (MoE cheap, dense pricey) → tune util empirically. _Archived 2026-08-05._ - `[2026-07-16]` **granite right-sized → ~10.5 GB freed on GPU1** (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536). LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped before 0.18 landed. `.env`-only, recreate `vllm-granite` alone (shared stack). [Superseded by the 07-16 GPU re-org above → 128K.] _Archived 2026-08-05._ - `[2026-07-15]` **image-bench eviction DONE (parked item closed).** Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM `image-judge`+`qwen-image-bench` → gen :8015 (judge samplers + thinking-off); comfy-dev pinged; backfilled the canonical char-rp-reasoning litellm block. Revert ~90 s. auto-memory `project_arbo_gen_switch_imagebench_evict`. _Archived 2026-08-05._ - `[2026-07-15]` **Homepage AI-tab revamp** — flat "AI Systems" group → dedicated AI tab, 6 role-based groups + AI-Dormant; committed `569e1af`, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.) _Archived 2026-08-05._ - `[2026-07-15]` **Home Assistant config repo created** (`vh/home-assistant-config`, private). UI-managed HA → allowlist model (YAML + curated secret-free `.storage` subset). git-in-place in `/config` on esh-docker-vm + scoped deploy key + local clone `~/development/home-assistant-config`. _Archived 2026-08-05._ - `[2026-07-15]` **char-rp-reasoning OOM rescue** — solo-restart on the packed GPU0 crash-looped; fixed via `expandable_segments:True` + util 0.39→0.38 + max-model-len 192K→150K. LESSON: `max-model-len` does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom. _Archived 2026-08-05._ - `[2026-07-15]` **soong-lab `SOONG_LAB_LIBRARY_DIR` made persistent** (corviduo-dev) — was on the redeploy-wiped code default; set to `/home/infra-ops/soong-lab-data/library`, restarted. Closed a queued no-rush item. _Archived 2026-08-05._ - `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost / threshold-colored ctx+rate. _Archived 2026-08-05._ ## Tried and abandoned (archived) — moved 2026-08-12 - `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%, N=4 −20%). Reserve for dedicated/interactive deployments. _Archived 2026-08-12._ - `[2026-07-02]` **irv-ml1 `/worktank` ROOT is root-owned — lkraven can't write there (irv-ml1 sudo needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-`. _Archived 2026-08-12._ ## Recent decisions (archived) - `[2026-07-18]` **worldtree-sdk 1.0.0 (Python) published to the internal vh Gitea PyPI** (wtsdk-dev request; the npm/TS side shipped prior session). Built from tag `python-v1.0.0` (clean worktree), `uv publish` → `https://gitea.phasefinal.com/api/packages/vh/pypi`; acceptance `uv pip install worldtree-sdk==1.0.0` (vh index as extra-index-url) resolves + imports, __version__ 1.0.0. Registry already existed (bifrost publishes there; soong-lab consumes it via `[[tool.uv.index]] name=gitea`). Publish cred = the vh `write:package` PAT the operator had already handed over (in `worldtree-sdk/.npmrc` `_authToken`) — Gitea `write:package` is package-type-agnostic, so the npm-publish token published PyPI too. Consumers install like bifrost (add the vh index + a read token). [[reference_worldtree_demo_key_mint]] _Archived 2026-08-15._ - `[2026-07-18]` **nh3-dev /tmp auto-clean enabled** — Debian ships /tmp with no tmpfiles age (`D /tmp 1777 root root -` → never cleans); this high-churn agent box had accreted **~190k stale temp dirs / 25G**. One-shot manual purge (194k→10k entries, 25G→1.7G; deleted top-level dirs/files >1d old, spared `/tmp/claude-*` by name + anything ≤1d). Then `/etc/tmpfiles.d/tmp.conf` = `D /tmp 1777 root root 3d` (daily `systemd-tmpfiles-clean.timer` removes >3d-untouched items; active files + socket dirs spared). Tunable via the age. Note the churn: ~10k /tmp entries/day here. _Archived 2026-08-15._ - `[2026-07-18]` **soong-lab containerize cutover — COMPLETE + LIVE on corviduo-dev.** Migrated soong-lab (Noonien Soong character-design studio) from a hand-built `soong-lab-studio.service` (systemd + git-pull-on-webhook) to a containerized deploy, image built by CI + pushed to the Gitea registry. soong-dev owns the in-repo artifacts (Dockerfile/compose/workflow/`docs/DEPLOY.md` = checklist); infra-ops owned the host cutover. Operator confirmed functional ("Soong works great" — a real Soong turn round-trips + saves) → cutover 100% closed. **Final state (corviduo-dev, 10.250.50.152):** - Container `soong-lab-soong-lab-1` LIVE + healthy on `0.0.0.0:8443`, image `gitea.phasefinal.com/vh/soong-lab:latest` (v0.3.24), `restart:unless-stopped` (survives reboot; no systemd unit needed — docker restart policy handles boot). - Deploy dir **`/home/infra-ops/soong-lab-deploy/`** — pull-based `compose.yaml` (image + env_file + `8443:8443` + named volumes; NO build/secrets stanza) + `.env` (copied from the live `soong-lab.env`, STRIPPED of the `SOONG_LAB_*_DIR` overrides so the container uses image defaults `/data/library` + `/data/portraits` + `/app/web` → the volumes). - Named volumes `soong-lab_soong-library` + `soong-lab_soong-portraits`, migrated from `/home/infra-ops/soong-lab-data/{library,portraits}` (2 saved designs incl. **Sindra** + 27 portraits), **chowned `10001:999`** (the container `soong` user) so it can read AND write new designs. - Old `soong-lab-studio.service` + `soong-webhook.service` (the `:9010` git-pull redeploy listener) both **stopped + disabled**. **Topology reality (≠ what DEPLOY.md assumed):** there is **NO TLS proxy**. WT-personal (`:8081`) and soong-lab are **co-located on corviduo-dev**, and the Bifrost callback is **plain-HTTP same-host** `http://10.250.50.152:8443` — the value of `SOONG_LAB_BIFROST_ENDPOINT_URL`, unchanged by the move, so the WT Bifrost host-allowlist stayed valid as-is. Nothing on the WT side needed touching. **Safety net:** data backup `/home/infra-ops/soong-lab-data-backup-20260718-091831.tar.gz` (35M) taken BEFORE migration. Verified pre-retire: `/api/version` 200 (0.3.24), SPA `/` 200, `POST /bifrost/tool-call` → 401 (route present + auth-gated), bidirectional WT↔soong reachability, container healthcheck green. **Ops commands:** - Redeploy a new image: `cd /home/infra-ops/soong-lab-deploy && sudo docker compose pull && sudo docker compose up -d`. (Auto-pull-on-`:latest` — watchtower or a deploy hook — is an open follow-up.) - Rollback: `sudo docker compose down` + `sudo systemctl enable --now soong-lab-studio.service soong-webhook.service`. - Homepage tile: manual `- Apps:` entry "Soong Lab" (href http://10.250.50.152:8443) in esh-docker-vm `/opt/docker/conf/homepage/services.yaml` — corviduo-dev isn't a Homepage-watched docker endpoint, so docker-label auto-discovery can't surface it (see [[2026-07-18-fleet-gitea-runner-build-recipe]] for the CI half). See [[reference_corviduo_dev_emergency_ops]], [[reference_claude_bot_gitea_creds]]. _Archived 2026-08-15._ - `[2026-07-18]` **zonos-gateway 0.2.1 — voice-resolved emotion presets baked (provisional) from the axes sweep.** After the axes sweep ([[reference_zonos_tts_stack]] + the `[2026-07-18] axes sweep` Recent-decisions entry) rescued angry and confirmed startled-happy, the operator green-lit baking the results as **provisional** gateway presets + docs. Shipped `vh/zonos-gateway` **0.2.1** (main `8f1885b`, tag `v0.2.1`, PUSHED; deployed live on irv-ml1 `:8890`). **Design — voice-resolved, NOT global.** `resolve_preset(name, voice)` picks the per-voice measured cell, because a single global preset is unsafe (dvalin ruling; BritishFemale's *named* angry misfires as fear). Presets: - `angry`, `happy`, `startled_happy` (+ aliases `surprised`, `startled` → startled_happy). All expressive (`accurate_mode:false`), cfg 1.5, pure-axes (no named sliders). - Calibrated cells (the 3 default voices): - angry: AmF v-0.4/a+1.0 s1.0 (emo0.53/id0.685); BrF v-0.4/a+0.8 s1.0 (emo0.99/id0.725, metric fear-clean); AmM **two-tier** — soft v-0.6/a+0.8 s1.0 (0.23/id0.654) + drama v-0.6/a+0.8 s1.2 (1.0/id0.616 clean; strength is NOT a smooth knob on AmM, 1.0→1.2 is the window, past that flips to disgust). - happy / startled_happy: AmF v+0.6/a+0.8; AmM v+0.3/a+1.0; BrF v+0.6/a+1.0 (happy~1.0, id 0.74-0.80; axes-happy keeps +0.15 id over the named happy slider). - `sad` = unchanged named-slider preset (not axes-tested). - Uncalibrated voices (Cora + the 4 clones) → mid-region fallback until measured. - Docs surface: `/v1/dials` exposes `voice_emotion_presets`; the FastAPI `/docs` description documents it; durable spec `docs/EMOTION-DIALS-SPEC.md` (moved INTO the repo — was mirror-only); README table. 44 tests green. **Repo-hygiene gotcha (fixed).** The local clone `~/development/zonos-gateway` and gitea `vh/zonos-gateway` had **TWO UNRELATED git histories** (no merge-base) — gitea held the voice-wav commits, the local clone held the code + no remote. Reconciled by resetting local→origin/main, overlaying the 7 bake files, `uv lock`, commit, push (fast-forward). Voices stay tracked; local now shares gitea's lineage + has origin wired. **The deployed irv-ml1 tree `/opt/docker/compose/zonos-gateway` is still NON-git** (hand-updated build context) — CI-wire remains an open follow-up. **Provisional pending** ear-validation on emotion-congruent text (the neutral-text audition was inconclusive: "they all sound different, hard to tell"). Follow-ups: sad axes/text pass on the 3 voices; congruent-text pass; clone-char emotion rows. Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio}.py` (run ON irv-ml1; scoring env `uv run --with resemblyzer --with funasr --with "numpy<2" --with soundfile --with requests --with "setuptools<80" --with torchaudio`). _Archived 2026-08-15._ - `[2026-07-18]` **Fleet Gitea-Actions build recipe + the `vh`-is-a-user package-write constraint** (learned the hard way across 3 failed soong-lab validation builds; reusable for ANY fleet CI image build or package publish). **The runner.** One `act_runner` (`gitea/act_runner`) on ana-docker, labels `pfi-fleet` / `ana-docker` → both map to job image **`node:20-bookworm-slim`**, which has **NO docker and NO git**. Config `/opt/docker/conf/gitea-runner/data/config.yaml`: `valid_volumes: []` (no socket propagated to job containers). So: - `actions/checkout@v4` fails (needs git); `docker/*` marketplace actions fail (need docker) — a workflow built on those dies at the first step (~15s). **The working recipe (mirror Worldtree `deploy.yml`).** Run the job in a docker-capable image + drive docker with RAW commands, not the JS actions: ```yaml runs-on: pfi-fleet container: image: docker:24.0.7-cli # has docker+buildx; add git+node steps: - run: apk add --no-cache git nodejs # so actions/checkout@v4 works - uses: actions/checkout@v4 - name: login # RAW, not docker/login-action run: echo "$REGISTRY_TOKEN" | docker login gitea.phasefinal.com -u "$REGISTRY_USER" --password-stdin - name: buildx builder run: docker buildx create --name X --driver docker-container --use; docker buildx inspect --bootstrap - name: build+push # RAW, not docker/build-push-action run: docker buildx build --secret id=,env= -t :latest --push . ``` The runner mounts the host docker socket into ITSELF; the docker:cli job reaches the daemon through that. The `docker/*` JS actions are unreliable on act_runner — raw commands are the fleet convention. **`vh` is a USER account, not an org.** Consequences that bit repeatedly: 1. `GET /api/v1/orgs/vh` → 404 "user redirect"; there are **no org teams** to add a service account to. 2. **User-owned packages are OWNER-WRITE-ONLY.** claude-bot (even repo admin-*collaborator* on `vh/soong-lab`, even with `write:package` scope + full basic-auth) gets **`401 unauthorized`** on `docker push` to `vh/soong-lab`, and `npm publish` to `vh/npm/` would 401 too. Only `vh` itself can write vh packages. → CI must authenticate AS `vh` for the push (a vh-owned `write:package` PAT as `REGISTRY_TOKEN` + `REGISTRY_USER=vh`), exactly how WT pushes `vh/worldtree`. claude-bot CAN still: clone/read repos, READ packages (pulled the image fine), dispatch workflows, mint demo Worldtree keys. 3. **Repo Actions secrets are OWNER-ONLY too** — `PUT .../actions/secrets/X` as claude-bot (repo admin-collab) → 403 "user should be the owner of the repo". Only `vh` can set a repo's secrets. **Other gotchas:** - Gitea **reserves the `GITEA_` secret-name prefix** — a secret named `GITEA_PYPI_TOKEN` is illegal; use e.g. `PYPI_TOKEN`. - Gitea **package auth is token-based / username-lenient** — `docker login` / PyPI basic-auth authenticate via the token; the username is nominal (tested `-u gitea` and `-u claude-bot` both 200 against the vh PyPI). So a Dockerfile hardcoding `UV_INDEX_GITEA_USERNAME=gitea` is fine with any valid token. - Homepage (esh-docker-vm) docker-label auto-discovery only covers the 5 endpoints in its `docker.yaml` (esh-vm-docker, ana-docker, ana-ml2, nh3-docker, irv-ml1); **corviduo-dev is NOT watched** → services there need a manual `services.yaml` entry, not labels. Applied in the soong-lab CI: [[2026-07-18-soong-lab-containerize-cutover]]. _Archived 2026-08-15._ - `[2026-07-18]` **Peer credential provisions — Wyrd conv-api key + wtsdk npm token, both delivered + closed.** Wyrd: demo Worldtree user-tier key (key_id `da7a0bdf`, user_id `wyrd-dev`) minted via `docker exec worldtree-worldtree-api-1 /admin/keys` (omit tier→user), drop-and-shred delivery. wtsdk: operator-minted vh `write:package` PAT relayed drop-and-shred → worldtree-sdk@1.0.0 published to `vh/npm/`. Secret-delivery pattern = drop to a mode-600 file on the peer's box, they collect+shred+confirm, then shred the holding copy; NEVER cleartext over althing. [[reference_worldtree_demo_key_mint]] _Archived 2026-08-15._ - `[2026-07-18]` **Axes sweep RESCUED angry; surprised-class dead but startled-happy ships.** Valence×arousal grid on the 3 calibrated defaults (AmericanFemale/Male, BritishFemale), exp/cfg1.5/strength1.0, 84 clips, emotion2vec + resemblyzer scored, graded vs dvalin's floor. **ANGRY rescued** (named direction was 0.004–0.15, British named-angry even misfired as fear 0.89): axes ship cells at **negative valence (−0.4..−0.8) + high arousal (+0.8..+1.0)** — BritishFemale v-0.4/a+0.8 angry=0.99/id0.725 SHIP, AmericanFemale v-0.4/a+1.0 angry=0.53/id0.685 SHIP; AmericanMale two-tier post-ladder (no single ship cell — best drama = v-0.6/a+0.8 str1.2 angry=1.0/id0.616 clean, soft = same cell str1.0 angry0.23/id0.654; cell A v-0.6/a+1.0 is a non-monotonic minefield, skip). BrF ship cell proxy-CLEAN of fear (str<1.0 just kills anger). **SURPRISED-class DEAD** (max 0.047 across all 84 cells) but **startled-happy** (happy-proxy) ships all 3 at high arousal + neutral/positive valence, with a **+0.17–0.20 identity LIFT** over the named-surprised route (named hits happy~1.0 but at id0.57–0.61, under floor; axes hits happy~1.0 at id0.74–0.80). Bonus: axes-happy retains ~0.10–0.15 more identity than the named happy slider too. Caveats: response surface non-monotonic/sharp-thresholded; angry region borders fear/disgust (bleed); emotion2vec saturates at 1.0 (needs ear-confirm); neutral text understates. Tooling `~/development/zonos-tools/axes_sweep.py`; per-clip JSON was `irv-ml1:/tmp/axes_sweep_results.json` (ephemeral). Sent dvalin msg `01KXT2ZB8G…`. NEXT = operator ear-confirm → bake presets. [[reference_zonos_tts_stack]] _Archived 2026-08-15._ - `[2026-07-18]` **Zonos2 emotion CANONICAL from an empirical sweep + the voice-cloning pipeline.** **Voice-cloning pipeline (established this session).** Source zips at `/mnt/smithy/voice_clones/.zip` (irv-ml1 NFS from nh3-nas; remount post-reboot) — each = diarized single-speaker podcast clips + `manifest.jsonl` (per-clip WhisperX `mean_score`, word timestamps, text) + `metadata.csv`. `~/development/zonos-tools/assemble_voice.py ` ranks by mean_score and concatenates top clips to ~15–24s (Zyphra's blessed clone-ref length; single clip if already ≥15s). Drop the assembled `.wav` into the gateway voices dir → `voice:"name"`. 4 characters cloned: **Emmie, Penny, Natalie, Miranda** (+ Zyphra defaults AmericanFemale/Male/British/Cora) = 8 voices in `zonos-gateway`. Clone is inline `speaker_audio_base64` (text-independent Qwen3 speaker embedding — NO transcript); `/tts/speakers` registration is session-scoped (needs `X-TTS-Session-ID`), so the gateway holds the ref wav and clones per-call. **Gateway voices are host-managed (bind-mount, added this session).** Added `./voices:/app/voices:ro` to `/opt/docker/compose/zonos-gateway/compose.yaml` (committed to `vh/zonos-gateway` + eshpfi mirror `438cd35`). So adding a voice = drop the wav + `docker compose restart zonos-gateway` (registry rebuilds at boot; NO image rebuild). This also un-stranded the other voices (deploy build context had only Cora before). Voice wavs committed to the repo for backup. **Emotion mechanism (Zyphra canonical, from their README @194c0a3).** Additive direction vectors: 4 named (happy/sad/angry/surprised) + valence/arousal axes. `emotion_strength` 1.0 = per-voice calibrated (calibration.json optimizes emotion2vec recognizability only, NOT identity). `accurate_mode` is THE trade-off: `true` = closer voice match (identity), `false` = expressive mode (emotion lands, identity drifts). Zyphra's strong recipe: `accurate_mode:false` + `cfg~1.5`. Single-emotion is blessed; mixing is unblessed (and degrades the clone — operator confirmed by ear). "deaf by 1.5" — cfg past 1.5 distorts + costs ~2× compute. **THE SWEEP (`~/development/zonos-tools/emotion_sweep.py`).** 4 cloned voices × 4 named emotions × {accurate,expressive}×{cfg 1.0,1.3,1.5} @ strength 1.0, single-emotion, neutral sentence + a neutral baseline per voice (~100 clips). Scored on TWO axes: **emotion-landing** = emotion2vec `iic/emotion2vec_plus_large` target-emotion prob [0-1]; **identity** = resemblyzer speaker-embedding cosine vs the clone reference (neutral baseline ~0.85). Scoring env: `uv run --with resemblyzer --with funasr --with "numpy<2" --with soundfile --with requests --with "setuptools<80" --with torchaudio` (setuptools<80 for webrtcvad's pkg_resources; torchaudio for funasr). **RESULTS (mean across the 4 voices) — emotion, best setting, emo/id:** - happy — **exp cfg1.5** 0.80/0.68 (soft: exp cfg1.0 0.76/0.69) → WORKS - sad — **exp cfg1.5** 0.53/0.57 (only working cell; id below the ~0.65 floor) → modest - angry — acc cfg1.3 / exp cfg1.5 tied at ~0.25 emo → WEAK (named ceiling ~0.25) - surprised — max ~0.015 across ALL settings → NON-FUNCTIONAL on the named direction Accurate + low cfg = identity/suppress regime (emo→0); expressive REQUIRED for emotion to land, at ~0.15–0.28 identity cost. **dvalin-smithy-dev synthesis (adopted, triaged genuine-adds; thread `01KXT12FN0AS5A3WMKEK06BVPS`):** 1. Treat **identity as a hard FLOOR (~0.65)**, not a free variable in emo×id. 2. **Two-regime policy** — Regime A (default, identity-critical dialogue): `accurate_mode:true, cfg 1.0, emotion off` (text carries it) or soft-happy (exp cfg1.0). Regime B (tagged drama beats): `accurate_mode:false, cfg 1.5`, single emotion or axes. Line-type→regime heuristic (exposition→A, grief→B+sad, confrontation→B+axes-angry, shock→B+axes-arousal). 3. **Axes-first for the broken emotions** — angry ≈ valence −0.6..−0.8 / arousal +0.5..+0.8; surprised ≈ valence +0.2..+0.4 / arousal +0.7..+1.0 (exp cfg1.5); or "startled-happy" (happy + high arousal) as a surprised stand-in. These are PROVISIONAL — the sweep did NOT test axes. **NEXT (highest VoI, operator to green-light):** an **axes sweep** for angry/surprised (valence×arousal grid) — the only path to rescue the two broken named emotions; then a strength ladder at the best cells + emotion-congruent text (neutral content understates landing) + per-voice tables + a 2nd emotion judge / human pairwise. Then bake the happy/sad canonical into gateway presets. I owe dvalin the axes-sweep numbers. See [[reference_zonos_tts_stack]]; dials-first spec at `vh/zonos-gateway` `docs/EMOTION-DIALS-SPEC.md`. _Archived 2026-08-15._ - `[2026-07-18]` **yt-voice-clipper: A6000-pin fix + v0.3.3 redeploy.** Fixed a latent misconfig — the host override *said* "pin worker to A6000" but `NVIDIA_VISIBLE_DEVICES` was `"0"` (the 3090); re-pinned worker+api to the A6000 by UUID (`GPU-9672f0d5`, 3090 is zonos2's). Then redeployed api+worker to v0.3.3 (`docker compose up -d --build`; SPA+Python; `max_gap` 0.6→1.2s; stderr surfaced in job.log). A6000 + version verified; yields test in-flight (job `f3ff746dbae9494d`). yt-voice-clipper-dev thread `01KXT0T6GYHB`. [[reference_ytvc_autodeploy]] _Archived 2026-08-15._ - `[2026-07-17]` **Worldtree #365 internal-comms config CLOSED (demo+personal → b125) + WT#368 cross-agent memory-leak forensics + PERSONAL agent-memory scrub.** #365: staged the internal-tiers/rules/gate on both instances' bind-mounts (byte-exact vs baked b125), both now live on b125. WT#368 (read-only): the operator's name was in NO recall store on demo; on PERSONAL it sat in `lofn.chroma` (old-code `saga-v1` seeding + legacy contamination), and a clean-slate marker test proved **current b125 code isolates character-session extraction correctly** — the leak is legacy data, not a live bug. Operator-directed → executed a full PERSONAL agent-memory scrub (backup `/opt/worldtree-personal/agent-memory-backup-20260717-181004.tar.gz`; conversations/mood/auth preserved). worldtree-dev owns the code-fix/data contract. [[reference_corviduo_dev_emergency_ops]] _Archived 2026-08-15._ - `[2026-07-17]` **Zonos emotion levers RESOLVED: text-priming is FLAT → the working lever is ZONOS2's native emotion-steering, which the gateway ALREADY exposes as presets.** The prosody-priming A/B (prime→generate→excise, silence-gap cut, parakeet-validated) was operator-judged FLAT on this checkpoint — text doesn't move it. Native `emotion_directions/` (happy/sad/angry/surprised + valence/arousal axes, per-speaker calibrated for AmericanFemale/Male/British) clearly WORKS (sad→slow/quiet, excited→fast/bright, etc.). **`zonos-gateway:0.2.0` (:8890) already wires it**: simplest caller path = `POST /v1/audio/speech {preset:"…"}` — presets neutral/warm/excited/sad/intense/whisper (defined in `~/zonos-gateway/src/zonos_gateway/dials.py`), reached via the **LiteLLM `ext-tts` alias** (engine-neutral swap point; consumers never call the gateway by name). RTF measured on 3090: cfg1.0 steering = FREE (~0.52 = neutral, additive vectors), cfg1.5 amplified ~0.625 (~+20%, still realtime). Captured the live gateway stack → `stacks/zonos-gateway/` (compose+env+README); ⚠️ gateway SOURCE at `~/zonos-gateway` on irv-ml1 is NOT in gitea (backup gap, follow-up); `stacks/zonos` (v0.1 Gradio) marked DEAD/superseded. Whisper is a composed preset (no whisper *direction*; escalation for hard affects = custom directions via `scripts/build_emotion_directions.py` or emotional-ref cloning `speaker_audio_base64`). Harnesses in scratchpad (not yet landed). [[reference_zonos_tts_stack]] _Archived 2026-08-15._ - `[2026-07-17]` **Zonos2 `:1920` engine → self-contained container (stays on 3090); prosody-priming is a SERVING-LAYER change (engine stays stock).** **Context.** The production Zonos TTS engine (irv-ml1 `:1920`, feeds asset-engine + gateway-chat via `zonos-gateway` :8890) was a bare native process — its real launch config existed ONLY in the running process argv (the committed `~/tts-audition/harness/zonos_server.sh` was STALE: said A6000/:1919/no perf flags; live is 3090/:1920 with `--cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3`). Captured to eshpfi `stacks/zonos-engine/` (README + corrected `zonos2-server.sh` + `.env.example`), commit **14a0004** (UNPUSHED as of the snapshot). **Decision 1 — containerize as a SELF-CONTAINED image** (not systemd — operator rejected; not a thin bind-mount wrapper — I walked that back: bind-mounting the host's CUDA-compiled `.venv` couples to the host's exact CUDA/glibc and is fragile + not reproducible). Shape: `FROM` a CUDA 12.8 base → `uv sync` against the repo's committed `uv.lock` (deterministic env) → mount the ~15 GB HF weights (`~/.cache/huggingface/hub/models--Zyphra--ZONOS2`, do NOT bake) → pin the **3090** (`NVIDIA_VISIBLE_DEVICES=0`) → `restart: unless-stopped` → CMD = the captured invocation. **Engine stays STOCK** Zyphra/Zonos2 @ commit `194c0a3` (no fork — the `zonos2` package ships its own server). **Build risk:** heavy compiled-CUDA deps (flashinfer / sgl_kernel / cutlass-dsl / apache-tvm-ffi / pynini) on torch 2.9.1+cu128 — mostly prebuilt wheels + the `uv.lock` make it tractable, expect a couple build iterations. **Cutover (in place on the 3090):** stop the native process (frees ~17 GB) → `docker compose up -d` (re-allocates ~17 GB, same footprint) → repoint `zonos-gateway`'s `ZONOS_URL` at the container (or keep the `:1920` host-port publish). One brief prod-TTS blip. **GPU = 3090 (operator 2026-07-17).** Keep it OFF the A6000 — the A6000 already OOMs under ComfyUI load (idle ~19 GB but spikes far higher during gen), so it can't host Zonos too. The 3090 already runs Zonos, so the containerize-in-place cutover changes nothing about placement. **Decision 2 — the prosody-priming hypothesis (operator's test; the reason for building fresh).** PRIME the autoregressive engine with an emotional sentence, then TRUNCATE it from delivery: prepend a primer → **generate "primer + real text" as ONE continuous utterance** (the AR model carries prosody forward across the boundary) → ASR-timestamp the primer's end (**parakeet**, already up on irv-ml1 `:8765`, word timestamps) → **clip the primer in the inter-sentence silence gap** (+ ~15 ms fade-in, no click) → deliver only the real text, now wearing the primed prosody. Examples: primer "I'm so EXCITED about this." → "This will be a lot of fun!" spoken excited; primer "I'm whispering this to you right now." → "I'm so glad to see you baby." whispered. **This is PURE serving-layer orchestration — the engine is untouched; it lives in the gateway adapter `stacks/zonos/adapter/server.py`.** Only fork the engine if the black-box approach fails. **THE CRUX the test resolves:** does AR prosody actually **carry across the sentence boundary**, or does Zonos reset at the period? → the harness A/Bs the **JOIN punctuation**: period (operator's examples) vs comma vs ellipsis vs none ("…excited about this, this will be…"). Everything else is plumbing. **Plan / design recs.** (a) Build the stock engine image (parallel track). (b) Stand up a priming TEST HARNESS against the NATIVE engine (fast iteration, seconds) + parakeet ASR: prime→generate→timestamp→gap-clip→out; compare primed-clipped vs plain on the two cases (subjective + a cheap objective proxy: pitch/energy variance for "excited", spectral-tilt/low-energy for "whisper"). Iterate on the join, then bake the winner into the gateway adapter. **Primer source:** caller-supplied for the harness (test arbitrary primers) → a curated emotion→primer library (`excited`/`whisper`/…) + optional caller override for production. **ASR:** parakeet primary; WhisperX forced-align fallback if parakeet word timestamps are coarse. See eshpfi `stacks/zonos-engine/README.md` + `stacks/zonos/` (the gateway adapter). _Archived 2026-08-15._ - `[2026-07-15]` **esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service`** (the prior `After=remote-fs.target` drop-in was silently defeated by `nofail`). Reached only after a REBOOT (D-state phantom containers uptime-kuma + paperless-web that no `docker`/`ctr`/daemon-restart could clear). Committed `21d9a07` + playbook updated. See Tried and abandoned. _Archived 2026-08-15._ ## Tried and abandoned (archived) - `[2026-07-15]` **`docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts** — `nofail` drops a mount out of remote-fs.target's blocking set, so the drop-in ordering is silently defeated (paperless still Exited(255) on reboot). Real fix = DIRECT mount->docker ordering via the fstab `x-systemd.before=docker.service` option (verify `systemctl show docker -p After` lists the mnt-*.mount units). esh-docker-vm. _Archived 2026-08-15._ - `[2026-07-15]` **The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT** — reconfirmed: `docker stop/rm -f`, `ctr -n moby task delete`, AND `systemctl restart docker` all fail to clear it; `docker exec` into a wedged container ALSO fails (`setns ... exit status 1`), so the in-place restart escape hatch is out. Worse, a daemon restart can HALF-KILL other healthy containers (knocked paperless's granian down + left it wedged). Process dead but dockerd won't reap -> phantom. NFS mounts are `_netdev,nofail` so the reboot is boot-safe. _Archived 2026-08-15._ - `[2026-07-15]` **vLLM `max-model-len` does NOT free GPU VRAM** — the KV cache POOL is sized by `gpu-memory-utilization`, not max-model-len. Lowering max-model-len only caps per-request context + drops max concurrency; the pool still fills the util budget. To actually free VRAM, lower `gpu-memory-utilization`. (Bit the char-rp-reasoning "drop KV to 150K" ask: the 150K applied but freed 0 VRAM until util dropped 0.39->0.38.) _Archived 2026-08-15._ - `[2026-07-15]` **Claude Code statusline `.cost.total_cost_usd` is per-SESSION** (Claude Code's own cache/model-aware session accounting), not a lifetime aggregate — the large value just reflects a long, multiple-times-summarized session. And the old statusline hardcoded Sonnet pricing ($3/$15) on an Opus session -> ~5x cost understatement. _Archived 2026-08-15._ - `[2026-07-14]` **MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround.** All crashed the same way (`qwen3_5_mtp.py:256` `param_data.shape == loaded_weight.shape` AssertionError — bf16 mtp head loaded into a quantized drafter param): (1) mtp excludes in `config.json` (WRONG file — vLLM modelopt reads `hf_quant_config.json`); (2) specific-unfused mtp names in hf_quant_config; (3) wildcards `mtp*`/`mtp.layers.0*` (`is_layer_skipped` is EXACT-membership, NOT glob — wildcards match nothing); (4) exact fused+unfused names in both `mtp.`/`model.` prefixes. Instrumenting `is_layer_skipped` proved the drafter's exclude list holds ONLY the main model's `linear_attn` entries — the mtp excludes never reach the draft-model quant config. ONLY fix = a mounted `sitecustomize` force-skipping `mtp.*`. LESSON: don't chase checkpoint-config fixes for the mtp-drafter crash; go straight to the runtime patch. Also `nvidia-modelopt[hf]==0.43` (AEON's producer version) is a trap — it pins transformers back to 4.57 which can't load `qwen3_5` at all; use 0.45 + the FusedMoE guard in `quant_modelopt.py`. _Archived 2026-08-15._ - `[2026-07-14]` **AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof.** `vllm-aeon-rp`'s .env → `AEON_RP_MODEL=pantheon-27b-mtp-nvfp4`, `AEON_RP_QUANT=compressed-tensors` — it LOADED (mtp silently skipped, `exited 0`) but never accelerated. Same vLLM image (`:latest` = `sha256:4091d55` = 0.24.0) as the failed Heretic2 test, so the "AEON ran on an older vLLM" theory was wrong. Don't treat a seat that "ran" as MTP-validated without checking its `SpecDecoding` acceptance. _Archived 2026-08-15._ - `[2026-07-14]` **NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept.** Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp weights off a compressed-tensors main model → `Parameter … not found in params_dict`, `Avg Draft acceptance rate: 0.0%`). Pantheon is identical — its "working NVFP4+MTP" was working *structure*, never real acceleration. Working native MTP needs the **modelopt** main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt. _Archived 2026-08-15._ - `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state. _Archived 2026-08-15._ - `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip. _Archived 2026-08-15._ - `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny ''` line) AND an instrumented receiver. _Archived 2026-08-15._ - `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction. _Archived 2026-08-15._ - `[2026-07-13]` `althing-cli reply ` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to `. Bit me several times this session. _Archived 2026-08-15._ - `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter, `has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs. (`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.) _Archived 2026-08-15._ - `[2026-07-09]` **Per-frame CPU SNAC decode is too slow for streaming** — per-call overhead × ~60 frames serialized → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx] window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s). _Archived 2026-08-15._ - `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4 (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY (temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models. Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going **GGUF (llama.cpp)** to sidestep the whole NVFP4-quant surface. _Archived 2026-08-15._ - `[2026-07-08]` **Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24).** (a) HF `tokenizer.json` for Mistral = **GARBAGE output** — the card's "use the official Mistral tokenizer" warning is REAL; must use the `tekken.json`/mistral tokenizer. (b) BUT `--tokenizer-mode mistral` + vision **CRASHES** (`Failed to apply PixtralProcessor on {'text': '[IMG]'}`; and with tekken.json present in auto mode, `CachedMistralCommonBackend has no attribute is_fast`). So it's **mistral-tokenizer OR vision, not both** on this vLLM. Text-only + mistral tokenizer serves clean (`--limit-mm-per-prompt '{"image": 0}'`). **GGUF/llama.cpp avoids all of this** (native mistral tokenizer + vision). _Archived 2026-08-15._ - `[2026-07-08]` **Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base.** The base (`llmfan46 heretic`) writes freely (thinking-off), but Gryphe distilled the reasoning traces from **DeepSeek 3.2** (safety-aligned) onto every turn (`preserve_thinking:true`) → the model reasons ITSELF into refusals in the `` phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified). **Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw abliterated base is cleaner** — this is WHY the pivot went to the llmfan46 heretic base for gen. _Archived 2026-08-15._ - `[2026-07-08]` **Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance.** MTP is a separate **bf16** head (`mtp.*`, in `model-auxiliary.safetensors`, 15 tensors); AEON preserved it by INJECTING the bf16 head into the quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors main + injected bf16 mtp + `text_config.mtp_num_hidden_layers=1` → vLLM detected the MTP but SKIPPED the bf16 self_attn weights → 0/192 draft tokens accepted. **The bf16 MTP head only loads on the MODELOPT main-model format (like AEON), not compressed-tensors.** (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.) _Archived 2026-08-15._ - `[2026-07-07]` **vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639).** Adapter loads HTTP 200 but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate `zc` adapter — correct per vLLM's `check_unexpected_modules` allowlist — loads clean but inert; the fused-key rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of `expand_packed_lora` — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no version-bump helps. Merge bakes deltas in (bypasses this) but is static. _Archived 2026-08-15._ - `[2026-07-07]` **SGLang generic image can't LOAD our NVFP4 AEON** — ModelOptModelLoader weight-shape/ packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated `qwen36-27b-nvfp4` dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was quant-loader-specific). _Archived 2026-08-15._ - `[2026-07-07]` **SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for** (invalid choice: 'in_proj_qkv'); `'all'` resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants its OWN packed layout (base r16 + `get_stacked_multiply=3`, NOT a pre-fused rank-48 qkv → the [48]-vs-[144] shape assert). A THIRD adapter format; version-exact source needed (`:latest`=0.5.13, NOT `main`). _Archived 2026-08-15._ - `[2026-07-07]` **Engine invocation footguns cost several wasted serve-bounces this session** — `docker run --rm` ate crash logs; duplicated `serve` (vLLM image entrypoint is already `["vllm","serve"]`); `--max-lora-rank 48` invalid (choices 1/8/16/32/64… → use 64); parens in `echo` inside `ssh host -c "…"` break the remote shell. LESSON: verify engine launch flags (`--help`, GPU-free) + never `--rm` a container whose crash logs you need, BEFORE bouncing a production serve. _Archived 2026-08-15._ - `[2026-07-04]` **LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge** → my deliberately-invalid `top_k=-5` forwarding-probe bled into a param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT caching (none configured), NOT a config change. **Never fire invalid/distinctive sampler values at a SHARED gateway alias with live consumers** — use a throwaway alias, or a `docker restart litellm` flushes residual carryover. `feedback_litellm_shared_param_mutation`. _Archived 2026-08-15._ ## Recent decisions (archived 2026-08-16 batch) - `[2026-07-15]` **arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake.** Operator-directed full switch (comfy-dev executed, live in prod). Established: gen (`qwen3.6-35b-a3b-heretic`) is vision-enabled and was image-bench's predecessor as arbo's hero-judge; image-judge actually serves 4 roles (vision quality-scoring + identity-scoring + bbox grounding + an uncensored text tier), not just grounding. comfy-dev spot-check: gen faster on every task, grounding within ~3px, uncensoring preserved, and it FIXED a bug (image-judge's reasoning preamble broke json_object + stalled the router). Sequencing = short prod bake then evict (~30 GB GPU1 reclaim); revert = flip `ARBO_VISION_MODEL`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`. _Archived 2026-08-16._ - `[2026-07-18]` **soong-lab auto-redeploy — DONE + VALIDATED** (was approved/queued; executed same day on fresh context — see AS-BUILT at the bottom). Vuong approved wiring auto-redeploy for soong-lab (relayed via soong-dev, thread `01KXT3A6C3908TA4V9THV3AMH7`): new images should go live on corviduo-dev without the manual `docker compose pull && up -d`. Host-side implementation is infra-ops's lane; mechanism is infra-ops's call per fleet conventions. Operator deferred execution — "we'll do soong on fresh context." **Chosen mechanism (recommended, agrees with soong-dev): Worldtree-style CI-deploy step** — NOT watchtower polling. - Add a deploy job/step to soong-lab's `.gitea/workflows/build-and-push.yml` that, after the build+push job succeeds, **SSHes from the pfi-fleet runner to corviduo-dev** and runs `cd /home/infra-ops/soong-lab-deploy && docker compose pull && docker compose up -d`, then a **health-gate** (`curl -fsS http://localhost:8443/api/version`). - This is exactly how WT deploys the demo instance to the SAME host: see `~/development/Worldtree/.gitea/workflows/deploy.yml` — the "Deploy to demo VM + health-gate" step uses `secrets.DEMO_VM_SSH_KEY` / `DEMO_VM_HOST` / `DEMO_VM_USER`. Explicit-over-implicit (visible in the run log, fires exactly on build success), one less always-on service than watchtower. **Constraints (from soong-dev):** deploy on CI success only; keep the trigger gated to `v*` tags + `workflow_dispatch` (as today); preserve the one-command rollback posture (`docker compose down` / pin a previous tag). **BLOCKER — needs from vh (owner-only):** a **runner→corviduo-dev deploy SSH key** as a repo secret (+ host/user), same class as WT's `DEMO_VM_SSH_KEY`. Likely **reuse WT's existing demo-deploy key** (WT's runner already SSHes to 10.250.50.152 as its deploy user). Repo secrets are vh-owner-only (see [[2026-07-18-fleet-gitea-runner-build-recipe]]). **Next-session steps:** (1) confirm/obtain the deploy SSH-key secret from vh (reuse WT's or mint fresh); (2) add the deploy job to build-and-push.yml (infra-ops has push on vh/soong-lab); (3) dispatch a build to verify it deploys + health-gates; (4) ping soong-dev so they sync DEPLOY.md's "open follow-up" note to the as-built mechanism. Auto-pull (watchtower) explicitly NOT chosen. See [[2026-07-18-soong-lab-containerize-cutover]]. ## AS-BUILT (2026-07-18, same-day execution) **Mechanism landed** exactly as planned: `build-and-push.yml` gained a `Deploy to corviduo-dev + health-gate` step (after build+push) that SSHes the host as `deploy` and runs `docker compose pull && up -d` from `/opt/soong-lab`, then polls `http://localhost:8443/api/version` for 120s and fails the job loud if unhealthy. No compose is shipped from CI (the in-repo `docker-compose.yml` is a BUILD compose; the host pull-compose is infra-ops-managed). Kept the `v*`-tag/`workflow_dispatch` trigger. Skipped WT's disk-watermark gate + health-gated-`:latest`-advance (low cadence, easy rollback). **Deploy identity = reuse WT's `deploy` account** (operator accepted the rec): - `deploy` (uid 1001, docker-group → no sudo) already owns `/opt/worldtree`; relocated soong-lab's deploy dir `/home/infra-ops/soong-lab-deploy` → **`/opt/soong-lab`** (deploy-owned), copied compose + `.env`. Named volumes (`soong-lab_soong-library`, `soong-lab_soong-portraits`) are project-scoped by compose `name: soong-lab` → followed the move untouched (dry-run `up -d` ADOPTED the running container, no recreate). Old dir **retired → `.retired-20260718`** (recoverable). Also lingering: `soong-lab-deploy.sh` / `.log` (dead pre-container webhook artifacts) — harmless, left in place. - **Dedicated soong-only ed25519 deploy key** minted (NOT literally WT's key — cleaner independent revocation), pubkey appended to `deploy`'s `authorized_keys` (fp `SHA256:MG7M3RiZJ176sLfblffb96V6W1qkRTgJ5dow1CpiY68`). Existing `deploy` key is plain/unrestricted, so parity held. **The secret gate (the friction point):** repo Actions secrets are **vh-owner-only** — claude-bot's token is `write:package,read:repository` (403 on secret-write), and the vh package-scoped PAT also 403'd on `PUT …/actions/secrets/…`. So `DEPLOY_SSH_KEY` / `DEPLOY_HOST` (10.250.50.152) / `DEPLOY_USER` (deploy) HAD to be set by the operator. First operator attempt produced a **bad key paste** — the deploy step died with `Load key … error in libcrypto` + `Permission denied (publickey)` (build+push were green; live Soong never moved). Fix: operator re-set the secret; the minted key path was pre-validated from nh3-dev (`ssh -i … deploy@… 'cd /opt/soong-lab && docker compose config -q'` → OK, health 200) so the re-set was the only variable. **Validation:** `workflow_dispatch` via claude-bot **basic auth** (its token lacks `write:repository` for the dispatch API; the account password works). Run #5 (task 1886) GREEN — live container recreated `sha256:…541f7730` → `…07526a08`, `StartedAt` fresh, health 200. `/api/version` now reports **0.3.25** (run #5 shipped soong-dev's 1c2f831 STYLE_WORKFLOWS re-pin as validation cargo). soong-dev synced `docs/DEPLOY.md` (commit `00b67c3`). NB: tag **v0.3.25 exists only locally** — pushing it would re-trigger a redundant build+deploy of the same commit (operator's discretion). **Ops now:** redeploy = tag `v*` or `workflow_dispatch` the CI (auto). Manual fallback = `sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'` (the `.env` is `deploy`-owned 600, so infra-ops needs `sudo -u deploy`, not a bare `cd`). _Archived 2026-08-16._ - `[2026-07-18]` **soong-lab auto-redeploy — DONE + VALIDATED** (was approved/queued; executed same day on fresh context — see AS-BUILT at the bottom). Vuong approved wiring auto-redeploy for soong-lab (relayed via soong-dev, thread `01KXT3A6C3908TA4V9THV3AMH7`): new images should go live on corviduo-dev without the manual `docker compose pull && up -d`. Host-side implementation is infra-ops's lane; mechanism is infra-ops's call per fleet conventions. Operator deferred execution — "we'll do soong on fresh context." **Chosen mechanism (recommended, agrees with soong-dev): Worldtree-style CI-deploy step** — NOT watchtower polling. - Add a deploy job/step to soong-lab's `.gitea/workflows/build-and-push.yml` that, after the build+push job succeeds, **SSHes from the pfi-fleet runner to corviduo-dev** and runs `cd /home/infra-ops/soong-lab-deploy && docker compose pull && docker compose up -d`, then a **health-gate** (`curl -fsS http://localhost:8443/api/version`). - This is exactly how WT deploys the demo instance to the SAME host: see `~/development/Worldtree/.gitea/workflows/deploy.yml` — the "Deploy to demo VM + health-gate" step uses `secrets.DEMO_VM_SSH_KEY` / `DEMO_VM_HOST` / `DEMO_VM_USER`. Explicit-over-implicit (visible in the run log, fires exactly on build success), one less always-on service than watchtower. **Constraints (from soong-dev):** deploy on CI success only; keep the trigger gated to `v*` tags + `workflow_dispatch` (as today); preserve the one-command rollback posture (`docker compose down` / pin a previous tag). **BLOCKER — needs from vh (owner-only):** a **runner→corviduo-dev deploy SSH key** as a repo secret (+ host/user), same class as WT's `DEMO_VM_SSH_KEY`. Likely **reuse WT's existing demo-deploy key** (WT's runner already SSHes to 10.250.50.152 as its deploy user). Repo secrets are vh-owner-only (see [[2026-07-18-fleet-gitea-runner-build-recipe]]). **Next-session steps:** (1) confirm/obtain the deploy SSH-key secret from vh (reuse WT's or mint fresh); (2) add the deploy job to build-and-push.yml (infra-ops has push on vh/soong-lab); (3) dispatch a build to verify it deploys + health-gates; (4) ping soong-dev so they sync DEPLOY.md's "open follow-up" note to the as-built mechanism. Auto-pull (watchtower) explicitly NOT chosen. See [[2026-07-18-soong-lab-containerize-cutover]]. ## AS-BUILT (2026-07-18, same-day execution) **Mechanism landed** exactly as planned: `build-and-push.yml` gained a `Deploy to corviduo-dev + health-gate` step (after build+push) that SSHes the host as `deploy` and runs `docker compose pull && up -d` from `/opt/soong-lab`, then polls `http://localhost:8443/api/version` for 120s and fails the job loud if unhealthy. No compose is shipped from CI (the in-repo `docker-compose.yml` is a BUILD compose; the host pull-compose is infra-ops-managed). Kept the `v*`-tag/`workflow_dispatch` trigger. Skipped WT's disk-watermark gate + health-gated-`:latest`-advance (low cadence, easy rollback). **Deploy identity = reuse WT's `deploy` account** (operator accepted the rec): - `deploy` (uid 1001, docker-group → no sudo) already owns `/opt/worldtree`; relocated soong-lab's deploy dir `/home/infra-ops/soong-lab-deploy` → **`/opt/soong-lab`** (deploy-owned), copied compose + `.env`. Named volumes (`soong-lab_soong-library`, `soong-lab_soong-portraits`) are project-scoped by compose `name: soong-lab` → followed the move untouched (dry-run `up -d` ADOPTED the running container, no recreate). Old dir **retired → `.retired-20260718`** (recoverable). Also lingering: `soong-lab-deploy.sh` / `.log` (dead pre-container webhook artifacts) — harmless, left in place. - **Dedicated soong-only ed25519 deploy key** minted (NOT literally WT's key — cleaner independent revocation), pubkey appended to `deploy`'s `authorized_keys` (fp `SHA256:MG7M3RiZJ176sLfblffb96V6W1qkRTgJ5dow1CpiY68`). Existing `deploy` key is plain/unrestricted, so parity held. **The secret gate (the friction point):** repo Actions secrets are **vh-owner-only** — claude-bot's token is `write:package,read:repository` (403 on secret-write), and the vh package-scoped PAT also 403'd on `PUT …/actions/secrets/…`. So `DEPLOY_SSH_KEY` / `DEPLOY_HOST` (10.250.50.152) / `DEPLOY_USER` (deploy) HAD to be set by the operator. First operator attempt produced a **bad key paste** — the deploy step died with `Load key … error in libcrypto` + `Permission denied (publickey)` (build+push were green; live Soong never moved). Fix: operator re-set the secret; the minted key path was pre-validated from nh3-dev (`ssh -i … deploy@… 'cd /opt/soong-lab && docker compose config -q'` → OK, health 200) so the re-set was the only variable. **Validation:** `workflow_dispatch` via claude-bot **basic auth** (its token lacks `write:repository` for the dispatch API; the account password works). Run #5 (task 1886) GREEN — live container recreated `sha256:…541f7730` → `…07526a08`, `StartedAt` fresh, health 200. `/api/version` now reports **0.3.25** (run #5 shipped soong-dev's 1c2f831 STYLE_WORKFLOWS re-pin as validation cargo). soong-dev synced `docs/DEPLOY.md` (commit `00b67c3`). NB: tag **v0.3.25 exists only locally** — pushing it would re-trigger a redundant build+deploy of the same commit (operator's discretion). **Ops now:** redeploy = tag `v*` or `workflow_dispatch` the CI (auto). Manual fallback = `sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'` (the `.env` is `deploy`-owned 600, so infra-ops needs `sudo -u deploy`, not a bare `cd`). _Archived 2026-08-16._ - `[2026-07-19]` **irv-ml1 ComfyUI — RTX VSR baked into canonical provisioning (comfy-dev ticket DONE).** RTXVideoSuperResolution node + `nvidia-vfx` dep were manual installs; documented both in the canonical `stacks/comfyui/README.md` runbook (this stack's provisioning IS the README — no automated provision script). Key durability insight: the **node** lives in `basedir/custom_nodes` (persistent, restic-included → durable) but the **`nvidia-vfx` wheel** lives in the venv under `run/` (disposable, restic-excluded → **dropped by any `rm -rf run/*` fresh-bootstrap**), so the pip step must re-run after every venv rebuild. Both steps run **as uid 1000** (root install → venv-ownership crash-loop, [[reference_irv_ml1_comfyui_mmartial]]); `--extra-index-url https://pypi.nvidia.com` kept **scoped to the nvidia-vfx install**, deliberately NOT a global compose `PIP_EXTRA_INDEX_URL` (would risk perturbing the pinned torch 2.12.1/SageAttention boot bootstrap). Node already live on the box; no host change, canonical runbook now replays it. comfy-dev informed. _Archived 2026-08-16._ - `[2026-07-19]` **vh private Gitea PyPI — consumer READ-access convention set + wyrd-dev provisioned.** Consuming agents read the internal vh PyPI (`https://gitea.phasefinal.com/api/packages/vh/pypi/simple/`) with a **shared read-only token** (operator call: shared, not per-consumer — read-only blast radius is small, per-agent Gitea identities aren't worth it). Minted a dedicated `read:package`-scoped PAT off **claude-bot** (`POST /users/claude-bot/tokens`, name `vh-pypi-read-consumers`; verified reads worldtree-sdk, write-probe 401), revocable/rotatable independently. uv auth = `UV_INDEX_GITEA_USERNAME=claude-bot` + `UV_INDEX_GITEA_PASSWORD=` (or `~/.netrc`); pyproject uses `[[tool.uv.index]] name=gitea … explicit=true` + `[tool.uv.sources] = { index = "gitea" }` (mirrors soong-lab's bifrost setup). Delivered to wyrd-dev (worldtree-sdk adoption) via mode-600 drop on nh3-dev, drop-and-shred. [[reference_claude_bot_gitea_creds]] _Archived 2026-08-16._ - `[2026-07-23]` **jackdaw-compose backend deployed as a persistent nh3-dev service (:8787).** Hosted for jackdaw-dev: thin stateless `bun server/index.ts` (from `~/development/jackdaw`) → LiteLLM `gen`, Origin-gated (INV-BK04/05), reached same-origin via their `:4500` bench's `/compose` proxy. `jackdaw-compose.service` (env/shared-key server-side, unit 0600, uncommitted). Also stood up + tore down a throwaway cloudflare quick-tunnel for their preview (`cloudflared` now installed at `~/bin`). In the nh3-dev README inventory (`cd4d52e`). _Archived 2026-08-16._ - `[2026-07-25]` **nh3-extdev herald installed — box is now a full v2 push participant.** forseti flagged (relaying operator): extdev had the `althing-herald` binary (`/usr/local/bin/`) but NO unit (skipped the whole v2 arc), so `herald-status` = "notifications suspended" and ldp-dev ran on the `althing-light-monitor` poll fallback. Installed `/etc/systemd/system/althing-herald.service` as a **SYSTEM unit mirroring the receiver** (`User=althing-svc`, `Group=althing`, `Environment=ALTHING_ROOT=/srv/althing`, `ExecStart=/usr/local/bin/althing-herald --poll 5`, enabled) via the **lkraven@ NOPASSWD path** (used under the then-mistaken belief infra-ops was sudo-less — **CORRECTION 2026-08-03: infra-ops has had full NOPASSWD sudo on extdev since 2026-06-25** per [[reference_nh3_extdev_althing_mesh]]; future extdev installs can self-serve as infra-ops without the lkraven@ hop). Verified: active / 0 restarts / `herald-status` flipped to "✓ herald up." No zellij routes on extdev → heartbeat + wake-FIFO poke only, no pane-dispatch; ldp-dev keeps light-monitor unless it opts into a wake-listener. _Archived 2026-08-16._ - `[2026-07-25]` **Booth v0.1.4 — booths are downloadable.** Verbatim `index.html` booths (e.g. edict-design-brief) were served raw with no download affordance. Added `/b//?download=1` (streams the whole booth as `.zip`, attachment) + `?dl=1` on the file route (forces Content-Disposition attachment so html/md/text saves instead of rendering inline) + ⬇ zip links on the index card (the accessible spot for verbatim booths) and the gallery header. `zip_booth()` helper, 31 tests green; verified live on nh3-dev :8090 (edict-design-brief.zip = index.html + ui-design-brief.md). eshpfi `91a031f` / tag `booth-v0.1.4`. _Archived 2026-08-16._ - `[2026-07-25]` **Kimi K3 wired into the LiteLLM gateway — CODING endpoint** (operator-directed; fulfills a Heid gateway request to add a 4th cross-frontier panel arm). **Primary `model_name: kimi-k3` → `openai/k3` @ `https://api.kimi.com/coding/v1`** (Kimi Code / Vivace membership; key `KIMI_CODE_API_KEY`). A general-endpoint variant `kimi-k3-gen-api` → `openai/kimi-k3` @ `https://api.moonshot.ai/v1` (key `MOONSHOT_API_KEY`) is kept alongside (originally wired then demoted when the operator corrected: the plan uses the CODING endpoint, not the general Moonshot API). Both keys in compose env + server `.env` (NOT committed) + `.env.example`. Both verified live through the gateway :4000 (17+25→"42", "PONG"). **k3 constraints on BOTH endpoints (config-pinned + commented):** accepts ONLY `temperature=1` (else 400 "only 1 is allowed"); REASONING model (CoT in `reasoning_content`, answer in `content` → tiny `max_tokens` returns EMPTY; Kimi Code adds thinking-effort tiers low/high/max). Coding lineup also carries `k3-256k` / `kimi-for-coding` / `kimi-for-coding-highspeed` (not wired). Reachable by any gateway key spanning all proxy models (incl. shared all-agents key → spends the paid Vivace/Moonshot quota). eshpfi `edaa9a9` (gen wiring) + `9e2f787` (coding correction). **OPEN:** Heid key-scoping — shared key reaches it (paid) vs a dedicated scoped key (asked in althing `01KYD63ZBY…`). _Archived 2026-08-16._ `[2026-07-25]` **infra-ops Worldtree config-as-code repo — SHIPPED + boundary AGREED.** **STATUS (2026-07-25, done this session):** `vh/worldtree-instance-configs` (private, gitea) built, pushed, validated; boundary agreement secured from worldtree-dev. - **Repo:** dir-per-instance `demo/` + `personal/` (5 files each: `defaults.yaml`, `policies.yaml`, `model_roles.yaml`, `providers.yaml`, `matrix.yaml`), seeded byte-exact from live `/opt//config`. `pinned/` = README stub only — **no `/app/config` bind-mount; config baked into frozen image `446e5807` (2026-05-13)**, so out-of-scope; deploy verb refuses it. - **Tool:** `scripts/deploy-wt-config ` — `diff` (read-only repo-vs-host), `deploy` (in-run host backup → `install -o vh -g vh -m 644` → restart **api+matrix** → health-gate api `/health` → auto-rollback), `capture` (host→repo reconcile). Instance table in-script (demo→`/opt/worldtree/config`+`worldtree-worldtree-{api,matrix}-1`; personal→`/opt/worldtree-personal/config`+`worldtree-personal-worldtree-{api,matrix}-1`). Matrix sidecar shares the config mount but has no healthcheck → restart both, gate on api. Env `WT_CONFIG_HOST` (default `infra-ops@10.250.50.152`), `WT_HEALTH_WAIT` (90s). Local clone `~/development/worldtree-instance-configs`. - **Gitea plumbing (reusable):** nh3-dev **403s the gitea HTTP API** (public fail2ban + internal `:3000` both 403). Repo CREATE went via **ana-docker localhost API** (`ssh infra-ops@10.250.50.70` → `curl localhost:3000/api/v1/user/repos`, vh token from `~/.config/tea/config.yml`, operator-authorized one-time). PUSH went over **internal git-SSH `ssh://git@10.250.50.70:222`** (works from nh3-dev; auths as vh). `git init` defaulted to `master` → renamed `main` to match repo default_branch. - **Boundary AGREED (worldtree-dev, althing thread `01KYCAECRWVEF16EVKQAGT2N80`):** no hand-edits to `/opt//config`; config changes route to infra-ops as deltas (worldtree-dev owns CONTENT + approval trail — the wyrd-grant shape — infra-ops lands+deploys). **Three-layer model:** image `config/` = baseline new instances seed from (theirs) → `vh/worldtree-instance-configs` = per-instance truth (ours) → host bind-mount = deploy target (written only by the tool). **Carve-out:** worldtree-dev's admin-API ops (`/admin/keys` mint, tier changes, session retirement, future runtime-grant surfaces) mutate instance **DATABASES not config files** → NOT config edits, stay in-band. If a future API writes config *files*, they flag at design time. b132 CONFIG BASELINE breadcrumb composes (INFO line = config-as-code diverges from image baseline, by design). - **No live deploy** done or needed — repo seeded == live (diff clean, capture round-trips zero-diff). Deploy path is dry-run-validated only; first real deploy needs operator per-change yes (managed box). --- _Original plan (2026-07-25, pre-build):_ `[2026-07-25]` **infra-ops to OWN a Worldtree per-deployment config repo + deploy tooling (operator-directed).** **Decision.** Vuong directed (2026-07-25, this session) that Worldtree instance config should be a *tracked change*, **managed and deployed by infra-ops — not worldtree-dev**. Model: worldtree-dev owns the app/image (+ the baked baseline defaults); **infra-ops owns config-as-code for every deployment** and deploys it. This is the durable fix for the root cause behind the whole #376 arc — config was edited live on host bind-mounts (`/opt//config/`) with zero version history, audit, or recovery. **What "no worldtree-dev involvement" does and does NOT cover** (clarified with the operator this session): - **Build + deploy = infra-ops-only.** Deploying config = write the host bind-mount file + restart the container (the *exact* procedure already run this session — backup → replace → restart → health-gate → rollback-on-unhealthy). No worldtree-dev in the deploy loop. Their CI only swaps the IMAGE; it does NOT resync the host config bind-mount (confirmed #376 finding). - **ONE load-bearing exception — a one-time boundary agreement, NOT per-deploy involvement:** for the repo to *own* config it must be the **only writer**. worldtree-dev "live-bridges" (hand-edits mounted config directly on the box). If the repo deploys config *and* they keep live-editing → **two writers fighting the same files** = #376 all over again. So secure a one-time "yes" from worldtree-dev: *the config repo is now authoritative; stop hand-editing `/opt//config`; route config changes through the repo.* (Five-minute agreement, not a design collab.) - **Standing coupling (not "involvement"):** the config *schema* is the app's, enforced by its boot validator (`core.config_validator`). infra-ops configs must stay schema-compatible with the deployed image; the boot gate is the loud backstop. **Build shape (recommended):** - Gitea repo `worldtree-instance-configs` (infra-ops-owned), **dir per instance** (`demo/`, `personal/`, `pinned/` — the three on corviduo-dev 10.250.50.152: demo `worldtree-worldtree-api-1` :8080, personal `worldtree-personal-worldtree-api-1` :8081, pinned `worldtree-pinned-worldtree-api-1` :8082). Config dirs: demo `/opt/worldtree/config`, personal `/opt/worldtree-personal/config`, pinned `/opt/worldtree-pinned/config` (verify pinned's mount). - **SEED FROM CURRENT MOUNTED STATE, don't author fresh** — capture each instance's live config (incl. legitimate live-bridged deltas: personal carries `agent_architect` role [Soong/soong-lab] in model_roles.yaml + `ratatoskr-affect-full-allow` in policies.yaml that are NOT in the app repo — the operator ruled these are BY DESIGN, keep them). Losing them = breakage (the affect-render one gates mood rendering). - Deploy script (e.g. `scripts/deploy-wt-config `): git = source of truth → push to host bind-mount + `docker restart` (same pinned image, no pull) + health-gate + auto-rollback. This is the proven-this-session procedure, scripted. - Files per instance: `policies.yaml`, `model_roles.yaml` (+ whatever else is bind-mounted — `defaults.yaml`, `providers.yaml`, `matrix.yaml` all live in `/opt//config`; decide scope — policies+model_roles are the authz/role layer, defaults/providers are heavier instance tunables). **Tracking surface:** operator-directed 2026-07-25, carried by this snapshot + `/tmp/infra-ops-handoff.md`. No issue filed (infra-ops-internal build). Related fleet idiom to reuse: canonical-sync (`.corviduo-canonicals.toml` / `canonical_sync.py`). Later scale option (deferred, needs worldtree-dev): base+overlay with a merge step in their pipeline. See [[2026-07-25-wt-376-per-instance-config-arc]] for the incident that produced this. Auto-memory: `reference_worldtree_perinstance_config`, `reference_corviduo_dev_emergency_ops`. _Archived 2026-08-16._ - `[2026-07-26]` **Demo `BIFROST_CLIENT_ALLOWED_HOSTS` += `10.100.10.50:8391`** (wyrd-dev's bifrost memory-store provider; operator-approved). **First live exercise of the #376 config-as-code boundary working as designed** — worldtree-dev routed the delta to infra-ops instead of hand-editing `/opt/demo`. Appended to `/opt/worldtree/.env:25` (now 4 netlocs), recreated ONLY `worldtree-api` (the gated conv-api path), health-gate green, container env verified. **REUSABLE FOOT-GUN:** an env-var change needs a container **RECREATE, not `docker restart`** (env is baked at create); and the demo `.env` defaults `WORLDTREE_IMAGE=:latest` while the box runs a specific SHA — so a naive `compose up` risks the documented stale-`:latest` crash. FIX = capture the running image live (`docker inspect …Config.Image` → `…:9eff09f007ba`) and `sudo env WORLDTREE_IMAGE= docker compose up -d worldtree-api`. Backup `/opt/worldtree/.env.bak-bifrost-20260726-221602`. **BOUNDARY SEAM:** this was a compose-`.env` var, NOT a `config.yaml` file in `vh/worldtree-instance-configs` — the `.env` holds secrets so it's deliberately not repo-tracked → env-deltas land directly on the box (config *files* are versioned, compose *env vars* aren't). [[reference_worldtree_instance_configs_repo]] _Archived 2026-08-16._ - `[2026-07-27]` **Zed edit-predictions: keyless FIM-completion route SHIPPED end-to-end.** Operator wants Zed's inline edit-prediction (which CANNOT send an auth header) to reach a FIM coder via `/v1/completions`. **Deep-research (106-agent workflow) picked `Qwen/Qwen2.5-Coder-1.5B`** (BASE, Apache-2.0; native FIM `<|fim_prefix|>/<|fim_suffix|>/<|fim_middle|>` IDs 151659/60/61; Zed `prompt_format:"qwen"`). Runner-up 3B = non-commercial Qwen-Research license; **no small dense Qwen3-Coder exists (all MoE, smallest 30B)**. **Stood up `vllm-coder`** on ana-ml2 **GPU1 :8020** (served-name `qwen2.5-coder-1.5b`, 8192 ctx, util 0.06, fp8 KV). To fit, **shrank granite (phasing out, operator-directed):** util 0.27→0.13, max-len 131072→16384, seqs 1024→256 (freed ~14 GB; the KV-≥-1×-max-len rule crash-looped it at util 0.12/32768 → settled 0.13/16384). **LiteLLM alias `coder-fast`** → :8020 (`mode: completion`). **Minted a `coder-fast`-SCOPED virtual key** (verified 403 on `gen` — the real blast-radius bound). **Built `zed-fim-proxy`** (ana-docker **:4141**, `network_mode: host`, stdlib-python, `stacks/zed-fim-proxy`): keyless POST `/v1/completions`, model-allowlist `coder-fast`, injects the scoped key → LiteLLM :4000; `GET /ping` anon liveness; wrong-model→403, wrong-path→404, `/chat/completions` rejected. Verified keyless FIM end-to-end ('a + b', finish `stop`). **Zed `api_url` = `http://10.250.50.70:4141/v1`, model `coder-fast`, prompt_format `qwen`.** **source-IP allowlist intentionally LEFT OFF (operator direction 2026-07-27) — do NOT tighten:** Zed roams the operator's WireGuard `10.0.0.0/8`, so a single-IP pin would break it. Blast-radius bound is the `coder-fast`-scoped key + model/path allowlist (keyless but coder-fast-only, internal-net-only). (The proxy does exact-IP matching; scoping to the `10.0.0.0/8` CIDR would need CIDR support — deliberately not added.) Canonical: `stacks/vllm` (coder + granite shrink), `stacks/litellm` (coder-fast), `stacks/zed-fim-proxy` (NEW). Server vllm compose.yaml has benign stale-comment drift vs canonical (didn't overwrite the newer canonical). _Archived 2026-08-16._ - `[2026-07-27]` **Muninn ingestion-watcher sidecar deployed on PERSONAL Worldtree (#377).** worldtree-dev request (research-wing ingest arc, personal-only per the 2026-07-16 topology ruling); operator-approved. Added a `worldtree-muninn` **compose sidecar** to `/opt/worldtree-personal/compose.yaml` — `<<: *worldtree-common` anchor inherits the api's image + full env + config/state/kb mounts; `command: python -m core.muninn --watch`; `restart: unless-stopped`; `stop_grace_period: 1h` (INV-377-7: max 2 concurrent × worst-case job, SIGTERM-drains). **Pinned to the running SHA `773866084af9`** (b146, ≥ b143 — dodges both the `:latest` trap AND the "pre-b143 ref resurrects deleted dispatch.py from stale bytecode" warning). Verified: running / 0 restarts / flock sole-runner (no rc3) / heartbeat live at `{ingestion_root=/data/state/ingestion}/.watcher-heartbeat` (poll 30s). Container `worldtree-personal-worldtree-muninn-1`; backup `compose.yaml.bak-muninn-20260727-081920`. **DURABILITY RESOLVED (worldtree-dev, same day):** Q1 was a LIVE FOOTGUN — `deploy-personal.yml` scp's the REPO compose.yaml over the box's + runs `up -d --remove-orphans`, so the box-local sidecar would've been clobbered AND orphan-removed at the next staging tag. worldtree-dev fixed at source: moved the sidecar into their repo compose.yaml gated behind a **`muninn` compose profile** (commit 5d7f6bd) — shared compose stays instance-identical, `.env` `COMPOSE_PROFILES` differentiates (demo watcher-less). **My action:** added `COMPOSE_PROFILES=muninn` to `/opt/worldtree-personal/.env` (backup `.bak-muninn-profile-20260727-082541`; no-op vs the current unprofiled box-local sidecar → seamless handover at next deploy). Q2: their deploy `up -d`'s the whole stack w/ `WORLDTREE_IMAGE` exported → sidecar version-tracks the api, no drift. **CONFIG-AS-CODE EXTENSION:** mirrored the non-secret delta as `personal/env.public` in `vh/worldtree-instance-configs` (repo `a9d091e`) — FIRST extension beyond config.yaml files to env-level config; the secret-laden `.env` stays box-only, `env.public` records only non-secret infra-ops-owned env deltas (record, not a deploy source — `deploy-wt-config` globs `*.yaml`). **BOUNDARY CLARIFIED:** compose.yaml = worldtree-dev's (their repo, instance-identical, scp'd on deploy); per-instance `.env` = infra-ops's differentiator. Deploy step of the #363/#377 arc. **#377 CLOSED — acceptance PASSED 2026-07-27:** worldtree-dev enqueued a test job via muninn-dispatch 0.1.0 in a one-shot ephemeral container (no docker-exec); the sidecar claimed it within one 30s poll, drove it to terminal (structure→summarize→complete), zero restarts/rc3, heartbeat fresh throughout — whole loop (request→deploy→durability fix→acceptance) in <2h. (Pre-existing pipeline bug #379 surfaced — `output.kb_notes=false` ignored → 1 inert test note in the research wing — worldtree-dev owns it, nothing infra-ops-side.) **⚠ OPERATOR-SURFACE (open):** the `env.public` overlay mechanism is a repo-scope call to bless/adjust. [[reference_worldtree_deploys_cicd]] [[reference_worldtree_instance_configs_repo]] [[project_worldtree_research_wing_ingest]] _Archived 2026-08-16._ ## Tried and abandoned (archived 2026-08-16 batch) - `[2026-07-18]` **Fleet Gitea CI foot-guns** (3 failed soong-lab builds): the pfi-fleet runner's `node:20-slim` job image has no docker/git so `actions/checkout` + `docker/*` marketplace actions all fail; `vh` is a USER so its packages are owner-write-only (claude-bot repo-admin-collab still 401s on push/publish, and can't set repo secrets — owner-only); `GITEA_`-prefixed secret names are reserved/illegal. Fixes in → `persistent-memory.d/2026-07-18-fleet-gitea-runner-build-recipe.md` _Archived 2026-08-16._ - `[2026-07-18]` **zonos-gateway local clone had NO git remote + a history unrelated to gitea's** — "committed to vh/zonos-gateway" was never pushed from that clone; two separate `git init` lineages, no merge-base. Reconcile = reset local→origin/main + overlay the changed files + push (NOT force — that erases gitea's voice-wav commits). Check `git remote -v` + `git merge-base` before assuming a clone is wired. _Archived 2026-08-16._ - `[2026-07-25]` **Peer green-light ≠ operator consent for a managed-box mutation.** Auto-mode guard blocked a config-replace+restart on the Worldtree-team demo box that was authorized only by worldtree-dev's althing message — correctly: a persistent change to shared infra needs the *operator's* yes for that specific change, not a peer's. Surface it; don't route around the guard. (The operator then stood the whole change down — the guard's hold was the right call.) _Archived 2026-08-16._ ## Recent decisions (archived) **Worldtree b168/#384/#385 arc — COMPLETE 2026-08-03.** A long peer-driven arc across worldtree-dev / muninn-dev / mimir-dev / ratatoskr-dev, all on corviduo-dev's demo+personal instances. Sequence: providers.yaml boot-gate pre-sync → b168 deploy → DCC #384 reindex → round-2 full re-ingest → #381 restart → operator-approved production dedup sweep. Landed clean; three of MY foot-guns along the way, each caught + hardened into a fleet runbook rule (see Tried-and-abandoned: `mv -t`, `docker exec -u 1000`, shared-containerd race). ## providers.yaml pre-sync (boot-gating config) b168 (commit `293f8f3`) added a `summarization` capability block that in-image `agents/muninn/config.yaml` references → boot-blocking if the host bind-mounted providers.yaml lacks it. Synced both hunks (summarization block + deep-reasoning desc) into demo+personal via `deploy-wt-config`; instance-configs commit `53349f8`. - **deploy-wt-config runbook:** `~/development/worldtree-instance-configs/scripts/deploy-wt-config {diff|deploy|capture} --file providers.yaml` (per-instance dirs demo/personal/pinned; `deploy` = host write + api/matrix restart + 90s health-gate + auto-rollback; `diff`/`capture` safe). demo+personal providers.yaml are byte-identical. - **GOTCHAS:** (1) an UNPUSHED source commit → `git show ` 404s and a gitea `raw?ref=` silently falls back to the default branch; verify the commit exists (`/git/commits/`) before trusting a fetch, else ask the peer to paste hunks. (2) a peer's hunk paste may be mis-indented (8-space vs the block's 4-space) → invalid YAML; always YAML-validate after a paste-sourced edit. - **Config-delta pre-sync rule (verified via `docker inspect`):** worldtree containers bind-mount ONLY `config/` host-side (`/opt/worldtree-*/config/` → providers/model_roles/matrix/policies/defaults/env.public = the pre-syncable set); `agents/` (schemas.yaml, prompts) + all code ship IN-IMAGE. So only a `config/*.yaml` change is boot-blocking-pre-syncable; an `agents/`-or-code delta needs NO host pre-sync (CI carries it). b169's schemas.yaml (#387) was correctly no-pre-sync. ## #384 reindex + #381 restart + verify DCC job `mimir-6351554e8e8f`. Reindex: `sudo docker exec -u 1000 worldtree-personal-worldtree-muninn-1 python -m core.muninn --reindex ` (⚠️ MUST `-u 1000` — default-root writes contaminate the uid-1000 KB tree; see Tried-and-abandoned). Then **#381 restart** (stale-Chroma-client fix): `sudo docker restart worldtree-personal-worldtree-api-1` (plain bounce, NO compose up / no image repoint) → healthz/readyz 200 ~25s. - **Chroma-verify runbook:** `sudo docker exec -i python -` (MUST pass `-i` or stdin never reaches `python -`) → `chromadb.PersistentClient('/data/kb/.chroma').get_collection('fiction').get(where={'job_id':}, include=['metadatas'])`. Chroma persists at container `/data/kb/.chroma` = host volume `worldtree-personal_worldtree-kb`. - **Retrieval-visibility check (NOT grounding — that's ratatoskr's):** a Mimir session — admin token `~/.config/worldtree/personal-admin-token` (wildcard scope) → POST `/sessions` (agent_id=`mimir`, `record_tool_intermediates=true`) → POST `/sessions/{id}/messages` (STREAMS SSE, not JSON) → parse SSE `tool_result` for `search_library` wing hits → DELETE session. ## Production dedup sweep (operator-approved) Deleted the 785 April-era DCC orphan rows (`job_id=b59c147c5ce0`, no wing/source_identity metadata → predate identity tracking) from the `main` collection. Supervised protocol: read-only verify count == 785, back up all rows (ids+docs+embeddings) to `corviduo-dev:/tmp/main-sweep-backup-b59c147c5ce0.json` (reversible), `main.delete(where={job_id})` (assert target==785 first), verify `main` 4009→3224, then **bounce the api** (a separate-process delete leaves the api's in-memory HNSW index holding the vectors until reload — the #381 pattern generalizes to deletes), confirm search now fiction-only. Backup left for /tmp natural cleanup (fiction wing is canonical; `~/archives` has the historical record). Result: fiction wing 166 → 1,372 concepts; three consumer verify rounds 0/5 → 5/5 → saturated; #385 budget fix validated (705 vs April's 785 control, extraction AND indexing, zero truncations). worldtree-dev filed #388 for a deploy concurrency-lock (the shared-containerd race fix). See [[2026-08-02-mimir-inbox-arc]]. _Archived 2026-08-18._ `[2026-08-02]` **The mimir-inbox / #377-read-path arc — deploy, four bugs found+fixed+verified, a cloned voice, all in one long session (2026-08-01→02).** The browser-facing half of the #377 Muninn ingestion arc, end to end: mimir-inbox stood up, the write path proven, the read path chased through four defects to a verified-working state, and a character voice cloned into the TTS zoo. Peers: mimir-dev (the app), muninn-dev (gate/watcher spec), worldtree-dev (Worldtree app layer + the #380/#381/#382/#383 fixes), ratatoskr-dev (a consumer + the rigorous verifier). ## mimir-inbox deployed (#377) - **New infra-ops stack, canonical eshpfi `stacks/mimir-inbox/`; live corviduo-dev `10.250.50.152:8091`** (co-located w/ muninn-gate :8090 + the worldtree-personal muninn watcher). Full deploy detail + procedures → auto-memory `reference_mimir_inbox_deploy`. - **Placement decision (operator, reversed):** 7-31 he ruled mimir-inbox stays OFF corviduo-dev (shared/NFS mount); 8-01 he REVERSED to CO-LOCATE. Trigger: muninn-dev's code-check showed staging is NOT same-fs-constrained (gate reads staging metadata + passes path strings; `os.replace` is inside `ingestion_root`) — staging's real constraint is **path-identity across writer/gate/watcher**, which co-location buys outright while dodging NFS failure modes. I HELD the reversal for the operator's direct word (data/hosting on a team-managed box, reversing his own ruling) even against 3 peer relays — vindicated as the right instinct; muninn-dev agreed. - Build: **`uv sync --no-dev --frozen`, SINGLE-STAGE** (project installs editable-linked to `src/`, so src/ MUST stay beside .venv — a multi-stage "copy only .venv" dies at import/404s assets). uid 1000, host-net bind 10.250.50.152:8091, TCP-liveness healthcheck (deliberately NOT gate-coupled). Redeploy = refresh build context (**preserve the on-server `.env`!**) → `docker build -t mimir-inbox:0.0.1 -t mimir-inbox: .` → `compose up -d`. Version stays 0.0.1 across dev commits → tag the image w/ the source SHA too. Live commit progression `0478452`→`c8ab38f`→`2dcc77e`→**`8ece117`** (3 redeploys). - mimir-inbox key on the gate bumped [read,submit]→**[read,submit,control]** (cancel/retry); brokered via a 0600 drop on nh3-dev (never on the althing bus). ## The read-path bug chain (worldtree-dev's, all found via this arc) - **#380 wing-blind indexing:** the book-ingest path upserted concepts into a hardcoded `main` Chroma collection while wing search reads the `fiction` collection → P&P written to disk but `search_library` returned total 0. A silent-success defect ("complete/69 indexed" was right about the WRITE, wrong drawer). Root-caused off MY physical evidence (files on disk + search empty). Fixed b164 + a one-shot `--reindex ` (re-upsert into the right wing collection + delete stray `main` rows). - **#381 stale Chroma client:** the personal api opens its Chroma client before the watcher's cross-process writes → **a freshly-ingested/re-indexed book is NOT queryable until the api is restarted.** Proven by my restart-diagnostic (pre-restart total 0 → post-restart hits, same index). Workaround until fixed: `docker restart worldtree-personal-worldtree-api-1` after any ingest/re-index. Filed as #381. - **#382 unreliable Mimir grounding (the subtle one):** post-#380-fix the index was correct, but Mimir's grounding was INTERMITTENT — some sessions navigated the opaque job-hash dir (`mimir-f3887c9b97b7`) to the content, others distrusted the correct vector hits and **silently answered from training knowledge** (worst of the looks-fine-isn't family). ratatoskr-dev caught it; I'd been over-confident ("Mimir read Austen back to you") having verified the INDEX, not the GROUNDING. Fixed b166 with BOTH shapes: a self-describing `_index.md` per wing job-dir (resolves the hash dir to its title) + a Mimir prompt rule (wing-scoped hits ARE library content, never discard on a name mismatch, never substitute training). **Verified: ratatoskr-dev re-ran 3× fresh sessions → 3/3 grounded**, citations in note-extracted language not raw Austen. #382 CLOSED. - **DCC (Dungeon Crawler Carl, job `b59c147c5ce0`) backfill:** `--reindex` FAILED ("job not found in any state dir" — predates state-tracking). SETTLED = **no re-file** (the b166 prompt rule already grounds it even without an `_index.md`; ratatoskr confirmed incidentally); an `_index.md` rides whenever DCC is next re-ingested. - **#377 mimir-inbox banner bug (mimir-dev's, `8ece117`):** `/health-banner` misattributed an unwritable `ingestion_root` to the WORKER, rendering "The worker is not running." for a running worker — a false lead pointed at infra-ops's half of #377. Fixed (guard split into two banners); I confirmed from the DEPLOYED handler (not just the test) that `ingestion_root_writable:False` now renders "The ingestion root is not writable." ## muninn-gate → muninn-dispatch 0.1.5 Rebuilt `muninn-gate` off `vh/muninn-gate` main `bc04c4c` (dispatch 0.1.4→0.1.5) so the gate serves the new `concept_schema`/`concept_schema_source` row fields (computed gate-side). Gate version unchanged 0.0.14 (dual-tag the SHA). Build needs the vh gitea token as a BuildKit secret (`--secret id=gitea_pw`, UV_INDEX_GITEA_USERNAME=vh, drop+shred). Recreate with `compose up -d` (NOT bare restart — needs the new image). Verified: P&P job serves `concept_schema='fiction'`, `concept_schema_source=null` (null correct — pre-b164 job). Registry tags by commit SHA — `v1.0.0bNNN` docker tags don't resolve; use the deployed SHA (confirm `--reindex` present before using an image for a data-op). ## donut voice (65-frost → Zonos gateway) Operator: "pick up 65-frost, use that bundle as a voice for a character named donut." 65-frost = a **Booth id** (`~/booth-data/65-frost/`) holding a curated yt-voice-clipper dataset (`dataset-…-curated.zip`: 4 clips + manifest, all SPEAKER_02 = Princess Donut). **Zonos gateway voice registry = a filesystem drop:** `.wav` in the voices dir (44.1kHz mono s16 PCM) auto-registers as `voice:""` on **startup** (needs a restart). The LIVE dir is the bind mount `/opt/docker/compose/zonos-gateway/voices/` (lkraven-writable), NOT the working tree. Built `Donut.wav` from seg000 (best clip), dropped it, restarted → `voice:"donut"` live in the gateway AND the Asset Engine's make form. Also copied to the build-source tree `~/zonos-gateway/voices/` for rebuild-durability (true canonical = the gitea repo, not yet CI-wired). Auditioned in booth `donut-voice`. **Expanded 2026-08-02 (onyx-58 bundle):** operator curated a 2nd Booth bundle `onyx-58` (`dataset-467d2cf8…curated.zip`, 3 Donut clips) as additions. Rebuilt the reference = **seg000 (65-frost) + seg101/seg110/seg148 (onyx-58)** ffmpeg-concat + resampled 24k→44.1k mono s16 = **52.0s**. `seg148` was diarized SPEAKER_03 but is Donut (operator-confirmed misdiarize → included). Assembly is NOT `assemble_voice.py` (that `-c copy` can't resample + caps ~15s); used a manual `aresample=44100,aformat=…,concat=n=4` filter. Backed up old ref → `irv-ml1:~/Donut.wav.pre-onyx58`; dropped to live bind-mount + build-source tree; `docker compose restart` (healthy 2s, `voice:"donut"` still 1 of 9). A/B booth `donut-onyx58` (A=old 16.3s ref, B=new 52s ref, same line). Longer ref is fine mechanically: gateway passes it as `speaker_audio_base64` → speaker *embedding*, not an audio prefix. **BUT auditioned → REVERTED same day:** pinned-seed neutral A/B (5 pairs, booth `donut-onyx58`) showed the single-clip seg000 (16.3s) beats the 52s 4-take concat on timbre — concatenating disparate takes muddied the embedding more than the range helped. Reverted both live + build-source to seg000-alone. Lessons (→ Tried-and-abandoned): more reference ≠ better when takes vary; and **emotion steering pulls output away from the clone fast** (operator craft rule) — keep clones emotion-neutral; bare `{input,voice}` calls send NO emotion (gateway only enables it on an explicit `emotion_*`/`preset` dial). ## Zonos streaming (no gateway change needed) ratatoskr wanted play-as-it-arrives. `/v1/audio/speech` ALREADY streams — chunked `StreamingResponse`, opens native `/tts/generate` with `stream=True`, wraps as a streaming int16 WAV with `0xFFFFFFFF` placeholder sizes (meant for progressive `