esh-pfi-infrastructure/archival-memory.md

# Archival memory — eshpfi-management

_Entries moved out of persistent-memory.md to keep the active file scannable. Read this when researching historical decisions or revisiting past foot-guns._

## Recent decisions (archived)

- `[2026-08-16]` **Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated.** brokkr-smithy-dev supplied the framing that reproduces (`01M05M48R4RSZF9D8KT7RR55EJ`): a **bare assistant-mode instruction** — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing **DS 0.0% / FF 0.0%** (n=75); plain character cards **DS 1.4% / FF 0.0%** (n=74); bare instruction **DS 92.5% (37/40) / FF 15.8% (6/38)**. Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→**0**, consensual 80→20, dubcon 80→**0**, self-harm 80→**0**. DS refused **25/25** on the five axes brokkr flagged. Root cause: `ReadyArt/Dark-Scarlett-v1.0-27B` is a plain finetune of stock `Qwen/Qwen3.6-27B` carrying **NO abliteration** — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-**ablated** (structural), which is why it holds. ⚠ **RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID.** His diff against his own artifact showed my `battery-instruct.yaml` reproduces only his **`creative` class — 8 of 16 axes**; it dropped all 5 `operational` (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 `meta` (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. **His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta.** Cause: I rebuilt his shape from his *message*, and the `class` field lives in the artifact, not the prose. **Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability.** ⚠ **Known battery bug left unfixed for comparability:** DS's arm-3 control gate failed at 11% because `ictrl-reunion` pairs "explicit / do not fade to black" with *brothers*, which DS reasonably read as an incest request; FF did not. `ictrl-storm` is the clean control. Commit `b9e68c3`.
  _Archived 2026-09-01._
- `[2026-08-15]` **Quant lessons consolidated into `docs/pfi/model-quantization-playbook.md` — the durable home; read it BEFORE any requant.** Survey found quant knowledge scattered across 18 files in 4 trees, with **three** documents having independently written overlapping "landmines" sections (the loader-class trap alone was rediscovered 3×). Playbook owns the **transferable** lessons (scheme choice, landmines, acceptance gate + its 3 measurement traps, hardware/co-residency); per-model artifacts are demoted to worked examples that link up. Carries a **superseded-claims table** — which immediately earned itself: the heretic2 runbook's "use modelopt, compressed-tensors can't load the BF16 MTP" is **false** (the cause was the missing `re:^mtp.*` ignore, not the format) and would have sent the next session down the modelopt dependency-hell path; that runbook now carries a stale-warning header. Maintenance rule in `CLAUDE.md`: model-agnostic → playbook, model-specific → stays put, wrong claim → dated superseded row, never a silent edit. Motivated by Qwen3.8 having just released — the next model swap needs a requant. Commit `a91cc3f`.
  _Archived 2026-09-01._
- `[2026-08-15]` **Operator ruling: the gen seat's +1.7% perplexity is an acceptable price for the speed — SETTLED, don't re-litigate.** Precise attribution for future reasoning: it is the **activation-quantization** cost (W4A4 MLPs + FP8 attention vs BF16 activations), not an MTP cost — PPL was measured with speculative decoding **off** on both builds, so MTP was not in the loop. Turning MTP off would not recover it; only reverting the quant would (rollback = one `.env` line, old build intact at `…/qwen38-27b-uncensored-nvfp4`).
  _Archived 2026-09-01._
- `[2026-08-15]` **gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed.** The queued "W4A8" (NVFP4 weights + FP8 activations) is **not servable** — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and `CompressedTensorsW4A8Fp8` is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter **per-layer-group**. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did **80.12** (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: `unsloth/Qwen3.8-27B-NVFP4` was already on-box → served as a probe, measured **+19.1% at identical acceptance**, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → **80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%**; surface 6/6 live, 7 aliases routing. char-rp had **no** tool parser at all (every tools request 400'd) → `gemma4` tool + reasoning parser + a **mandatory** `enable_thinking:false` (the parser defaults it True → null `content` for all RP prose; proven byte-identical prompt before deploying). Commits `b8f0f4c`, `74f596b`. Foot-guns banked (llm-compressor prunes unmatched `ignore` entries → the 0%-MTP bug, **fired on this run**; prompt_logprobs uniform under spec-decode; 0600 `.env` silently no-ops compose; GPU0 is zero-sum). → `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`
  _Archived 2026-09-01._
- `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
  _Archived 2026-09-01._

- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md`
  _Archived 2026-09-01._
- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md`
  _Archived 2026-09-01._

- `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`.
  _Archived 2026-08-28. Its deferred item — the nh3-extdev herald 2.1.2 upgrade — is closed: extdev went 2.1.0 -> 3.0.0 -> 3.1.1 at the v3 cutover, so the staged v2.1.2 wheel is moot._


# eRP dual-seat overhaul — MeroMero-v2 + Dark-Scarlett, NVFP4A16 @ 256K on ana-ml2

`[2026-08-12]` Replaced the two legacy char-rp seats with home-quantized NVFP4A16 vLLM
seats. Operator-driven, end to end this session.

## What landed

| Seat (LiteLLM alias) | Model | Role | GPU | Context |
|---|---|---|---|---|
| `char-rp` (:8016) | **G4-MeroMero-v2-31B** (Gemma-4) | non-thinking PROSE, **multimodal (vision)** | GPU0 | 256K @ 2.07× (util 0.52) |
| `char-rp-reasoning` (:8018) | **Dark-Scarlett-v1.0-27B** (Qwen3.6) | THINKING (default) | GPU1 | 256K @ 1.62× (util 0.44) |

- Both **NVFP4A16 weight-only** (llm-compressor, `compressed-tensors`), `--kv-cache-dtype fp8`.
- Replace: `char-rp-gguf` (Magidonia-24B GGUF/llama.cpp, :8016) + `heretic2-charrp-reasoning`
  (DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP, :8018). Old stacks/containers **stopped +
  retained** for rollback.
- Compose-ified: `stacks/meromero-charrp` + `stacks/darkscarlett-charrp-reasoning` (ana-ml2
  `/opt/docker/compose/`, mirrored to eshpfi, commit **`f08b6cb`**) → survive reboot.
- Research that drove picks: `docs/pfi/erp-thinking-finetunes-2026.md` (from the `gecko-65` Booth).

## Load-bearing lessons (the whole point of this file)

1. **Load via the ConditionalGeneration WRAPPER class, never `AutoModelForCausalLM`.** For a
   multimodal-capable base (Gemma-4, Qwen3.6), `AutoModelForCausalLM.from_pretrained` +
   `save_pretrained` writes a FLAT text config (`Qwen3_5TextConfig`, `model.layers.*`) that
   **both vLLM AND SGLang reject** (SGLang: "Qwen3_5ForCausalLM has no SGLang implementation";
   vLLM wants `Qwen3_5ForConditionalGeneration`). Loading via `Qwen3_5ForConditionalGeneration` /
   `Gemma4ForConditionalGeneration` keeps the wrapper config they accept. **This was the DS
   blocker** — re-quant via the wrapper fixed it (`Dark-Scarlett-...-NVFP4A16-wrapper`).
2. **NVFP4A16 is weight-only → DATA-FREE.** llm-compressor infers `DataFreePipeline`; calibration
   data is unused (only matters for W4A4 activation quant). W4A16 chosen per NVIDIA's sm_120
   long-context guidance (W4A4 KLD 2-4× worse past ~10k ctx).
3. **Load on CPU (`device_map=None`)** so llm-compressor onloads one layer at a time. `device_map=
   "auto"` packs the whole model onto the GPU and OOMs when the card isn't fully free.
4. **Both models are KV-EFFICIENT — the "dense = KV-hungry" worry was WRONG.** MeroMero (Gemma-4)
   uses **sliding-window attention** (most layers cache only a bounded window); DS (Qwen3.6) uses
   **hybrid GatedDeltaNet linear-attention** (3:1 linear:full, linear layers carry no KV). Both
   hit full native 256K easily. (MeroMero KV pool ~542K tokens at util 0.52.)
5. **MeroMero vision reconstruction.** The finetune ships `processor_config.json` (image_processor
   inline, `Gemma4ImageProcessor`) but NOT `preprocessor_config.json` — the old-format file vLLM's
   feature-extractor loader wants. **Even google/gemma-4-31B-it (ungated!) ships only
   processor_config.json.** FIX: extract the `image_processor` section → write
   `preprocessor_config.json` verbatim, serve WITHOUT `--language-model-only`. Verified (model
   correctly ID'd a red circle). Audio is config-declared but WEIGHTLESS (0 audio tensors).
6. **GPU placement.** Match the KV-heavier model to the roomier GPU. GPU0 (gen neighbor, ~54GB
   free) > GPU1 (utility cluster, ~45GB free). Swapped MeroMero→GPU0, DS→GPU1. Pins via compose
   `deploy.resources.reservations.devices`.

## Dead ends (tried + abandoned)

- **DS via llm-compressor `AutoModelForCausalLM`** → flat config vLLM/SGLang reject. → wrapper class.
- **DS via NVIDIA ModelOpt** → modelopt↔transformers **version deadlock**: current transformers
  supports `qwen3_5` but crashes modelopt's sparse-moe plugin (`issubclass()` on a non-class);
  modelopt 0.43.0 pulls an old transformers that can't load `qwen3_5` at all. Abandoned.
- **DS via SGLang** → `Qwen3_5ForCausalLM has no SGLang implementation`. Abandoned, but it REVEALED
  that both engines need the wrapper (→ the fix in lesson 1).
- **`device_map="auto"` for the quant** → CUDA OOM in the weight observer. → `device_map=None`.

## granite retired + gateway repoint

- `vllm-granite` (granite-4.1-8b, fleet summarizer, GPU1) **`docker stop`ped** (reversible) to
  reclaim ~13.6GB GPU1 for RP context.
- LiteLLM (`ana-docker:/opt/docker/conf/litellm/config.yaml`, backed up
  `.bak-pre-granite-down-*`): **`granite-4.1-8b` alias RETIRED** — commented out, now 404s cleanly
  (the `*` wildcard→llama-swap was decommissioned 2026-06-20, so no fallthrough). **`summarizer` +
  `classifier` REPOINTED to gen** (`hosted_vllm/qwen3.6-35b-a3b-heretic` @ :8015,
  `enable_thinking:false`) — both verified. ⚠ This LiteLLM change is **server-only / not
  version-controlled** (a follow-up).

## MTP — deferred

DS's MTP heads were dropped by the CausalLM loader; **deferred, not restored** (spec-decode is
net-negative at RP temps: ~38-52% accept at temp 0.8-1.25, below vLLM's 0.5 cutoff). The
splice-back path (`splice_mtp.py` in the heretic2 work dir) exists if ever wanted. MeroMero
(Gemma-4) has no MTP by architecture.

## On-disk / where things live

- Quant pipelines: `ana-ml2:/tank/aimodels/meromero-v2-nvfp4-work/` +
  `/tank/aimodels/darkscarlett-nvfp4-work/` (scripts, BF16 source, NVFP4 outputs).
- Compose stacks: `ana-ml2:/opt/docker/compose/{meromero-charrp,darkscarlett-charrp-reasoning}/`.
- Gateway aliases (unchanged, port-based): `char-rp`→:8016, `char-rp-reasoning`→:8018. (char-rp was
  also fixed from the stale `magidonia-24b-v4.3` backend model name → `char-rp`.)

## Open follow-ups

1. LiteLLM granite/repoint change NOT version-controlled (server + backup only).
2. eshpfi unpushed (many commits this session incl. `f08b6cb`, `7bd7375`, `398b58a`).
3. MTP deferred (see above).
4. DS thinks verbosely (~13:1 reasoning:content) — eval item; consumers need generous `max_tokens`.
5. MeroMero full 256K needs util 0.55 (GPU0 ~1.8GB free, tight); ran at 0.52 for headroom (~4.6GB).
_Archived 2026-08-27._

- `[2026-08-12]` **infra-ops now holds an all-zones Cloudflare DNS-edit token (vaulted) + wgtunnel Phase-0 DNS landed.** Operator handed over a `Zone·DNS·Edit` (all zones) CF token → `secret put nh3-dev/.config/cloudflare/infra-ops-dns-token` (round-trip verified; /tmp drop shredded). Fleet DNS is now self-serve for infra-ops (⚠ HIGH blast radius — all zones). First use: created `boring.phasefinal.com` CNAME → `ana-srv1.phasefinal.com`, **DNS-only** (proxied:false), verified resolving to 38.120.12.44 on both authoritative NS (louis/wren) + 1.1.1.1 — NOT Cloudflare-proxied. Unblocks wgtunnel's wstunnel ACME cert. phasefinal.com zone id `f812ba74ed9a75cf21bbe7ce9188db50`. auto-memory `reference_infra_ops_cloudflare_dns_token`. (Earlier gap: the only prior vaulted CF token, jackdaw's, had `zone:read`+`worker:edit` but no `dns_records:edit`.)
_Archived 2026-08-27._

- `[2026-08-12]` **wgtunnel stood up as its own repo (`vh/wgtunnel`, private) after a live endpoint-verification pass.** Operator directed own-repo (mirrors stonehenge-park/tts-stack). Verified off the fleet before seeding: `ana-wg` WG server = **UDP/31337** (not 51820), subnet 10.30.10.0/24, MTU 1420, active roaming peer proves the public UDP DNAT works; traefik on ana-docker **terminates TLS :443** (ACME `anaprod` http-challenge, docker+file providers, CrowdSec bouncer) → confirms the clean design (wstunnel container on `traefik-net`, Host-routed, WS→UDP to `ana-wg:31337`); edge `38.120.12.44` direct-A, `tunnel.phasefinal.com` free (⚠ must be **direct**, NOT Cloudflare-proxied like vaultwarden). Repo pre-seeded (README/CLAUDE/persistent-memory/ROADMAP + `docs/verified-infrastructure.md` = ground truth) + pushed; commit `9584d38`, Vuong-attributed. vh gitea token pulled from the vault (`secret get`), not persisted to `.git/config`. **NEXT = `/vor-plan` or `/vor` (operator's call, interactive).** Deps to line up in the plan: DNS A-record, FortiGate :443 host-routing, a new ana-wg peer for the laptop, client tooling.
_Archived 2026-08-27._

`[2026-08-11]` **stonehenge-park — new fleet `/park` service repo stood up + designed.**

**What.** A separate greenfield repo (`~/development/stonehenge-park`, gitea `vh/stonehenge-park`,
pushed) for a self-contained `/park` service: one durable place to park any idea (repo-born OR
personal), find it by search, and have it **actively resurface** (by due-date or staleness) until
acted on — so parked ideas stop dying when a repo goes cold. NOT part of eshpfi; this is a pointer.

**Design (via `/vor-plan`, converged + persisted to `docs/design/`):** four contract-sized units —
**U1** core store+API (SQLite+FTS5, slug minting, bearer auth, REST) — the tracer, build first; **U2**
scheduler+notifier (in-process; due/stale → statusline `due-count` + althing push to a dedicated
**assistant channel**; keep-surfacing until promote/drop/re-snooze); **U3** `park` CLI (mirrors the
`secret` CLI); **U4** browse UI. `/vor-ui` ran too (U4 brief persisted).

**Locked decisions (operator):** SQLite, self-contained, ONE container, no external DB ("don't want
to troubleshoot it when a database upgrade happens") — a hard `[OPS]` invariant; system-minted
title-derived slugs + short ID (addressable as `park/<slug>`); active keep-surfacing resurfacing with
**re-snooze as the anti-nag valve**; bearer key, LAN/WG-internal; host nh3-docker; `/park` **replaces**
the global ROADMAP parking-lot discipline (deferred ideas → `/park`, `source`-tagged; ROADMAP keeps
only the v1 target) as a **fast-follow after v1** incl. migrating existing lots.

**Deferred (in the plan):** the althing assistant-channel handle **name** (decide at U2 contract
time); staleness threshold + re-push cadence (env-tunable defaults ~30d/~daily); design U2's emit
structured/consumable so a future **mission-control (Ledger→orchestrator)** can read it — park does
NOT build the orchestrator.

**State.** Pre-seeded for a fresh agent (CLAUDE/persistent-memory/ROADMAP/README + the design docs),
committed (`294ee98`), pushed. Next build task lives in that repo: the **U1 tracer contract** under
the House Code Discipline. Auto-memory candidate not yet written (repo is self-documenting).
_Archived 2026-08-27._

- `[2026-08-12]` **Global `~/.claude/CLAUDE.md`: `secret`/vault tool entry + "store in AND pull from the vault" standing directive** (dotfiles `9db703b`, pushed); statusline reset-countdowns + a latent tab-collapse parse-bug fix, now tracked in the dotfiles stow tree. Dogfooded the directive: created `vh/stonehenge-park` pulling the gitea token via `secret get`. (dotfiles + global config, not eshpfi.)
_Archived 2026-08-27._

- `[2026-08-11]` **TTS stack extracted to its own repo (`tts-stack`) + eshpfi stood down on TTS dev.** Operator: hand all TTS tuning/dev to a separate agent with a self-contained repo (knowledge + infra access + a live knowledge list), and move the voice corpus in. New repo `~/development/tts-stack` (commit `9ee3288`) carries: dots-tts stack (canonical intent), `voices/` corpus (MOVED out of eshpfi), `KNOWLEDGE.md` (engine landscape + prosody findings + foot-guns), `docs/infrastructure.md` (irv-ml1 access + gated deploy runbook + rollback), CLAUDE/persistent-memory/ROADMAP, `tools/` (pause-probe + Booth render). Followed the **chatterbox-fast precedent**: eshpfi `stacks/dots-tts/` reduced to a POINTER README; the ~15 experimental TTS compose wrappers stay here as reference (catalogued in tts-stack KNOWLEDGE). Blast-radius check: no eshpfi playbook/script reads the canonical corpus (other `voices/` refs = unrelated host paths). **Reverses** the earlier "Corpus home = eshpfi `voices/` (keep-here)" call. ⚠ tts-stack is LOCAL-ONLY until pushed — needs a gitea remote (`vh/tts-stack`) + push before the separate agent can clone (operator's call — outward-facing + repo-create creds).
_Archived 2026-08-27._

- `[2026-08-10]` **dots-tts v3 — clause-break → period pause mapping.** Operator: v2 "sounds good" but donut won't pause at semicolons/dashes. ROOT CAUSE (measured via a pause-probe A/B — synth duration over N runs, non-determinism averaged out): dots' prosody honors a real pause **only for ellipsis (~+0.43s) and period (~+0.3s, capitalization-independent)**; comma/semicolon/colon/dash all run **flat (~+0.03s vs no-punct)**. Two distinct sub-causes: **dashes regressed in v2** (the `—`→`-` fold made em-dashes read as word-joiners), while **semicolons were NEVER a v2 change** — dots ignores them natively, only newly noticeable because v2 made everything else clean. Operator call: ellipsis "too much" → **map `;`, clause `:`, and em-dash `—` → period** in `_sanitize` (believable ~0.3s clause break). GUARDS (pinned by 11 unit tests, `stacks/dots-tts/test_sanitize.py`): digit-guarded colon `(?<!\d)\s*:\s*(?!\d)` so times `3:45` / ratios `2:1` survive; en-dash `–`→hyphen KEPT (numeric-range `10–20` safety — em-dash breaks, en-dash ranges, different jobs); genuine ellipsis left at full strength (author meant a long pause). Gated deploy (redeploy2 pattern → v3): build → throwaway :8199 test container + **pause-gate** (semicolon sentence must run ≥0.12s longer than baseline; measured **+0.427s**) → only then cut live over. LIVE + healthy `local/dots-tts:v3` on :8198. **rollback = `sed -i 's/^DOTS_TAG=.*/DOTS_TAG=v2/' .env + docker compose up -d dots-tts`** (v2 image retained). Booth `dots-pauses` (A=old-flat / C=ellipsis-too-much / D=live-v3). [[reference_chatterbox_fast_repo]]
_Archived 2026-08-27._

- `[2026-08-10]` **dots-tts v2 — contraction fix (curly-sanitize) + sentence-chunking + dependency-pin recovery.** Operator: donut read contractions wrong ("you're"→"you ree", "donut's"→"donut ess"). ROOT CAUSE (isolated via A/B booth): **curly/typographic apostrophes** (`’` U+2019 from ratatoskr's LLM) — dots' tokenizer mispronounces them; STRAIGHT apostrophes read clean under `normalize_text=True`. FIX (`app.py`): fold curly→ASCII (`str.maketrans`) before synth, **KEEP `normalize_text=True`** (operator call — retains number/date expansion). Also added **server-side sentence-chunking** (pack ≤280 chars): dots caps one `generate()` at ~500 patches/~40s, so long RP turns (the Zev monologue = 160s audio) truncated; chunking stitches them (verified full 160.3s, not 40s-cut). **⚠ BUILD FOOT-GUNS (both bit this redeploy):** (1) upstream dots.tts `constraints/recommended.txt` now pins **`gradio==6.17.0` — phantom, not on PyPI** → fresh `pip install dots.tts` unsatisfiable; FIX = pin `dots.tts==0.2.1` + **DROP** the `-c recommended.txt` constraints (0.2.1 pulls working gradio 6.17.3). (2) pinning only `torch==2.8.0` let **torchaudio float to 2.11.0 → dots.tts refuses to load** (minor-version match check); FIX = pin `torchaudio==2.8.0`. **⚠ DEPLOY LESSON:** `docker compose up -d` to a new tag swaps the LIVE container BEFORE any health check — a broken image crash-loops production (**ratatoskr TTS down ~1-2min this session**). NEW PATTERN = build → test in a THROWAWAY container on an alt port (:8199) → health+verify → only THEN cut live over (redeploy2.sh). v2 LIVE + healthy on irv-ml1:8198, **CONSUMER-CONFIRMED clean** (ratatoskr verified end-to-end on their :8765 — apostrophe string reads clean, /api/tts 200 @ 48kHz, no client change; the ~1-2min blip didn't hit them, their concurrent auto-audio issue was client-side localStorage). **rollback = `sed DOTS_TAG=v1 + docker compose up -d dots-tts`** (v1 image retained). Also: deployed container GPU crept ~6→13.9GB over 8h serving (cache accumulation; a redeploy resets it — watch item). [[reference_chatterbox_fast_repo]]
_Archived 2026-08-27._


- `[2026-08-07]` **Personal-Worldtree kb-contamination incident (WT #394) diagnosed; attribution CLOSED UNRESOLVED.** A reconcile `WingStore._embed` full-tree walk (kb `fs_root=KB_PATH` root, sibling wings nested) swept 5,354 fiction+main rows into personal's `knowledge_base` (2 superseded generations served as current). Fixed by WT #394 (aca39a1, kb walks exclude sibling wings; ships b182). Trigger un-attributable — peer reconcile via the SHARED infra-ops identity + 0 dockerd exec-logging = fingerprint-less. Durable finding → auto-memory `infra_ops_shared_identity_attribution_gap`, PARKED (operator ruled A) into [[project_migrate_infra_access_to_claude_credentials]]. Evidence hold on the 5,354 rows until operator sequences cleanup (w/ Brokkr, on #394's agenda).
  _Archived 2026-08-22._

- `[2026-05-12]` corviduo-dev (Worldtree-team dev VM, 10.250.50.152,
  CT 106 on pfi-pve) added to `servers/` inventory. Treat like SF
  client hosts: PFI hosts + provides emergency-ops backstop;
  Worldtree team owns OS config + deploys + backup decisions.
  _Archived 2026-05-27._

- `[2026-05-12]` Worldtree `:latest` tag drift bug — fixed by
  health-gated `:latest` advance in vh/worldtree's deploy workflow
  (architect commit `8ef3801`): only tag `:latest` AFTER the new
  container's `/health` probe passes. Build-on-host stacks here
  don't have this problem because the playbook always builds the
  SHA-tagged image from a `git reset --hard <ref>` checkout.
  _Archived 2026-05-27._

- `[2026-05-12]` asset-engine stack scaffolded LAN-direct at
  `http://10.250.50.70:8200`. Initially included Traefik labels for
  public hostname; user pulled them out (internal tool, no public
  TLS surface needed). Pattern: internal tools default LAN-direct;
  Traefik wiring only when external/TLS required.
  _Archived 2026-05-27._

- `[2026-05-12]` asset-engine catalog gains `lifecycle: { stack,
  vram_gb, gpu_device_id }` per irv-ml1 service for the
  orchestrator feature. SSH keypair scaffolded at
  `ana-docker:/opt/docker/conf/asset-engine/ssh/` for
  asset-engine container → irv-ml1 orchestration via dedicated
  ed25519 key.
  _Archived 2026-05-27._

- `[2026-05-13]` `pull-hf-repo.yaml` is the canonical HF-fetch
  playbook on ana-ml2. Supports `--var repo_type=model|dataset|space`.
  Replaces ad-hoc `huggingface_hub.snapshot_download` calls.
  _Archived 2026-05-27._

- `[2026-05-13]` Selene-1-Mini-Llama-3.1-8B added to llama-swap as
  judge model. mradermacher `i1-Q6_K` imatrix quant (~6.5GB). AtlaAI
  reward/eval model — temp 0.01, ctx 32K, q8_0 KV cache. New JUDGE /
  EVAL MODELS section in `stacks/llama-swap/conf/config.yaml`.
  _Archived 2026-05-27._

- `[2026-05-13]` `vllm-qwen3` → `vllm` stack rename. Added
  `vllm-reward` service (Skywork-Reward-V2-Llama-3.1-8B-AWQ classifier).
  Three vLLM services share GPU 1 (embed 0.20, rerank 0.20, reward 0.30
  utilization; 30% headroom). All use `--runner pooling`; classification
  drives via model's `architectures: [LlamaForSequenceClassification]`
  in config.json, NOT `--task classify` (deprecated in vLLM 0.19.1).
  _Archived 2026-05-27._

- `[2026-05-13]` /tend-docs first pass deletions: `stacks/infinity/`
  removed (retired by vllm). Archived
  `docs/asset-engine/design-brief.md` → `docs/archive/asset-engine/`
  with archival header. Fixed `pfi-pve` VM list to full
  `qm list` enumeration. Dropped stale weak-password section from
  `pfi-postgres` (rotation done 2026-04-23).
  _Archived 2026-05-27._

- `[2026-05-14]` althing-chamber stack scaffolded: chamber + forseti.
  Internal LAN-only at port 7881 (chamber default 7878 collides with
  task-board). Two-service compose, shared SQLite bind-mount,
  build-on-host pattern via vh/althing's gitea-workflow. Forseti is the
  canonical dev for this stack (galdrabok is on a different project).
  _Archived 2026-05-31._

- `[2026-05-16]` althing-chamber Phase 2: added `althing-agent-runner`
  as third compose service (worldtree-driver agent dispatcher).
  All three althing services use the same image; `command:` selects
  entrypoint. Safe to enable preemptively (sleeps when no
  driver=worldtree handles declared).
  _Archived 2026-05-31._

- `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8
  alpine as a sibling compose service in `stacks/althing-chamber/`,
  redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` /
  `msg_start` / `msg_complete` event kinds. DB bridge keeps
  `msg_curated` + `floor_grant` (structured / canonical). Two-channel
  architecture, no overlap. chamber + agent-runner `depends_on:
  valkey: service_healthy`.
  _Archived 2026-05-31._

- `[2026-05-17]` Worldtree admin workflow shift (per vh): infra-ops
  gets its own permanent admin-tier key (`61419c92`, stored at
  `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`).
  Future admin ops route through this key, not the bootstrap admin
  via docker-as-root.
  _Archived 2026-05-31._

- `[2026-05-17]` Worldtree env-var addition checklist: anytime
  introducing `os.environ.get("FOO")` in worldtree code, update BOTH
  `.env.example` AND `compose.yaml`'s `&worldtree-env` anchor in the
  same PR. Same Z_AI_API_KEY-shape footgun bit
  `BIFROST_CLIENT_ALLOWED_HOSTS` (#170) until worldtree-dev added the
  passthrough line in `08f02b2`.
  _Archived 2026-05-31._

- `[2026-05-18]` Volva systemd install complete after three-stage
  debug. Final unit at `/etc/systemd/system/volva.service` runs as
  `User=lkraven` with `ProtectHome=read-only` +
  `ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex`
  carve-outs for state writes. `VOLVA_ALTHING_CLI=/home/lkraven/
  .local/bin/althing-cli` + `ALTHING_HANDLE=volva` both pinned in
  env.sh.
  _Archived 2026-05-31._

- `[2026-05-19]` Worldtree CD disk-hygiene strategy: watermark gate
  (env-tunable threshold + window, fail-loud on still-low post-prune)
  + eager post-deploy prune (only after `:latest` advance succeeds,
  uses `docker image prune -a --filter "until=24h"` which respects
  in-use semantic — protects pinned + personal images automatically).
  Combined: demo VM holds ~24h of deploy history instead of unbounded
  accumulation. Shipped in vh/Worldtree PR #184 (`306cd61` +
  `613dac2` + `bd91df5`).
  _Archived 2026-05-31._

- `[2026-05-19]` Skaldsong CD shape: shape (1) of three operator
  options — container + Gitea registry + pull-restart, matching
  Worldtree's pattern. Target host ana-docker (NOT nh3-dev where
  skaldsong-dev runs for iteration). SHA-pin only for now; health-gated
  `:latest` advance is a follow-up once `/health` exercises Worldtree
  + Kokoro reachability.
  _Archived 2026-05-31._

- `[2026-05-19]` Skaldsong prod (ana-docker) switched from demo
  Worldtree (:8080) to personal (:8081). Same `user_id=skaldsong`
  as the nh3-dev hand-launch key — shared Heimdall agent slot
  (`skaldsong:wizard-v2`), different `key_id`s for independent
  rotation. Demo Worldtree stays for isolation; personal becomes
  the multi-consumer dev iteration instance.
  _Archived 2026-05-31._

- `[2026-05-19]` mead-hall Bifrost v0.3 end-to-end smoke green.
  Closed task #32 (althing thread `01KRV1M2KW6N6HBEXGTH72QXCA`).
  Wire layer (handshake + binding + dispatch) + data-flow
  (per-dispatch JWT claims → `ctx.session_id` populated → real
  session-scoped data) + agent-loop (LLM reads + quotes back) all
  proven. Resolves the "stalled mid-Worldtree" state from the
  2026-05-17 snapshot.
  _Archived 2026-05-31._

- `[2026-05-25]` v0.25.3 lofn tuning: `temperature 0.6 → 1.0` +
  `repetition_penalty 1.0 → 1.15` on default+fast profiles.
  Heretic-abliterated qwen3.6 was locking into degenerate attractors
  at the model's thinking-mode floor (0.6). Pattern:
  abliterated/uncensored Qwen variants need higher temp + non-trivial
  rep-penalty than base, NOT the model-card's documented floors.
  _Archived 2026-05-31._

- `[2026-05-25]` Worldtree #205 v0.25.2 ships `/app/config/` as
  bind-mount + root-then-drop entrypoint shim (gosu). Operators
  get persistent per-instance config without container-rebuild.
  Same bind-mount pattern hit twice subsequently in v0.27.0 (selene)
  and v0.29.9 (echo) — bind-mount shadows image-baked defaults, so
  every new required key surfaces as a crash-loop on existing
  deployments. The v0.29.12 canonical example files close this loop.
  _Archived 2026-05-31._

- `[2026-05-26]` Worldtree v0.27.0/v0.27.1 fixes Tier 3 GET
  visibility. `available_agents()` helper was over-applied to
  `GET /agents/<id>`, masking ALL Tier 3 agents regardless of row
  state. Bug only visible as "agent not found" via GET; storage was
  fine (silent-2xx PATCHes had persisted correctly). v0.27.1 added
  fail-fast hardening for the startup pre-resolve fragility class.
  _Archived 2026-05-31._

- `[2026-05-26]` Skaldsong v0.30.7 defensive 409→PATCH fallback.
  v0.30.6's GET-then-define-or-patch path crash-looped against
  pre-v0.27.0 Worldtree's GET-visibility bug (GET 404 phantom →
  define 409 conflict). v0.30.7 catches the 409 and falls through
  to PATCH (which silently 2xx'd on the pre-fix Worldtree).
  _Archived 2026-05-31._

- `[2026-05-27]` Worldtree v0.29.x landed full saga→echo refactor +
  config-validator hardening (v0.29.10 create_provider family-before-regex;
  v0.29.11 collect-then-raise echo startup validators; v0.29.12 ships
  providers.yaml.example/defaults.yaml.example canonical configs; v0.29.13
  reasoning_content extraction + catalog family lookup). Operator-asked,
  worldtree-dev-shipped, CI-deployed.
  _Archived 2026-06-01._

- `[2026-05-27]` artemis-31b-v1i added to llama-swap + worldtree personal.
  BeaverAI Gemma 4 31B Q6_K (~28.6GB), 128K ctx, `--reasoning-format deepseek`
  (gemma format unsupported in deployed llama.cpp). Worldtree catalog
  `family: gemma` so GemmaProvider routes reasoning tokens.
  _Archived 2026-06-01._

- `[2026-05-27]` Skaldsong streaming TTS v0.32.0→v0.32.2: chunked-batch SSE
  (one Kokoro POST per paragraph); defensive event_stream catch-all; NDJSON
  parsing for Kokoro /dev/captioned_speech multi-line responses.
  _Archived 2026-06-01._

- `[2026-05-31]` Dia2 deployed as **two fixed-model instances**
  (`dia2-2b` :8200, `dia2-1b` :8202) from `local/dia:v2`, retiring
  legacy Dia 1.6B; catalog `dia` entry **removed → dia2-2b + dia2-1b**
  (breaking for asset-engine). Rationale: the devnen wrapper is
  single-model and IGNORES the OpenAI `model` field (verified on its live
  OpenAPI), so the only way to offer both Dia2 models as real per-request
  asset-engine choices is one fixed endpoint per model. `3139e81`
  (deploy), `db15638` (catalog swap).
  _Archived 2026-06-03._

- `[2026-05-31]` Both dia2 catalog entries route to the wrapper's richer
  **`/tts`** endpoint (not `/v1/audio/speech`) to expose the full control
  surface (cfg_scale/temperature/top_p/cfg_filter_top_k/voice_mode/clone);
  all defaults sourced from the wrapper's `CustomTTSRequest` Pydantic
  blessed values. Voice default is `voice_mode: clone` +
  `clone_reference_filename: Abigail.wav` so a stable (non-random-gender)
  voice is the out-of-box behavior. `55602b7`, `5c47843`.
  _Archived 2026-06-03._

- `[2026-05-31]` Zonos REST adapter (`stacks/zonos/adapter/`,
  `local/zonos-api`) — thin OpenAI-ish `/v1/audio/speech` FastAPI in front
  of the Gradio-only Zonos SDK; JSON-envelope `{audio, audio_format,
  seed}` (Zonos is the fleet's first seedable TTS). Port 8203 (moved off
  8201 — collided with csm). Built; NOT deployed (stack down for VRAM).
  Also fixed the upstream image's missing CMD (`71df6f7`). `81efa8d`.
  _Archived 2026-06-03._

- `[2026-05-31]` Catalog schema regenerated: added `CatalogLifecycle` +
  `reproducibility.seed_field` (`b7b2130`). Resolves the stale-schema
  hand-off; catalog now validates clean. (asset_engine `catalog.py`
  Pydantic already supported both — schema file was just behind.)
  _Archived 2026-06-03._

- `[2026-05-31]` TTS bench expanded with `stacks/{dia,zonos,csm}`
  (`666f7f3` dia+zonos, `a4b8c2a` csm). The bench already had Fish
  S2-Pro / Chatterbox-Turbo / IndexTTS-2 / CosyVoice3 / Kokoro /
  VibeVoice / Qwen3-TTS / Kyutai. (csm since removed 2026-06-01.)
  _Archived 2026-06-03._

- `[2026-05-31]` Remote browser/iPad/Vision-Pro driver seat for the
  agent-fleet zellij session `Claude` stood up on nh3-dev (ttyd behind
  Caddy, network-gated). Out of this repo — full architecture + the
  HTTP2/OSC52/Safari-auth gotchas in auto-memory `reference_ttyd_fleet_seat`.
  _Archived 2026-06-03._

- `[2026-05-30]` esh-docker-vm NFS boot-ordering fix:
  `playbooks/fix-esh-nfs-boot-ordering.yaml` (`c0458d9`, + `53157b1`
  drop-in filename-collision fix) adds `_netdev,nofail` to the four
  10.0.50.50 NFS mounts + a docker `After=remote-fs.target` drop-in —
  resolves paperless `Exited(255)` on reboot. traefik also gained
  `restart: unless-stopped`. Full incident → auto-memory
  `incident_esh_docker_nfs_boot_race`.
  _Archived 2026-06-03._

- `[2026-06-01]` **Fish cloning VERIFIED competent (ECAPA-TDNN)** — retracting
  the earlier "weak cloner" call. Isolated test: Imogen-referenced clone ~0.79
  cosine to the real `Imogen.wav` vs ~0.10 for the no-reference default;
  transcript condition (correct 0.787 / bogus 0.778 / empty 0.738) barely moves
  identity (affects pronunciation, not timbre). Root cause of "Imogen sounds
  nothing like British" = the `"undefined"` select bug feeding Fish its default
  voice, NOT Fish. So the entire Fish-Imogen saga was the undefined bug; no
  Fish-side fix needed. (Methodology lessons → Tried and abandoned.)
  _Archived 2026-06-04._

- `[2026-06-01]` **CSM (Sesame csm-1b) torn down entirely** — removed from
  catalog, `stacks/csm/`, `playbooks/deploy-csm.yaml`, and host
  (`c54ab13`). Two reasons: (1) deep-research verdict — the acclaimed
  Maya/Miles demo is a fine-tuned, larger variant Sesame never open-sourced;
  the open 1B base ships no usable voices, can't generate text, English-only,
  no real-time streaming, absent from leaderboards, dominated by
  Kokoro/Dia2/Fish/IndexTTS. (2) The upstream image is rotted (see Tried and
  abandoned). asset-engine re-synced (`306aae9`).
  _Archived 2026-06-04._

- `[2026-06-01]` **Chatterbox catalog v1→v2** (`93cedd6`) — re-point to the
  devnen wrapper's `/tts` (was the thin OpenAI `/v1/audio/speech` exposing
  none of the emotion knobs → poor output). Default `exaggeration: 0.5` per
  Resemble's README ("works well for most"); the server SHIPS demo-tuned 1.3
  which over-acts — deliberately NOT adopted. 28 predefined voices via
  `/get_predefined_voices` (default Emily.wav). Corrected seedable false→true.
  _Archived 2026-06-04._

- `[2026-06-01]` **Fish-S2 catalog v1→v6** (`f3c260e`..`38d9e3b`) — was a
  blank 2-field form; now exposes the full fish-speech ServeTTSRequest surface
  + a 32-voice `reference_id` "Voice" dropdown (STATIC list — Fish has no
  /voices API). `reference_id`-by-name works ONLY when each voice is staged as
  a `references/<name>/` dir with `<name>.wav` + `<name>.lab` — was silently a
  no-op until that layout fix on 2026-06-01 (see the two-root-causes entry
  above); the inline-path-as-filesystem-path `references` shape 500s (see Tried
  and abandoned). Corrected vram
  4→24 GB, cold_start 8→240s, seedable false→true. Added 3 consenting
  British-female VCTK voices (Imogen=p225/Eleanor=p228/Beatrice=p229).
  _Archived 2026-06-04._

- `[2026-06-01]` **`blendable: false` catalog flag — DEFERRED**, pending
  asset-engine-dev honoring it in the form template. Clone-voice selects
  (chatterbox/fish/dia2) must render as plain dropdowns, not the Kokoro blend
  widget (blending reference clones is nonsensical). I ship the catalog flag +
  schema field once they read it. Tracked: althing msg
  `01KT2K2SY9N7AY69R9V0B4RXSW` to asset-engine-dev.
  _Archived 2026-06-04._

- `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — Populated
  `references/<name>/<name>.wav`+`.lab` for all 32 voices; playbook gained
  normalize-step + A/B smoke gate. glados got a real transcript (ASR'd via
  Parakeet): the Portal "Welcome to test chamber 4" lines.
  _Archived 2026-06-11._

- `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast`
  streaming container.** Workload = single-stream interactive. GPU placement:
  3090 (device 0) if it fits else A6000 (device 1). Cutover: parallel catalog
  entry, burn in beside live `chatterbox`, then flip. Streaming = adaptive
  buffer-ratchet chunking (native frame-streaming abandoned). Since extracted to
  its own repo `vh/chatterbox-fast` (auto-memory `reference_chatterbox_fast_repo`).
  _Archived 2026-06-11._

- `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** Claimed
  naive sentence-level streaming has "zero quality loss" — WRONG. The T3 AR backbone
  conditions prosody on the WHOLE text; splitting loses cross-sentence prosodic
  context even though voice timbre stays (reference-conditioned). No *artifacts* ≠
  no *quality loss*. Hence adaptive-chunk design (maximize context per chunk subject
  to latency budget), not fixed per-sentence splits.
  _Archived 2026-06-11._

- `[2026-06-03]` **Declined worldtree v0.30.4 staging deploy** — that's worldtree-team's
  CI/CD lane (a developer `staging/vX.Y.Z` git-tag promote), not infra-ops. They self-corrected
  to the same conclusion independently.
  _Archived 2026-06-13._

- `[2026-06-03]` **R14 scope = (a) provision-only.** infra-ops provides box + CUDA env +
  engines + weights + NFS; brokkr/dev wires `arms.py` + runs — keeps infra-ops OFF the
  VIVAE-processing path (VIVAE = Variably Intense Vocalizations of Affect/Emotion, CHARTER §4
  highest-liability; operator authorized R&D-eval-only, quarantined). Box = irv-ml1 (A6000
  free; ana-ml2 GPU-saturated). Per-engine venvs (divergent torch stacks); A6000 = `cuda:0`
  NATIVE (≠ docker `=1`).
  _Archived 2026-06-13._

- `[2026-06-03]` **yt-voice-clipper push-to-deploy via gitea webhook** (operator-directed,
  after 6 manual rebuilds in ~40 min). Webhook (not poll) — gitea CAN reach the WG IP per the
  operator. The proxy env + Homepage labels live in the **host-specific override** (untracked
  → survive the auto-deploy's `git reset --hard`), NOT yt-voice-clipper-dev's image. Runbook
  `d4f180d`.
  _Archived 2026-06-13._

- `[2026-06-03]` **yt-voice-clipper bot-gate fix = route yt-dlp through NH3 residential
  egress, NOT cookies/PO-token.** YouTube hard-flags the Irvine colo IP (LOGIN_REQUIRED on a
  public video even with no cookies). Cookies + the bgutil PO-token + deno JS-runtime all
  loaded fine — the gate is pure IP reputation. Operator chose proxy-via-nh3-dev → durable
  dante proxy → proven. The egress proxy is a reusable fleet lever for any datacenter-IP-gated
  service.
  _Archived 2026-06-13._

- `[2026-06-04]` **`brokkr` user (no-sudo) on irv-ml1; R14/R15/R16 substrate moved to /home/brokkr.** Persistent box services there need SYSTEM systemd units (see Tried).
  _Archived 2026-06-14._

- `[2026-06-04]` **ollama upgraded 0.9.0→0.30.4 on irv-ml1** (Ministral-3 is a Dec-2025 model the old engine refused); A6000 pinned by **UUID** not index (native fastest-first ≠ nvidia-smi PCI).
  _Archived 2026-06-14._

- `[2026-06-04]` **Worldtree demo/pinned/personal deploys are ALL CI/CD, not infra-ops** — a "deploy vX.Y.Z" request to infra-ops is MISROUTED → point them back to their pipeline. (`d8d776c`, auto-memory `reference_worldtree_deploys_cicd`)
  _Archived 2026-06-14._

- `[2026-06-04]` **infra-ops NOPASSWD-sudo identity commissioned, scoped to PFI boxes** (+esh-docker-vm by operator override) — so infra-ops completes DevOps end-to-end vs handing the operator sudo steps. Dedicated key, sudo log_output, key-gated. (`8c32a05`, auto-memory `reference_infra_ops_sudo_identity`)
  _Archived 2026-06-14._

- `[2026-06-04]` **phi4 ships the CANONICAL/official Phi-4 chat template, NOT Ollama's** (Ollama's omits the system `<|end|>`). Applied an Ollama-matching override then reverted — ship correct, not the benchmark quirk. (`90e08f0`→`27eb537`; "headgun" lesson in Tried.)
  _Archived 2026-06-14._

- `[2026-06-04]` **phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer; granite-4-small retired** from llama-swap. 50K ctx + FP8 KV. (`40a374b`) [superseded by the 2026-06-05 granite-4.1-8b decision]
  _Archived 2026-06-14._

- `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons; Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid `12aa5a9`); volva.service was crash-looping 203/EXEC. (`6e2f80e`)
  _Archived 2026-06-16._

- `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer.** Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not Q4/AWQ** — primary workload is high-concurrency, where FP8 scales ~linearly (2010 tok/s @ C=32). vLLM `vllm-granite` :8004 GPU 1, official IBM compressed-tensors FP8. (Then on Ada; box has since gone Blackwell.) (`34a43a0`, auto-memory `reference_ana_ml2_vllm_granite`)
  _Archived 2026-06-16._

- `[2026-06-05]` **Langfuse v3 on ana-docker (:3001) as the gateway trace UI**; LiteLLM `success_callback:[langfuse]` live. Pretty traces + tok/s dashboard. NOT a prerequisite (spend_logs already capture tokens+latency). (`9171e6a`)
  _Archived 2026-06-16._

- `[2026-06-05]` **Ollama BANNED fleet-wide** (operator directive) — never stand one up; tear down any found; serve via llama-swap or vLLM. (auto-memory `feedback_avoid_ollama`)
  _Archived 2026-06-16._

- `[2026-06-05]` **ComfyUI / FLUX.2 work split to `~/development/comfy-dev`** (dedicated repo + agent). eshpfi keeps the `comfyui`/`arbo` stack compose; comfy-dev owns the model/workflow knowledge. (auto-memory `reference_irv_ml1_ampere_quant`)
  _Archived 2026-06-16._

- `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is the structured-output profile, ON HOLD, no live consumer). Bind-mount CONFIGS (providers.yaml, vh-owned) ARE infra-ops's to apply directly — only the app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`)
  _Archived 2026-06-16._

- `[2026-06-13]` **Arbo prod bearer auth turned OFF — WireGuard is the access boundary** (operator decision; reverses ADR-0001's "closed the open-auth hole"). ENGINE_TOKEN must be **ABSENT, not empty** (empty-string still gates) — removed from BOTH the host `.env` AND the compose `environment:` injection line. Original token backed up at `irv-ml1:/opt/docker/compose/arbo/.env.pre-auth-off.bak`; comfy-dev updated their ADR-0001. Tracking: `db97899` + `playbooks/arbo-disable-engine-token.yaml`.
  _Archived 2026-06-16._

- `[2026-06-13]` **Storetank image-models archive DECOMMISSIONED; arbo is the single live ComfyUI model tree (502 G).** Curated `/storetank/image-models/comfy` (was 919 G, the native `/opt/ComfyUI/models` symlink target) → killed everything superseded by arbo's current gen (Hunyuan, WAN2.1, FLUX.1, Chroma, SD3.5, orphaned umt5+llava ≈ 739 G) + migrated the keepers (gen-agnostic utilities + the SDXL/Pony stack, 177 G) into `/storetank/arbo/models` (same-fs move, skip-existing protects prod). Tracking: `docs/storetank-image-models-archive.md` + `docs/arbo-comfyui-model-catalog.md` (`1902425`→`5007ec1`).
  _Archived 2026-06-16._

- `[2026-06-13]` **GRANITE_KEY provisioned to comfy-dev's nh3-dev dev env** at `~/.arbo_granite_key` (0600) for the hero gen+judge script — verbatim copy of the prod `arbo-prompt-enhance` vkey (now extended to reach BOTH `granite-4.1-8b` AND `qwen3.5-9b-fp8`); nothing minted. The vkey README's "granite-only" wording was stale → corrected (`f32c6dd`).
  _Archived 2026-06-16._

- `[2026-06-13]` **ana-ml2 upgraded Ada → dual RTX PRO 6000 Blackwell Max-Q** (96 GB each, cc 12.0 / sm_120; was dual RTX 6000 Ada 48 GB / cc 8.9 — confirmed live via `nvidia-smi`). Unlocks NVFP4 (FP4 tensor cores) and doubles VRAM headroom. CLAUDE.md GPU-spec doc-fix **LANDED `355a240`** (operator). Tracking: `19a07b9`/`1e2a3a1` ("Blackwell 96GB").
  _Archived 2026-06-16._

- `[2026-06-13]` **NVFP4-W4A4 is infeasible for Granite — FP8 stays the Granite-on-Blackwell format.** W4A4 collapses at 30k context, proven **producer-independent** (modelopt AND llm-compressor both clean-NONE from the same BF16 base + wikitext-2k calib). No 4-bit wins both axes: W4A4 = quality collapse; W4A16-NVFP4/AWQ = weight-only dequant → bf16 (no FP4-core speedup). **30B retired**. (auto-memory `reference_nvfp4_w4a4_granite_infeasible`)
  _Archived 2026-06-16._

- `[2026-06-13]` **Qwen3.5-9B VL (FP8) deployed on ana-ml2 GPU 1** — `qwen35-vl` stack, :8007, gateway alias `qwen3.5-9b-fp8`. **Pinned nightly digest, not `:latest`**: the stable release quantizes the VL *vision tower* under `--quantization fp8` → garbage vision (LM fine); the nightly correctly excludes it. Re-pin + drop the pin once that exclusion lands stable. (`2e3dcc2`)
  _Archived 2026-06-16._

- `[2026-06-13]` **comfyui 325 G model tree migrated worktank → `/storetank/arbo`** (worktank 97% → 26%). `arbo` is the consuming app; overlay bind-mount via `COMFYUI_MODELS_DIR`. (`38186be`) (See the 2026-06-13 archive-decommission decision above — this tree later absorbed the storetank-archive keepers, reaching 502 G.)
  _Archived 2026-06-16._

- `[2026-06-13]` **GPU layout settled on the Blackwell box.** GPU 0 held free for large-model hot-loads (llama-swap pinned, `edf0f91`); GPU 1 steady-tenant — granite 131k ctx, qwen 65k, embed/rerank/reward trio, ~3.5 GB free (`1e2a3a1`, `19a07b9`; trio re-floored for 96 GB, 20×-parallel-stable). embed/rerank left at floor — long docs chunked BEFORE embedding. max-model-len is a ceiling not a reservation.
  _Archived 2026-06-16._

- `[2026-06-13]` **Prefix caching pinned explicit on granite + qwen** — benched ~6.5× faster TTFT on a shared ~4.5k-token summarizer template; soft/evictable, neutral when prefixes don't repeat. vLLM v1 defaults it ON (granite) but the qwen nightly defaults OFF — pin both. (`a9a2be7`)
  _Archived 2026-06-16._

- `[2026-06-13]` **granite-4.1-8b listed as the always-available summarizer/classifier + a shared all-agents key minted** (operator-directed). Global `~/.claude/CLAUDE.md` Global-tools entry; key alias `all-agents-local`, scoped to the FREE local models only (granite + qwen-vision + embed/rerank, NOT paid GLM), internal-gateway-only, rotatable. (auto-memory `reference_litellm_gateway`)
  _Archived 2026-06-16._

- `[2026-06-13]` **arbo engine + frontend stack stood up** (ADR-0001) — irv-ml1 co-located inference engine (`ee57e69`), python-based healthcheck (slim image, no curl/wget, `bdb3312`), frontend ro-mounted from the checkout (`922e8ad`, ADR-0001 D2).
  _Archived 2026-06-16._

- `[2026-06-11]` **GLM thinking inverted at the LiteLLM gateway** (operator call): `glm-5.1` defaults thinking-OFF; `glm-5.1-reasoning` = same z.ai upstream, thinking ON. Mechanism: `litellm_params.extra_body:{thinking:{type:disabled}}` — `drop_params` strips a top-level `thinking`/`reasoning_effort` but forwards `extra_body` verbatim to z.ai. Shared-gateway change. (`95b2701`, auto-memory `reference_litellm_gateway`)
  _Archived 2026-06-16._

- `[2026-06-11]` **pi coding agent installed on nh3-dev as a GLM 5.1 harness** — `@earendil-works/pi-coding-agent` via **bun** (npm's global prefix is `/usr` → needs sudo, bun avoids it). Config `~/.pi/agent/models.json`, launcher `~/.local/bin/glm`.
  _Archived 2026-06-16._

- `[2026-06-11]` **z.ai web-tools (regin) = z.ai hosted MCP path, NOT the `/paas/v4` Tool API.** WORKS: MCP streamable-HTTP at `https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp`, `Authorization: Bearer $ZAI_API_KEY` (the **MCP** key, distinct from `Z_AI_API_KEY` the LLM key). Reference impl = Worldtree's Leif agent.
  _Archived 2026-06-16._

- `[2026-06-10]` **Mac Pro migration framed: workstation-only** (M2 Ultra ARM, racked NH3 on-subnet); sidecars stay Linux. `migration-plan.md`. (See in-flight.)
  _Archived 2026-06-16._

- `[2026-06-10]` **Worldtree deployed-config propagation is infra-ops's OWNED lane** (operator ruling). worldtree-dev pings the config delta pre-merge; infra-ops syncs `config/*.yaml` from MERGED canonical to the `/opt/worldtree*/config` bind-mounts on demo+personal. The v0.33.8 9-HOUR demo outage (a `model_roles.yaml` startup-dep that never reached the VM) is the failure mode this prevents. providers.yaml stays hand-tuned. corviduo emergency-ops = `ssh vh@10.250.50.152`, docker no-sudo. (auto-memory `reference_worldtree_deploys_cicd`, `reference_corviduo_dev_emergency_ops`)
  _Archived 2026-06-16._

- `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr-smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev — granite, later extended to qwen-vision). Mint via `/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus. (auto-memory `reference_litellm_gateway`)
  _Archived 2026-06-16._

- `[2026-06-14]` **ana-ml2 GPU-1 vision upgraded: Qwen3.5-9B → Qwen3.6-35B-A3B (official FP8), served under its TRUE name only.** `qwen36-vl` replaces `qwen35-vl` on :8007 (`a0fed13`). The stale `qwen3.5-9b-fp8` name is KILLED at vLLM AND the litellm gateway (404/400) — a model is NEVER aliased under a prior model's name (silent substitution = downstream footgun; operator directive). Consumer comfy-dev/arbo migrated; arbo vkeys → all-proxy-models; shared `all-agents-local` key repointed qwen3.5-9b-fp8 → qwen3.6-35b-a3b. GPU-1 rebalanced for the ~34 GB FP8 weights (granite 0.35→0.24/64K; embed/rerank 0.05→0.03, reclaimed ~4 GB util-waste). Validated: vision correct, 20-concurrent = no OOM. (auto-memory `feedback_no_false_model_aliases`)
  _Archived 2026-06-18._

- `[2026-06-14]` **NVFP4 was the lighter fit (~21 GB) but is BLOCKED on vLLM — FP8 is the working vision path.** `nvidia/Qwen3.6-35B-A3B-NVFP4` won't load: the ModelOpt-NVFP4-MoE loader errors on expert/lm_head scale keys across 0.19.1 (`w2_input_scale`) AND 0.22.0 (`lm_head.input_scale`, vllm #44081) — a pattern across modelopt NVFP4 MoEs. Revisit NVFP4 (frees ~13 GB on GPU 1) once fixed; the 21 GB checkpoint stays cached on ana-ml2. **(SUPERSEDED 2026-06-16 — it loads on vLLM 0.23.0; qwen36 swapped to NVFP4. See the top of this section.)**
  _Archived 2026-06-18._

- `[2026-06-14]` **llama-swap qwen3.5-9b GPU-0 pin DROPPED; GPU 0 reserved for a creative-writing model (pick DEFERRED by operator).** Deep-research (this session) on big-fast-uncensored creative for a 96 GB Blackwell: **GLM-Steam-106B-A12B** (already in the llama-swap config — balanced default) vs **TheDrummer/Behemoth-X-123B-v2** (prose-tier, tops UGI writing+willingness) vs XORTRON-123B (max willingness, weak prose); GGUF-on-llama-swap is the serving path. Tracking: this session + llama-swap config (GLM-Steam present, `untracked by operator choice`).
  _Archived 2026-06-18._

- `[2026-06-14]` **R16 splice-pivot yield probe executed** (infra-ops ran the irv-ml1 inference for brokkr; brokkr owns design + analysis). See Current state. Tracking: althing thread `01KV010WGS…`, `gen_yield_probe.py` in `irv-ml1:~/r16-vmoan-harness`.
  _Archived 2026-06-18._

- `[2026-06-14]` **R16 vmoan inline-generation arc CLOSED — v1 at default decode (rep1.2/temp0.8) is the final Chatterbox-tag inline artifact.** Operator's ear rejected every alternative: v2/v3 windowing (omission vs coherence-loss), v4 multi-tag (cohesion held but lost to capacity-competition), emergent inline-token modulation (degenerates, not modulates), and the gen-time decode-polish sweep (soft tamers cut the NVV itself — same omission family as v2; p0 baseline beat p1). All adapters v1–v4 + `tokenizer.json.v3bak` preserved on `irv-ml1:~/r16-vmoan-harness`. Likely-next direction (deferred, NOT formalized): generate→bin→splice + one-shot-clone NVV pipeline routing around the inline-coherence wall. Tracking: brokkr R16 journal + althing thread `01KV010WGSSMPWRNCPAGSPK15Y`.
  _Archived 2026-06-18._

- `[2026-06-14]` **Arbo deploy pipeline fixed, hardened, and version-controlled.** Prod rebuilt v0.11.1 → **v0.11.6** backend; the webhook machinery (`arbo-deploy.sh` + `arbo-webhook.py`, :9009 HMAC listener) is now repo-tracked at `stacks/arbo/` (was host-only = recoverability foot-gun). Deploy reaches gitea via the INTERNAL route (`10.250.50.70:222`) and restarts the engine ONLY on `catalog/` changes (graphs/frontend per-request; warn on `src/`|`Dockerfile` only — pyproject/uv.lock churn every commit). Operator kept arbo stack ownership in **eshpfi** (not migrated to comfy-dev's repo). Secret + `.env` stay host-only. Tracking: `6d66bc2`, `6e58e57`, `stacks/arbo/README` Q5.
  _Archived 2026-06-18._

- `[2026-06-15]` **qwen36 thinking-split at the gateway** (operator) — base
  `qwen3.6-35b-a3b` = thinking-OFF default (the vision judge needs it), `…-a3b-thinking`
  = opt-in reasoning. One hybrid checkpoint + `--reasoning-parser qwen3`; mirrors the
  glm-5.1 split. The judge's enable_thinking=false is what makes the NVFP4 swap
  transparent. (`6de0844`)
  _Archived 2026-06-19._

- `[2026-06-15]` **arbo engine rebuilt v0.11.6 → v0.11.22 from main; catalog re-applied (v0.11.23) — curated `/workflows` footer live.** src/ schema changes (Model/Lora.summary, Workflow.blurb/keywords + the /workflows endpoint) need a MANUAL image rebuild (the webhook warns-only on `src/`/`Dockerfile`). Built from `/worktank/arbo/repo`, bumped `ARBO_IMAGE` in the host .env, recreated + verified healthy (auto-rollback armed, unneeded). comfy-dev's v0.11.20 catalog push had first crash-looped prod (old baked schema `extra=forbid` vs new git-mounted catalog fields → ValidationError); they reverted the catalog, so rebuild-from-main loaded clean, then they re-applied (v0.11.23). comfyui is a SEPARATE compose project — the rebuild didn't touch it.
  _Archived 2026-06-19._

- `[2026-06-15]` **althing-core refreshed to v0.14.1 box-wide on nh3-dev** (forseti + heid both routed it; central-sequenced per DevOps-delegation since `uv tool install --force` swaps the shared binary all handles use). Fixes the monitor non-truncating-lock bug so `stop-monitor`/`--stop` reliably recovers orphans. The editable install (`uv tool install --force --editable .`) covers every nh3-dev handle; `--version` trued up 0.14.0→0.14.1; verified `--stop` works against a real orphan.
  _Archived 2026-06-19._

- `[2026-06-15]` **ratatoskr Tier-3 affect live smoke GREEN** (consumer-confirmed). Brokered/minted consumer_id="ratatoskr" Heimdall HS256 key on personal Worldtree (the key IS BOTH the create-session Bearer AND ratatoskr's JWT verifier secret — plain utf-8 string, NOT base64/raw); injected ENV-ONLY to `~/.config/ratatoskr/provider.env` (mode 600) on nh3-dev; added `10.100.10.50:8390` to personal's `BIFROST_CLIENT_ALLOWED_HOSTS`; drove define→session (handshake 200)→turn→affect.emit (200, snapshot persisted). Worldtree signs Bifrost with the caller's raw Bearer verbatim. Drive commands + claims contract in althing thread `01KV4ZYSE3…`.
  _Archived 2026-06-19._

- `[2026-06-15]` **Dense Qwen3-VL-32B-NVFP4 A/B'd as arbo's anatomy hero-judge vs the qwen3.6-35b-a3b MoE → dense LOST, pilot torn down, GPU 0 reclaimed.** Dense NVFP4 loads clean on vLLM where the MoE NVFP4 doesn't (no expert-scale keys); ~44 tok/s on Blackwell GPU 0; vision tower stays high-precision. But comfy-dev's ground-truth A/B: MoE wins (faster + dual-use as general LLM + slight fine-detail edge), and the subtle-proportion tier is below ALL VLs. **Don't re-propose a dense-VL judge swap.** (auto-memory `project-qwen3vl-dense-pilot`)
  _Archived 2026-06-19._

- `[2026-06-15]` **MastMed prototype containerized + exposed publicly via cloudflared.** Self-contained static SPA (bun→caddy multi-stage build), basic-auth gated (operator's gating call), ana-docker:8089, cloudflared quick tunnel. See in-flight for the ephemeral-URL caveat + named-tunnel upgrade path.
  _Archived 2026-06-19._

- `[2026-06-15]` **infra-ops bootstrapped onto corviduo-dev** (operator-directed) — infra-ops now has SSH + NOPASSWD sudo there, OVERRIDING the fleet bootstrap script's deliberate partner-box exclusion (explicit host arg). corviduo config edits no longer need `vh` + a sudo password. (auto-memory `reference_corviduo_dev_emergency_ops`)
  _Archived 2026-06-19._

- `[2026-06-15]` **R18 clip+caption placement = ana-ml2** (operator: "ana-docker… ana-ml2 unless irv-ml1 has room"; irv-ml1 lacked free VRAM for the 30B-A3B captioner). Models staged + stub deployed; see in-flight + the hard real-voice gate.
  _Archived 2026-06-19._

- `[2026-06-14]` **LiteLLM infra-ops admin key provisioned** (operator) — resolves the LiteLLM half of the credential-migration directive; use it for ALL gateway ops (NOT `sk-corvid`). Value at `~/.config/litellm/infra-ops-key` (mode 600); gateway reachable directly from nh3-dev at 10.250.50.70:4000. (auto-memory `reference_litellm_infra_ops_key`)
  _Archived 2026-06-19._

- `[2026-06-16]` **litellm `strip_empty_tools` pre-call hook shipped** (`d1bea13`) — an empty `tools:[]` 500s vLLM ("tools must not be an empty array"); a global `litellm_settings.callbacks` CustomLogger pops it (+ orphaned `tool_choice`) before forwarding, so it covers EVERY vLLM model, not one. `drop_params` only drops unsupported PARAMS, not empty VALUES. Mounts beside config.yaml (litellm resolves callbacks relative to the config dir). Verified live across granite/mistral/stream. (`stacks/litellm/conf/strip_empty_tools.py`)
  _Archived 2026-06-20._

- `[2026-06-16]` **single-file `gateway-chat.html` playground shipped** (`984ca3d`, `tools/`) — zero-dep browser chat straight to the gateway (`:4000`, CORS open), system-prompt box, streaming SSE, renders `reasoning_content`, NEVER sends `tools`. Built because the LiteLLM admin-UI playground can't test vLLM-backed models (see Tried-and-abandoned). Serve on-request via `python3 -m http.server -d tools`.
  _Archived 2026-06-20._

- `[2026-06-16]` **claude-bot granted ADMIN on vh/arbo** (operator) — arbo CI/CD gitea ops now route through the service account, not the operator's personal `vh` (the credential-migration directive). Operator added claude-bot as Administrator collaborator; claude-bot self-serves via its `gitea-password` (basic-auth = FULL perms; its stored token is package-write-only, can't change repo settings). (auto-memory `reference_claude_bot_gitea_creds`)
  _Archived 2026-06-20._

- `[2026-06-16]` **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven.** The missing server-side step was the per-PORT `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist (`:8391`; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the `score_breakdown` shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), not `agent_self_enabled`. (tracking: c99aa49 + auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED:** the `score_breakdown`-shape root-cause was disproven; real cause = cold-recall `scope_filter` adds an `agent_self` axis → ratatoskr's conjunctive INV-005 match drops the `{end_user}`-only chunks → 0 hits. Now Worldtree #297. (auto-memory `project_worldtree_295_recall_rootcause`)
  _Archived 2026-06-20._

- `[2026-06-16]` **infra-ops = the DURABLE admin path on corviduo-dev** (operator granted standing admin; extends the 2026-06-15 bootstrap) — added a `Host corviduo-dev → User infra-ops` `~/.ssh/config` alias + repointed `servers/corviduo-dev/ssh-target` to `infra-ops@`; `ssh corviduo-dev` / `elway corviduo-dev` now land as infra-ops w/ NOPASSWD (vh = explicit-only human path). elway resolves via ssh-config, NOT the ssh-target file. (tracking: c99aa49 + auto-memory `reference_corviduo_dev_emergency_ops`)
  _Archived 2026-06-20._

- `[2026-06-16]` **demo + personal `character` model flipped qwen3.6-35-a3b → mistral-small-4** (operator) — in `model_roles.yaml` the FIRST `catalog_id` in a role's `binds:` is the default; reordered mistral-first (qwen retained in the switch-allowlist, non-destructive), applied via the PIN-SAFE api recreate (derive `WORLDTREE_IMAGE` from the matrix sibling — never a bare `compose up`, which falls to the crash-blocking stale `:latest`), verified a fresh character agent resolves model=mistral-small-4. (auto-memory `reference_corviduo_dev_emergency_ops`)
  _Archived 2026-06-20._

- `[2026-06-16]` **NVFP4 ModelOpt MoE now LOADS on vLLM 0.23.0 — SUPERSEDES the
  2026-06-14 "blocked" finding (#44081 fixed).** Confirmed the cached
  `nvidia/Qwen3.6-35B-A3B-NVFP4` loads clean + generates on 0.23.0; swapped prod
  qwen36 FP8→NVFP4 (~13 GB reclaimed on GPU 1, fp16 KV). Validated by brokkr (speed:
  faster single-stream, ≥parity batched) + comfy-dev (vision anatomy-judge A/B: PASS).
  One W4A4 tell: minor multi-step chained-reasoning slip — doesn't bite the vision
  judge. Mistral's NVFP4 is a SEPARATE compressed-tensors path (day-0), distinct from
  the ModelOpt one. (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`)
  _Archived 2026-06-20._

- `[2026-06-16]` **Mistral Small 4 = the GPU-0 tenant** (operator) — fills the slot
  the deferred creative-writing pick held, and supersedes llama-swap (now down).
  NVFP4 119B MoE, pinned **vLLM v0.22.0** for working vision (0.23.0 breaks it),
  256K context. Operator's creative-writing model; **abliteration planned** → then it
  becomes the uncensored-creative model the persona substrate needs. Reasoning via
  `reasoning_effort` (BINARY none|high — no medium). (tracking: `c77a9aa`/`9a49963`)
  _Archived 2026-06-20._

- `[2026-06-16]` **Selene judge restored on vLLM FP8, NOT NVFP4** — W4A4 is too
  aggressive for a precision judge validated at Q6_K; FP8 ≥ Q6_K fidelity, and Selene
  is text-only Llama-3.1 so dynamic `--quantization fp8` has no vision tower to noise.
  `selene-1-mini-8b` on GPU 1 :8011. Worldtree's Domari role repointed to it. (`c985ede`)
  _Archived 2026-06-20._

- `[2026-06-16]` **GPU-1 buffer put to work: selene 0.13→0.17 + qwen36 0.32→0.34**
  — the two KV-bound generative models that gained Worldtree consumers; granite + the
  pooling models under-use their util, so growing THEM is wasted. (`e124a2f`)
  _Archived 2026-06-20._

- `[2026-06-16]` **claude-bot Gitea service account stood up; corviduo-ORG conversion
  TABLED** (operator). claude-bot is the credential-migration foundation. The proper
  fix for scoped agent package-write is converting vh (a USER) → a `corviduo` ORG —
  blast radius assessed (org creation = free; arbo-first migration = cheap, no registry
  images; full fleet = incremental, ~8 active vh images / 50 repos) — but tabled for
  later. Registry-push deferred until then. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
  _Archived 2026-06-20._

- `[2026-06-16]` **arbo engine builds → comfy-dev OWNERSHIP** (operator approved) +
  Gitea Actions runner stood up on irv-ml1 (host-executor as lkraven). infra-ops keeps
  box-level escalation (runner, package:write PAT pending corviduo-org, webhook). See
  the runner entry in Current state. (tracking: this snapshot + arbo thread `01KV70AB2B4B…`)
  _Archived 2026-06-20._

- `[2026-06-16]` **Worldtree persona substrate migrated qwen→Mistral via capability-
  named profiles** (v0.35.8→v0.35.10; demo + personal). PRE-SYNC-FIRST is now the
  standing pattern for any providers.yaml/profile change: sync config to the bind-mount
  BEFORE the deploy image-swap restart, so new agents + new config land together (the
  v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread `01KV6X0M…`)
  _Archived 2026-06-20._

- `[2026-06-19]` **litellm task-aliases: `classifier`→granite-4.1-8b, `summarizer-large`→gen/qwopus** (`summarizer`→granite already existed). Also added then REMOVED `gen-nt`/`gen-reasoning-nt` (redundant with the global `strip_empty_tools` hook). (ef45f6d, 2e5ab72, 75bd4c3)
  _Archived 2026-06-21._

- `[2026-06-19]` **gateway-chat: model-smoking web chat → PERSISTENT container.** `/v1/models` auto-discovery + image-upload; `stacks/gateway-chat/` (nginx:alpine, ana-docker:8091, homepage tile). (4c40b9f, 740bcae)
  _Archived 2026-06-21._

- `[2026-06-19]` **pi on nh3-dev wired to `gen`** — `~/.pi/agent/models.json` + a `gen` launcher mirroring `glm`. Local box config, not a repo artifact. (untracked by operator choice)
  _Archived 2026-06-21._

- `[2026-06-19]` **`gen` model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8.** ana-ml2 GPU 0, served-name `qwen3.5-122-a10b`. STABLE vLLM + fp8 KV reaches 262144 — nightly+turboquant-4bit-KV was UNNECESSARY. 92.7 tok/s warm. (20e796c, 5b06514)
  _Archived 2026-06-21._

- `[2026-06-19]` **TP=2 across the two ana-ml2 Blackwells REJECTED** — `PIX` (PCIe, NO NVLink) → all-reduce-bound → SLOWER. One-model-per-card is optimal; PP=2 only if a >96 GB model is ever wanted. (untracked by operator choice)
  _Archived 2026-06-21._

- `[2026-06-19]` **qwen-image-bench (T2I judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1**, aliased `image-judge`. (bfae924, 5dfce04)
  _Archived 2026-06-21._

- `[2026-06-18]` **heretic abliterated Mistral Small 4 NVFP4 built + LIVE as `mistral-small-4`** (in-house NVFP4, drop-in stack, A/B'd vs official, "heretic stays"). DISPLACED from GPU 0 on 2026-06-19 for the gen/Qwopus swap. (dd3a5c9, f566f61, `tools/mistral-small4-nvfp4/`)
  _Archived 2026-06-21._

- `[2026-06-18]` **irv-ml1 VRAM consolidation + comfy-dev torch-pin** — ComfyUI pinned to the A6000 (48 GB), audio zoo → 3090; `DISABLE_UPGRADES=true` @ torch 2.12.1. (a8550ad)
  _Archived 2026-06-21._

- `[2026-06-18]` **ComfyUI acceleration set (9 nodes) installed for comfy-dev** on irv-ml1.
  _Archived 2026-06-21._

- `[2026-06-17]` **ana-ml2 `vm.overcommit_memory=1` made durable** (sysctl drop-in) — overcommit=0 + zero swap caps CommitLimit; large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff)
  _Archived 2026-06-21._

- `[2026-06-17]` **GLM 5.2 wired into litellm** (`glm-5.2` + `-reasoning`, z.ai passthrough, `extra_body.thinking.type` toggle). (fe77a35)
  _Archived 2026-06-21._

- `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (successor to nh3-ansible); hosts `/opt/externs` pi-on-GLM-5.2 client workspaces. (a841eab)
  _Archived 2026-06-21._

- `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-` prefix (operator) — reuse the existing capability convention.** gen/gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge (passthrough)/gen-frontier/-reasoning. The **swap-transparency rule**: embedding stays model-named (`qwen3-embedding`) because vectors aren't swap-transparent; everything else can swap behind a stable alias. (d372103, 7819f96)
  _Archived 2026-06-25._

- `[2026-06-20]` **Removed the litellm `*` wildcard → llama-swap** (operator; llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now 404 cleanly instead of a misleading "Connection error" — closes the long-standing footgun. (d0eb09c)
  _Archived 2026-06-25._

- `[2026-06-20]` **Langfuse retired** (operator) — redundant with LiteLLM native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse spew). Callbacks dropped, stack torn down. (f8eda1c)
  _Archived 2026-06-25._

- `[2026-06-20]` **OmniVoice streaming `/tts` — vendor chatterbox-fast's scheduler, not its whole package** (operator-approved vendor-copy). Diffusion TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox-fast remains the lowest-latency front-end. (288d085, cd92b85)
  _Archived 2026-06-25._

- `[2026-06-20]` **Daily image+buildx prune cron on ana-docker** (the missing automation behind the recurring image bloat) — `lkraven` crontab, prune unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context.
  _Archived 2026-06-25._


- `[2026-06-25]` **althing re-architected to the lean multi-machine bus; nh3-extdev stood up as a
  MODEL B mesh peer** (dedicated `althing-svc` + group-shared `/srv/althing`). `reference_nh3_extdev_althing_mesh`.
  _Archived 2026-07-08._

- `[2026-06-23]` **zellij native web client piloted on nh3-dev** (`zellij-web.service` :8443)
  alongside ttyd. `reference_zellij_web_seat`.
  _Archived 2026-07-08._

- `[2026-06-22]` **Worldtree persona-render config arc (#314/#322/#317) pre-synced + deployed green
  on demo+personal** — #317 a boot-blocking config REMOVAL. `reference_corviduo_dev_emergency_ops`.
  _Archived 2026-07-08._

- `[2026-06-20]` **R22 (brokkr/dwarves) stood down to gateway-only; full-access R22 key minted;
  Phase B CANCELLED** (Worldtree model-agnostic → no deploy path). Key at `/home/lkraven/.r22-gateway-key`
  (persistent mode-600, carries paid GLM, don't delete). MUT = free `qwen3.5-122-a10b` (`gen`).
  Operator steer: R22 research is gated on a pragmatic/deployable outcome, not advancing-the-art.
  _Archived 2026-07-08._

- `[2026-06-20]` **claude-bot issue-scope token minted for worldtree-dev self-serve** (id 16,
  `write:repository`+`write:issue`); old token revoked. Advances the credential-migration directive.
  _Archived 2026-07-08._

- `[2026-06-20]` **rest-server-ana recovered + backup prevention shipped + worldtree-dev admin keys
  provisioned** (demo d113207c / personal f4f75adb). Cred rotation (5 rest-server pw) BELAYED.
  _Archived 2026-07-08._

- `[2026-06-20]` **claude-bot → ADMIN on vh/Worldtree** (operator-authorized) — self-serves WT
  deploys/tokens henceforth.
  _Archived 2026-07-08._

- `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.**
  (auto-memory `project_migrate_infra_access_to_claude_credentials`)
  _Archived 2026-07-13._

- `[2026-07-01]` **Worldtree #332 embed-recall diagnosed + scoped-log view/tunnel provisioned + fix
  verified** (15×→1.01× re-embed). `reference_wt_gateway_scoped_log_view`.
  _Archived 2026-07-13._

- `[2026-07-01]` **qwopus native MTP speculative-decode tested on `gen` → NOT kept** (+12% single-stream,
  −15–20% aggregate at concurrency, silently drops min_p/logit_bias). Banked for T1. `reference_gen_qwopus_122b`.
  _Archived 2026-07-13._

- `[2026-07-01]` **Deckard trial → reverted to qwopus (`gen`)** (won writing "in every way" but ~36 vs
  ~90 tok/s; spec-decode rescue ruled out). git `b63c48b`→`681eb70`. Deckard kept staged as T1's
  writing benchmark.
  _Archived 2026-07-13._

- `[2026-07-02]` **mtf-dev granite harness-spike ran GREEN — MECHANICAL only, efficacy DEFERRED to
  the T1 run.** Trainer TRL SFT→DPO→eval seam proven end-to-end on a synthetic fixture (not the E-RP
  corpus); operator DECIDED no intermediate real-efficacy granite spike (uninterpretable proxy —
  arch gap + abliteration axis). `reference_gen_qwopus_122b`.
  _Archived 2026-07-13._

- `[2026-07-03]` **ratatoskr-dev DEMO Heimdall key provisioned (R30 φ0).** Minted a tier-user key on
  the demo via `POST /admin/keys` (bootstrap admin key), mirroring their personal base consumer (no
  character-binding); base-agent affect reads work ungated. `reference_worldtree_demo_key_mint`.
  _Archived 2026-07-13._

- `[2026-07-04]` **character-rp: gateway-forwarding proven + role pre-staged + #344 shipped.**
  Empirically confirmed per-request `extra_body` (top_k/repetition_penalty) forwards through the
  `gen-reasoning` LiteLLM alias to vLLM + standard params override the alias defaults — no gateway
  cap needed (I over-built a dedicated alias, operator corrected, reverted with zero fleet impact).
  Pre-staged the `character-rp` role into demo+personal bind-mount `model_roles.yaml` (byte-verified
  on b18; caught the cached-registry ordering). worldtree-dev shipped **#344 (v1.0.0b19)** for the
  durable-agent override-drop. spend_logs spot-check queued (ratatoskr's timestamp ping).
  _Archived 2026-07-13._

- `[2026-07-04]` **althing v2 herald+receiver formalized as systemd on nh3-dev.** `althing-herald.service`
  (Restart=always, **Environment=PATH incl ~/.cargo/bin** — the pane-dispatch fix) + `althing-receiver.service`
  (v2 → pillar-3 `/owner/*` live); stale forseti unit removed; both on v2.0.0b2, canonicalized by
  forseti. `reference_nh3_dev_althing_herald`.
  _Archived 2026-07-13._

- `[2026-07-04]` **LitBench-RM respun (irv-ml1 A6000, comfyui displaced)** for T1's reward ensemble;
  operator sole comfyui consumer, holding image-gen until LitBench done. `reference_litbench_rm_irv_ml1`.
  _Archived 2026-07-13._

- `[2026-07-05]` **T1 training venue: CLOUD recommended; operator chose smoke-first on ana-ml2.**
  On-prem ruled out (ana-ml2 full — both 96G GPUs ~93G used): keep-serving = NVMe offload ~6-8 DAYS;
  full ana-ml2 shutdown = CPU offload ~1 DAY but a whole-fleet outage. Cloud Vast.ai 8×80GB (no
  offload → ~3-6h, ~$60-500, zero fleet impact) is the clean alt (mtf-dev + infra-ops both rec;
  Vast for its no-content-AUP marketplace + likely-existing VastBlue account). Operator's next step
  = the ana-ml2 CPU-offload SMOKE (~60 steps) to get real samples/sec before the full-outage-vs-cloud
  call. HF base verified public (zero-auth pull). Runbook + gotchas in `reference_t1_cloud_train_plan`.
  _Archived 2026-07-13._

- `[2026-07-05]` **glm-5.2 canonical limits recorded** (probed live vs z.ai): **1,048,576 (1M) input
  context / 131,072 (128K) max output**; NO gateway-side cap (pure passthrough → z.ai's limits are
  effective). Written to the config comment (commit `624a07e`) + `reference_litellm_gateway`.
  _Archived 2026-07-13._

- `[2026-07-06]` **infra-ops NOPASSWD sudo confirmed on irv-ml1 + ana-ml2** (the arbo memory's "NOT
  infra-ops on irv-ml1" was stale) — enabled the worker deploy + every ana-ml2 hot-swap smoke.
  _Archived 2026-07-13._

- `[2026-07-07]` **SGLang zc hot-swap: 2-line patch RAN → PARTIAL. Cleared startup + self-attn E2E, GDN
  in_proj hit 2 MORE name-routing gaps → MERGE recommended.** Root cause was a NAME-routing gap (not the
  feared shape-assert): `lora/utils.py:235` hardcoded `params_mapping` lacked `in_proj_qkv/in_proj_z →
  in_proj_qkvz`. Patched those 2 lines (overlay bind-mount) → `init_lora_shapes` passed, server up, self-attn
  loaded end-to-end. But the SAME missing GDN mapping bit 2 MORE resolvers: adapter-load skipped 192 in_proj
  weights, and inference `utils.py:300 get_target_module_name` (substring match) RAISED → 500 + scheduler
  died. So GDN LoRA name-routing is incomplete in ≥3 SGLang paths (fixed 1). NOT shape-intractable (self-attn
  proves the separate-key→fused-buffer machinery works), but NOT "one targeted fix" either → per operator's
  LAST-shot rule → MERGE. mtf-dev looped + closed (thread 01KWZDZFYN…); B/A shard-math never reached.
  Supersedes the "SGLang wants a THIRD format" framing — no new adapter format needed.
  _Archived 2026-07-13._

- `[2026-07-07]` **T1 (retargeted qwopus-122B→AEON-27B) SFT COMPLETE on ana-ml2** — all-TRL torch-GDN,
  batch1/accum8/GC-off, 1 epoch ~9h @ 0.25 s/s, loss 2.445, adapter banked. Supersedes the 2026-07-05
  cloud-vs-smoke venue decision: AEON-27B's 52G bf16 fits one Blackwell → ran on-prem in ~9h (no cloud,
  no full-fleet outage). GC is NOT the throughput lever (batch1-no-GC == batch4+GC at 0.25; dvalin's 0.565
  was warmup-noise from a 30-sample smoke); ~0.25 s/s is the torch-GDN floor (only FLA beats it, sm_120-
  blocked #913). `reference_t1_cloud_train_plan`.
  _Archived 2026-07-13._

- `[2026-07-07]` **T1 hot-swap on vLLM NVFP4 is DEAD — vLLM 0.24.0 qwen3_5 LoRA-application silent no-op**
  (#47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4+FP8) AND
  adapter-format-agnostic (separate `zc` loads-but-inert; fused rejected); #47640 guard-patch overlay didn't
  fix. Root-caused by a 3-peer dwarf panel (eitri/regin/dvalin); their vLLM issue refs verified real via
  GitHub API. Engine mirror-image: vLLM wants SEPARATE keys, SGLang wants FUSED.
  _Archived 2026-07-13._

- `[2026-07-07]` **HARD REQUIREMENT restated: hot-swappable LoRA or ABANDON.** Merge is NOT a solution for
  this requirement (static; reserved for a later RP-quant track). SGLang is the last hot-swap shot; if it
  can't apply, park the adapter + file upstream repros. `reference_lora_training_worker` (adjacent).
  _Archived 2026-07-13._

- `[2026-07-07]` **comfy-dev in-arbo LoRA-training worker DONE** — built/deployed/validated Phases 1+2
  (first Sindra train succeeded, publish-step + `train_id`), Phase 2.5 GC contract (shape A) confirmed +
  impl deferred. `services/lora-training-worker/`, commits 888ba6a→b617a8b. `reference_lora_training_worker`.
  _Archived 2026-07-13._

- `[2026-08-08]` **worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev.** worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared `~/snapshots/worldtree-400-fiction-decomp` (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). **LEFT INTACT:** `rex393-fiction-index`/`rex393-fiction-snapshot` (separate operator KEEP word, unchanged) + `r42-gate-*`. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (`01KZJ9GMCC…`).
  _Archived 2026-08-24._

- `[2026-08-07]` **chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed).** Long saga, operator-driven clean diagnosis. **Symptom:** ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. **NOT** German-leak (Turbo `generate()` has NO language param — plain AutoTokenizer, no `language_id`; the multilingual `language_id="en"` lever lives only in the separate `ChatterboxMultilingualTTS`), **NOT** OOM alone. **Real cause:** the Chatterbox **Turbo T3 model OVER-RUNS its generation tail** — a long single `generate()` degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens `<6561` + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). **ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE** — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). **Diagnosis method** (deterministic, no ears-only): single-shot length sweep + **amplitude-gated voiced-ZCR** (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. **FIX:** server-side `max_chunk_chars=250` cap on the scheduler (`:v2` image, `CBF_MAX_CHUNK_CHARS=250` env) — bounds each generation to just under the ~300-char onset → clean **3-4 sentence** chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; **chatterbox's low emotiveness keeps chunk joins smooth** (the harsh joins that got Zonos rejected are absent — operator's key call). **ratatoskr TODO (relayed msg `01KZER9X7S`):** revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. **Cap value tunable** per-request (`max_chunk_chars`) + env. **Deeper prosody** (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). **⚠ FOOT-GUNS:** (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) **build-context vs image drift** — the `:v2` image was built from cap source, but after a `:v1` rollback the build context held `:v1` source → a `docker compose build` would've silently produced a cap-less `:v2`; re-synced the flat cap source to `/opt/docker/compose/chatterbox-fast/` (rebuild-verified). **⚠ DIVERGENCE (follow-up):** deployed build context is FLAT (`app.py`/`scheduler.py`, `from scheduler import`, thin-overlay `FROM local/chatterbox:v1`, cap-only) vs the `vh/chatterbox-fast` REPO which is PACKAGE-layout (`chatterbox_fast/`, `from chatterbox_fast.scheduler`, self-contained Dockerfile) + has `norm_loudness` (repo commit `6bc7bf0` = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: `.bak-cap-20260807-104850` backups on irv-ml1 + `:v1` image both retained. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]]
  _Archived 2026-08-24._

- `[2026-08-07]` **Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY.** Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. **⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init.** GPU memory was held by the `--multiprocessing-fork` CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). **RESTORE CMD** (from irv-ml1, user lkraven): `cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 &` then `docker start zonos-gateway`. **Consumers that lost Zonos:** asset-engine + gateway-chat (via LiteLLM `ext-tts` alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). [[reference_zonos_tts_stack]]
  _Archived 2026-08-24._

- `[2026-08-07]` **chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos).** Operator-directed. Copied `zonos-gateway/voices/Donut.wav` → chatterbox `/refs` (`/worktank/chatterbox/reference_audio/donut.wav` — the reference_audio SUBDIR is lkraven-owned so no sudo despite `/worktank` root; container globs `/refs` live → **NO restart**), exposed as `voice:"donut"` (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at `http://10.100.10.50:8090/b/donut-chatterbox/`. Answered ratatoskr's 8-question contract ask from the live gateway (`local/chatterbox-fast:v1`) + source: **NOT OpenAI-shaped** (`POST /tts`; body `text`/`voice`/`format`/`stream`, not `input`/`model`/`response_format`); **NO affect dials** (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; **Zonos stays the only fleet TTS with real emotion steering**); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. **FYI-worthy (operator):** ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory `reference_chatterbox_fast_repo` enriched w/ the live contract. [[reference_zonos_tts_stack]]
  _Archived 2026-08-24._

- `[2026-08-07]` **Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43).** The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM `reranker` repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); **R42 v13 gate PASSED first-ever** (56/90→90/90). Incumbent kept warm :8002 (rollback via `qwen3-reranker` alias), A4 fallback :8014. Full arc + rollback runbook `docs/pfi/reranker-selection-ledger.md`; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes.
  _Archived 2026-08-24._

## Tried and abandoned (archived)

- `[2026-04-30]` task-board workflow with
  `container: image: debian:bookworm-slim` — fails:
  `actions/checkout@v4` needs `node` at runtime, slim image lacks it.
  Switched to `node:20-bookworm-slim` (has node + apt) or runner-label
  default. (Pattern revisited 2026-05-17 for skaldsong-dev: container
  override needs `nodejs` apt-installed unless it IS the default.)
  _Archived 2026-05-27._

- `[2026-04-30]` Dropping the `container:` directive before runner
  re-registration with docker-schema labels — runner silently falls
  back to **host mode** (jobs run inside the alpine `act_runner`
  container itself, no apt). The `:host` suffix in startup logs
  (`labels updated to: [pfi-fleet:host ana-docker:host]`) is the
  giveaway. Fix: register with `pfi-fleet:docker://<image>` schema
  labels.
  _Archived 2026-05-27._

- `[2026-04-30]` Updating runner labels by editing `.env` and bouncing
  — doesn't take. The `.runner` registration cache pins labels at
  first registration; env-var updates are read each start but the
  stored token + UUID are tied to the original label set on the gitea
  side. Fix: stop runner, delete `.runner`, generate new admin
  registration token, redeploy.
  _Archived 2026-05-27._

- `[2026-04-30]` `git reset --hard origin/<sha>` in
  `deploy-task-board.yaml` (and the in-repo nevermore playbook before
  fix) — invalid syntax: `origin/` prefix only works for branch refs.
  SHAs need `git reset --hard <sha>` directly. Resolved with
  `git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first,
  then bare `"{{ ref }}^{commit}"` fallback.
  _Archived 2026-05-27._

- `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after
  task-board wiring — it was actually only repo-scope on
  `vh/task-board`. vor's first CI run failed with empty SSH key
  (`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user
  scope at `gitea.phasefinal.com/user/settings/actions/secrets`.
  _Archived 2026-05-27._

- `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity —
  leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a
  follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1<redacted>/"`
  still leaked the first 4 chars. Lesson: when probing secret-bearing
  files, use field-by-field SELECTIVE inspection
  (`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any
  password line, `grep -c` (existence) or `test -n "$(...)"`
  (non-empty), never `cat` or value-printing.
  _Archived 2026-05-27._

- `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT
  timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero
  matches and led to a wrong "no /v2/ traffic in 12 days" conclusion.
  **Gitea logs in PDT, Traefik logs in UTC** — same host, different
  timezones. Always normalize timezones (UTC) when correlating logs
  across services on the same box. Cost: ~30 min in the wrong
  direction.
  _Archived 2026-05-27._

- `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` /
  `PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on
  `/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern
  **response writes**, not request body reads. `unexpected EOF` from
  Go's HTTP server means the client closed mid-body-upload; not a
  knob Gitea exposes server-side.
  _Archived 2026-05-27._

- `[2026-05-12]` Defaulting asset-engine to Traefik-routed
  (`asset-engine.phasefinal.com` with `anaprod` cert resolver) on
  first scaffold — user pulled it back to LAN-direct. Internal
  tools default LAN-direct; only add Traefik when an external/TLS
  surface is actually needed.
  _Archived 2026-05-31._

- `[2026-05-12]` Routing althing thread replies through `galdrabok`
  when the actual dev handle is `forseti` — bus rejected
  `to=forseti` initially because thread participants list was
  `[galdrabok, infra]`. Solved by starting a new thread with
  `forseti` as the direct recipient. Lesson: when the bus
  auto-resolves a sender handle that doesn't match the actual dev
  role, start a fresh thread rather than fighting the participant list.
  _Archived 2026-05-31._

- `[2026-05-13]` Initial Voxtral default voice `alloy` (OpenAI-compat
  naming) — vLLM-Omni serving Voxtral does NOT translate aliases.
  Native presets are `<register>_<gender>` shape (`neutral_female`,
  `casual_male`, etc.). Always live-probe `/v1/audio/voices` for the
  exact wrapper-deployed preset names before setting a catalog
  default. Same caveat for Qwen3-TTS (wrapper exposes 15 voices: 9
  Qwen presets + 6 OpenAI aliases) and Kyutai-TTS (NillPointer
  wrapper has NO voice-listing endpoint at all; voices are
  filesystem paths under the `kyutai/tts-voices` HF repo).
  _Archived 2026-05-31._

- `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 —
  flag was deprecated. Use `--runner pooling`; the model's
  `architectures: [LlamaForSequenceClassification]` in config.json
  drives the classification head. Surfaced as `vllm: error:
  unrecognized arguments: --task classify` in container logs.
  _Archived 2026-05-31._

- `[2026-05-17]` Trusting that `.env` edit alone propagates a new env
  var into a worldtree container — `compose.yaml`'s `&worldtree-env`
  anchor must explicitly declare the passthrough or the value silently
  doesn't land. Same footgun bit `Z_AI_API_KEY` (2026-05-12) AND
  `BIFROST_CLIENT_ALLOWED_HOSTS` (2026-05-17). Cost ~10 min of "why is
  env empty?" diagnosis each time. Worldtree-side fix in
  `vh/worldtree@08f02b2`.
  _Archived 2026-05-31._

- `[2026-05-17]` `--force-recreate --pull never` from the docker:cli
  sandbox without explicit `-e WORLDTREE_IMAGE=<sha>` re-pins the
  container to `:latest`, even when a newer SHA-tagged image is on
  disk. Symptom: container "recreated" but actually reverted to a
  stale image. Pass `-e WORLDTREE_IMAGE=...:<sha>` to the docker run
  invocation. Worldtree-dev's `8ef3801` health-gated `:latest`
  advance is the long-term fix.
  _Archived 2026-05-31._

- `[2026-05-18]` Volva env.sh.template `$HOME` in commented examples
  — systemd's `EnvironmentFile=` parser doesn't expand `$HOME`;
  uncommenting lands the literal `$HOME/...` string. Volva-dev's
  `f4dda73` swapped to `/home/<svc-user>/...` placeholders.
  _Archived 2026-05-31._

- `[2026-05-18]` Initial Volva systemd unit's `ProtectHome=read-only`
  without `ReadWritePaths=` — althing-cli's SQLite (`~/.althing/
  althing.db`) and codex's session state (`~/.codex/`) both need to
  write. Container started but every poll failed with "db path not
  writable". Surgical fix: `ReadWritePaths=/home/lkraven/.althing
  /home/lkraven/.codex` (preserves the hardening intent, only carves
  out the specific dirs).
  _Archived 2026-05-31._

- `[2026-05-18]` Trusting that env.sh's `export VOLVA_ROOT="$(cd
  "$(dirname "${BASH_SOURCE[0]}")" && pwd)"` template line works
  under systemd — `EnvironmentFile=` parser aborts on the first
  unparseable line (command substitution), and `VOLVA_ALTHING_CLI`
  declared below silently never lands. Symptom: `Environment=`
  property empty, daemon error "althing-cli not found at
  'althing-cli'". Fix: replace command-substitution with literal
  path. Volva-dev's `d436c3c` dropped VOLVA_ROOT entirely upstream.
  _Archived 2026-05-31._

- `[2026-05-19]` Naive `docker rmi worldtree:<old-sha> --force`
  for CD SHA cleanup — would untag pinned/personal worldtree images
  since all three deployments share corviduo-dev. Use
  `docker image prune -a --filter "until=Xh"` instead — respects
  in-use semantic (Docker won't remove an image referenced by any
  container on the host), so pinned/personal protected automatically.
  _Archived 2026-05-31._

- `[2026-05-19]` Skaldsong CD first attempt: `docker pull` step
  failed with 401 unauthorized. ana-docker had no `docker login`
  for `gitea.phasefinal.com`. My playbook prereq note ("docker login
  has been done at least once") was an unverified assumption. One-time
  manual login persists in `~/.docker/config.json`; architectural fix
  (workflow-side `ssh ana-docker 'docker login ...'` step using
  `REGISTRY_USER`/`REGISTRY_TOKEN` secrets) flagged as v2.
  _Archived 2026-05-31._

- `[2026-05-19]` `SKALDSONG_HOST_CORS_ORIGINS=http://10.250.50.70:8300`
  as a bare URL — pydantic-settings parses complex env vars via
  `json.loads()`; first-boot crashloop with `SettingsError: error
  parsing value for field "cors_origins"`. Must be JSON array literal:
  `SKALDSONG_HOST_CORS_ORIGINS=["http://..."]`.
  _Archived 2026-05-31._

- `[2026-05-19]` `SKALDSONG_HOST_STATIC_ASSETS_PATH=/app/web/dist`
  in compose — mismatched Dockerfile reality. The Dockerfile COPYs
  SvelteKit build output flat into `/app/spa` (not `/app/spa/dist`).
  Lifted the path from skaldsong-dev's CD-ask message ("`/app/web/dist`")
  rather than verifying against the actual Dockerfile they shipped.
  Lesson: when encoding container-internal paths in compose, verify
  against the Dockerfile, not the design-doc.
  _Archived 2026-05-31._

- `[2026-05-19]` Playbook verify step `docker ps | grep healthy`
  racing the container's `start_period` (30s in compose's
  healthcheck). Verify ran 0.09s after `compose up -d --force-recreate`
  — well before docker's healthcheck could flip the status from
  `(health: starting)` to `(healthy)`. False-negative; container was
  operationally up (the earlier `/health` poll verify already
  confirmed). Fix: grep `^Up` not `healthy`. /health-200 IS the
  liveness check; docker's `(healthy)` is just a delayed echo.
  _Archived 2026-05-31._

- `[2026-05-20]` `SKALDSONG_DB_PATH` + `SKALDSONG_RUNS_DIR` in
  compose env block — names skaldsong's app doesn't read. App reads
  `SKALDSONG_HOST_SQLITE_PATH` + `SKALDSONG_HOST_RUNS_ROOT` (per
  Dockerfile ENV defaults). Wrong names = silently no-op; app fell
  back to Dockerfile defaults pointing at `/app/data/...` which the
  compose's bind mount did NOT cover (target was `/app/state/...`).
  Result: every `--force-recreate` wiped the SQLite DB. Caught by
  skaldsong-dev (althing thread `01KS4DPF6SXTBP4Q360JZVWPNT`). Fix in
  `52e98fa`. Lesson: verify env var NAMES against the Dockerfile/app,
  not against design-doc shorthand.
  _Archived 2026-05-31._

- `[2026-05-25]` First selene-block patch put the block under
  `saga_allowed_models:` instead of top-level `models:` — used
  `text.replace("models:\n", ...)` which substring-matched the
  `saga_allowed_models:\n` line first. Caused YAML parse error. Fix:
  anchored regex `re.compile(r"^models:\n", re.MULTILINE)`. Pattern:
  substring replace on YAML top-level keys WILL match suffix-containing
  keys.
  _Archived 2026-05-31._

- `[2026-05-27]` `docker compose up -d` inside the `docker:cli` sandbox:
  `${VAR:-./config}` defaults resolve `./config` to the sandbox CWD, but the
  Docker daemon interprets the path against the HOST filesystem → auto-creates
  an empty dir → entrypoint reseeded image-baked defaults (lost host-side
  providers.yaml patches). Fix: pass `-e WORLDTREE_CONFIG_DIR=/abs/path`.
  Folded into the docker-as-root convention note.
  _Archived 2026-06-01._

- `[2026-05-27]` `:latest`-pinned compose + private gitea registry + sandboxed
  pull = recreate on ancient cached `:latest` (deploy pulls by SHA so the tag
  never advances; sandbox can't pull). Fix: retag SHA→`:latest` on host, then
  `--pull never`. Better: pin SHA in `.env`, advance in CI.
  _Archived 2026-06-01._

- `[2026-05-27]` Container recreate during in-flight skaldsong gen kills the
  runner. With deploys every ~10min and stories >5min, structural not
  incidental. Roadmap (skaldsong-dev): pre-shutdown signal handler, per-scene
  resume-from-checkpoint, /api/admin/quiesce. None shipped.
  _Archived 2026-06-01._

- `[2026-05-27]` `--reasoning-format gemma` on artemis-31b-v1i — unsupported in
  the deployed llama.cpp (accepts none|deepseek|deepseek-legacy). `deepseek`
  populates the `reasoning_content` SSE delta Worldtree GemmaProvider checks.
  _Archived 2026-06-01._

- `[2026-05-27]` `head -c N` piped after a streaming curl SIGPIPEs the curl,
  killing the request early. Use file-write + separate read.
  _Archived 2026-06-01._

- `[2026-05-31]` Building the dia2-capable image surfaced THREE upstream
  packaging quirks: (1) `pip install -e nari-labs/dia2` fails — no PEP 660
  `build_editable` hook; (2) plain `pip install` builds an empty
  `UNKNOWN-0.0.0` wheel (base setuptools 59.6 < dia2's required ≥70); (3)
  `--no-deps` leaves `transformers`/`sphn`/`whisper-timestamped` missing. Fix
  (`local/dia:v2`): copy the pure-python `dia2/` package into site-packages +
  install ONLY those 3 deps; base torch/numpy already satisfy Dia2.
  _Archived 2026-06-03._

- `[2026-05-31]` Dia2 predefined voices (43, baked at `/app/voices`) are NOT
  reachable from the `/tts` clone path — it resolves `clone_reference_filename`
  against the reference_audio dir ONLY. The OpenAI `/v1/audio/speech` `voice`
  param auto-resolves them (separate code path), which masked the gap. Fix:
  stage `/app/voices/*` into `/worktank/dia/reference_audio`. Lesson: verify on
  the endpoint the catalog ACTUALLY targets.
  _Archived 2026-06-03._

- `[2026-05-31]` `voice_mode=clone` with an empty `clone_reference_filename`
  → asset-engine serializes it as the literal string `"undefined"` → `/tts`
  404. First observed on dia2; worked around in the catalog (default the field
  to a real voice). **[2026-06-01] root cause found** — the Kokoro voice-blend
  widget reading Shoelace `.value` before hydration (see Current state); the
  real fix is asset-engine-side and is escalated.
  _Archived 2026-06-03._

- `[2026-05-31]` asset-engine `services.schema.json` is DERIVED (regen from the
  Pydantic model via `dump_schema.py`) and had DRIFTED — rejected the
  `lifecycle` field 12/14 services use. RESOLVED: regenerated with
  `CatalogLifecycle` + `reproducibility.seed_field` (`b7b2130`). Lesson:
  hand-editing `services.yaml` shape without regenerating re-introduces drift.
  _Archived 2026-06-03._

- `[2026-05-31]` ttyd-over-TLS forces HTTP/2 (kills ttyd's terminal WebSocket
  → blank screen); Safari/WebKit never sends HTTP basic-auth on WS upgrades.
  Both solved for the fleet seat (Caddy forces HTTP/1.1; auth → network-gating)
  — detail in auto-memory `reference_ttyd_fleet_seat`.
  _Archived 2026-06-03._

- `[2026-05-30]` esh-docker-vm: `hard` NFS mounts from 10.0.50.50 froze a
  container worker in UNKILLABLE D-state when the NAS stalled — only a host
  reboot clears it. Separately, `fstab defaults` (no `_netdev`) made NFS-bind
  containers `Exited(255)` on reboot. → auto-memory
  `incident_esh_docker_nfs_boot_race`.
  _Archived 2026-06-03._

- `[2026-06-02]` **Naive cumulative re-decode for streaming** — decode the
  growing token prefix each chunk + emit the delta, assuming the causal flow
  gives a stable prefix. It does NOT (full-context attention, see above);
  maxdiff 0.30. Also the high-level `s3gen.inference` re-applies `trim_fade` at
  the start every call. Don't go this way.
  _Archived 2026-06-04._

- `[2026-06-01]` **CSM bring-up** — upstream `phildougherty/sesame_csm_openai`
  Dockerfile pins NO `huggingface_hub` version, so it now resolves to 1.17.0
  where `huggingface-cli` is fully removed (replaced by `hf`) → the build dies
  at `huggingface-cli login`/`download`. Passing the HF token as a build-arg
  got the token in (the model downloads at BUILD time, not runtime — our
  compose only wired it as runtime env, the original failure) but then hit the
  rotted CLI. Fixing would need vendoring + patching the Dockerfile; not worth
  it for a weak model → CSM removed.
  _Archived 2026-06-04._

- `[2026-06-01]` **Fish path-form references** (`{audio:"/app/references/x.wav"}`)
  → HTTP 500 on this build, despite the stack README showing that shape. The
  working clone paths are `reference_id` (by name → resolves the DIRECTORY
  `references/<name>/` containing `<name>.wav` + `<name>.lab`; the flat
  `references/<name>.wav` is ignored — see the two-root-causes entry above) or
  inline base64 `references`. The catalog uses `reference_id`.
  _Archived 2026-06-04._

- `[2026-06-01]` **Reference transcript barely affects Fish clone IDENTITY**
  (disproving my mid-session theory). I'd blamed a bogus provenance-note `.txt`
  for poor cloning, but the ECAPA re-test showed correct (0.787) / bogus (0.778)
  / empty (0.738) transcripts all clone Imogen about equally — the transcript
  affects PRONUNCIATION (phoneme disambiguation per the docs), not who it sounds
  like. The real culprit for "not British" was the `"undefined"` select bug, not
  the transcript. (A correct transcript still marginally helps pronunciation —
  cheap to stage, not load-bearing.)
  _Archived 2026-06-04._

- `[2026-06-01]` **Pitch-shift register control** (rubberband, to deepen Imogen
  to contralto/mezzo) — Fish ignores small reference shifts and overshoots
  large ones; `-3st` WITHOUT formant preservation → "baritone through a tuba"
  (formants collapsed); `-2/-3st` WITH `--formant` → too subtle / still bad.
  Abandoned at every depth; all variants deleted. Finer independent
  pitch/formant control needs praat (not installed). Future clones = operator's.
  _Archived 2026-06-04._

- `[2026-06-01]` **resemblyzer is too weak for cloning-fidelity scoring** — its
  dated 2019 LSTM encoder + a synthetic-vs-natural domain gap scored the Imogen
  clone CLOSER to the default than to real-Imogen, which led me to a WRONG "Fish
  is a weak cloner" call. ECAPA-TDNN (speechbrain `spkrec-ecapa-voxceleb`) on the
  same clips gave the correct answer (clone 0.79 to real Imogen, 0.10 to
  default). Use ECAPA, not resemblyzer, for speaker-verification.
  _Archived 2026-06-04._

- `[2026-06-02]` **Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED
  (sub-realtime).** Benched RTF on A6000: 0.72x/0.82x/0.86x, mean 0.80x = sub-realtime,
  so client-side chunking starves. Root cause: Fish only chunks on `<|speaker:X|>` tags;
  plain text → batches=[whole text] → all semantic tokens generate before any audio
  (chunk_length inert). Plus kui/ASGI StreamResponse doesn't flush (fix = anti-buffering
  headers, kept on file). STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands → bench
  fp4-quantized Fish; if RTF > ~1.5x give it the chatterbox-fast treatment. For now Fish is
  a buffered catalog entry (great for SAVED gens, not live-audition).
  _Archived 2026-06-11._

- `[2026-06-02]` **Context-priming at chunk joins (chatterbox-fast §1.6) — ABANDONED
  (discard-cut leaks the prefix).** Prepend prior sentence, generate `prefix+content`,
  discard prefix audio. Built + shipped (`d707439`), live-A/B'd, reverted (`090e70a`).
  Kill: `generate()` returns one waveform with NO marker where the prefix ends, and the
  model renders the same prefix with different timing solo vs followed-by-content — so
  locating the cut is a guess that left a whole clause in the output (operator caught
  "...without a trace of sarcasm" spoken twice). Reliable cut needs token-level boundaries
  or per-chunk forced-alignment (heavy). Coherence loss at joins = ACCEPTED limitation.
  _Archived 2026-06-11._

- `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED (turbo
  isn't built for streaming).** (1) `S3GenStreamer` is docstring-only (unimplemented).
  (2) `flow_inference(finalize=False)` is buggy (lookahead trim removes frames from `h`
  not `h_masks`/conds → shape mismatch); a 1-line patch makes it RUN. (3) BUT the flow
  encoder uses full-context attention (`static_chunk_size=0`) → incremental decode is
  prefix-unstable (adding tokens re-attends, shifts earlier mel ~0.30-0.39 vs one-shot).
  (4) Forcing `static_chunk_size>0` didn't stabilize it. True sub-second frame-streaming
  needs deep attention surgery with quality risk — not worth it. → adaptive-chunking instead.
  _Archived 2026-06-11._

- `[2026-06-03]` **`uv pip install .` fails on SmoothKen/knn-svc** (and similar script-repos)
  — it's analysis scripts + a poetry pyproject, no buildable package (setuptools
  package-discovery error). Install the pyproject deps directly, don't build the "package".
  _Archived 2026-06-13._

- `[2026-06-03]` **Backgrounding `althing-cli monitor` with an inline shell `&` (instead of
  the Bash-tool `run_in_background`) orphans it** — it survives the shell exit, holds the
  per-handle flock UNTRACKED (won't notify the session), and `stop-monitor` doesn't detect it.
  Fix: find + kill the orphan PID (verify cwd=this repo / handle first — nh3-dev is shared, other
  agents' monitors run there too), then re-arm via run_in_background. Always re-arm tracked.
  _Archived 2026-06-13._

- `[2026-06-03]` **torch-2.12 venvs need `uv pip install torchcodec`** — torchaudio 2.12
  defaults to the TorchCodec backend for `.load`; without it, real audio I/O throws "TorchCodec
  is required" — and it ONLY surfaces at actual conversion, NOT at import/model-load. Lesson:
  validate real I/O, not just import, when provisioning ML engine envs. (seed-vc on torch 2.4
  uses the legacy backend, exempt.)
  _Archived 2026-06-13._

- `[2026-06-03]` **gitea webhook to a private IP is denied by `webhook.ALLOWED_HOST_LIST`**
  (anti-SSRF; default `external` blocks private/loopback). Symptom: delivery shows
  `dial tcp ...: webhook can only call allowed HTTP servers`. Fix = APPEND the target net to
  ALLOWED_HOST_LIST in gitea's app.ini (keep `external`; scope tight, never `*`/`private`) +
  restart gitea (act_runner job containers survive a restart). gitea runs as a container on
  ana-docker (`gitea_gitea_data` volume, `/data/gitea/conf/app.ini`).
  _Archived 2026-06-13._

- `[2026-06-04]` **homepage "crash" ≠ always NFS** — a wedged container in unkillable D-state can come from dead `siteMonitor` widget targets (retired ESH firewall IPs) hanging the node event loop into `exit_mmap`, needing a host reboot. Check homepage's siteMonitors against retired hosts. (`incident_esh_docker_nfs_boot_race`)
  _Archived 2026-06-14._

- `[2026-06-04]` **pyworld needs `setuptools<81`** (imports the removed `pkg_resources`); and **R/soundgen `-lgfortran` fails** on irv-ml1 (default gcc is gcc-11 but only gfortran-12 present) → install `libgfortran-11-dev`.
  _Archived 2026-06-14._

- `[2026-06-04]` **Persistent services on irv-ml1 need SYSTEM systemd units** — the box reaps user-session processes on ssh disconnect, and `--user` systemd isn't reachable over non-login ssh, so nohup/setsid/`screen -dmS`/`systemd-run --user` all die (even with enable-linger). Use `/etc/systemd/system/`.
  _Archived 2026-06-14._

- `[2026-06-04]` **GPU pin by INDEX is ambiguous on irv-ml1** — native CUDA orders fastest-first (A6000=0) but nvidia-smi/docker use PCI order (A6000=1). Pin by **UUID** (`CUDA_VISIBLE_DEVICES=GPU-…`); verify via nvidia-smi compute-apps.
  _Archived 2026-06-14._

- `[2026-06-04]` **Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF — the "headgun" lesson.** Serve each model's canonical `tokenizer.apply_chat_template`, not the bundled template (bundled ones corrupt baselines). Verify via vLLM `/tokenize`→`/detokenize`. (`90e08f0`/`27eb537`)
  _Archived 2026-06-14._

- `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()` requires torchcodec → dies AFTER separating** (0 stems, rc=1). Fix = pin `torch==torchaudio==2.4.1` + `UV_LINK_MODE=copy`. Validate the SAVE path, not just import + GPU inference, on a bleeding-edge torch.
  _Archived 2026-06-16._

- `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — fills KV to the `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs AFTER weights+KV load (crash-loops). Fix: free co-tenant room OR `--enforce-eager`. FP8 single-stream is batch-1 GEMV (memory-bound) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`)
  _Archived 2026-06-16._

- `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres rows; clone a default + swap the measure. tok/s is the `outputTokensPerSecond` MEASURE (metrics-API/dashboard query time), not a per-generation field. langfuse-web needs `HOSTNAME=0.0.0.0`. Host 3000 is gitea's → langfuse on 3001.
  _Archived 2026-06-16._

- `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh + corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are vh-owned, the `.env` is deploy-owned 600.
  _Archived 2026-06-16._

- `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL`; defaults to claude-haiku in code, opt-in via config not `.env`. Inspect the live container env + vh-owned config files first.
  _Archived 2026-06-16._

- `[2026-06-13]` **Loading an old LoRA adapter after a vocab bump fails on embedding size.** The harness config + `tokenizer.json` are now at `new_vocab_size=50279` (v4 multi-tag); the v1/v2/v3 adapters are 50277. To load v1 (the accepted artifact), set `cfg.new_vocab_size=50277` before `load_finetuned_engine_lora` (else PeftModel state_dict size mismatch). `tokenizer.json.v3bak` is the 50277 tokenizer for a clean restore.
  _Archived 2026-06-16._

- `[2026-06-13]` **Heavy from-source compile (`MAX_JOBS=128`) on the shared PROD GPU box PINS it** — load hit 187, prod vLLM restarted, killed an in-flight quant. ana-ml2 hosts live inference; never run a big build there at full parallelism. Cap `MAX_JOBS≤32`, build off-box, or cgroup-constrain.
  _Archived 2026-06-16._

- `[2026-06-13]` **`--quantization fp8` on a VL model can quantize the VISION TOWER → garbage vision** (Qwen3.5-VL on stable vLLM: gray-grid output; LM answers text fine, so it "looks" healthy). The nightly excludes the vision tower. Validate the VISION path on a quantized VLM, not just text — and pin the engine digest with the exclusion.
  _Archived 2026-06-16._

- `[2026-06-13]` **vLLM's `--gpu-memory-utilization` is checked against FREE VRAM at startup, not total** — on a shared card, growing one service before trimming a co-tenant OOMs. Trim the shrinking service FIRST, then grow. Size to the FREE budget.
  _Archived 2026-06-16._

- `[2026-06-13]` **The `vllm/vllm-openai` entrypoint is already `["vllm","serve"]`** — compose `command:` supplies the model as the first POSITIONAL arg + flags; a second `serve`/`--model X` → "unrecognized arguments". Same-class: `tee` masks the real exit code (use `>`); HF `datasets` rejects bare `wikitext` (needs `Salesforce/wikitext`).
  _Archived 2026-06-16._

- `[2026-06-13]` **Chatterbox-Turbo LoRA finetune: the repo's `setup.py` loads the WRONG tokenizer** — pulls gpt2-medium + a grapheme merge file instead of the chatterbox-turbo GPT2 tokenizer (vocab.json+merges.txt, len 50276). Fix = override + delete the grapheme `tokenizer.json`; `[vmoan]` → new_vocab_size 50277 (1-row resize), lora_r 64 / alpha 128, modules_to_save=[text_emb,text_head]. Unique-stem corpus collision needs `{index}_{stem}` IDs. (`irv-ml1:~/r16-vmoan-harness`)
  _Archived 2026-06-16._

- `[2026-06-11]` **A completion-poll `while pgrep -f <scriptname>` SELF-MATCHES its own remote shell argv** — its own `pgrep -f` always finds itself → the loop never exits. Use a match pattern ABSENT from the poll command (the python stage, or a sentinel file), not the driver's own name.
  _Archived 2026-06-16._

- `[2026-06-14]` **vLLM ModelOpt-NVFP4-MoE loader is broken for current multimodal MoEs.** `nvidia/Qwen3.6-35B-A3B-NVFP4` fails weight-load: `KeyError: layers.0.mlp.experts.w2_input_scale` on 0.19.1, `lm_head.input_scale not registered` on 0.22.0 (vllm #44081); same class hits Gemma-4 MoE / Qwen3-30B-A3B NVFP4. The arch + quant ARE recognized (gets past arch resolution + vision-processor load) — it's the per-expert/lm_head scale-key mapping. Don't chase nightlies; use official FP8 until fixed.
  _Archived 2026-06-18._

- `[2026-06-14]` **vLLM sampler-warmup OOMs on a shared GPU even when weights fit** — it warms the sampler with `max_num_seqs` (default **1024**) dummy requests, and a big vocab (Qwen3.6 = 248K) makes that a huge transient logits tensor. A vision endpoint doesn't need 1024-way concurrency: set `--max-num-seqs 32`. Separately, post-load `ValueError: No available memory for the cache blocks` means util is too thin (weights+activation+graph ate it) — for 34 GB FP8 weights, util ≥ ~0.45 to leave KV room.
  _Archived 2026-06-18._

- `[2026-06-14]` **Recreating multiple vLLM services concurrently races the memory-profiling assertion** — `AssertionError: Error in memory profiling. Initial free memory X / current Y … other processes … release GPU memory while vLLM is profiling`. Recreate co-tenant vLLM services ONE AT A TIME (force-recreate one, wait healthy, next).
  _Archived 2026-06-18._

- `[2026-06-14]` **embed/rerank (0.6B) at util 0.05 reserve ~5.5 GB each — mostly util-reservation WASTE, not need.** A 0.6B model needs ~1.2 GB weights + ~2.5 GB CUDA/torch context; util 0.03 (~3.6 GB) fits with room, reclaiming ~4 GB (vLLM reserves the util fraction regardless of actual KV; embedding models barely use KV). Real-need floor ~3 GB — don't go to 0.02.
  _Archived 2026-06-18._

- `[2026-06-14]` **Chatterbox-Turbo decode-knob foot-guns** (R16 v1-polish + emergent probes): the turbo length cap is `max_gen_len` (default 1000) on `t3.inference_turbo`, NOT `max_new_tokens` — and `tts_turbo.generate` does NOT forward it (wrap inference_turbo to cap). `rep_pen 2.0 / temp 0.5` BACKFIRES (degenerate 24 s run-on). Soft decode tamers cut the NVV ITSELF, not just the run-on tail (operator: "p1 trims the moaning too") — gen-time polish can't beat v1's defaults. Inline base-NVV tokens DEGENERATE (moan-cascade + gibberish), they don't modulate the surrounding words.
  _Archived 2026-06-18._

- `[2026-06-15]` **A comfyui restart-loop with `exit 0` + `OOMKilled=false` is NOT OOM — suspect a native SEGFAULT.** `docker inspect` shows only the LAST exit code; a GPU OOM is a catchable `RuntimeError: CUDA out of memory`, NEVER `Fatal Python error: Segmentation fault`. Diagnostic: `RestartCount>0` = same container restarted in-place → the docker json-log ACCUMULATES across all cycles, so grep the full `docker logs` for the crash signature + the lines right before each restart banner. comfyui's was at VAE decode (sampling completes, then segfault loading `AutoencodingEngine`) — gen-triggered but native, not VRAM. Suspects logged: torch/cu130 mismatch, an injected `aimdo` cuda-funchooks lib, the VAE.
  _Archived 2026-06-19._

- `[2026-06-15]` **NEVER blanket-kill `althing-light-monitor` by PID** — they're per-handle and MANY agents run their own on the shared nh3-dev box (heid / ratatoskr-dev / worldtree-dev / bifrost-dev each have one). A `kill $(pgrep -f light-monitor)` downs four peers' monitors (near-miss caught by checking `/proc/<pid>/environ` `ALTHING_HANDLE` + cwd first). Recover your OWN via `althing-cli stop-monitor` (v0.14.1+); a raw `&`-backgrounded monitor is untracked (no harness fire-notification) — always arm via `run_in_background`. (extends the pkill-self-match lesson below)
  _Archived 2026-06-19._

- `[2026-06-15]` **Manual `docker compose up` on corviduo Worldtree DOWNGRADES to a stale local `:latest` → crash-loop (config/image mismatch).** Compose `image:` is `…/worldtree:latest`, but local `:latest` lagged ~2wk (`21c90d5eeb2d`); CI/CD deploys NEWER SHA images, and a real pull 401s (daemon has no gitea-registry creds — that's CI/CD's). `--pull never` then used the stale latest → personal crash-looped on `ConfigurationError: echo.default_model missing` → **~2-3 min personal outage**. RECOVER: retag `:latest` to the image a HEALTHY SIBLING runs (`docker inspect worldtree-worldtree-api-1 --format '{{.Image}}'` → `docker tag …`), then `up -d --pull never`; verify `Application startup complete`. Don't manually recreate Worldtree on corviduo without pinning the correct image — image+config are CI/CD-coupled. (auto-memory `reference_corviduo_dev_emergency_ops`)
  _Archived 2026-06-19._

- `[2026-06-15]` **`.claude.json` (Claude Code config) truncates on ENOSPC → invalid JSON → CC resets to defaults on next start.** nh3-dev hit 100% disk (a peer's R19 dataset pull staged ~147G to local /home instead of the 7TB NAS); the in-flight `.claude.json` write was cut mid-key. Repair = strip the dangling token + close the open braces, validate `json.load`. CC keeps its OWN rolling backups at `~/.claude/backups/` + writes a `.claude.json.corrupted.*` snapshot when it self-detects. After ANY ENOSPC on a box running CC/agents: integrity-check `.claude.json` (json.load) AND the althing bus DB (`PRAGMA integrity_check`).
  _Archived 2026-06-19._

- `[2026-06-15]` **`pkill -f althing-light-monitor` SELF-MATCHES the killing shell** (the pattern is in the command's own argv) → kills itself mid-run (exit 144/truncated output). Stop the light-monitor via `althing-cli stop-monitor` or a captured PID — never `pkill -f <string-present-in-this-command>`. The singleton lock can also RACE to 2 live monitors during re-arm churn; keep exactly one tracked (run_in_background) monitor, and a raw `&` monitor is untracked (no harness fire-notification — don't use it).
  _Archived 2026-06-19._

- `[2026-06-14]` **Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP.** `gitea.phasefinal.com` = public `38.120.12.44` (ana-srv1); gitea is a container on ana-docker, git-SSH `10.250.50.70:222` + HTTP `:3000`. A fleet host egressing to public `:22` gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (`git fetch` times out under `set -euo pipefail`, aborts before reset). Bit irv-ml1's arbo deploy. `:22` on `10.250.50.70` is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented `docs/orientation.md` (`6e58e57`).
  _Archived 2026-06-19._

- `[2026-06-16]` **litellm 500 `Router.acompletion()/aembedding() missing 'messages'/'input'` = a request missing `Content-Type: application/json`, NOT a gateway outage.** curl `-d` defaults to form-encoding → litellm can't parse the JSON body → `data` reaches the router without `messages`/`input` → 500 (should be a 400; litellm #16993). My own diagnostic calls dropped the header → I misread it as a gateway outage and needlessly bounced the gateway ~4× chasing a phantom (image/version/config were fine throughout; a malformed UI-added "Mistral Story Eval" model in the DB was a red herring I deleted). ALWAYS send `-H "Content-Type: application/json"` testing litellm; reproduce with a header'd call before declaring a litellm incident.
  _Archived 2026-06-20._

- `[2026-06-16]` **LiteLLM admin-UI playground can't test vLLM-backed models** — it auto-sends empty `tools:[]`, vLLM 400s (litellm #6228); the gateway `strip_empty_tools` hook is a PROXY hook and structurally can't reach the UI's in-process `litellm.completion()` call. Off-ramp = `tools/gateway-chat.html`. (Langfuse playground also out: its SSRF guard blocks internal-IP LLM connections, wontfix Langfuse #13097.) (auto-memory `reference_litellm_ui_playground_vllm_deadend`)
  _Archived 2026-06-20._

- `[2026-06-16]` **A gitea Actions run that "looks like it never fired" (no image, untouched build dir) usually DID fire then skipped or failed-fast.** Bit arbo TWICE: run #3 looked-never-fired but SKIPPED (change-detection `engine=0`); run #4 looked-never-fired but FAILED in ~7s (early-step). Check the run LIST + status via claude-bot (`GET /repos/{o}/{r}/actions/tasks`) BEFORE a runner-connection/dispatch theory. CORRECTION (2026-06-16, proven): action logs ARE readable via API at the per-JOB endpoint — `GET /api/v1/repos/{o}/{r}/actions/runs/{run}/jobs` → take a job `.id` → `GET /api/v1/repos/{o}/{r}/actions/jobs/{job_id}/logs` returns the full plain-text log (claude-bot basic-auth, internal route `10.250.50.70:3000`). Only the RUN-level `/runs/{id}/logs` 404s; the per-job one works, so no UI needed. Gotcha when reading: gitea marks **every** step `conclusion:failure` once any step fails — the per-step conclusions mislead; trust the log body + step *timestamps* (the step with real duration is the culprit) not the conclusions. (Used 2026-06-16 to diagnose arbo run #5/task 1175 = test-gate `uv run pytest` → `Failed to spawn: pytest` = pytest absent from the fresh-checkout venv sync, NOT the SSH-clone comfy-dev suspected.)
  _Archived 2026-06-20._

- `[2026-06-16]` **vLLM 0.23.0 breaks Mistral MULTIMODAL at startup** (`MistralCommonImageProcessor
  has no attribute fetch_images`, #44911 regression ~0.22.1+; same class hit Mistral-3.1/
  Devstral). The dummy-image profiling crashes. Workarounds: `--limit-mm-per-prompt
  '{"image":0,"video":0}'` loads text-only; for VISION, pin **v0.22.0** (last pre-regression).
  Tradeoff: v0.22.0 *buffers* reasoning (doesn't stream reasoning_content cleanly); a newer
  build streams it but breaks vision — vision won (Mistral is the abliteration-baseline).
  _Archived 2026-06-20._

- `[2026-06-16]` **Mistral `-reasoning` TTFT variance = Triton kernel JIT-compile-during-
  inference, NOT reasoning time.** `jit_monitor` logs the spike (`_compute_slot_mapping_kernel`,
  `apply_token_bitmask_inplace_kernel` ← the tool-call/structured-output path). vLLM's startup
  warmup covers only SOME shapes; a first-hit uncovered shape JITs a kernel (seconds), then
  CACHES it — warm until container recreate (the Triton cache is IN-container, no volume, so
  recreate = cold again). Secondary: GPU idle-down to P8/180 MHz between requests. Also:
  Mistral `reasoning_effort` is BINARY (none|high only — medium/low → HTTP 400).
  _Archived 2026-06-20._

- `[2026-06-16]` **Worldtree old-baseline instances need the FULL config set synced, not the
  changeset delta.** Personal (a PRE-mistral baseline) crash-looped (41 restarts) on
  `DanglingBindingError: role 'echo' binds missing catalog entry 'qwen3.6-35-a3b-heretic'` —
  its `model_roles.yaml` (2026-06-10, never migrated) still bound the retired heretic that the
  synced v0.35.10 providers removed. The "model_roles unchanged — skip" was correct only for
  demo (already at the v0.35.9 baseline). Old-baseline → sync providers + model_roles + defaults.
  _Archived 2026-06-20._

- `[2026-06-16]` **vh is a Gitea USER, not an org → no scoped package-write for a service
  account.** Gitea user-namespace packages are owner/admin-only, so claude-bot can't push to
  `vh/arbo` without being admin. The fix is converting vh → a `corviduo` ORG (tabled). Also:
  vh's `tea` token lacks `write:admin` — create gitea users via the server-side `gitea admin
  user create` CLI (`docker exec -u git gitea …`), not the admin API.
  _Archived 2026-06-20._

- `[2026-06-16]` **`docker compose up -d` on a multi-service vLLM stack recreates ALL its
  services** (shared `.env` → all configs re-hash) → concurrent memory-profiling race
  ("No available memory for the cache blocks"). Target one: `docker compose up -d <service>`.
  (auto-memory `feedback_compose_up_recreates_whole_stack`)

_76 older entries archived to archival-memory.md._
  _Archived 2026-06-20._


- `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Only a process restart drops a config model. The INVERSE works: `/model/new` live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce.
  _Archived 2026-06-21._

- `[2026-06-19]` **the litellm `*` wildcard routed to DECOMMISSIONED llama-swap → a stale/typo'd model name returned a misleading `Connection error`, not `model not found`.** Bit a brokkr call to the renamed `qwen-image-judge`. **→ RESOLVED 2026-06-20: wildcard removed (d0eb09c) — unknown models now 404 cleanly.**
  _Archived 2026-06-21._

- `[2026-06-19]` **A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace (~3.09 GB) is the floor**, allocated OUTSIDE `gpu-memory-utilization`. FIX: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` + leave ~3.2 GB free (util <= ~0.95 on a tight MoE).
  _Archived 2026-06-21._

- `[2026-06-19]` **vLLM decode tok/s: ALWAYS discard the first generation (cold warmup).** Cold ~ 24.8 tok/s vs warm 92.7 — a cold read undersells decode ~3-4x. Measure run 2+ over a >=256-token output.
  _Archived 2026-06-21._

- `[2026-06-19]` **For full native 256K on one 96 GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE.** Stable fp8 KV already fits 262144 (hybrid SSM+attn caches KV only on attn layers → small pool). Reach for fp8 first.
  _Archived 2026-06-21._

- `[2026-06-18]` **mmartial `comfyui-nvidia-docker`: root pip installs CRASH-LOOP the container.** `docker exec -u 0 pip install` leaves root-owned files in the uid-1000 venv → the boot script (uid 1000) fails its torch step → crash loop. FIX: `chown -R 1000:1000 /comfy/mnt/venv`; pin with `DISABLE_UPGRADES=true`. (auto-memory `reference_irv_ml1_comfyui_mmartial`)
  _Archived 2026-06-21._

- `[2026-06-18]` **ComfyUI custom nodes break on version-assumption drift** — box runs 0.24.1; TeaCache imports `precompute_freqs_cis` at MODULE level (guard the LTX-only import). `pip install tensorrt` defaults to cu13 → use `tensorrt-cu12`.
  _Archived 2026-06-21._

- `[2026-06-17]` **Mistral HF→NVFP4 quant: the placement knob is the whole game.** `device_map="cpu"` (CPU-resident, sequential onload) works; `"auto"` OOMs, `max_memory` offloads to meta. Read shards with plain `read()` + `safetensors.torch.load(bytes)`, NOT `safe_open` (mmaps the 50 GB shard → ENOMEM on `/tank` ZFS).
  _Archived 2026-06-21._

- `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — no HF `Mistral4` backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first.
  _Archived 2026-06-21._

- `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree (worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea` on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur).
  _Archived 2026-06-25._

- `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell (vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3` arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1 fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager` (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 (non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` + `playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending operator OK to free.
  _Archived 2026-06-25._

- `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere.** No per-stack `logging:` limit and no docker-daemon default → ClickHouse's verbose stdout grew unbounded to ~94 GB and filled ana-docker's `/` (healthchecks then failed fleet-wide on `runc /tmp: no space left`). Lesson: on a full disk, the symptom (all-unhealthy) is the FORK-can't-write healthcheck, not the containers; find the runaway via per-container-dir `du`, not `docker system df`. The systemic fix is a daemon-default `log-opts` (`max-size`/`max-file`) — NOT just per-service patching.
  _Archived 2026-06-25._

- `[2026-06-20]` **OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk streaming is a net loss.** Measured ~2.8× (32-step) / 5.6× (16-step) on the 3090; every chunk costs ~the same regardless of length, so naive per-sentence streaming triples total gen time + starves. Use few/aggressive chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end.
  _Archived 2026-06-25._

- `[2026-06-20]` **Named buildx builders accumulate cache the standard prunes MISS.** `docker image prune -a` + `docker builder prune` don't touch `buildx_buildkit_<name>-builder0_state` volumes (per-builder buildkit state). Prune them with `docker buildx prune --builder <name> -af`, or `docker volume rm` orphaned ones (builder already removed).
  _Archived 2026-06-25._


- `[2026-06-25]` **althing "unreachable: <machine>" can MASK an app-level 500.** Raw network was
  clean; root cause = receiver DB agents-table not synced with the config roster → delivery 500'd
  "unknown to: <handle>", MAPPED to "unreachable". Diagnose: raw curl to :8087 + connect-probe ⇒ NOT
  network. Fixed in althing v0.17.1. `reference_nh3_extdev_althing_mesh`.
  _Archived 2026-07-08._

- `[2026-06-20]` **rest-server `.htpasswd: permission denied` = the ana-nas NFS mount FAILED (ghost
  file on the local mount point), NOT a decommission.** `mnt-backup.mount` stuck `failed` (fstab bare
  `defaults`) → rest-server serves an empty local dir. Recovery in disaster-recovery.md.
  _Archived 2026-07-08._

- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — but `ssh
  infra-ops@ana-docker` HAS NOPASSWD root.** A `sudo cp` as lkraven silently failed → nearly punted
  the rest-server recovery. Reach for `infra-ops@ana-docker` for sudo ops.
  _Archived 2026-07-08._
## Archived 2026-08-02 — Recent decisions (archived)

### 2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session

- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` +
  `envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal),
  YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my
  parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.)

_Archived 2026-08-02._

### 2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b

- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker
  docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB
  registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the
  2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to
  remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because
  daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never
  reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd
  opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`,
  `rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the
  frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime
  (zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares
  `live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start,
  never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers
  once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon;
  log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running
  containers).

_Archived 2026-08-02._

### 2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped

- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into
  gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17,
  vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack
  (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829);
  max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and
  `repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture
  no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone
  voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514).
  **Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not
  A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s).

_Archived 2026-08-02._

### 2026-07-09-granite-gen-memory-extractor-bind-host-synced-on

- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed,
  #335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides
  intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on
  `summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit
  both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen**
  (b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted
  `worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL
  NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1`
  after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config
  INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven
  safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed).
  StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON:
  the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never
  updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) —
  pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass`
  parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider
  description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix
  (operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING
  defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs
  when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway
  config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD;
  live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09):
  character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving
  character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping
  principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the
  dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent
  but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44,
  UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly
  from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) +
  mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's
  `load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character +
  character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window
  VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev
  to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked
  config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly
  re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard
  + character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning
  downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying
  (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37**
  (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09):
  mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx
  131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45
  canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push
  of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity
  block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev),
  monitor armed. See [[reference_corviduo_dev_emergency_ops]].

_Archived 2026-08-02._

### 2026-07-09-granite-gen-memory-extractor-bind-green-lit-for

- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).**
  Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty
  idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching).
  Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08).
  **This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the
  operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread
  01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]].

_Archived 2026-08-02._

## Archived 2026-08-02 — Tried and abandoned (archived)

### 2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s

- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized"
  is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but
  fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push
  finishes. FIX: re-run once built, or gate on `workflow_run: completed`.

_Archived 2026-08-02._

### 2026-07-04-on-prem-t1-train-that-keeps-any-ana

- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe
  ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast
  on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the
  566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`.

_Archived 2026-08-02._

### 2026-07-04-a-systemd-user-daemon-that-shells-out-to

- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools
  needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The
  althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO
  routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`.

_Archived 2026-08-02._

### 2026-07-09-fp8-breaks-morpheus-audio-token-generation

- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC
  frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks-
  TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the
  bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real.

_Archived 2026-08-02._

### 2026-07-09-sentence-chunking-tts-loses-prosody

- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that
  spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning
  (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path).

_Archived 2026-08-02._

### 2026-07-09-hf-whisper-datasets-aren-t-actually-whispered

- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8
  (not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the
  cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) —
  that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting.

_Archived 2026-08-02._


## Recent decisions (archived 2026-08-03 batch)

### 2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed

- `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug.

_Archived 2026-08-03._

### 2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt

- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section.

_Archived 2026-08-03._

### 2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning

- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`.

_Archived 2026-08-03._

### 2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code

- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]].

_Archived 2026-08-03._

### 2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause

- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528.

_Archived 2026-08-03._

### 2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway

- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1.

_Archived 2026-08-03._

### 2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s

- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot.

_Archived 2026-08-03._

### 2026-07-13-wt-355-residual-300s-hang-localized-to-our

- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]**

_Archived 2026-08-03._

### 2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree

- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize.

_Archived 2026-08-03._

### 2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min

- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate).

_Archived 2026-08-03._

### 2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on

- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated).

_Archived 2026-08-03._

### 2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree

- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev
  R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml`
  (`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347
  `session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's
  personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based
  `affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347.
  R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe
  → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string
  insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own
  yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows
  baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo
  untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they
  want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]].

_Archived 2026-08-03._

### 2026-07-10-comfyui-v0-27-1-success-on-irv-ml1

- `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1,
  SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch
  2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt`
  FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came
  only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt`
  as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any
  transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment).
  `docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB
  migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3
  needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a
  mmartial ComfyUI version bump, `git diff <old> <new> -- requirements.txt` — if torch is unpinned, bump the CODE
  without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid
  1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See
  [[reference_irv_ml1_comfyui_mmartial]].

_Archived 2026-08-03._

### 2026-07-10-comfyui-0-25-x-bump-on-irv-ml1

- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev
  requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`,
  0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the
  VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` =
  `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump
  broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch
  2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke
  the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot
  `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention),
  re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at
  `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):**
  comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention
  rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY
  the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) →
  rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1),
  lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.

_Archived 2026-08-03._

### 2026-07-10-biweekly-open-weight-releases-scan-cron-set-up

- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable
  systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00`
  America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh`
  runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in
  `~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH
  footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS
  new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode
  ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0):
  web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input +
  caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle
  brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor);
  registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired
  step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default
  + `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled
  1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending.
  **NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start
  brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session
  handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 +
  ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token
  feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo).

_Archived 2026-08-03._

### 2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b

- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD
  Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.**
  Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0
  KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf);
  (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a
  DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over
  RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp
  template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the
  live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively
  opens `<think>`/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap
  b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).**

_Archived 2026-08-03._

### 2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated

- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.**
  Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision,
  util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps
  ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled
  refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path);
  Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting
  ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway
  sampling-defaults wiring still PENDING.

_Archived 2026-08-03._

### 2026-07-08-dpo-was-silently-running-3-epochs-harness-gap

- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3
  epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass
  `num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs
  + _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the
  off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a
  ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run.
  FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig,
  float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`.
  GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min
  train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step;
  train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45
  banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0):
  ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is
  COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral
  literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift,
  arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep
  on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval +
  the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO
  adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).

_Archived 2026-08-03._

### 2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes

- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.**
  Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path
  finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied
  VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before
  `from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does
  `model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers
  5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the
  DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED
  un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO
  confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93%
  util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit.
  gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.

_Archived 2026-08-03._

### 2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved

- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.**
  Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the
  SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo
  --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank
  -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml
  --stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is
  `["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash.
  Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer`
  (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→
  immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→
  **llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→
  (more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate
  1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0)
  but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image.
  Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX,
  expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.

_Archived 2026-08-03._


## Recent decisions (archived 2026-08-05 batch)

- `[2026-07-16]` **GPU re-org: char-rp→GPU1 + both cards re-optimized for max context.** Moved char-rp (Magidonia-24B) GPU0→GPU1, then maxed context: char-rp-reasoning 150K→256K (util 0.46, 1.56x), gen→256K + seqs 16→32 (util 0.42, 5.43x), granite 64K→**128K full-chapter** (util 0.27, 1.50x). FINAL: GPU0 ~14 G reserve (both seats 256K native), GPU1 ~6.7 G headroom. All healthy. LESSON: KV must hold ≥1× max-len (util-floor crashes) + per-model KV cost varies ~8× (MoE cheap, dense pricey) → tune util empirically.
  _Archived 2026-08-05._

- `[2026-07-16]` **granite right-sized → ~10.5 GB freed on GPU1** (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536). LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped before 0.18 landed. `.env`-only, recreate `vllm-granite` alone (shared stack). [Superseded by the 07-16 GPU re-org above → 128K.]
  _Archived 2026-08-05._

- `[2026-07-15]` **image-bench eviction DONE (parked item closed).** Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM `image-judge`+`qwen-image-bench` → gen :8015 (judge samplers + thinking-off); comfy-dev pinged; backfilled the canonical char-rp-reasoning litellm block. Revert ~90 s. auto-memory `project_arbo_gen_switch_imagebench_evict`.
  _Archived 2026-08-05._

- `[2026-07-15]` **Homepage AI-tab revamp** — flat "AI Systems" group → dedicated AI tab, 6 role-based groups + AI-Dormant; committed `569e1af`, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.)
  _Archived 2026-08-05._

- `[2026-07-15]` **Home Assistant config repo created** (`vh/home-assistant-config`, private). UI-managed HA → allowlist model (YAML + curated secret-free `.storage` subset). git-in-place in `/config` on esh-docker-vm + scoped deploy key + local clone `~/development/home-assistant-config`.
  _Archived 2026-08-05._

- `[2026-07-15]` **char-rp-reasoning OOM rescue** — solo-restart on the packed GPU0 crash-looped; fixed via `expandable_segments:True` + util 0.39→0.38 + max-model-len 192K→150K. LESSON: `max-model-len` does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom.
  _Archived 2026-08-05._

- `[2026-07-15]` **soong-lab `SOONG_LAB_LIBRARY_DIR` made persistent** (corviduo-dev) — was on the redeploy-wiped code default; set to `/home/infra-ops/soong-lab-data/library`, restarted. Closed a queued no-rush item.
  _Archived 2026-08-05._

- `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost / threshold-colored ctx+rate.
  _Archived 2026-08-05._

## Tried and abandoned (archived) — moved 2026-08-12

- `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%, N=4 −20%). Reserve for dedicated/interactive deployments.
  _Archived 2026-08-12._

- `[2026-07-02]` **irv-ml1 `/worktank` ROOT is root-owned — lkraven can't write there (irv-ml1 sudo needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-<uuid>`.
  _Archived 2026-08-12._


## Recent decisions (archived)

- `[2026-07-18]` **worldtree-sdk 1.0.0 (Python) published to the internal vh Gitea PyPI** (wtsdk-dev request; the npm/TS side shipped prior session). Built from tag `python-v1.0.0` (clean worktree), `uv publish` → `https://gitea.phasefinal.com/api/packages/vh/pypi`; acceptance `uv pip install worldtree-sdk==1.0.0` (vh index as extra-index-url) resolves + imports, __version__ 1.0.0. Registry already existed (bifrost publishes there; soong-lab consumes it via `[[tool.uv.index]] name=gitea`). Publish cred = the vh `write:package` PAT the operator had already handed over (in `worldtree-sdk/.npmrc` `_authToken`) — Gitea `write:package` is package-type-agnostic, so the npm-publish token published PyPI too. Consumers install like bifrost (add the vh index + a read token). [[reference_worldtree_demo_key_mint]]
_Archived 2026-08-15._

- `[2026-07-18]` **nh3-dev /tmp auto-clean enabled** — Debian ships /tmp with no tmpfiles age (`D /tmp 1777 root root -` → never cleans); this high-churn agent box had accreted **~190k stale temp dirs / 25G**. One-shot manual purge (194k→10k entries, 25G→1.7G; deleted top-level dirs/files >1d old, spared `/tmp/claude-*` by name + anything ≤1d). Then `/etc/tmpfiles.d/tmp.conf` = `D /tmp 1777 root root 3d` (daily `systemd-tmpfiles-clean.timer` removes >3d-untouched items; active files + socket dirs spared). Tunable via the age. Note the churn: ~10k /tmp entries/day here.
_Archived 2026-08-15._

- `[2026-07-18]` **soong-lab containerize cutover — COMPLETE + LIVE on corviduo-dev.**

Migrated soong-lab (Noonien Soong character-design studio) from a hand-built
`soong-lab-studio.service` (systemd + git-pull-on-webhook) to a containerized
deploy, image built by CI + pushed to the Gitea registry. soong-dev owns the
in-repo artifacts (Dockerfile/compose/workflow/`docs/DEPLOY.md` = checklist);
infra-ops owned the host cutover. Operator confirmed functional ("Soong works
great" — a real Soong turn round-trips + saves) → cutover 100% closed.

**Final state (corviduo-dev, 10.250.50.152):**
- Container `soong-lab-soong-lab-1` LIVE + healthy on `0.0.0.0:8443`, image
  `gitea.phasefinal.com/vh/soong-lab:latest` (v0.3.24), `restart:unless-stopped`
  (survives reboot; no systemd unit needed — docker restart policy handles boot).
- Deploy dir **`/home/infra-ops/soong-lab-deploy/`** — pull-based `compose.yaml`
  (image + env_file + `8443:8443` + named volumes; NO build/secrets stanza) +
  `.env` (copied from the live `soong-lab.env`, STRIPPED of the `SOONG_LAB_*_DIR`
  overrides so the container uses image defaults `/data/library` + `/data/portraits`
  + `/app/web` → the volumes).
- Named volumes `soong-lab_soong-library` + `soong-lab_soong-portraits`, migrated
  from `/home/infra-ops/soong-lab-data/{library,portraits}` (2 saved designs incl.
  **Sindra** + 27 portraits), **chowned `10001:999`** (the container `soong` user)
  so it can read AND write new designs.
- Old `soong-lab-studio.service` + `soong-webhook.service` (the `:9010` git-pull
  redeploy listener) both **stopped + disabled**.

**Topology reality (≠ what DEPLOY.md assumed):** there is **NO TLS proxy**.
WT-personal (`:8081`) and soong-lab are **co-located on corviduo-dev**, and the
Bifrost callback is **plain-HTTP same-host** `http://10.250.50.152:8443` — the
value of `SOONG_LAB_BIFROST_ENDPOINT_URL`, unchanged by the move, so the WT
Bifrost host-allowlist stayed valid as-is. Nothing on the WT side needed touching.

**Safety net:** data backup `/home/infra-ops/soong-lab-data-backup-20260718-091831.tar.gz`
(35M) taken BEFORE migration. Verified pre-retire: `/api/version` 200 (0.3.24),
SPA `/` 200, `POST /bifrost/tool-call` → 401 (route present + auth-gated),
bidirectional WT↔soong reachability, container healthcheck green.

**Ops commands:**
- Redeploy a new image: `cd /home/infra-ops/soong-lab-deploy && sudo docker compose pull && sudo docker compose up -d`.
  (Auto-pull-on-`:latest` — watchtower or a deploy hook — is an open follow-up.)
- Rollback: `sudo docker compose down` + `sudo systemctl enable --now soong-lab-studio.service soong-webhook.service`.
- Homepage tile: manual `- Apps:` entry "Soong Lab" (href http://10.250.50.152:8443)
  in esh-docker-vm `/opt/docker/conf/homepage/services.yaml` — corviduo-dev isn't
  a Homepage-watched docker endpoint, so docker-label auto-discovery can't surface
  it (see [[2026-07-18-fleet-gitea-runner-build-recipe]] for the CI half).

See [[reference_corviduo_dev_emergency_ops]], [[reference_claude_bot_gitea_creds]].
_Archived 2026-08-15._

- `[2026-07-18]` **zonos-gateway 0.2.1 — voice-resolved emotion presets baked (provisional) from the axes sweep.**

After the axes sweep ([[reference_zonos_tts_stack]] + the `[2026-07-18] axes sweep`
Recent-decisions entry) rescued angry and confirmed startled-happy, the operator
green-lit baking the results as **provisional** gateway presets + docs. Shipped
`vh/zonos-gateway` **0.2.1** (main `8f1885b`, tag `v0.2.1`, PUSHED; deployed live
on irv-ml1 `:8890`).

**Design — voice-resolved, NOT global.** `resolve_preset(name, voice)` picks the
per-voice measured cell, because a single global preset is unsafe (dvalin ruling;
BritishFemale's *named* angry misfires as fear). Presets:
- `angry`, `happy`, `startled_happy` (+ aliases `surprised`, `startled` →
  startled_happy). All expressive (`accurate_mode:false`), cfg 1.5, pure-axes
  (no named sliders).
- Calibrated cells (the 3 default voices):
  - angry: AmF v-0.4/a+1.0 s1.0 (emo0.53/id0.685); BrF v-0.4/a+0.8 s1.0
    (emo0.99/id0.725, metric fear-clean); AmM **two-tier** — soft v-0.6/a+0.8 s1.0
    (0.23/id0.654) + drama v-0.6/a+0.8 s1.2 (1.0/id0.616 clean; strength is NOT a
    smooth knob on AmM, 1.0→1.2 is the window, past that flips to disgust).
  - happy / startled_happy: AmF v+0.6/a+0.8; AmM v+0.3/a+1.0; BrF v+0.6/a+1.0
    (happy~1.0, id 0.74-0.80; axes-happy keeps +0.15 id over the named happy slider).
  - `sad` = unchanged named-slider preset (not axes-tested).
- Uncalibrated voices (Cora + the 4 clones) → mid-region fallback until measured.
- Docs surface: `/v1/dials` exposes `voice_emotion_presets`; the FastAPI `/docs`
  description documents it; durable spec `docs/EMOTION-DIALS-SPEC.md` (moved INTO
  the repo — was mirror-only); README table. 44 tests green.

**Repo-hygiene gotcha (fixed).** The local clone `~/development/zonos-gateway` and
gitea `vh/zonos-gateway` had **TWO UNRELATED git histories** (no merge-base) — gitea
held the voice-wav commits, the local clone held the code + no remote. Reconciled
by resetting local→origin/main, overlaying the 7 bake files, `uv lock`, commit,
push (fast-forward). Voices stay tracked; local now shares gitea's lineage + has
origin wired. **The deployed irv-ml1 tree `/opt/docker/compose/zonos-gateway` is
still NON-git** (hand-updated build context) — CI-wire remains an open follow-up.

**Provisional pending** ear-validation on emotion-congruent text (the neutral-text
audition was inconclusive: "they all sound different, hard to tell"). Follow-ups:
sad axes/text pass on the 3 voices; congruent-text pass; clone-char emotion rows.
Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio}.py`
(run ON irv-ml1; scoring env `uv run --with resemblyzer --with funasr --with "numpy<2"
--with soundfile --with requests --with "setuptools<80" --with torchaudio`).
_Archived 2026-08-15._

- `[2026-07-18]` **Fleet Gitea-Actions build recipe + the `vh`-is-a-user package-write constraint** (learned the hard way across 3 failed soong-lab validation builds; reusable for ANY fleet CI image build or package publish).

**The runner.** One `act_runner` (`gitea/act_runner`) on ana-docker, labels
`pfi-fleet` / `ana-docker` → both map to job image **`node:20-bookworm-slim`**,
which has **NO docker and NO git**. Config `/opt/docker/conf/gitea-runner/data/config.yaml`:
`valid_volumes: []` (no socket propagated to job containers). So:
- `actions/checkout@v4` fails (needs git); `docker/*` marketplace actions fail
  (need docker) — a workflow built on those dies at the first step (~15s).

**The working recipe (mirror Worldtree `deploy.yml`).** Run the job in a
docker-capable image + drive docker with RAW commands, not the JS actions:
```yaml
runs-on: pfi-fleet
container:
  image: docker:24.0.7-cli        # has docker+buildx; add git+node
steps:
  - run: apk add --no-cache git nodejs   # so actions/checkout@v4 works
  - uses: actions/checkout@v4
  - name: login   # RAW, not docker/login-action
    run: echo "$REGISTRY_TOKEN" | docker login gitea.phasefinal.com -u "$REGISTRY_USER" --password-stdin
  - name: buildx builder
    run: docker buildx create --name X --driver docker-container --use; docker buildx inspect --bootstrap
  - name: build+push   # RAW, not docker/build-push-action
    run: docker buildx build --secret id=<name>,env=<TOKEN> -t <img>:latest --push .
```
The runner mounts the host docker socket into ITSELF; the docker:cli job reaches
the daemon through that. The `docker/*` JS actions are unreliable on act_runner —
raw commands are the fleet convention.

**`vh` is a USER account, not an org.** Consequences that bit repeatedly:
1. `GET /api/v1/orgs/vh` → 404 "user redirect"; there are **no org teams** to add
   a service account to.
2. **User-owned packages are OWNER-WRITE-ONLY.** claude-bot (even repo
   admin-*collaborator* on `vh/soong-lab`, even with `write:package` scope + full
   basic-auth) gets **`401 unauthorized`** on `docker push` to `vh/soong-lab`, and
   `npm publish` to `vh/npm/` would 401 too. Only `vh` itself can write vh packages.
   → CI must authenticate AS `vh` for the push (a vh-owned `write:package` PAT as
   `REGISTRY_TOKEN` + `REGISTRY_USER=vh`), exactly how WT pushes `vh/worldtree`.
   claude-bot CAN still: clone/read repos, READ packages (pulled the image fine),
   dispatch workflows, mint demo Worldtree keys.
3. **Repo Actions secrets are OWNER-ONLY too** — `PUT .../actions/secrets/X` as
   claude-bot (repo admin-collab) → 403 "user should be the owner of the repo".
   Only `vh` can set a repo's secrets.

**Other gotchas:**
- Gitea **reserves the `GITEA_` secret-name prefix** — a secret named
  `GITEA_PYPI_TOKEN` is illegal; use e.g. `PYPI_TOKEN`.
- Gitea **package auth is token-based / username-lenient** — `docker login` /
  PyPI basic-auth authenticate via the token; the username is nominal (tested
  `-u gitea` and `-u claude-bot` both 200 against the vh PyPI). So a Dockerfile
  hardcoding `UV_INDEX_GITEA_USERNAME=gitea` is fine with any valid token.
- Homepage (esh-docker-vm) docker-label auto-discovery only covers the 5 endpoints
  in its `docker.yaml` (esh-vm-docker, ana-docker, ana-ml2, nh3-docker, irv-ml1);
  **corviduo-dev is NOT watched** → services there need a manual `services.yaml`
  entry, not labels.

Applied in the soong-lab CI: [[2026-07-18-soong-lab-containerize-cutover]].
_Archived 2026-08-15._

- `[2026-07-18]` **Peer credential provisions — Wyrd conv-api key + wtsdk npm token, both delivered + closed.** Wyrd: demo Worldtree user-tier key (key_id `da7a0bdf`, user_id `wyrd-dev`) minted via `docker exec worldtree-worldtree-api-1 /admin/keys` (omit tier→user), drop-and-shred delivery. wtsdk: operator-minted vh `write:package` PAT relayed drop-and-shred → worldtree-sdk@1.0.0 published to `vh/npm/`. Secret-delivery pattern = drop to a mode-600 file on the peer's box, they collect+shred+confirm, then shred the holding copy; NEVER cleartext over althing. [[reference_worldtree_demo_key_mint]]
_Archived 2026-08-15._

- `[2026-07-18]` **Axes sweep RESCUED angry; surprised-class dead but startled-happy ships.** Valence×arousal grid on the 3 calibrated defaults (AmericanFemale/Male, BritishFemale), exp/cfg1.5/strength1.0, 84 clips, emotion2vec + resemblyzer scored, graded vs dvalin's floor. **ANGRY rescued** (named direction was 0.004–0.15, British named-angry even misfired as fear 0.89): axes ship cells at **negative valence (−0.4..−0.8) + high arousal (+0.8..+1.0)** — BritishFemale v-0.4/a+0.8 angry=0.99/id0.725 SHIP, AmericanFemale v-0.4/a+1.0 angry=0.53/id0.685 SHIP; AmericanMale two-tier post-ladder (no single ship cell — best drama = v-0.6/a+0.8 str1.2 angry=1.0/id0.616 clean, soft = same cell str1.0 angry0.23/id0.654; cell A v-0.6/a+1.0 is a non-monotonic minefield, skip). BrF ship cell proxy-CLEAN of fear (str<1.0 just kills anger). **SURPRISED-class DEAD** (max 0.047 across all 84 cells) but **startled-happy** (happy-proxy) ships all 3 at high arousal + neutral/positive valence, with a **+0.17–0.20 identity LIFT** over the named-surprised route (named hits happy~1.0 but at id0.57–0.61, under floor; axes hits happy~1.0 at id0.74–0.80). Bonus: axes-happy retains ~0.10–0.15 more identity than the named happy slider too. Caveats: response surface non-monotonic/sharp-thresholded; angry region borders fear/disgust (bleed); emotion2vec saturates at 1.0 (needs ear-confirm); neutral text understates. Tooling `~/development/zonos-tools/axes_sweep.py`; per-clip JSON was `irv-ml1:/tmp/axes_sweep_results.json` (ephemeral). Sent dvalin msg `01KXT2ZB8G…`. NEXT = operator ear-confirm → bake presets. [[reference_zonos_tts_stack]]
_Archived 2026-08-15._

- `[2026-07-18]` **Zonos2 emotion CANONICAL from an empirical sweep + the voice-cloning pipeline.**

**Voice-cloning pipeline (established this session).** Source zips at
`/mnt/smithy/voice_clones/<name>.zip` (irv-ml1 NFS from nh3-nas; remount
post-reboot) — each = diarized single-speaker podcast clips + `manifest.jsonl`
(per-clip WhisperX `mean_score`, word timestamps, text) + `metadata.csv`.
`~/development/zonos-tools/assemble_voice.py <dir>` ranks by mean_score and
concatenates top clips to ~15–24s (Zyphra's blessed clone-ref length; single
clip if already ≥15s). Drop the assembled `<Name>.wav` into the gateway voices
dir → `voice:"name"`. 4 characters cloned: **Emmie, Penny, Natalie, Miranda**
(+ Zyphra defaults AmericanFemale/Male/British/Cora) = 8 voices in
`zonos-gateway`. Clone is inline `speaker_audio_base64` (text-independent Qwen3
speaker embedding — NO transcript); `/tts/speakers` registration is
session-scoped (needs `X-TTS-Session-ID`), so the gateway holds the ref wav and
clones per-call.

**Gateway voices are host-managed (bind-mount, added this session).** Added
`./voices:/app/voices:ro` to `/opt/docker/compose/zonos-gateway/compose.yaml`
(committed to `vh/zonos-gateway` + eshpfi mirror `438cd35`). So adding a voice =
drop the wav + `docker compose restart zonos-gateway` (registry rebuilds at
boot; NO image rebuild). This also un-stranded the other voices (deploy build
context had only Cora before). Voice wavs committed to the repo for backup.

**Emotion mechanism (Zyphra canonical, from their README @194c0a3).** Additive
direction vectors: 4 named (happy/sad/angry/surprised) + valence/arousal axes.
`emotion_strength` 1.0 = per-voice calibrated (calibration.json optimizes
emotion2vec recognizability only, NOT identity). `accurate_mode` is THE trade-off:
`true` = closer voice match (identity), `false` = expressive mode (emotion lands,
identity drifts). Zyphra's strong recipe: `accurate_mode:false` + `cfg~1.5`.
Single-emotion is blessed; mixing is unblessed (and degrades the clone — operator
confirmed by ear). "deaf by 1.5" — cfg past 1.5 distorts + costs ~2× compute.

**THE SWEEP (`~/development/zonos-tools/emotion_sweep.py`).** 4 cloned voices × 4
named emotions × {accurate,expressive}×{cfg 1.0,1.3,1.5} @ strength 1.0,
single-emotion, neutral sentence + a neutral baseline per voice (~100 clips).
Scored on TWO axes: **emotion-landing** = emotion2vec `iic/emotion2vec_plus_large`
target-emotion prob [0-1]; **identity** = resemblyzer speaker-embedding cosine vs
the clone reference (neutral baseline ~0.85). Scoring env:
`uv run --with resemblyzer --with funasr --with "numpy<2" --with soundfile
--with requests --with "setuptools<80" --with torchaudio` (setuptools<80 for
webrtcvad's pkg_resources; torchaudio for funasr).

**RESULTS (mean across the 4 voices) — emotion, best setting, emo/id:**
- happy — **exp cfg1.5** 0.80/0.68 (soft: exp cfg1.0 0.76/0.69) → WORKS
- sad — **exp cfg1.5** 0.53/0.57 (only working cell; id below the ~0.65 floor) → modest
- angry — acc cfg1.3 / exp cfg1.5 tied at ~0.25 emo → WEAK (named ceiling ~0.25)
- surprised — max ~0.015 across ALL settings → NON-FUNCTIONAL on the named direction
Accurate + low cfg = identity/suppress regime (emo→0); expressive REQUIRED for
emotion to land, at ~0.15–0.28 identity cost.

**dvalin-smithy-dev synthesis (adopted, triaged genuine-adds; thread
`01KXT12FN0AS5A3WMKEK06BVPS`):**
1. Treat **identity as a hard FLOOR (~0.65)**, not a free variable in emo×id.
2. **Two-regime policy** — Regime A (default, identity-critical dialogue):
   `accurate_mode:true, cfg 1.0, emotion off` (text carries it) or soft-happy
   (exp cfg1.0). Regime B (tagged drama beats): `accurate_mode:false, cfg 1.5`,
   single emotion or axes. Line-type→regime heuristic (exposition→A, grief→B+sad,
   confrontation→B+axes-angry, shock→B+axes-arousal).
3. **Axes-first for the broken emotions** — angry ≈ valence −0.6..−0.8 / arousal
   +0.5..+0.8; surprised ≈ valence +0.2..+0.4 / arousal +0.7..+1.0 (exp cfg1.5);
   or "startled-happy" (happy + high arousal) as a surprised stand-in. These are
   PROVISIONAL — the sweep did NOT test axes.

**NEXT (highest VoI, operator to green-light):** an **axes sweep** for
angry/surprised (valence×arousal grid) — the only path to rescue the two broken
named emotions; then a strength ladder at the best cells + emotion-congruent text
(neutral content understates landing) + per-voice tables + a 2nd emotion judge /
human pairwise. Then bake the happy/sad canonical into gateway presets. I owe
dvalin the axes-sweep numbers.

See [[reference_zonos_tts_stack]]; dials-first spec at `vh/zonos-gateway`
`docs/EMOTION-DIALS-SPEC.md`.
_Archived 2026-08-15._

- `[2026-07-18]` **yt-voice-clipper: A6000-pin fix + v0.3.3 redeploy.** Fixed a latent misconfig — the host override *said* "pin worker to A6000" but `NVIDIA_VISIBLE_DEVICES` was `"0"` (the 3090); re-pinned worker+api to the A6000 by UUID (`GPU-9672f0d5`, 3090 is zonos2's). Then redeployed api+worker to v0.3.3 (`docker compose up -d --build`; SPA+Python; `max_gap` 0.6→1.2s; stderr surfaced in job.log). A6000 + version verified; yields test in-flight (job `f3ff746dbae9494d`). yt-voice-clipper-dev thread `01KXT0T6GYHB`. [[reference_ytvc_autodeploy]]
_Archived 2026-08-15._

- `[2026-07-17]` **Worldtree #365 internal-comms config CLOSED (demo+personal → b125) + WT#368 cross-agent memory-leak forensics + PERSONAL agent-memory scrub.** #365: staged the internal-tiers/rules/gate on both instances' bind-mounts (byte-exact vs baked b125), both now live on b125. WT#368 (read-only): the operator's name was in NO recall store on demo; on PERSONAL it sat in `lofn.chroma` (old-code `saga-v1` seeding + legacy contamination), and a clean-slate marker test proved **current b125 code isolates character-session extraction correctly** — the leak is legacy data, not a live bug. Operator-directed → executed a full PERSONAL agent-memory scrub (backup `/opt/worldtree-personal/agent-memory-backup-20260717-181004.tar.gz`; conversations/mood/auth preserved). worldtree-dev owns the code-fix/data contract. [[reference_corviduo_dev_emergency_ops]]
_Archived 2026-08-15._

- `[2026-07-17]` **Zonos emotion levers RESOLVED: text-priming is FLAT → the working lever is ZONOS2's native emotion-steering, which the gateway ALREADY exposes as presets.** The prosody-priming A/B (prime→generate→excise, silence-gap cut, parakeet-validated) was operator-judged FLAT on this checkpoint — text doesn't move it. Native `emotion_directions/` (happy/sad/angry/surprised + valence/arousal axes, per-speaker calibrated for AmericanFemale/Male/British) clearly WORKS (sad→slow/quiet, excited→fast/bright, etc.). **`zonos-gateway:0.2.0` (:8890) already wires it**: simplest caller path = `POST /v1/audio/speech {preset:"…"}` — presets neutral/warm/excited/sad/intense/whisper (defined in `~/zonos-gateway/src/zonos_gateway/dials.py`), reached via the **LiteLLM `ext-tts` alias** (engine-neutral swap point; consumers never call the gateway by name). RTF measured on 3090: cfg1.0 steering = FREE (~0.52 = neutral, additive vectors), cfg1.5 amplified ~0.625 (~+20%, still realtime). Captured the live gateway stack → `stacks/zonos-gateway/` (compose+env+README); ⚠️ gateway SOURCE at `~/zonos-gateway` on irv-ml1 is NOT in gitea (backup gap, follow-up); `stacks/zonos` (v0.1 Gradio) marked DEAD/superseded. Whisper is a composed preset (no whisper *direction*; escalation for hard affects = custom directions via `scripts/build_emotion_directions.py` or emotional-ref cloning `speaker_audio_base64`). Harnesses in scratchpad (not yet landed). [[reference_zonos_tts_stack]]
_Archived 2026-08-15._

- `[2026-07-17]` **Zonos2 `:1920` engine → self-contained container (stays on 3090); prosody-priming is a SERVING-LAYER change (engine stays stock).**

**Context.** The production Zonos TTS engine (irv-ml1 `:1920`, feeds asset-engine + gateway-chat via `zonos-gateway` :8890) was a bare native process — its real launch config existed ONLY in the running process argv (the committed `~/tts-audition/harness/zonos_server.sh` was STALE: said A6000/:1919/no perf flags; live is 3090/:1920 with `--cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3`). Captured to eshpfi `stacks/zonos-engine/` (README + corrected `zonos2-server.sh` + `.env.example`), commit **14a0004** (UNPUSHED as of the snapshot).

**Decision 1 — containerize as a SELF-CONTAINED image** (not systemd — operator rejected; not a thin bind-mount wrapper — I walked that back: bind-mounting the host's CUDA-compiled `.venv` couples to the host's exact CUDA/glibc and is fragile + not reproducible). Shape: `FROM` a CUDA 12.8 base → `uv sync` against the repo's committed `uv.lock` (deterministic env) → mount the ~15 GB HF weights (`~/.cache/huggingface/hub/models--Zyphra--ZONOS2`, do NOT bake) → pin the **3090** (`NVIDIA_VISIBLE_DEVICES=0`) → `restart: unless-stopped` → CMD = the captured invocation. **Engine stays STOCK** Zyphra/Zonos2 @ commit `194c0a3` (no fork — the `zonos2` package ships its own server). **Build risk:** heavy compiled-CUDA deps (flashinfer / sgl_kernel / cutlass-dsl / apache-tvm-ffi / pynini) on torch 2.9.1+cu128 — mostly prebuilt wheels + the `uv.lock` make it tractable, expect a couple build iterations. **Cutover (in place on the 3090):** stop the native process (frees ~17 GB) → `docker compose up -d` (re-allocates ~17 GB, same footprint) → repoint `zonos-gateway`'s `ZONOS_URL` at the container (or keep the `:1920` host-port publish). One brief prod-TTS blip.

**GPU = 3090 (operator 2026-07-17).** Keep it OFF the A6000 — the A6000 already OOMs under ComfyUI load (idle ~19 GB but spikes far higher during gen), so it can't host Zonos too. The 3090 already runs Zonos, so the containerize-in-place cutover changes nothing about placement.

**Decision 2 — the prosody-priming hypothesis (operator's test; the reason for building fresh).** PRIME the autoregressive engine with an emotional sentence, then TRUNCATE it from delivery: prepend a primer → **generate "primer + real text" as ONE continuous utterance** (the AR model carries prosody forward across the boundary) → ASR-timestamp the primer's end (**parakeet**, already up on irv-ml1 `:8765`, word timestamps) → **clip the primer in the inter-sentence silence gap** (+ ~15 ms fade-in, no click) → deliver only the real text, now wearing the primed prosody. Examples: primer "I'm so EXCITED about this." → "This will be a lot of fun!" spoken excited; primer "I'm whispering this to you right now." → "I'm so glad to see you baby." whispered. **This is PURE serving-layer orchestration — the engine is untouched; it lives in the gateway adapter `stacks/zonos/adapter/server.py`.** Only fork the engine if the black-box approach fails.

**THE CRUX the test resolves:** does AR prosody actually **carry across the sentence boundary**, or does Zonos reset at the period? → the harness A/Bs the **JOIN punctuation**: period (operator's examples) vs comma vs ellipsis vs none ("…excited about this, this will be…"). Everything else is plumbing.

**Plan / design recs.** (a) Build the stock engine image (parallel track). (b) Stand up a priming TEST HARNESS against the NATIVE engine (fast iteration, seconds) + parakeet ASR: prime→generate→timestamp→gap-clip→out; compare primed-clipped vs plain on the two cases (subjective + a cheap objective proxy: pitch/energy variance for "excited", spectral-tilt/low-energy for "whisper"). Iterate on the join, then bake the winner into the gateway adapter. **Primer source:** caller-supplied for the harness (test arbitrary primers) → a curated emotion→primer library (`excited`/`whisper`/…) + optional caller override for production. **ASR:** parakeet primary; WhisperX forced-align fallback if parakeet word timestamps are coarse.

See eshpfi `stacks/zonos-engine/README.md` + `stacks/zonos/` (the gateway adapter).
_Archived 2026-08-15._

- `[2026-07-15]` **esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service`** (the prior `After=remote-fs.target` drop-in was silently defeated by `nofail`). Reached only after a REBOOT (D-state phantom containers uptime-kuma + paperless-web that no `docker`/`ctr`/daemon-restart could clear). Committed `21d9a07` + playbook updated. See Tried and abandoned.
_Archived 2026-08-15._

## Tried and abandoned (archived)

- `[2026-07-15]` **`docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts** — `nofail` drops a mount out of remote-fs.target's blocking set, so the drop-in ordering is silently defeated (paperless still Exited(255) on reboot). Real fix = DIRECT mount->docker ordering via the fstab `x-systemd.before=docker.service` option (verify `systemctl show docker -p After` lists the mnt-*.mount units). esh-docker-vm.
_Archived 2026-08-15._

- `[2026-07-15]` **The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT** — reconfirmed: `docker stop/rm -f`, `ctr -n moby task delete`, AND `systemctl restart docker` all fail to clear it; `docker exec` into a wedged container ALSO fails (`setns ... exit status 1`), so the in-place restart escape hatch is out. Worse, a daemon restart can HALF-KILL other healthy containers (knocked paperless's granian down + left it wedged). Process dead but dockerd won't reap -> phantom. NFS mounts are `_netdev,nofail` so the reboot is boot-safe.
_Archived 2026-08-15._

- `[2026-07-15]` **vLLM `max-model-len` does NOT free GPU VRAM** — the KV cache POOL is sized by `gpu-memory-utilization`, not max-model-len. Lowering max-model-len only caps per-request context + drops max concurrency; the pool still fills the util budget. To actually free VRAM, lower `gpu-memory-utilization`. (Bit the char-rp-reasoning "drop KV to 150K" ask: the 150K applied but freed 0 VRAM until util dropped 0.39->0.38.)
_Archived 2026-08-15._

- `[2026-07-15]` **Claude Code statusline `.cost.total_cost_usd` is per-SESSION** (Claude Code's own cache/model-aware session accounting), not a lifetime aggregate — the large value just reflects a long, multiple-times-summarized session. And the old statusline hardcoded Sonnet pricing ($3/$15) on an Opus session -> ~5x cost understatement.
_Archived 2026-08-15._

- `[2026-07-14]` **MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround.** All crashed the same way (`qwen3_5_mtp.py:256` `param_data.shape == loaded_weight.shape` AssertionError — bf16 mtp head loaded into a quantized drafter param): (1) mtp excludes in `config.json` (WRONG file — vLLM modelopt reads `hf_quant_config.json`); (2) specific-unfused mtp names in hf_quant_config; (3) wildcards `mtp*`/`mtp.layers.0*` (`is_layer_skipped` is EXACT-membership, NOT glob — wildcards match nothing); (4) exact fused+unfused names in both `mtp.`/`model.` prefixes. Instrumenting `is_layer_skipped` proved the drafter's exclude list holds ONLY the main model's `linear_attn` entries — the mtp excludes never reach the draft-model quant config. ONLY fix = a mounted `sitecustomize` force-skipping `mtp.*`. LESSON: don't chase checkpoint-config fixes for the mtp-drafter crash; go straight to the runtime patch. Also `nvidia-modelopt[hf]==0.43` (AEON's producer version) is a trap — it pins transformers back to 4.57 which can't load `qwen3_5` at all; use 0.45 + the FusedMoE guard in `quant_modelopt.py`.
_Archived 2026-08-15._

- `[2026-07-14]` **AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof.** `vllm-aeon-rp`'s .env → `AEON_RP_MODEL=pantheon-27b-mtp-nvfp4`, `AEON_RP_QUANT=compressed-tensors` — it LOADED (mtp silently skipped, `exited 0`) but never accelerated. Same vLLM image (`:latest` = `sha256:4091d55` = 0.24.0) as the failed Heretic2 test, so the "AEON ran on an older vLLM" theory was wrong. Don't treat a seat that "ran" as MTP-validated without checking its `SpecDecoding` acceptance.
_Archived 2026-08-15._

- `[2026-07-14]` **NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept.** Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp weights off a compressed-tensors main model → `Parameter … not found in params_dict`, `Avg Draft acceptance rate: 0.0%`). Pantheon is identical — its "working NVFP4+MTP" was working *structure*, never real acceleration. Working native MTP needs the **modelopt** main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt.
_Archived 2026-08-15._

- `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state.
_Archived 2026-08-15._

- `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip.
_Archived 2026-08-15._

- `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny '<ip>'` line) AND an instrumented receiver.
_Archived 2026-08-15._

- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction.
_Archived 2026-08-15._

- `[2026-07-13]` `althing-cli reply <THREAD_id>` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to <peer>`. Bit me several times this session.
_Archived 2026-08-15._

- `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter,
  `has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs.
  (`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.)
_Archived 2026-08-15._

- `[2026-07-09]` **Per-frame CPU SNAC decode is too slow for streaming** — per-call overhead × ~60 frames serialized
  → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx]
  window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).
_Archived 2026-08-15._

- `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4
  (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY
  (temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models.
  Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going **GGUF (llama.cpp)** to sidestep the
  whole NVFP4-quant surface.
_Archived 2026-08-15._

- `[2026-07-08]` **Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24).** (a) HF `tokenizer.json`
  for Mistral = **GARBAGE output** — the card's "use the official Mistral tokenizer" warning is REAL; must use the
  `tekken.json`/mistral tokenizer. (b) BUT `--tokenizer-mode mistral` + vision **CRASHES** (`Failed to apply
  PixtralProcessor on {'text': '[IMG]'}`; and with tekken.json present in auto mode, `CachedMistralCommonBackend has
  no attribute is_fast`). So it's **mistral-tokenizer OR vision, not both** on this vLLM. Text-only + mistral
  tokenizer serves clean (`--limit-mm-per-prompt '{"image": 0}'`). **GGUF/llama.cpp avoids all of this** (native
  mistral tokenizer + vision).
_Archived 2026-08-15._

- `[2026-07-08]` **Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base.** The base
  (`llmfan46 heretic`) writes freely (thinking-off), but Gryphe distilled the reasoning traces from **DeepSeek 3.2**
  (safety-aligned) onto every turn (`preserve_thinking:true`) → the model reasons ITSELF into refusals in the
  `<think>` phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified).
  **Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw
  abliterated base is cleaner** — this is WHY the pivot went to the llmfan46 heretic base for gen.
_Archived 2026-08-15._

- `[2026-07-08]` **Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance.** MTP is a separate **bf16** head
  (`mtp.*`, in `model-auxiliary.safetensors`, 15 tensors); AEON preserved it by INJECTING the bf16 head into the
  quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors
  main + injected bf16 mtp + `text_config.mtp_num_hidden_layers=1` → vLLM detected the MTP but SKIPPED the bf16
  self_attn weights → 0/192 draft tokens accepted. **The bf16 MTP head only loads on the MODELOPT main-model format
  (like AEON), not compressed-tensors.** (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.)
_Archived 2026-08-15._

- `[2026-07-07]` **vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639).** Adapter loads HTTP 200
  but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate `zc`
  adapter — correct per vLLM's `check_unexpected_modules` allowlist — loads clean but inert; the fused-key
  rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of
  `expand_packed_lora` — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no
  version-bump helps. Merge bakes deltas in (bypasses this) but is static.
_Archived 2026-08-15._

- `[2026-07-07]` **SGLang generic image can't LOAD our NVFP4 AEON** — ModelOptModelLoader weight-shape/
  packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated
  `qwen36-27b-nvfp4` dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was
  quant-loader-specific).
_Archived 2026-08-15._

- `[2026-07-07]` **SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for**
  (invalid choice: 'in_proj_qkv'); `'all'` resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants
  its OWN packed layout (base r16 + `get_stacked_multiply=3`, NOT a pre-fused rank-48 qkv → the [48]-vs-[144]
  shape assert). A THIRD adapter format; version-exact source needed (`:latest`=0.5.13, NOT `main`).
_Archived 2026-08-15._

- `[2026-07-07]` **Engine invocation footguns cost several wasted serve-bounces this session** — `docker run
  --rm` ate crash logs; duplicated `serve` (vLLM image entrypoint is already `["vllm","serve"]`);
  `--max-lora-rank 48` invalid (choices 1/8/16/32/64… → use 64); parens in `echo` inside `ssh host -c "…"`
  break the remote shell. LESSON: verify engine launch flags (`--help`, GPU-free) + never `--rm` a container
  whose crash logs you need, BEFORE bouncing a production serve.
_Archived 2026-08-15._

- `[2026-07-04]` **LiteLLM (this gateway version) mutates the SHARED deployment config in-place on
  per-request sampler-param merge** → my deliberately-invalid `top_k=-5` forwarding-probe bled into a
  param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT
  caching (none configured), NOT a config change. **Never fire invalid/distinctive sampler values at
  a SHARED gateway alias with live consumers** — use a throwaway alias, or a `docker restart litellm`
  flushes residual carryover. `feedback_litellm_shared_param_mutation`.
_Archived 2026-08-15._


## Recent decisions (archived 2026-08-16 batch)

- `[2026-07-15]` **arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake.** Operator-directed full switch (comfy-dev executed, live in prod). Established: gen (`qwen3.6-35b-a3b-heretic`) is vision-enabled and was image-bench's predecessor as arbo's hero-judge; image-judge actually serves 4 roles (vision quality-scoring + identity-scoring + bbox grounding + an uncensored text tier), not just grounding. comfy-dev spot-check: gen faster on every task, grounding within ~3px, uncensoring preserved, and it FIXED a bug (image-judge's reasoning preamble broke json_object + stalled the router). Sequencing = short prod bake then evict (~30 GB GPU1 reclaim); revert = flip `ARBO_VISION_MODEL`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`.
  _Archived 2026-08-16._

- `[2026-07-18]` **soong-lab auto-redeploy — DONE + VALIDATED** (was approved/queued; executed same day on fresh context — see AS-BUILT at the bottom).

Vuong approved wiring auto-redeploy for soong-lab (relayed via soong-dev, thread
`01KXT3A6C3908TA4V9THV3AMH7`): new images should go live on corviduo-dev without
the manual `docker compose pull && up -d`. Host-side implementation is infra-ops's
lane; mechanism is infra-ops's call per fleet conventions. Operator deferred
execution — "we'll do soong on fresh context."

**Chosen mechanism (recommended, agrees with soong-dev): Worldtree-style
CI-deploy step** — NOT watchtower polling.
- Add a deploy job/step to soong-lab's `.gitea/workflows/build-and-push.yml` that,
  after the build+push job succeeds, **SSHes from the pfi-fleet runner to
  corviduo-dev** and runs `cd /home/infra-ops/soong-lab-deploy && docker compose
  pull && docker compose up -d`, then a **health-gate** (`curl -fsS
  http://localhost:8443/api/version`).
- This is exactly how WT deploys the demo instance to the SAME host: see
  `~/development/Worldtree/.gitea/workflows/deploy.yml` — the "Deploy to demo VM +
  health-gate" step uses `secrets.DEMO_VM_SSH_KEY` / `DEMO_VM_HOST` / `DEMO_VM_USER`.
  Explicit-over-implicit (visible in the run log, fires exactly on build success),
  one less always-on service than watchtower.

**Constraints (from soong-dev):** deploy on CI success only; keep the trigger
gated to `v*` tags + `workflow_dispatch` (as today); preserve the one-command
rollback posture (`docker compose down` / pin a previous tag).

**BLOCKER — needs from vh (owner-only):** a **runner→corviduo-dev deploy SSH key**
as a repo secret (+ host/user), same class as WT's `DEMO_VM_SSH_KEY`. Likely
**reuse WT's existing demo-deploy key** (WT's runner already SSHes to 10.250.50.152
as its deploy user). Repo secrets are vh-owner-only (see
[[2026-07-18-fleet-gitea-runner-build-recipe]]).

**Next-session steps:** (1) confirm/obtain the deploy SSH-key secret from vh (reuse
WT's or mint fresh); (2) add the deploy job to build-and-push.yml (infra-ops has
push on vh/soong-lab); (3) dispatch a build to verify it deploys + health-gates;
(4) ping soong-dev so they sync DEPLOY.md's "open follow-up" note to the as-built
mechanism. Auto-pull (watchtower) explicitly NOT chosen. See
[[2026-07-18-soong-lab-containerize-cutover]].

## AS-BUILT (2026-07-18, same-day execution)

**Mechanism landed** exactly as planned: `build-and-push.yml` gained a `Deploy to
corviduo-dev + health-gate` step (after build+push) that SSHes the host as `deploy`
and runs `docker compose pull && up -d` from `/opt/soong-lab`, then polls
`http://localhost:8443/api/version` for 120s and fails the job loud if unhealthy. No
compose is shipped from CI (the in-repo `docker-compose.yml` is a BUILD compose; the
host pull-compose is infra-ops-managed). Kept the `v*`-tag/`workflow_dispatch` trigger.
Skipped WT's disk-watermark gate + health-gated-`:latest`-advance (low cadence, easy
rollback).

**Deploy identity = reuse WT's `deploy` account** (operator accepted the rec):
- `deploy` (uid 1001, docker-group → no sudo) already owns `/opt/worldtree`; relocated
  soong-lab's deploy dir `/home/infra-ops/soong-lab-deploy` → **`/opt/soong-lab`**
  (deploy-owned), copied compose + `.env`. Named volumes (`soong-lab_soong-library`,
  `soong-lab_soong-portraits`) are project-scoped by compose `name: soong-lab` → followed
  the move untouched (dry-run `up -d` ADOPTED the running container, no recreate). Old dir
  **retired → `.retired-20260718`** (recoverable). Also lingering: `soong-lab-deploy.sh` /
  `.log` (dead pre-container webhook artifacts) — harmless, left in place.
- **Dedicated soong-only ed25519 deploy key** minted (NOT literally WT's key — cleaner
  independent revocation), pubkey appended to `deploy`'s `authorized_keys`
  (fp `SHA256:MG7M3RiZJ176sLfblffb96V6W1qkRTgJ5dow1CpiY68`). Existing `deploy` key is
  plain/unrestricted, so parity held.

**The secret gate (the friction point):** repo Actions secrets are **vh-owner-only** —
claude-bot's token is `write:package,read:repository` (403 on secret-write), and the vh
package-scoped PAT also 403'd on `PUT …/actions/secrets/…`. So `DEPLOY_SSH_KEY` /
`DEPLOY_HOST` (10.250.50.152) / `DEPLOY_USER` (deploy) HAD to be set by the operator.
First operator attempt produced a **bad key paste** — the deploy step died with
`Load key … error in libcrypto` + `Permission denied (publickey)` (build+push were green;
live Soong never moved). Fix: operator re-set the secret; the minted key path was
pre-validated from nh3-dev (`ssh -i … deploy@… 'cd /opt/soong-lab && docker compose config
-q'` → OK, health 200) so the re-set was the only variable.

**Validation:** `workflow_dispatch` via claude-bot **basic auth** (its token lacks
`write:repository` for the dispatch API; the account password works). Run #5 (task 1886)
GREEN — live container recreated `sha256:…541f7730` → `…07526a08`, `StartedAt` fresh,
health 200. `/api/version` now reports **0.3.25** (run #5 shipped soong-dev's 1c2f831
STYLE_WORKFLOWS re-pin as validation cargo). soong-dev synced `docs/DEPLOY.md`
(commit `00b67c3`). NB: tag **v0.3.25 exists only locally** — pushing it would re-trigger
a redundant build+deploy of the same commit (operator's discretion).

**Ops now:** redeploy = tag `v*` or `workflow_dispatch` the CI (auto). Manual fallback =
`sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`
(the `.env` is `deploy`-owned 600, so infra-ops needs `sudo -u deploy`, not a bare `cd`).
  _Archived 2026-08-16._

- `[2026-07-18]` **soong-lab auto-redeploy — DONE + VALIDATED** (was approved/queued; executed same day on fresh context — see AS-BUILT at the bottom).

Vuong approved wiring auto-redeploy for soong-lab (relayed via soong-dev, thread
`01KXT3A6C3908TA4V9THV3AMH7`): new images should go live on corviduo-dev without
the manual `docker compose pull && up -d`. Host-side implementation is infra-ops's
lane; mechanism is infra-ops's call per fleet conventions. Operator deferred
execution — "we'll do soong on fresh context."

**Chosen mechanism (recommended, agrees with soong-dev): Worldtree-style
CI-deploy step** — NOT watchtower polling.
- Add a deploy job/step to soong-lab's `.gitea/workflows/build-and-push.yml` that,
  after the build+push job succeeds, **SSHes from the pfi-fleet runner to
  corviduo-dev** and runs `cd /home/infra-ops/soong-lab-deploy && docker compose
  pull && docker compose up -d`, then a **health-gate** (`curl -fsS
  http://localhost:8443/api/version`).
- This is exactly how WT deploys the demo instance to the SAME host: see
  `~/development/Worldtree/.gitea/workflows/deploy.yml` — the "Deploy to demo VM +
  health-gate" step uses `secrets.DEMO_VM_SSH_KEY` / `DEMO_VM_HOST` / `DEMO_VM_USER`.
  Explicit-over-implicit (visible in the run log, fires exactly on build success),
  one less always-on service than watchtower.

**Constraints (from soong-dev):** deploy on CI success only; keep the trigger
gated to `v*` tags + `workflow_dispatch` (as today); preserve the one-command
rollback posture (`docker compose down` / pin a previous tag).

**BLOCKER — needs from vh (owner-only):** a **runner→corviduo-dev deploy SSH key**
as a repo secret (+ host/user), same class as WT's `DEMO_VM_SSH_KEY`. Likely
**reuse WT's existing demo-deploy key** (WT's runner already SSHes to 10.250.50.152
as its deploy user). Repo secrets are vh-owner-only (see
[[2026-07-18-fleet-gitea-runner-build-recipe]]).

**Next-session steps:** (1) confirm/obtain the deploy SSH-key secret from vh (reuse
WT's or mint fresh); (2) add the deploy job to build-and-push.yml (infra-ops has
push on vh/soong-lab); (3) dispatch a build to verify it deploys + health-gates;
(4) ping soong-dev so they sync DEPLOY.md's "open follow-up" note to the as-built
mechanism. Auto-pull (watchtower) explicitly NOT chosen. See
[[2026-07-18-soong-lab-containerize-cutover]].

## AS-BUILT (2026-07-18, same-day execution)

**Mechanism landed** exactly as planned: `build-and-push.yml` gained a `Deploy to
corviduo-dev + health-gate` step (after build+push) that SSHes the host as `deploy`
and runs `docker compose pull && up -d` from `/opt/soong-lab`, then polls
`http://localhost:8443/api/version` for 120s and fails the job loud if unhealthy. No
compose is shipped from CI (the in-repo `docker-compose.yml` is a BUILD compose; the
host pull-compose is infra-ops-managed). Kept the `v*`-tag/`workflow_dispatch` trigger.
Skipped WT's disk-watermark gate + health-gated-`:latest`-advance (low cadence, easy
rollback).

**Deploy identity = reuse WT's `deploy` account** (operator accepted the rec):
- `deploy` (uid 1001, docker-group → no sudo) already owns `/opt/worldtree`; relocated
  soong-lab's deploy dir `/home/infra-ops/soong-lab-deploy` → **`/opt/soong-lab`**
  (deploy-owned), copied compose + `.env`. Named volumes (`soong-lab_soong-library`,
  `soong-lab_soong-portraits`) are project-scoped by compose `name: soong-lab` → followed
  the move untouched (dry-run `up -d` ADOPTED the running container, no recreate). Old dir
  **retired → `.retired-20260718`** (recoverable). Also lingering: `soong-lab-deploy.sh` /
  `.log` (dead pre-container webhook artifacts) — harmless, left in place.
- **Dedicated soong-only ed25519 deploy key** minted (NOT literally WT's key — cleaner
  independent revocation), pubkey appended to `deploy`'s `authorized_keys`
  (fp `SHA256:MG7M3RiZJ176sLfblffb96V6W1qkRTgJ5dow1CpiY68`). Existing `deploy` key is
  plain/unrestricted, so parity held.

**The secret gate (the friction point):** repo Actions secrets are **vh-owner-only** —
claude-bot's token is `write:package,read:repository` (403 on secret-write), and the vh
package-scoped PAT also 403'd on `PUT …/actions/secrets/…`. So `DEPLOY_SSH_KEY` /
`DEPLOY_HOST` (10.250.50.152) / `DEPLOY_USER` (deploy) HAD to be set by the operator.
First operator attempt produced a **bad key paste** — the deploy step died with
`Load key … error in libcrypto` + `Permission denied (publickey)` (build+push were green;
live Soong never moved). Fix: operator re-set the secret; the minted key path was
pre-validated from nh3-dev (`ssh -i … deploy@… 'cd /opt/soong-lab && docker compose config
-q'` → OK, health 200) so the re-set was the only variable.

**Validation:** `workflow_dispatch` via claude-bot **basic auth** (its token lacks
`write:repository` for the dispatch API; the account password works). Run #5 (task 1886)
GREEN — live container recreated `sha256:…541f7730` → `…07526a08`, `StartedAt` fresh,
health 200. `/api/version` now reports **0.3.25** (run #5 shipped soong-dev's 1c2f831
STYLE_WORKFLOWS re-pin as validation cargo). soong-dev synced `docs/DEPLOY.md`
(commit `00b67c3`). NB: tag **v0.3.25 exists only locally** — pushing it would re-trigger
a redundant build+deploy of the same commit (operator's discretion).

**Ops now:** redeploy = tag `v*` or `workflow_dispatch` the CI (auto). Manual fallback =
`sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`
(the `.env` is `deploy`-owned 600, so infra-ops needs `sudo -u deploy`, not a bare `cd`).
  _Archived 2026-08-16._

- `[2026-07-19]` **irv-ml1 ComfyUI — RTX VSR baked into canonical provisioning (comfy-dev ticket DONE).** RTXVideoSuperResolution node + `nvidia-vfx` dep were manual installs; documented both in the canonical `stacks/comfyui/README.md` runbook (this stack's provisioning IS the README — no automated provision script). Key durability insight: the **node** lives in `basedir/custom_nodes` (persistent, restic-included → durable) but the **`nvidia-vfx` wheel** lives in the venv under `run/` (disposable, restic-excluded → **dropped by any `rm -rf run/*` fresh-bootstrap**), so the pip step must re-run after every venv rebuild. Both steps run **as uid 1000** (root install → venv-ownership crash-loop, [[reference_irv_ml1_comfyui_mmartial]]); `--extra-index-url https://pypi.nvidia.com` kept **scoped to the nvidia-vfx install**, deliberately NOT a global compose `PIP_EXTRA_INDEX_URL` (would risk perturbing the pinned torch 2.12.1/SageAttention boot bootstrap). Node already live on the box; no host change, canonical runbook now replays it. comfy-dev informed.
  _Archived 2026-08-16._

- `[2026-07-19]` **vh private Gitea PyPI — consumer READ-access convention set + wyrd-dev provisioned.** Consuming agents read the internal vh PyPI (`https://gitea.phasefinal.com/api/packages/vh/pypi/simple/`) with a **shared read-only token** (operator call: shared, not per-consumer — read-only blast radius is small, per-agent Gitea identities aren't worth it). Minted a dedicated `read:package`-scoped PAT off **claude-bot** (`POST /users/claude-bot/tokens`, name `vh-pypi-read-consumers`; verified reads worldtree-sdk, write-probe 401), revocable/rotatable independently. uv auth = `UV_INDEX_GITEA_USERNAME=claude-bot` + `UV_INDEX_GITEA_PASSWORD=<token>` (or `~/.netrc`); pyproject uses `[[tool.uv.index]] name=gitea … explicit=true` + `[tool.uv.sources] <pkg> = { index = "gitea" }` (mirrors soong-lab's bifrost setup). Delivered to wyrd-dev (worldtree-sdk adoption) via mode-600 drop on nh3-dev, drop-and-shred. [[reference_claude_bot_gitea_creds]]
  _Archived 2026-08-16._

- `[2026-07-23]` **jackdaw-compose backend deployed as a persistent nh3-dev service (:8787).** Hosted for jackdaw-dev: thin stateless `bun server/index.ts` (from `~/development/jackdaw`) → LiteLLM `gen`, Origin-gated (INV-BK04/05), reached same-origin via their `:4500` bench's `/compose` proxy. `jackdaw-compose.service` (env/shared-key server-side, unit 0600, uncommitted). Also stood up + tore down a throwaway cloudflare quick-tunnel for their preview (`cloudflared` now installed at `~/bin`). In the nh3-dev README inventory (`cd4d52e`).
  _Archived 2026-08-16._

- `[2026-07-25]` **nh3-extdev herald installed — box is now a full v2 push participant.** forseti flagged (relaying operator): extdev had the `althing-herald` binary (`/usr/local/bin/`) but NO unit (skipped the whole v2 arc), so `herald-status` = "notifications suspended" and ldp-dev ran on the `althing-light-monitor` poll fallback. Installed `/etc/systemd/system/althing-herald.service` as a **SYSTEM unit mirroring the receiver** (`User=althing-svc`, `Group=althing`, `Environment=ALTHING_ROOT=/srv/althing`, `ExecStart=/usr/local/bin/althing-herald --poll 5`, enabled) via the **lkraven@ NOPASSWD path** (used under the then-mistaken belief infra-ops was sudo-less — **CORRECTION 2026-08-03: infra-ops has had full NOPASSWD sudo on extdev since 2026-06-25** per [[reference_nh3_extdev_althing_mesh]]; future extdev installs can self-serve as infra-ops without the lkraven@ hop). Verified: active / 0 restarts / `herald-status` flipped to "✓ herald up." No zellij routes on extdev → heartbeat + wake-FIFO poke only, no pane-dispatch; ldp-dev keeps light-monitor unless it opts into a wake-listener.
  _Archived 2026-08-16._

- `[2026-07-25]` **Booth v0.1.4 — booths are downloadable.** Verbatim `index.html` booths (e.g. edict-design-brief) were served raw with no download affordance. Added `/b/<name>/?download=1` (streams the whole booth as `<name>.zip`, attachment) + `?dl=1` on the file route (forces Content-Disposition attachment so html/md/text saves instead of rendering inline) + ⬇ zip links on the index card (the accessible spot for verbatim booths) and the gallery header. `zip_booth()` helper, 31 tests green; verified live on nh3-dev :8090 (edict-design-brief.zip = index.html + ui-design-brief.md). eshpfi `91a031f` / tag `booth-v0.1.4`.
  _Archived 2026-08-16._

- `[2026-07-25]` **Kimi K3 wired into the LiteLLM gateway — CODING endpoint** (operator-directed; fulfills a Heid gateway request to add a 4th cross-frontier panel arm). **Primary `model_name: kimi-k3` → `openai/k3` @ `https://api.kimi.com/coding/v1`** (Kimi Code / Vivace membership; key `KIMI_CODE_API_KEY`). A general-endpoint variant `kimi-k3-gen-api` → `openai/kimi-k3` @ `https://api.moonshot.ai/v1` (key `MOONSHOT_API_KEY`) is kept alongside (originally wired then demoted when the operator corrected: the plan uses the CODING endpoint, not the general Moonshot API). Both keys in compose env + server `.env` (NOT committed) + `.env.example`. Both verified live through the gateway :4000 (17+25→"42", "PONG"). **k3 constraints on BOTH endpoints (config-pinned + commented):** accepts ONLY `temperature=1` (else 400 "only 1 is allowed"); REASONING model (CoT in `reasoning_content`, answer in `content` → tiny `max_tokens` returns EMPTY; Kimi Code adds thinking-effort tiers low/high/max). Coding lineup also carries `k3-256k` / `kimi-for-coding` / `kimi-for-coding-highspeed` (not wired). Reachable by any gateway key spanning all proxy models (incl. shared all-agents key → spends the paid Vivace/Moonshot quota). eshpfi `edaa9a9` (gen wiring) + `9e2f787` (coding correction). **OPEN:** Heid key-scoping — shared key reaches it (paid) vs a dedicated scoped key (asked in althing `01KYD63ZBY…`).
  _Archived 2026-08-16._

`[2026-07-25]` **infra-ops Worldtree config-as-code repo — SHIPPED + boundary AGREED.**

**STATUS (2026-07-25, done this session):** `vh/worldtree-instance-configs` (private, gitea) built, pushed, validated; boundary agreement secured from worldtree-dev.

- **Repo:** dir-per-instance `demo/` + `personal/` (5 files each: `defaults.yaml`, `policies.yaml`, `model_roles.yaml`, `providers.yaml`, `matrix.yaml`), seeded byte-exact from live `/opt/<instance>/config`. `pinned/` = README stub only — **no `/app/config` bind-mount; config baked into frozen image `446e5807` (2026-05-13)**, so out-of-scope; deploy verb refuses it.
- **Tool:** `scripts/deploy-wt-config <verb> <instance>` — `diff` (read-only repo-vs-host), `deploy` (in-run host backup → `install -o vh -g vh -m 644` → restart **api+matrix** → health-gate api `/health` → auto-rollback), `capture` (host→repo reconcile). Instance table in-script (demo→`/opt/worldtree/config`+`worldtree-worldtree-{api,matrix}-1`; personal→`/opt/worldtree-personal/config`+`worldtree-personal-worldtree-{api,matrix}-1`). Matrix sidecar shares the config mount but has no healthcheck → restart both, gate on api. Env `WT_CONFIG_HOST` (default `infra-ops@10.250.50.152`), `WT_HEALTH_WAIT` (90s). Local clone `~/development/worldtree-instance-configs`.
- **Gitea plumbing (reusable):** nh3-dev **403s the gitea HTTP API** (public fail2ban + internal `:3000` both 403). Repo CREATE went via **ana-docker localhost API** (`ssh infra-ops@10.250.50.70` → `curl localhost:3000/api/v1/user/repos`, vh token from `~/.config/tea/config.yml`, operator-authorized one-time). PUSH went over **internal git-SSH `ssh://git@10.250.50.70:222`** (works from nh3-dev; auths as vh). `git init` defaulted to `master` → renamed `main` to match repo default_branch.
- **Boundary AGREED (worldtree-dev, althing thread `01KYCAECRWVEF16EVKQAGT2N80`):** no hand-edits to `/opt/<instance>/config`; config changes route to infra-ops as deltas (worldtree-dev owns CONTENT + approval trail — the wyrd-grant shape — infra-ops lands+deploys). **Three-layer model:** image `config/` = baseline new instances seed from (theirs) → `vh/worldtree-instance-configs` = per-instance truth (ours) → host bind-mount = deploy target (written only by the tool). **Carve-out:** worldtree-dev's admin-API ops (`/admin/keys` mint, tier changes, session retirement, future runtime-grant surfaces) mutate instance **DATABASES not config files** → NOT config edits, stay in-band. If a future API writes config *files*, they flag at design time. b132 CONFIG BASELINE breadcrumb composes (INFO line = config-as-code diverges from image baseline, by design).
- **No live deploy** done or needed — repo seeded == live (diff clean, capture round-trips zero-diff). Deploy path is dry-run-validated only; first real deploy needs operator per-change yes (managed box).

---

_Original plan (2026-07-25, pre-build):_

`[2026-07-25]` **infra-ops to OWN a Worldtree per-deployment config repo + deploy tooling (operator-directed).**

**Decision.** Vuong directed (2026-07-25, this session) that Worldtree instance config should be a *tracked change*, **managed and deployed by infra-ops — not worldtree-dev**. Model: worldtree-dev owns the app/image (+ the baked baseline defaults); **infra-ops owns config-as-code for every deployment** and deploys it. This is the durable fix for the root cause behind the whole #376 arc — config was edited live on host bind-mounts (`/opt/<instance>/config/`) with zero version history, audit, or recovery.

**What "no worldtree-dev involvement" does and does NOT cover** (clarified with the operator this session):
- **Build + deploy = infra-ops-only.** Deploying config = write the host bind-mount file + restart the container (the *exact* procedure already run this session — backup → replace → restart → health-gate → rollback-on-unhealthy). No worldtree-dev in the deploy loop. Their CI only swaps the IMAGE; it does NOT resync the host config bind-mount (confirmed #376 finding).
- **ONE load-bearing exception — a one-time boundary agreement, NOT per-deploy involvement:** for the repo to *own* config it must be the **only writer**. worldtree-dev "live-bridges" (hand-edits mounted config directly on the box). If the repo deploys config *and* they keep live-editing → **two writers fighting the same files** = #376 all over again. So secure a one-time "yes" from worldtree-dev: *the config repo is now authoritative; stop hand-editing `/opt/<instance>/config`; route config changes through the repo.* (Five-minute agreement, not a design collab.)
- **Standing coupling (not "involvement"):** the config *schema* is the app's, enforced by its boot validator (`core.config_validator`). infra-ops configs must stay schema-compatible with the deployed image; the boot gate is the loud backstop.

**Build shape (recommended):**
- Gitea repo `worldtree-instance-configs` (infra-ops-owned), **dir per instance** (`demo/`, `personal/`, `pinned/` — the three on corviduo-dev 10.250.50.152: demo `worldtree-worldtree-api-1` :8080, personal `worldtree-personal-worldtree-api-1` :8081, pinned `worldtree-pinned-worldtree-api-1` :8082). Config dirs: demo `/opt/worldtree/config`, personal `/opt/worldtree-personal/config`, pinned `/opt/worldtree-pinned/config` (verify pinned's mount).
- **SEED FROM CURRENT MOUNTED STATE, don't author fresh** — capture each instance's live config (incl. legitimate live-bridged deltas: personal carries `agent_architect` role [Soong/soong-lab] in model_roles.yaml + `ratatoskr-affect-full-allow` in policies.yaml that are NOT in the app repo — the operator ruled these are BY DESIGN, keep them). Losing them = breakage (the affect-render one gates mood rendering).
- Deploy script (e.g. `scripts/deploy-wt-config <instance>`): git = source of truth → push to host bind-mount + `docker restart` (same pinned image, no pull) + health-gate + auto-rollback. This is the proven-this-session procedure, scripted.
- Files per instance: `policies.yaml`, `model_roles.yaml` (+ whatever else is bind-mounted — `defaults.yaml`, `providers.yaml`, `matrix.yaml` all live in `/opt/<instance>/config`; decide scope — policies+model_roles are the authz/role layer, defaults/providers are heavier instance tunables).

**Tracking surface:** operator-directed 2026-07-25, carried by this snapshot + `/tmp/infra-ops-handoff.md`. No issue filed (infra-ops-internal build). Related fleet idiom to reuse: canonical-sync (`.corviduo-canonicals.toml` / `canonical_sync.py`). Later scale option (deferred, needs worldtree-dev): base+overlay with a merge step in their pipeline.

See [[2026-07-25-wt-376-per-instance-config-arc]] for the incident that produced this. Auto-memory: `reference_worldtree_perinstance_config`, `reference_corviduo_dev_emergency_ops`.
  _Archived 2026-08-16._

- `[2026-07-26]` **Demo `BIFROST_CLIENT_ALLOWED_HOSTS` += `10.100.10.50:8391`** (wyrd-dev's bifrost memory-store provider; operator-approved). **First live exercise of the #376 config-as-code boundary working as designed** — worldtree-dev routed the delta to infra-ops instead of hand-editing `/opt/demo`. Appended to `/opt/worldtree/.env:25` (now 4 netlocs), recreated ONLY `worldtree-api` (the gated conv-api path), health-gate green, container env verified. **REUSABLE FOOT-GUN:** an env-var change needs a container **RECREATE, not `docker restart`** (env is baked at create); and the demo `.env` defaults `WORLDTREE_IMAGE=:latest` while the box runs a specific SHA — so a naive `compose up` risks the documented stale-`:latest` crash. FIX = capture the running image live (`docker inspect …Config.Image` → `…:9eff09f007ba`) and `sudo env WORLDTREE_IMAGE=<sha> docker compose up -d worldtree-api`. Backup `/opt/worldtree/.env.bak-bifrost-20260726-221602`. **BOUNDARY SEAM:** this was a compose-`.env` var, NOT a `config.yaml` file in `vh/worldtree-instance-configs` — the `.env` holds secrets so it's deliberately not repo-tracked → env-deltas land directly on the box (config *files* are versioned, compose *env vars* aren't). [[reference_worldtree_instance_configs_repo]]
  _Archived 2026-08-16._

- `[2026-07-27]` **Zed edit-predictions: keyless FIM-completion route SHIPPED end-to-end.** Operator wants Zed's inline edit-prediction (which CANNOT send an auth header) to reach a FIM coder via `/v1/completions`. **Deep-research (106-agent workflow) picked `Qwen/Qwen2.5-Coder-1.5B`** (BASE, Apache-2.0; native FIM `<|fim_prefix|>/<|fim_suffix|>/<|fim_middle|>` IDs 151659/60/61; Zed `prompt_format:"qwen"`). Runner-up 3B = non-commercial Qwen-Research license; **no small dense Qwen3-Coder exists (all MoE, smallest 30B)**. **Stood up `vllm-coder`** on ana-ml2 **GPU1 :8020** (served-name `qwen2.5-coder-1.5b`, 8192 ctx, util 0.06, fp8 KV). To fit, **shrank granite (phasing out, operator-directed):** util 0.27→0.13, max-len 131072→16384, seqs 1024→256 (freed ~14 GB; the KV-≥-1×-max-len rule crash-looped it at util 0.12/32768 → settled 0.13/16384). **LiteLLM alias `coder-fast`** → :8020 (`mode: completion`). **Minted a `coder-fast`-SCOPED virtual key** (verified 403 on `gen` — the real blast-radius bound). **Built `zed-fim-proxy`** (ana-docker **:4141**, `network_mode: host`, stdlib-python, `stacks/zed-fim-proxy`): keyless POST `/v1/completions`, model-allowlist `coder-fast`, injects the scoped key → LiteLLM :4000; `GET /ping` anon liveness; wrong-model→403, wrong-path→404, `/chat/completions` rejected. Verified keyless FIM end-to-end ('a + b', finish `stop`). **Zed `api_url` = `http://10.250.50.70:4141/v1`, model `coder-fast`, prompt_format `qwen`.** **source-IP allowlist intentionally LEFT OFF (operator direction 2026-07-27) — do NOT tighten:** Zed roams the operator's WireGuard `10.0.0.0/8`, so a single-IP pin would break it. Blast-radius bound is the `coder-fast`-scoped key + model/path allowlist (keyless but coder-fast-only, internal-net-only). (The proxy does exact-IP matching; scoping to the `10.0.0.0/8` CIDR would need CIDR support — deliberately not added.) Canonical: `stacks/vllm` (coder + granite shrink), `stacks/litellm` (coder-fast), `stacks/zed-fim-proxy` (NEW). Server vllm compose.yaml has benign stale-comment drift vs canonical (didn't overwrite the newer canonical).
  _Archived 2026-08-16._

- `[2026-07-27]` **Muninn ingestion-watcher sidecar deployed on PERSONAL Worldtree (#377).** worldtree-dev request (research-wing ingest arc, personal-only per the 2026-07-16 topology ruling); operator-approved. Added a `worldtree-muninn` **compose sidecar** to `/opt/worldtree-personal/compose.yaml` — `<<: *worldtree-common` anchor inherits the api's image + full env + config/state/kb mounts; `command: python -m core.muninn --watch`; `restart: unless-stopped`; `stop_grace_period: 1h` (INV-377-7: max 2 concurrent × worst-case job, SIGTERM-drains). **Pinned to the running SHA `773866084af9`** (b146, ≥ b143 — dodges both the `:latest` trap AND the "pre-b143 ref resurrects deleted dispatch.py from stale bytecode" warning). Verified: running / 0 restarts / flock sole-runner (no rc3) / heartbeat live at `{ingestion_root=/data/state/ingestion}/.watcher-heartbeat` (poll 30s). Container `worldtree-personal-worldtree-muninn-1`; backup `compose.yaml.bak-muninn-20260727-081920`. **DURABILITY RESOLVED (worldtree-dev, same day):** Q1 was a LIVE FOOTGUN — `deploy-personal.yml` scp's the REPO compose.yaml over the box's + runs `up -d --remove-orphans`, so the box-local sidecar would've been clobbered AND orphan-removed at the next staging tag. worldtree-dev fixed at source: moved the sidecar into their repo compose.yaml gated behind a **`muninn` compose profile** (commit 5d7f6bd) — shared compose stays instance-identical, `.env` `COMPOSE_PROFILES` differentiates (demo watcher-less). **My action:** added `COMPOSE_PROFILES=muninn` to `/opt/worldtree-personal/.env` (backup `.bak-muninn-profile-20260727-082541`; no-op vs the current unprofiled box-local sidecar → seamless handover at next deploy). Q2: their deploy `up -d`'s the whole stack w/ `WORLDTREE_IMAGE` exported → sidecar version-tracks the api, no drift. **CONFIG-AS-CODE EXTENSION:** mirrored the non-secret delta as `personal/env.public` in `vh/worldtree-instance-configs` (repo `a9d091e`) — FIRST extension beyond config.yaml files to env-level config; the secret-laden `.env` stays box-only, `env.public` records only non-secret infra-ops-owned env deltas (record, not a deploy source — `deploy-wt-config` globs `*.yaml`). **BOUNDARY CLARIFIED:** compose.yaml = worldtree-dev's (their repo, instance-identical, scp'd on deploy); per-instance `.env` = infra-ops's differentiator. Deploy step of the #363/#377 arc. **#377 CLOSED — acceptance PASSED 2026-07-27:** worldtree-dev enqueued a test job via muninn-dispatch 0.1.0 in a one-shot ephemeral container (no docker-exec); the sidecar claimed it within one 30s poll, drove it to terminal (structure→summarize→complete), zero restarts/rc3, heartbeat fresh throughout — whole loop (request→deploy→durability fix→acceptance) in <2h. (Pre-existing pipeline bug #379 surfaced — `output.kb_notes=false` ignored → 1 inert test note in the research wing — worldtree-dev owns it, nothing infra-ops-side.) **⚠ OPERATOR-SURFACE (open):** the `env.public` overlay mechanism is a repo-scope call to bless/adjust. [[reference_worldtree_deploys_cicd]] [[reference_worldtree_instance_configs_repo]] [[project_worldtree_research_wing_ingest]]
  _Archived 2026-08-16._

## Tried and abandoned (archived 2026-08-16 batch)

- `[2026-07-18]` **Fleet Gitea CI foot-guns** (3 failed soong-lab builds): the pfi-fleet runner's `node:20-slim` job image has no docker/git so `actions/checkout` + `docker/*` marketplace actions all fail; `vh` is a USER so its packages are owner-write-only (claude-bot repo-admin-collab still 401s on push/publish, and can't set repo secrets — owner-only); `GITEA_`-prefixed secret names are reserved/illegal. Fixes in → `persistent-memory.d/2026-07-18-fleet-gitea-runner-build-recipe.md`
  _Archived 2026-08-16._

- `[2026-07-18]` **zonos-gateway local clone had NO git remote + a history unrelated to gitea's** — "committed to vh/zonos-gateway" was never pushed from that clone; two separate `git init` lineages, no merge-base. Reconcile = reset local→origin/main + overlay the changed files + push (NOT force — that erases gitea's voice-wav commits). Check `git remote -v` + `git merge-base` before assuming a clone is wired.

  _Archived 2026-08-16._

- `[2026-07-25]` **Peer green-light ≠ operator consent for a managed-box mutation.** Auto-mode guard blocked a config-replace+restart on the Worldtree-team demo box that was authorized only by worldtree-dev's althing message — correctly: a persistent change to shared infra needs the *operator's* yes for that specific change, not a peer's. Surface it; don't route around the guard. (The operator then stood the whole change down — the guard's hold was the right call.)
  _Archived 2026-08-16._

## Recent decisions (archived)

**Worldtree b168/#384/#385 arc — COMPLETE 2026-08-03.** A long peer-driven arc across worldtree-dev / muninn-dev / mimir-dev / ratatoskr-dev, all on corviduo-dev's demo+personal instances. Sequence: providers.yaml boot-gate pre-sync → b168 deploy → DCC #384 reindex → round-2 full re-ingest → #381 restart → operator-approved production dedup sweep. Landed clean; three of MY foot-guns along the way, each caught + hardened into a fleet runbook rule (see Tried-and-abandoned: `mv -t`, `docker exec -u 1000`, shared-containerd race).

## providers.yaml pre-sync (boot-gating config)
b168 (commit `293f8f3`) added a `summarization` capability block that in-image `agents/muninn/config.yaml` references → boot-blocking if the host bind-mounted providers.yaml lacks it. Synced both hunks (summarization block + deep-reasoning desc) into demo+personal via `deploy-wt-config`; instance-configs commit `53349f8`.
- **deploy-wt-config runbook:** `~/development/worldtree-instance-configs/scripts/deploy-wt-config {diff|deploy|capture} <inst> --file providers.yaml` (per-instance dirs demo/personal/pinned; `deploy` = host write + api/matrix restart + 90s health-gate + auto-rollback; `diff`/`capture` safe). demo+personal providers.yaml are byte-identical.
- **GOTCHAS:** (1) an UNPUSHED source commit → `git show <sha>` 404s and a gitea `raw?ref=<sha>` silently falls back to the default branch; verify the commit exists (`/git/commits/<sha>`) before trusting a fetch, else ask the peer to paste hunks. (2) a peer's hunk paste may be mis-indented (8-space vs the block's 4-space) → invalid YAML; always YAML-validate after a paste-sourced edit.
- **Config-delta pre-sync rule (verified via `docker inspect`):** worldtree containers bind-mount ONLY `config/` host-side (`/opt/worldtree-*/config/` → providers/model_roles/matrix/policies/defaults/env.public = the pre-syncable set); `agents/` (schemas.yaml, prompts) + all code ship IN-IMAGE. So only a `config/*.yaml` change is boot-blocking-pre-syncable; an `agents/`-or-code delta needs NO host pre-sync (CI carries it). b169's schemas.yaml (#387) was correctly no-pre-sync.

## #384 reindex + #381 restart + verify
DCC job `mimir-6351554e8e8f`. Reindex: `sudo docker exec -u 1000 worldtree-personal-worldtree-muninn-1 python -m core.muninn --reindex <job>` (⚠️ MUST `-u 1000` — default-root writes contaminate the uid-1000 KB tree; see Tried-and-abandoned). Then **#381 restart** (stale-Chroma-client fix): `sudo docker restart worldtree-personal-worldtree-api-1` (plain bounce, NO compose up / no image repoint) → healthz/readyz 200 ~25s.
- **Chroma-verify runbook:** `sudo docker exec -i <muninn> python -` (MUST pass `-i` or stdin never reaches `python -`) → `chromadb.PersistentClient('/data/kb/.chroma').get_collection('fiction').get(where={'job_id':<job>}, include=['metadatas'])`. Chroma persists at container `/data/kb/.chroma` = host volume `worldtree-personal_worldtree-kb`.
- **Retrieval-visibility check (NOT grounding — that's ratatoskr's):** a Mimir session — admin token `~/.config/worldtree/personal-admin-token` (wildcard scope) → POST `/sessions` (agent_id=`mimir`, `record_tool_intermediates=true`) → POST `/sessions/{id}/messages` (STREAMS SSE, not JSON) → parse SSE `tool_result` for `search_library` wing hits → DELETE session.

## Production dedup sweep (operator-approved)
Deleted the 785 April-era DCC orphan rows (`job_id=b59c147c5ce0`, no wing/source_identity metadata → predate identity tracking) from the `main` collection. Supervised protocol: read-only verify count == 785, back up all rows (ids+docs+embeddings) to `corviduo-dev:/tmp/main-sweep-backup-b59c147c5ce0.json` (reversible), `main.delete(where={job_id})` (assert target==785 first), verify `main` 4009→3224, then **bounce the api** (a separate-process delete leaves the api's in-memory HNSW index holding the vectors until reload — the #381 pattern generalizes to deletes), confirm search now fiction-only. Backup left for /tmp natural cleanup (fiction wing is canonical; `~/archives` has the historical record).

Result: fiction wing 166 → 1,372 concepts; three consumer verify rounds 0/5 → 5/5 → saturated; #385 budget fix validated (705 vs April's 785 control, extraction AND indexing, zero truncations). worldtree-dev filed #388 for a deploy concurrency-lock (the shared-containerd race fix). See [[2026-08-02-mimir-inbox-arc]].
_Archived 2026-08-18._

`[2026-08-02]` **The mimir-inbox / #377-read-path arc — deploy, four bugs found+fixed+verified, a cloned voice, all in one long session (2026-08-01→02).**

The browser-facing half of the #377 Muninn ingestion arc, end to end: mimir-inbox stood up, the write path proven, the read path chased through four defects to a verified-working state, and a character voice cloned into the TTS zoo. Peers: mimir-dev (the app), muninn-dev (gate/watcher spec), worldtree-dev (Worldtree app layer + the #380/#381/#382/#383 fixes), ratatoskr-dev (a consumer + the rigorous verifier).

## mimir-inbox deployed (#377)
- **New infra-ops stack, canonical eshpfi `stacks/mimir-inbox/`; live corviduo-dev `10.250.50.152:8091`** (co-located w/ muninn-gate :8090 + the worldtree-personal muninn watcher). Full deploy detail + procedures → auto-memory `reference_mimir_inbox_deploy`.
- **Placement decision (operator, reversed):** 7-31 he ruled mimir-inbox stays OFF corviduo-dev (shared/NFS mount); 8-01 he REVERSED to CO-LOCATE. Trigger: muninn-dev's code-check showed staging is NOT same-fs-constrained (gate reads staging metadata + passes path strings; `os.replace` is inside `ingestion_root`) — staging's real constraint is **path-identity across writer/gate/watcher**, which co-location buys outright while dodging NFS failure modes. I HELD the reversal for the operator's direct word (data/hosting on a team-managed box, reversing his own ruling) even against 3 peer relays — vindicated as the right instinct; muninn-dev agreed.
- Build: **`uv sync --no-dev --frozen`, SINGLE-STAGE** (project installs editable-linked to `src/`, so src/ MUST stay beside .venv — a multi-stage "copy only .venv" dies at import/404s assets). uid 1000, host-net bind 10.250.50.152:8091, TCP-liveness healthcheck (deliberately NOT gate-coupled). Redeploy = refresh build context (**preserve the on-server `.env`!**) → `docker build -t mimir-inbox:0.0.1 -t mimir-inbox:<sha> .` → `compose up -d`. Version stays 0.0.1 across dev commits → tag the image w/ the source SHA too. Live commit progression `0478452`→`c8ab38f`→`2dcc77e`→**`8ece117`** (3 redeploys).
- mimir-inbox key on the gate bumped [read,submit]→**[read,submit,control]** (cancel/retry); brokered via a 0600 drop on nh3-dev (never on the althing bus).

## The read-path bug chain (worldtree-dev's, all found via this arc)
- **#380 wing-blind indexing:** the book-ingest path upserted concepts into a hardcoded `main` Chroma collection while wing search reads the `fiction` collection → P&P written to disk but `search_library` returned total 0. A silent-success defect ("complete/69 indexed" was right about the WRITE, wrong drawer). Root-caused off MY physical evidence (files on disk + search empty). Fixed b164 + a one-shot `--reindex <job_id>` (re-upsert into the right wing collection + delete stray `main` rows).
- **#381 stale Chroma client:** the personal api opens its Chroma client before the watcher's cross-process writes → **a freshly-ingested/re-indexed book is NOT queryable until the api is restarted.** Proven by my restart-diagnostic (pre-restart total 0 → post-restart hits, same index). Workaround until fixed: `docker restart worldtree-personal-worldtree-api-1` after any ingest/re-index. Filed as #381.
- **#382 unreliable Mimir grounding (the subtle one):** post-#380-fix the index was correct, but Mimir's grounding was INTERMITTENT — some sessions navigated the opaque job-hash dir (`mimir-f3887c9b97b7`) to the content, others distrusted the correct vector hits and **silently answered from training knowledge** (worst of the looks-fine-isn't family). ratatoskr-dev caught it; I'd been over-confident ("Mimir read Austen back to you") having verified the INDEX, not the GROUNDING. Fixed b166 with BOTH shapes: a self-describing `_index.md` per wing job-dir (resolves the hash dir to its title) + a Mimir prompt rule (wing-scoped hits ARE library content, never discard on a name mismatch, never substitute training). **Verified: ratatoskr-dev re-ran 3× fresh sessions → 3/3 grounded**, citations in note-extracted language not raw Austen. #382 CLOSED.
- **DCC (Dungeon Crawler Carl, job `b59c147c5ce0`) backfill:** `--reindex` FAILED ("job not found in any state dir" — predates state-tracking). SETTLED = **no re-file** (the b166 prompt rule already grounds it even without an `_index.md`; ratatoskr confirmed incidentally); an `_index.md` rides whenever DCC is next re-ingested.
- **#377 mimir-inbox banner bug (mimir-dev's, `8ece117`):** `/health-banner` misattributed an unwritable `ingestion_root` to the WORKER, rendering "The worker is not running." for a running worker — a false lead pointed at infra-ops's half of #377. Fixed (guard split into two banners); I confirmed from the DEPLOYED handler (not just the test) that `ingestion_root_writable:False` now renders "The ingestion root is not writable."

## muninn-gate → muninn-dispatch 0.1.5
Rebuilt `muninn-gate` off `vh/muninn-gate` main `bc04c4c` (dispatch 0.1.4→0.1.5) so the gate serves the new `concept_schema`/`concept_schema_source` row fields (computed gate-side). Gate version unchanged 0.0.14 (dual-tag the SHA). Build needs the vh gitea token as a BuildKit secret (`--secret id=gitea_pw`, UV_INDEX_GITEA_USERNAME=vh, drop+shred). Recreate with `compose up -d` (NOT bare restart — needs the new image). Verified: P&P job serves `concept_schema='fiction'`, `concept_schema_source=null` (null correct — pre-b164 job). Registry tags by commit SHA — `v1.0.0bNNN` docker tags don't resolve; use the deployed SHA (confirm `--reindex` present before using an image for a data-op).

## donut voice (65-frost → Zonos gateway)
Operator: "pick up 65-frost, use that bundle as a voice for a character named donut." 65-frost = a **Booth id** (`~/booth-data/65-frost/`) holding a curated yt-voice-clipper dataset (`dataset-…-curated.zip`: 4 clips + manifest, all SPEAKER_02 = Princess Donut). **Zonos gateway voice registry = a filesystem drop:** `<Name>.wav` in the voices dir (44.1kHz mono s16 PCM) auto-registers as `voice:"<name>"` on **startup** (needs a restart). The LIVE dir is the bind mount `/opt/docker/compose/zonos-gateway/voices/` (lkraven-writable), NOT the working tree. Built `Donut.wav` from seg000 (best clip), dropped it, restarted → `voice:"donut"` live in the gateway AND the Asset Engine's make form. Also copied to the build-source tree `~/zonos-gateway/voices/` for rebuild-durability (true canonical = the gitea repo, not yet CI-wired). Auditioned in booth `donut-voice`. **Expanded 2026-08-02 (onyx-58 bundle):** operator curated a 2nd Booth bundle `onyx-58` (`dataset-467d2cf8…curated.zip`, 3 Donut clips) as additions. Rebuilt the reference = **seg000 (65-frost) + seg101/seg110/seg148 (onyx-58)** ffmpeg-concat + resampled 24k→44.1k mono s16 = **52.0s**. `seg148` was diarized SPEAKER_03 but is Donut (operator-confirmed misdiarize → included). Assembly is NOT `assemble_voice.py` (that `-c copy` can't resample + caps ~15s); used a manual `aresample=44100,aformat=…,concat=n=4` filter. Backed up old ref → `irv-ml1:~/Donut.wav.pre-onyx58`; dropped to live bind-mount + build-source tree; `docker compose restart` (healthy 2s, `voice:"donut"` still 1 of 9). A/B booth `donut-onyx58` (A=old 16.3s ref, B=new 52s ref, same line). Longer ref is fine mechanically: gateway passes it as `speaker_audio_base64` → speaker *embedding*, not an audio prefix. **BUT auditioned → REVERTED same day:** pinned-seed neutral A/B (5 pairs, booth `donut-onyx58`) showed the single-clip seg000 (16.3s) beats the 52s 4-take concat on timbre — concatenating disparate takes muddied the embedding more than the range helped. Reverted both live + build-source to seg000-alone. Lessons (→ Tried-and-abandoned): more reference ≠ better when takes vary; and **emotion steering pulls output away from the clone fast** (operator craft rule) — keep clones emotion-neutral; bare `{input,voice}` calls send NO emotion (gateway only enables it on an explicit `emotion_*`/`preset` dial).

## Zonos streaming (no gateway change needed)
ratatoskr wanted play-as-it-arrives. `/v1/audio/speech` ALREADY streams — chunked `StreamingResponse`, opens native `/tts/generate` with `stream=True`, wraps as a streaming int16 WAV with `0xFFFFFFFF` placeholder sizes (meant for progressive `<audio>`). Verified TTFB 0.44s vs 6.84s total, `transfer-encoding: chunked`, dials preserved. ratatoskr's proxy was rewriting the placeholder header → forced buffering. Fix was theirs (pass chunks through); shipped + confirmed (TTFB 0.46s progressive). The Asset Engine (ana-docker:8200) IS the fleet "TTS zoo" (~20 audio svcs w/ irv-ml1 endpoints); zonos-gateway registered there, state=ready.

## Lessons (also in Tried-and-abandoned)
- **Verifying the INDEX (search returns hits) is NOT verifying GROUNDING** (does the agent trust+use them vs. silently answer from training). Check that citations are note-extracted, not model-knowledge. ratatoskr caught this after my over-confident "it works."
- **Reading the DEPLOYED artifact > trusting the test** for "is the fix live" — the test proves the source is right; reading the running code proves the artifact is, which is what an on-call actually meets.
- Held a boundary-box/data reversal for the operator's DIRECT word against 3 peer relays — the right call (peer relay ≠ operator consent; the placement guard was vindicated).

See also: [[2026-07-31-muninn-gate-deploy]]. auto-memory: `reference_mimir_inbox_deploy`, `reference_muninn_gate_deploy`, `reference_muninn_gate_staging_path`, `reference_zonos_tts_stack`, `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
_Archived 2026-08-18._

- `[2026-07-27]` **jackdaw-compose.service DECOMMISSIONED** (jackdaw-dev request; the JackDAW AI Composer was cut from v1 by operator decision 2026-07-27). Stopped + disabled the nh3-dev `:8787` user service (no client calls it — ai/server/AiChat deleted from main, `/compose` proxy removed); unit **archived not deleted** → `~/.config/systemd/user/jackdaw-compose.service.decommissioned-20260727` (revival = rename + `daemon-reload`). **No credential revoked** — the unit used the SHARED all-agents LiteLLM key (`sk-eA_XOd…`, model `gen`), not a dedicated one. Code preserved on jackdaw `origin/ai-composer-preserved`; treat as permanent. The `:4500` HTTPS audition bench is untouched. (Supersedes the 2026-07-23 stand-up line below.)
_Archived 2026-08-18._


## Tried and abandoned (archived)

- `[2026-08-02]` **donut voice multi-clip reference (onyx-58 expansion) — TRIED, REVERTED.** Folded the `onyx-58` bundle's 3 Donut clips (seg101/seg110/seg148) in alongside the original seg000 → a 52.0s 4-take concat reference, hoping a longer ref → more robust speaker embedding. A pinned-seed A/B (5 pairs, varied registers, booth `donut-onyx58`) showed the **original single-clip seg000 (16.3s) sounds better** — concatenating disparate takes muddied the timbre more than the extra range helped. Reverted to seg000-alone (live + build-source). **Two durable lessons:** (1) for a faithful clone, a single clean representative take can beat a longer multi-take concat — more reference audio is NOT automatically better when the takes vary. (2) **Emotion steering pulls the output AWAY from the cloned voice fast** (operator's craft rule) — keep donut (and clones) emotion-neutral for fidelity; the gateway only enables emotion when an `emotion_*`/`preset` dial is explicitly sent, so bare `{input,voice}` calls stay pure-clone. `seg148` was diarized SPEAKER_03 but IS Donut (operator-confirmed misdiarize). onyx-58 curated bundle lives in booth `onyx-58` (24h TTL — stash to `/mnt/smithy/voice_clones/` if a future middle-ref experiment is wanted).
_Archived 2026-08-18._

- `[2026-08-02]` **Verifying the INDEX is not verifying GROUNDING** (#382). A `search_library` returning wing=fiction hits proves the content is *retrievable*; it does NOT prove the agent (Mimir) *trusts and uses* those hits vs. silently answering from training. I reported "Mimir read Austen back to you" off a grounded-*looking* answer; ratatoskr-dev caught that grounding was intermittent (some sessions discarded the correct hits and substituted training knowledge). Test the harder claim — are the citations note-extracted or model-knowledge? — and reading the DEPLOYED artifact beats trusting the test for "is the fix live."
_Archived 2026-08-18._

- `[2026-07-30]` **brokkr's WebSearch "verification" CONFIRMED a hallucination — 3 phantom `microsoft/Mage-Flow-{Base,Turbo,Edit}` repo IDs.** brokkr-smithy-dev handed 3 gated-looking repo IDs for an operator-directed model pull; they don't exist (its own web-search fabricated an arXiv ID + project page, twice). Lesson: the HF **registry API is ground truth** — an unauth 401 ≠ exists (`{"error":"Invalid username or password"}` masks private/gated/nonexistent alike), an authed 404 = phantom, and `author=X&search=Y` refutes existence. API-verify every repo ID before a pull; LLM-summarized web fetches confabulate. auto-memory `reference_verify_hf_repo_ids_before_pull`.
_Archived 2026-08-18._

- `[2026-07-30]` **magpie TTS serving — evaluated, ABANDONED.** Pulled `magpie_tts_multilingual_357m` (the one real repo of brokkr's batch) to NFS, stood it up on irv-ml1 (ephemeral NeMo-Speech-`main` container — stock PyPI/NGC NeMo can't load v2607), A/B'd vs Zonos → Zonos wins expressive English decisively, multilingual not needed. Not served; `magpie-nemo` torn down. `.nemo` KEPT on NFS as brokkr's fine-tuning base. auto-memory `project_magpie_tts_eval_rejected`.
_Archived 2026-08-18._

## Recent decisions (archived 2026-08-19 batch)

`[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap; fix relayed to heid, NOT applied gateway-side.**

heid reported that `kimi-k3` (the primary route = Kimi Code coding endpoint `openai/k3` @ `api.kimi.com/coding/v1`) silently degraded its cross-frontier panel: on large/reasoning-heavy dispatches, `completion_tokens: 16381` **exactly**, `content` empty, `reasoning_content` ~64KB, `finish_reason: **stop**` (a truncation mislabeled as a clean stop). `max_tokens: 100000` in the request was not honored.

**Investigation arc (a clean cross-frontier-triage + verify-on-the-wire case):**
1. My first read: a flat ~16384 OUTPUT cap; fix = a LiteLLM `stop→length` relabel callback (heid's fallback ask). Confirmed the cap isn't in our LiteLLM config (no `max_tokens` clamp on the route).
2. Operator routed a fix-research pass to **dvalin-smithy-dev + bil-smithy-dev** (independent). Both CONVERGED (docs-based): `max_tokens` is a deprecated alias on Kimi/Moonshot; the canonical field is `max_completion_tokens` (default 131072, max 1M); the coding endpoint defaults output to 16384; fix = send `max_completion_tokens` + `reasoning_effort` via `extra_body` (drop_params-safe).
3. **heid's live data REFUTED the docs hypothesis:** a later dispatch hit `completion_tokens: 18455` (ABOVE 16384) cleanly, with `reasoning_tokens: 16198` (just under 16384) and content present. So COMPLETION is uncapped; the bound is on **REASONING at ~16384**. When a hard task's thinking exhausts that budget, nothing's left for content → empty answer under `stop`.
4. **I proved it on the wire** — ran heid's real 500KB failing bundle direct at both endpoints (bypassing LiteLLM so `reasoning_effort` isn't dropped): default effort → 504/timeout (the failure); **`reasoning_effort: low` → reasoning ~12–13.5k (under the ceiling), content returns (6–7.6k chars)**, on BOTH coding AND general endpoints. So re-routing to the general endpoint buys nothing — the fix is the effort param, and it works on the wire.

**THE FIX (caller-side, no shared-gateway change/restart):** send `reasoning_effort` via **`extra_body`** on kimi-k3 dispatches (`low` for large bundles). LiteLLM `drop_params: true` strips the top-level `reasoning_effort` — which is exactly why heid's earlier `reasoning_effort: low` was a no-op. `extra_body` survives drop_params (the house GLM-thinking pattern). Tradeoff: low effort = shallower reasoning, but a complete answer beats today's empty one.

**Relayed to heid to validate on a real round** (the one unconfirmed hop is whether `extra_body` survives OUR LiteLLM). **Backstop if it doesn't:** add `allowed_openai_params: ["reasoning_effort"]` to the `kimi-k3` route in the gateway config — that IS a shared-gateway change + a ~10s restart (blips all consumers), so it needs a heads-up.

Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/litellm/conf/config.yaml` (see Recent-decisions `[2026-07-25]` Kimi K3 wiring). No gateway change was made this session. Failing dispatch on record: `01KYTASKTY3T` (jackdaw-dev bug-hunt).
  _Archived 2026-08-19._

- `[2026-07-25]` **bil-smithy-dev wired as an althing zellij-window-ping (pane route).** She's a `driver: human` dwarf peer (pane `bil-smithy` already live alongside eitri/dvalin/regin-smithy in the `Claude` zellij session) but had no delivery route → smoke messages posted to the bus but never reached her window. **Mechanism (reusable for any pane-route handle):** `~/.althing/config.yaml` → `zellij_sessions.Claude.agents[]` maps `handle` → `target` (a zellij pane **TITLE**, matched via `list-panes -j` in `althing/zellij.py:resolve_pane_id`) → `command` (herald `write-chars` + CR into that pane). The **herald loads config ONCE at startup** (`herald.py main()`), so **`systemctl --user restart althing-herald.service`** after editing. Added bil (`target: bil-smithy`), restarted, verified: herald delivered the pending smoke `01KYD7W7CF…` (available→attempted→**delivered**). ⚠️ Noticed pre-existing pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range", empty msg_ids — likely `render_command messages[0]` on an empty list; NOT caused by this change, bil works) — worth a herald look.
  _Archived 2026-08-19._

## Tried and abandoned (archived 2026-08-19 batch)

- `[2026-08-02]` **`docker exec` into worldtree containers defaults to ROOT — root writes contaminate the uid-1000 (vh) KB tree.** My `sudo docker exec … --reindex` on personal ran as ROOT (muninn app = uid 1000); its wing git-commit + atomic note-swap left root-owned files in the `worldtree-personal_worldtree-kb` volume: a root-owned `.old-<job>` backup dir (blocked the uid-1000 retry's `rmtree` → Errno 13, because unlink needs write on the DIR and it was root:root 755) AND **60 root-owned loose git objects** in `.git/objects/`. Fix (host-side, corviduo-dev): `sudo rm -rf` the superseded `.old-` dir (tar'd aside to /tmp first) + `sudo find … -user 0 -exec chown 1000:1000` the objects (ownership-only, git-content-safe; the `.git/objects/XX/` dirs were vh-owned so these weren't a hard blocker, but violated "clean tree"). **RUNBOOK RULE (worldtree-dev, ADOPTED):** any `docker exec` into worldtree containers that WRITES pipeline state runs **`-u 1000`**, never default-root — same genus as the mv footgun (acting without matching the target's constraints; 3rd such slip in one session). **GOTCHA that hid the scope:** `find … -user 0 | head -20` TRUNCATED (the `.old-` dir alone had 153 files, so the first page was all `.old-`) → I "verified clean" off a partial list. Never `head` a scope-defining find; count first (`| wc -l`). **Related blind-spot (muninn-dev):** a root-owned job SUBDIR passes every requeue guard (job_row/dispatch/list_jobs render fine) AND `/health` (contract's `os.access(ingestion_root, W_OK)` tests only the ROOT dir, so a foreign-owned subdir under `pending/` still reports `ingestion_root_writable: true`) — then the uid-1000 gate can't write into it. "Clean board + green /health + failure at next mutation." muninn-dev added an OWNERSHIP column to the standing post-move check to catch it; two green signals both miss a foreign-owned subdir otherwise.
  _Archived 2026-08-19._

- `[2026-07-25]` **Chaining the althing wake-listener arm orphans it.** `reply && althing-wake-listener &` (or spawning `althing-wake-listener` with `&` *inside* a `run_in_background` task) → the `&`-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a *plain operator turn* (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn `althing-wake-listener` as its OWN `run_in_background` task, and re-arm ONLY after a real fire (`<task-notification> completed rc0`). Reclaim an orphan with `althing-cli stop-monitor` then re-arm.
  _Archived 2026-08-19._

## Recent decisions (archived 2026-08-20 batch)

- `[2026-08-05]` **Booth — 3 features shipped, live on `:8090` + tagged.** (1) verbatim-`index.html` booths get a floating top-right "‹ all booths" chip + inherited favicon, doctype/charset-safe byte-injection (`booth-v0.1.5`, `8577e7e`); (2) `.md` renders + `.txt`/`.log` view in-booth without downloading via the `/b/<n>/view` route + a `markdown` dep + `doc.html` (`booth-v0.1.6`, `315faac`); (3) prev/next arrows in the image zoom viewer — wrap-around + keyboard ←/→, hidden for single-image booths (`booth-v0.1.7`, `c37a425`). Canonical `services/booth/`; deploy = `systemctl --user restart booth.service` on nh3-dev (runs from the checkout's `.venv`; `uv pip install` new deps into it first); 47 tests. `uv.lock` gitignored (`348c5c1`).
  _Archived 2026-08-20._

- `[2026-07-31]` **worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred.** memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on `vh/worldtree-sdk` (source pulled via the **Gitea API archive** — git-HTTP 403s on that repo); publishing to the vh USER namespace **can't be delegated** (401 `reqPackageAccess` even with `write:package`) so it needs an owner token — operator saved a **FULL vh site-admin token at `~/.config/gitea/vh-token` (0600)** for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
  _Archived 2026-08-20._

## Tried and abandoned (archived 2026-08-20 batch)

- `[2026-08-03]` **corviduo-dev shared containerd: a concurrent-pull race fails ONE instance's deploy; DON'T "prune to fix" — the image is in-use by the instance that won the race.** b169 personal deploy failed at `docker compose pull` (`Lchown … no such file or directory` on the big torch layer → looked like disk pressure / corrupt snapshot). ACTUAL: NOT disk (56G free, inodes 7%). demo + personal + pinned share ONE `/var/lib/containerd` on corviduo-dev; demo (from main) and personal (from staging tag) extracted b169's shared torch layer simultaneously → personal's hit a partial snapshot mid-race and aborted while demo's completed. The image `6e34a87` was FULLY VALID — demo was RUNNING it healthy. Fix = just re-run the failed deploy (image already materialized; compose pull finds it present). **NEAR-MISS:** worldtree-dev's suggested "prune unused images/snapshots" would have rmi'd `6e34a87` = the image the running demo depends on → demo outage. **Lesson: before any prune/rmi "cleanup," `docker ps` the running images — an "unused" image may be a co-tenant's live one; and verify the failure's REAL cause (disk? inode? in-use? race?) before applying the suggested remedy.** (Pipeline fix, deferred: serialize demo-from-main + personal-from-staging, or a per-image pull lock, to avoid the shared-layer extraction race.)
  _Archived 2026-08-20._

- `[2026-08-02]` **`mv <job> complete/ → failed/` RENAMED the job to `failed` because failed/ didn't exist.** worldtree-dev's round-2 unblock command (`mv /data/state/ingestion/complete/<job> /data/state/ingestion/failed/`) assumed `failed/` existed; on PERSONAL muninn it did NOT (fresh instance — root was `active/ complete/ pending/ sources/`, no `failed/`). `mv src nonexistent/` **renames** src→nonexistent, so job1 became the `failed` dir and job2 nested inside it. Caught on post-move `ls` (failed/ held job *contents*, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt `failed/` (worldtree:worldtree 755) — NO data loss. **Lessons:** (1) before `mv X into-dir/`, verify the dir EXISTS (`[ -d dir ]`) — an empty `ls dir/ 2>/dev/null` is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is **`mv -t <targetdir> <src>`** (`--target-directory`): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a **trailing slash does NOT protect** — `mv src failed/` with `failed/` missing STILL silently renames to `failed` (rc=0); "just add the slash" is a false guard. (`mkdir -p failed/` first also works, but `mv -t` inverts the failure from silent-wrong to loud-safe in one flag.) Container `sh` is dash — no `(` in echo strings. **SILENT failure mode (muninn-dev carry-forward):** a misplaced ingestion-state move doesn't crash anything — `list_jobs()` stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (`job_row`→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (`job_row` found + guards pass), don't trust mv exit codes — and confirm `job.dispatch.json` survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. **DON'T TIDY (round-2 pending):** both DCC + P&P jobs currently REST in personal `failed/` with manifests reading `state: complete` until round-2 requeue runs — deliberate + load-bearing (`requeue` keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. **Round-2 sequencing:** the requeue is **mimir-dev's** browser flow (pending their operator's board-vs-API ruling); **muninn-dev** is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); **infra-ops** = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z. **ARC COMPLETE (2026-08-03 ~05:49):** both books terminal — DCC `mimir-6351554e8e8f` 705 concepts + P&P `mimir-f3887c9b97b7` 667, extracted AND indexed, 5/5 phases, 0 failures/truncations (validates the #385 budget fix vs April's 785 control); **#381 restart-after-ingest FIRED** (personal api, healthz/readyz 200 ~25s), retrieval-visibility confirmed (search_library returns DCC+P&P from fiction post-restart); handed ratatoskr-verify go to worldtree-dev. **Delete-sweep precondition NOW MET** — the stale DCC rows in `main` are genuine duplicates of live `fiction` rows, so worldtree-dev's supervised sweep of the ~785 April orphans is unblocked (still comes to me supervised: snapshot + operator-in-loop).
  _Archived 2026-08-20._