# Persistent memory — eshpfi-management

_Last updated: 2026-06-02_

## Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical
Docker Compose stacks, ops playbooks, and conventions. Authoritative
copies of compose files live on the servers under
`/opt/docker/compose/<stack>/`; this repo mirrors them for version
control, editing, planning, and CI-driven deploys.

## Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

| Repo | Role | CI status |
|---|---|---|
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) |
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
| `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) |
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
| `vh/worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration | push-to-main → CI deploys |
| `vh/volva` | Codex peer agent on althing bus (single-turn oracle, systemd daemon on nh3-dev) | manual install via `deploy/volva.service` (2026-05-18) |

- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
  fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
  VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
  cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
  for the blast-radius matrix.

- **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
  model/dataset onto ana-ml2's shared cache at
  `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns.

- **Worldtree admin auth — per-instance.** Each Worldtree deployment
  (demo :8080, personal :8081, pinned :8082) has its own Heimdall
  registry and its own bootstrap admin key. Infra-ops's stored
  long-lived admin key (`key_id 61419c92`) at
  `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
  auths against **demo only**. For personal-instance admin ops, fetch
  the bootstrap admin per-op via
  `docker exec worldtree-personal-worldtree-api-1 printenv WORLDTREE_BOOTSTRAP_ADMIN_KEY`
  on corviduo-dev. Used for `POST /admin/keys`, admin diagnostics
  (`/admin/sessions/<id>/{bifrost,tools}`, etc.).

- **Per-project user keys against personal Worldtree** (issued
  2026-05-19): `skaldsong:79744637` (nh3-dev iteration),
  `skaldsong:7c1dbbbe` (ana-docker prod), `althing:50d85460`,
  `mead-hall:a360822d`. Same `user_id=skaldsong` across both
  skaldsong keys → shared Heimdall agent slot; different `key_id`
  → independently rotatable. Pattern: mint via `/admin/keys`, drop
  value to `/tmp/wt-personal-<name>.key` mode 600, dev collects +
  shreds (DO NOT cat to chat transcript).

- **Skaldsong CD pattern (registry-pull).** Differs from althing /
  asset-engine which build-on-host. vh/skaldsong's CI builds and
  pushes `gitea.phasefinal.com/vh/skaldsong:<sha>` + `:latest`;
  `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates.
  SHA-pin only (no `:latest` health-gated advance yet). Prereq: host
  needs `docker login gitea.phasefinal.com` once (read:package PAT) —
  not currently in the workflow.

- **docker-as-root pattern** (for ops that have no admin API, e.g.
  `SqliteUserStore.set_bifrost_credentials`): on hosts where the SSH
  user is in the `docker` group but lacks passwordless sudo, run
  `docker run --rm -v <target-dir>:/wt -v /var/run/docker.sock:/var/run/docker.sock docker:cli sh -c "..."` to edit deploy-owned files
  without sudo. Documented with security warning in
  `servers/corviduo-dev/README.md`. docker-group membership is
  effectively root via bind-mount; treat as a sudo-equivalent grant.
  **Foot-gun: when running `docker compose` inside this sandbox,
  any relative path in compose.yaml (e.g. `${WORLDTREE_CONFIG_DIR:-./config}`)
  resolves against the sandbox CWD, but Docker daemon interprets the
  resulting path against the HOST filesystem. Always pass `-e VAR=/abs/path`
  to the docker run invocation for any relative-default config dir.**

- **`scripts/elway` sudo handling** — elway prompts for the sudo password
  ONCE via `getpass` before the first `sudo: true` step. That prompt is
  interactive → elway can't run unattended from a non-TTY tool if any step
  needs sudo. For sudo-free playbooks (no `sudo: true` steps) it runs fully
  non-interactive over key SSH. To create root-owned dirs WITHOUT host sudo,
  use the docker-daemon-root trick: `docker run --rm -v /worktank:/mnt alpine
  sh -c 'mkdir -p /mnt/<x> && chown -R 1000:1000 /mnt/<x>'`.

## Current state / in-flight

_As of 2026-06-02:_

- **PRIMARY FOCUS — building `chatterbox-fast`, a custom streaming TTS
  container; Chatterbox is becoming our MAIN TTS engine.** Operator-authorized
  high-effort build (incl. custom container from source). **Plan-of-attack:
  `docs/design/chatterbox-fast-plan.md`** (durable; carries the full executable
  detail). The old `docs/design/chatterbox-fast.md` was KILLED 2026-06-01 — its
  §5 windowed-token-streaming was the abandoned native arc; superseded by the
  adaptive-chunk plan. Phase 1 built: `stacks/chatterbox-fast/` (scheduler +
  app + sim tests).
  - **Goal:** cut time-to-first-audio from ~2.5s → sub-second via streaming,
    keep turbo quality. Workload = single-stream interactive.
  - **Chosen approach = adaptive buffer-ratchet chunking** (operator's idea):
    generate the FIRST sentence alone for instant response (~0.66s first-audio
    measured), then while it plays, generate the largest sentence-aligned chunk
    that fits in ~80% of the bought playback time; chunks grow ~3× each (driven
    by Chatterbox's ~3.8× realtime headroom), so after 2-3 chunks the rest of
    the paragraph is one big chunk with near-full context. Context loss confined
    to 2-3 joins at natural sentence-pause boundaries. THIS is "sentence-level
    done right" and supersedes naive per-sentence splitting (which loses
    cross-sentence prosodic context = real quality loss, operator-corrected).
    Only works because RTF>1 (Fish at <1× realtime starves — can't use this).
  - **Native frame-level streaming on turbo = ABANDONED** (see Tried/abandoned).
  - **Also to build for A/B:** base-chatterbox + davidbrowne17 streaming fork
    (true frame-stream but base-model quality) — not yet installed.
  - **A/B samples** (GLaDOS voice) on nh3-dev `~/chatterbox-ab/`: sentence-level
    turbo, chunked-native (artifacty), chunked-oneshot.
  - **Dev/test pattern:** one-off GPU-1 container from `local/chatterbox:v1`,
    mount `/worktank/chatterbox/{cache,reference_audio}` + the test script;
    `docker exec -i chatterbox python -` for lib introspection. lkraven is in
    the `docker` group on irv-ml1 — NO sudo needed for docker.
- **TTS fleet on irv-ml1 reworked this session; asset-engine catalog now
  17 services** (CSM removed). Canonical `docs/asset-engine/services.yaml`
  + vendored `vh/asset-engine` both pushed (eshpfi `38d9e3b`; asset-engine
  `0b552bd`). Per-engine live state:
  - **dia2-2b (:8200) + dia2-1b (:8202)** — LIVE + healthy (restarted this
    session; `local/dia:v2`, voices + cache intact). `status: ready`.
  - **Chatterbox (:8196)** — LIVE. Catalog **v2**: routes to `/tts`, exposes
    emotion levers (exaggeration 0.5, cfg_weight 0.5, temperature 0.8), 28
    built-in predefined voices (default `Emily.wav`) + clone. Fast + competent
    cloner (it audibly cloned Imogen). Plain `Imogen.wav` uploaded to its clone
    dropdown (`/get_reference_files` now: Imogen.wav + glados_25s.wav).
  - **Fish-S2 (:8195)** — LIVE but **HEAVY**: ~240s torch.compile warmup on
    cold start, ~24 GB on the A6000; first call after any restart eats the
    warmup. Catalog **v6**: full `/tts` ServeTTSRequest surface
    (temperature/top_p/repetition_penalty/latency/seed/format) + `reference_id`
    is a 32-voice static "Voice" dropdown (default Emily) + advanced base64
    clone. NOT OpenAI-compat (only `/v1/tts`, no `/openapi.json`, no `/voices`
    API). Research/non-commercial license.
  - **zonos (:8203)** down (adapter built, not deployed). **ComfyUI :8188**
    catalog-deferred link.
- **"undefined" select bug — RESOLVED.** asset-engine-dev shipped form-select
  hardening (v0.1.9/.10) + a durable **per-field enable-toggle** (`togglable`,
  v0.1.14/.16) — the real fix for the "form submits untouched fields" family.
  My catalog half: opted fish-s2 `references` into `togglable: true`
  (`catalog_version 1→2`, schema gained the `togglable` boolean; `c98a12b`,
  pushed to origin). The earlier `blendable` idea was superseded by `togglable`.
  dia2/chatterbox clone fields deliberately NOT toggled (dia2 defaults clone-on
  as its stable voice). asset-engine CI was briefly red because the commit was
  local-only until pushed — lesson: push catalog commits promptly (their CI
  drift-checks against the remote).
- **Fish "not British" had TWO independent root causes — BOTH now fixed.**
  The ECAPA-TDNN re-test (2026-06-01) showed Fish's cloning *engine* works: an
  Imogen reference scores **~0.79 cosine vs real `Imogen.wav`** vs **~0.10 vs
  the no-reference default** — but that test fed the reference via the inline
  base64 `references` path, which masked the real defect. **`reference_id`-by-
  name (the path the catalog + form actually use) was silently a NO-OP.** Proof
  (2026-06-01): `reference_id=Abigail`, `=Imogen`, and no-reference all returned
  **byte-identical** audio (md5 `40b6ac73…`, same text+seed) → reference_id was
  being ignored entirely, so every dropdown voice collapsed to Fish's default
  speaker. Cause: fish-speech resolves `reference_id=<name>` against the
  **directory** `references/<name>/` (audio + same-basename `.lab`), but the
  voices were staged as **flat** `references/<name>.wav` + `.txt` with the
  per-name dirs left **empty** → nothing resolved. **FIX (2026-06-01):**
  populated `references/<name>/<name>.wav` + `<name>.lab` for all 32 voices;
  re-test → Imogen/Eleanor/Beatrice/Abigail/no-ref all 5 **distinct**.
  reference_id now works; British accents come through. Hardened: the
  deploy playbook gained a normalize-layout step + an A/B smoke gate
  (`reference_id` MUST change output) so the empty-dir regression can't ship
  silently. So the "not British" saga was (A) the `"undefined"` select bug
  (asset-engine, still belt-and-suspenders) AND (B) this reference_id no-op
  (server-side, FIXED) — B alone would have kept every voice identical even
  with A fixed. Earlier "reference_id-by-name is THE working path, verified"
  claim RETRACTED. Pitch-shift deepening abandoned; future clones operator-
  handled.
- **On-host consented voice library** — ~992 real-person clips cached in the
  kyutai tts-voices repo (`/worktank/kyutai-tts/.../snapshots/.../`): VCTK
  (CC BY 4.0, accent-tagged speaker IDs), Unmute voice-donations (CC0), EARS +
  expresso (CC BY-NC). Source for future clone voices. British-female Southern
  England speakers p225/p228/p229 staged into Fish as Imogen/Eleanor/Beatrice.
- **Parakeet ASR (:8765) is now UP** (brought online 2026-06-02 to transcribe
  the glados clip; operator wants it kept online). CPU-only (`gpu_device_id:
  null`), `restart=unless-stopped`, healthy. API: POST multipart `file` to
  `/transcribe` → `{"text":...}`. Image `local/parakeet:sherpa-onnx-v2`
  pre-built; `docker compose up -d` in `/opt/docker/compose/parakeet`.
- **NFS share for Smithy — DONE.** `/volume1/smithy` on nh3-nas → `/mnt/smithy`
  on nh3-dev (single export, RW, map-to-admin, scoped to nh3-dev only, fstab
  `_netdev,nofail,x-systemd.automount`). Subdirs `datasets/{raw,manifest-store,
  derived,holdout,quarantine}`; rename-atomicity verified. Smithy wired
  storage-roots.yaml + E2E-verified. **Backup DEFERRED** per operator (datasets
  regenerable; raw + manifest-store are the irreplaceable surface for when it's
  wired — durable tier should land OFF nh3-nas/cross-site, since the NH3 restic
  repo lives on the same NAS as the source).
- **Worldtree healthy v0.29.13** (last-known); **Skaldsong v0.32.2**
  (ana-docker:8300, Kokoro SSE streaming); **artemis-31b-v1i** live on
  llama-swap + worldtree personal; **ttyd fleet driver seat** on nh3-dev
  (RO :7681 / drive :7682 via Caddy) — all out of this repo, last-known.
- **Disclosed-keys hygiene queue** — rotate at convenience: HF token `hf_HBl…`
  (lkraven's HF account) leaked into BuildKit logs during the CSM build attempt
  (logs shredded, never committed — low urgency);
  `/tmp/wt-personal-skaldsong-prod.key` on nh3-dev; mead-hall's prior Worldtree
  bearer (superseded by `a360822d`); Worldtree `Z_AI_API_KEY`/`ZAI_API_KEY`;
  chamber `forseti`/`agent_runner` api_keys (superseded by `50d85460`); Gitea
  runner registration token (`a1135753…`).
- **Still open from prior sessions:** rotate `MINIFLUX_PASSWORD` (leaked
  twice); clean up legacy `news-digest` detritus on ana-docker; watch nh3-nas
  `/volume1` (was 65%; recheck before ~80%); the `docker push 60s client-side
  ceiling` mystery remains uninstrumented.

## Recent decisions

- `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast`
  streaming container.** Workload = single-stream interactive. **GPU placement:
  3090 (device 0) if it fits else A6000 (device 1)** — shared dev stack, 20.5 GB
  3090-idle is expected residency, not a blocker. **Cutover: parallel catalog
  entry**, burn in beside live `chatterbox`, then flip. **Streaming approach:
  adaptive buffer-ratchet chunking** (see in-flight). Native frame-streaming
  abandoned (Tried/abandoned). Tracked: `docs/design/chatterbox-fast-plan.md`.

- `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** I
  claimed naive sentence-level streaming has "zero quality loss" — WRONG. The
  T3 AR backbone conditions prosody on the WHOLE text; splitting loses
  cross-sentence prosodic context (contextual delivery, declination, affect
  continuity) even though voice timbre stays (reference-conditioned). No
  *artifacts* ≠ no *quality loss*. Hence the adaptive-chunk design (maximize
  context per chunk subject to latency budget), not fixed per-sentence splits.

- `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — see
  in-flight + Tried/abandoned. Populated `references/<name>/<name>.wav`+`.lab`
  for all 32 voices; playbook gained normalize-step + A/B smoke gate. glados got
  a real transcript (ASR'd via Parakeet): the Portal "Welcome to test chamber 4"
  lines.

- `[2026-06-01]` **Fish cloning VERIFIED competent (ECAPA-TDNN)** — retracting
  the earlier "weak cloner" call. Isolated test: Imogen-referenced clone ~0.79
  cosine to the real `Imogen.wav` vs ~0.10 for the no-reference default;
  transcript condition (correct 0.787 / bogus 0.778 / empty 0.738) barely moves
  identity (affects pronunciation, not timbre). Root cause of "Imogen sounds
  nothing like British" = the `"undefined"` select bug feeding Fish its default
  voice, NOT Fish. So the entire Fish-Imogen saga was the undefined bug; no
  Fish-side fix needed. (Methodology lessons → Tried and abandoned.)

- `[2026-06-01]` **CSM (Sesame csm-1b) torn down entirely** — removed from
  catalog, `stacks/csm/`, `playbooks/deploy-csm.yaml`, and host
  (`c54ab13`). Two reasons: (1) deep-research verdict — the acclaimed
  Maya/Miles demo is a fine-tuned, larger variant Sesame never open-sourced;
  the open 1B base ships no usable voices, can't generate text, English-only,
  no real-time streaming, absent from leaderboards, dominated by
  Kokoro/Dia2/Fish/IndexTTS. (2) The upstream image is rotted (see Tried and
  abandoned). asset-engine re-synced (`306aae9`).

- `[2026-06-01]` **Chatterbox catalog v1→v2** (`93cedd6`) — re-point to the
  devnen wrapper's `/tts` (was the thin OpenAI `/v1/audio/speech` exposing
  none of the emotion knobs → poor output). Default `exaggeration: 0.5` per
  Resemble's README ("works well for most"); the server SHIPS demo-tuned 1.3
  which over-acts — deliberately NOT adopted. 28 predefined voices via
  `/get_predefined_voices` (default Emily.wav). Corrected seedable false→true.

- `[2026-06-01]` **Fish-S2 catalog v1→v6** (`f3c260e`..`38d9e3b`) — was a
  blank 2-field form; now exposes the full fish-speech ServeTTSRequest surface
  + a 32-voice `reference_id` "Voice" dropdown (STATIC list — Fish has no
  /voices API). `reference_id`-by-name works ONLY when each voice is staged as
  a `references/<name>/` dir with `<name>.wav` + `<name>.lab` — was silently a
  no-op until that layout fix on 2026-06-01 (see the two-root-causes entry
  above); the inline-path-as-filesystem-path `references` shape 500s (see Tried
  and abandoned). Corrected vram
  4→24 GB, cold_start 8→240s, seedable false→true. Added 3 consenting
  British-female VCTK voices (Imogen=p225/Eleanor=p228/Beatrice=p229).

- `[2026-06-01]` **`blendable: false` catalog flag — DEFERRED**, pending
  asset-engine-dev honoring it in the form template. Clone-voice selects
  (chatterbox/fish/dia2) must render as plain dropdowns, not the Kokoro blend
  widget (blending reference clones is nonsensical). I ship the catalog flag +
  schema field once they read it. Tracked: althing msg
  `01KT2K2SY9N7AY69R9V0B4RXSW` to asset-engine-dev.

- `[2026-05-31]` Dia2 deployed as **two fixed-model instances**
  (`dia2-2b` :8200, `dia2-1b` :8202) from `local/dia:v2`, retiring
  legacy Dia 1.6B; catalog `dia` entry **removed → dia2-2b + dia2-1b**
  (breaking for asset-engine). Rationale: the devnen wrapper is
  single-model and IGNORES the OpenAI `model` field (verified on its live
  OpenAPI), so the only way to offer both Dia2 models as real per-request
  asset-engine choices is one fixed endpoint per model. `3139e81`
  (deploy), `db15638` (catalog swap).

- `[2026-05-31]` Both dia2 catalog entries route to the wrapper's richer
  **`/tts`** endpoint (not `/v1/audio/speech`) to expose the full control
  surface (cfg_scale/temperature/top_p/cfg_filter_top_k/voice_mode/clone);
  all defaults sourced from the wrapper's `CustomTTSRequest` Pydantic
  blessed values. Voice default is `voice_mode: clone` +
  `clone_reference_filename: Abigail.wav` so a stable (non-random-gender)
  voice is the out-of-box behavior. `55602b7`, `5c47843`.

- `[2026-05-31]` Zonos REST adapter (`stacks/zonos/adapter/`,
  `local/zonos-api`) — thin OpenAI-ish `/v1/audio/speech` FastAPI in front
  of the Gradio-only Zonos SDK; JSON-envelope `{audio, audio_format,
  seed}` (Zonos is the fleet's first seedable TTS). Port 8203 (moved off
  8201 — collided with csm). Built; NOT deployed (stack down for VRAM).
  Also fixed the upstream image's missing CMD (`71df6f7`). `81efa8d`.

- `[2026-05-31]` Catalog schema regenerated: added `CatalogLifecycle` +
  `reproducibility.seed_field` (`b7b2130`). Resolves the stale-schema
  hand-off; catalog now validates clean. (asset_engine `catalog.py`
  Pydantic already supported both — schema file was just behind.)

- `[2026-05-31]` TTS bench expanded with `stacks/{dia,zonos,csm}`
  (`666f7f3` dia+zonos, `a4b8c2a` csm). The bench already had Fish
  S2-Pro / Chatterbox-Turbo / IndexTTS-2 / CosyVoice3 / Kokoro /
  VibeVoice / Qwen3-TTS / Kyutai. (csm since removed 2026-06-01.)

- `[2026-05-31]` Remote browser/iPad/Vision-Pro driver seat for the
  agent-fleet zellij session `Claude` stood up on nh3-dev (ttyd behind
  Caddy, network-gated). Out of this repo — full architecture + the
  HTTP2/OSC52/Safari-auth gotchas in auto-memory `reference_ttyd_fleet_seat`.

- `[2026-05-30]` esh-docker-vm NFS boot-ordering fix:
  `playbooks/fix-esh-nfs-boot-ordering.yaml` (`c0458d9`, + `53157b1`
  drop-in filename-collision fix) adds `_netdev,nofail` to the four
  10.0.50.50 NFS mounts + a docker `After=remote-fs.target` drop-in —
  resolves paperless `Exited(255)` on reboot. traefik also gained
  `restart: unless-stopped`. Full incident → auto-memory
  `incident_esh_docker_nfs_boot_race`.

_25 older entries archived to archival-memory.md._

## Tried and abandoned

- `[2026-06-02]` **Context-priming at chunk joins (chatterbox-fast §1.6) —
  ABANDONED (discard-cut leaks the prefix).** To give a chunk backward prosodic
  context, prepend the prior sentence, generate `prefix+content` together, then
  discard the prefix audio. Built + opt-in shipped (commit d707439), live-A/B'd,
  reverted (090e70a). The kill: `generate()` returns one finished waveform with
  NO marker for where the prefix ends, and the model renders the same prefix with
  different timing solo vs followed-by-content — so locating the cut (generate
  prefix solo → measure duration → snap to nearest energy-min pause within ±0.4s)
  is a guess that left a whole clause of prefix in the output ("...without a trace
  of sarcasm," spoken twice; operator caught it). A reliable cut needs token-level
  boundaries (= the abandoned native-streaming arc) or per-chunk ASR/forced-
  alignment (heavy, imperfect, eats the latency budget). → Coherence loss at joins
  stays an ACCEPTED limitation; cold adaptive-chunk streaming judged "really good".
  Scheduler-side work that DID land + survive: affordability-gated priming math
  (a 2nd pass can't starve the buffer) — sound, but moot without a working cut.

- `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED
  (turbo isn't built for streaming).** Long R&D arc; record so it's not
  re-derived. (1) The model's flow is CosyVoice2-derived but `S3GenStreamer` is
  referenced-in-docstring-only (not implemented). (2) The lib's
  `flow_inference(finalize=False)` is BUGGY: the lookahead trim removes
  `pre_lookahead_len(3)*token_mel_ratio(2)=6` frames from `h` but NOT from
  `h_masks`/conds → decoder shape mismatch (e.g. 656 vs 662). A 1-line patch
  (`h_masks = h_masks[:, :, :-pre*ratio]` after the `h` trim) + sizing the
  meanflow noise to the trimmed length makes finalize=False RUN. (3) BUT the
  flow encoder uses FULL-context attention (`static_chunk_size=0`), so
  incremental/cumulative decode is **prefix-unstable** — adding tokens
  re-attends and shifts earlier mel (maxdiff ~0.30-0.39 vs one-shot,
  irrespective of fixed-noise slicing or emit-margin). (4) Forcing
  `static_chunk_size>0` on the 2 modules that carry the attr did NOT stabilize
  it (decoding_chunk_size is a forward-arg, not settable via attribute). Verdict:
  true sub-second frame-streaming on turbo needs deep model-attention surgery
  with quality risk — not worth it. Matches research ("turbo+streaming
  unsolved"; vLLM-turbo outputs noise; davidbrowne17 streaming fork is
  BASE-only). → Use adaptive-chunking instead.

- `[2026-06-02]` **Naive cumulative re-decode for streaming** — decode the
  growing token prefix each chunk + emit the delta, assuming the causal flow
  gives a stable prefix. It does NOT (full-context attention, see above);
  maxdiff 0.30. Also the high-level `s3gen.inference` re-applies `trim_fade` at
  the start every call. Don't go this way.

- `[2026-06-01]` **CSM bring-up** — upstream `phildougherty/sesame_csm_openai`
  Dockerfile pins NO `huggingface_hub` version, so it now resolves to 1.17.0
  where `huggingface-cli` is fully removed (replaced by `hf`) → the build dies
  at `huggingface-cli login`/`download`. Passing the HF token as a build-arg
  got the token in (the model downloads at BUILD time, not runtime — our
  compose only wired it as runtime env, the original failure) but then hit the
  rotted CLI. Fixing would need vendoring + patching the Dockerfile; not worth
  it for a weak model → CSM removed.

- `[2026-06-01]` **Fish path-form references** (`{audio:"/app/references/x.wav"}`)
  → HTTP 500 on this build, despite the stack README showing that shape. The
  working clone paths are `reference_id` (by name → resolves the DIRECTORY
  `references/<name>/` containing `<name>.wav` + `<name>.lab`; the flat
  `references/<name>.wav` is ignored — see the two-root-causes entry above) or
  inline base64 `references`. The catalog uses `reference_id`.

- `[2026-06-01]` **Reference transcript barely affects Fish clone IDENTITY**
  (disproving my mid-session theory). I'd blamed a bogus provenance-note `.txt`
  for poor cloning, but the ECAPA re-test showed correct (0.787) / bogus (0.778)
  / empty (0.738) transcripts all clone Imogen about equally — the transcript
  affects PRONUNCIATION (phoneme disambiguation per the docs), not who it sounds
  like. The real culprit for "not British" was the `"undefined"` select bug, not
  the transcript. (A correct transcript still marginally helps pronunciation —
  cheap to stage, not load-bearing.)

- `[2026-06-01]` **Pitch-shift register control** (rubberband, to deepen Imogen
  to contralto/mezzo) — Fish ignores small reference shifts and overshoots
  large ones; `-3st` WITHOUT formant preservation → "baritone through a tuba"
  (formants collapsed); `-2/-3st` WITH `--formant` → too subtle / still bad.
  Abandoned at every depth; all variants deleted. Finer independent
  pitch/formant control needs praat (not installed). Future clones = operator's.

- `[2026-06-01]` **resemblyzer is too weak for cloning-fidelity scoring** — its
  dated 2019 LSTM encoder + a synthetic-vs-natural domain gap scored the Imogen
  clone CLOSER to the default than to real-Imogen, which led me to a WRONG "Fish
  is a weak cloner" call. ECAPA-TDNN (speechbrain `spkrec-ecapa-voxceleb`) on the
  same clips gave the correct answer (clone 0.79 to real Imogen, 0.10 to
  default). Use ECAPA, not resemblyzer, for speaker-verification.

- `[2026-05-31]` Building the dia2-capable image surfaced THREE upstream
  packaging quirks: (1) `pip install -e nari-labs/dia2` fails — no PEP 660
  `build_editable` hook; (2) plain `pip install` builds an empty
  `UNKNOWN-0.0.0` wheel (base setuptools 59.6 < dia2's required ≥70); (3)
  `--no-deps` leaves `transformers`/`sphn`/`whisper-timestamped` missing. Fix
  (`local/dia:v2`): copy the pure-python `dia2/` package into site-packages +
  install ONLY those 3 deps; base torch/numpy already satisfy Dia2.

- `[2026-05-31]` Dia2 predefined voices (43, baked at `/app/voices`) are NOT
  reachable from the `/tts` clone path — it resolves `clone_reference_filename`
  against the reference_audio dir ONLY. The OpenAI `/v1/audio/speech` `voice`
  param auto-resolves them (separate code path), which masked the gap. Fix:
  stage `/app/voices/*` into `/worktank/dia/reference_audio`. Lesson: verify on
  the endpoint the catalog ACTUALLY targets.

- `[2026-05-31]` `voice_mode=clone` with an empty `clone_reference_filename`
  → asset-engine serializes it as the literal string `"undefined"` → `/tts`
  404. First observed on dia2; worked around in the catalog (default the field
  to a real voice). **[2026-06-01] root cause found** — the Kokoro voice-blend
  widget reading Shoelace `.value` before hydration (see Current state); the
  real fix is asset-engine-side and is escalated.

- `[2026-05-31]` asset-engine `services.schema.json` is DERIVED (regen from the
  Pydantic model via `dump_schema.py`) and had DRIFTED — rejected the
  `lifecycle` field 12/14 services use. RESOLVED: regenerated with
  `CatalogLifecycle` + `reproducibility.seed_field` (`b7b2130`). Lesson:
  hand-editing `services.yaml` shape without regenerating re-introduces drift.

- `[2026-05-31]` ttyd-over-TLS forces HTTP/2 (kills ttyd's terminal WebSocket
  → blank screen); Safari/WebKit never sends HTTP basic-auth on WS upgrades.
  Both solved for the fleet seat (Caddy forces HTTP/1.1; auth → network-gating)
  — detail in auto-memory `reference_ttyd_fleet_seat`.

- `[2026-05-30]` esh-docker-vm: `hard` NFS mounts from 10.0.50.50 froze a
  container worker in UNKILLABLE D-state when the NAS stalled — only a host
  reboot clears it. Separately, `fstab defaults` (no `_netdev`) made NFS-bind
  containers `Exited(255)` on reboot. → auto-memory
  `incident_esh_docker_nfs_boot_race`.

_29 older entries archived to archival-memory.md._