memory: snapshot — CSM teardown, Chatterbox v2 + Fish v6 reworks, undefined-bug root cause, dia2 restart; archive 8 (2026-05-27) entries
This commit is contained in:
@@ -174,6 +174,25 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
||||
to PATCH (which silently 2xx'd on the pre-fix Worldtree).
|
||||
_Archived 2026-05-31._
|
||||
|
||||
- `[2026-05-27]` Worldtree v0.29.x landed full saga→echo refactor +
|
||||
config-validator hardening (v0.29.10 create_provider family-before-regex;
|
||||
v0.29.11 collect-then-raise echo startup validators; v0.29.12 ships
|
||||
providers.yaml.example/defaults.yaml.example canonical configs; v0.29.13
|
||||
reasoning_content extraction + catalog family lookup). Operator-asked,
|
||||
worldtree-dev-shipped, CI-deployed.
|
||||
_Archived 2026-06-01._
|
||||
|
||||
- `[2026-05-27]` artemis-31b-v1i added to llama-swap + worldtree personal.
|
||||
BeaverAI Gemma 4 31B Q6_K (~28.6GB), 128K ctx, `--reasoning-format deepseek`
|
||||
(gemma format unsupported in deployed llama.cpp). Worldtree catalog
|
||||
`family: gemma` so GemmaProvider routes reasoning tokens.
|
||||
_Archived 2026-06-01._
|
||||
|
||||
- `[2026-05-27]` Skaldsong streaming TTS v0.32.0→v0.32.2: chunked-batch SSE
|
||||
(one Kokoro POST per paragraph); defensive event_stream catch-all; NDJSON
|
||||
parsing for Kokoro /dev/captioned_speech multi-line responses.
|
||||
_Archived 2026-06-01._
|
||||
|
||||
## Tried and abandoned (archived)
|
||||
|
||||
- `[2026-04-30]` task-board workflow with
|
||||
@@ -383,3 +402,32 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
||||
substring replace on YAML top-level keys WILL match suffix-containing
|
||||
keys.
|
||||
_Archived 2026-05-31._
|
||||
|
||||
- `[2026-05-27]` `docker compose up -d` inside the `docker:cli` sandbox:
|
||||
`${VAR:-./config}` defaults resolve `./config` to the sandbox CWD, but the
|
||||
Docker daemon interprets the path against the HOST filesystem → auto-creates
|
||||
an empty dir → entrypoint reseeded image-baked defaults (lost host-side
|
||||
providers.yaml patches). Fix: pass `-e WORLDTREE_CONFIG_DIR=/abs/path`.
|
||||
Folded into the docker-as-root convention note.
|
||||
_Archived 2026-06-01._
|
||||
|
||||
- `[2026-05-27]` `:latest`-pinned compose + private gitea registry + sandboxed
|
||||
pull = recreate on ancient cached `:latest` (deploy pulls by SHA so the tag
|
||||
never advances; sandbox can't pull). Fix: retag SHA→`:latest` on host, then
|
||||
`--pull never`. Better: pin SHA in `.env`, advance in CI.
|
||||
_Archived 2026-06-01._
|
||||
|
||||
- `[2026-05-27]` Container recreate during in-flight skaldsong gen kills the
|
||||
runner. With deploys every ~10min and stories >5min, structural not
|
||||
incidental. Roadmap (skaldsong-dev): pre-shutdown signal handler, per-scene
|
||||
resume-from-checkpoint, /api/admin/quiesce. None shipped.
|
||||
_Archived 2026-06-01._
|
||||
|
||||
- `[2026-05-27]` `--reasoning-format gemma` on artemis-31b-v1i — unsupported in
|
||||
the deployed llama.cpp (accepts none|deepseek|deepseek-legacy). `deepseek`
|
||||
populates the `reasoning_content` SSE delta Worldtree GemmaProvider checks.
|
||||
_Archived 2026-06-01._
|
||||
|
||||
- `[2026-05-27]` `head -c N` piped after a streaming curl SIGPIPEs the curl,
|
||||
killing the request early. Use file-write + separate read.
|
||||
_Archived 2026-06-01._
|
||||
|
||||
+176
-220
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-05-31_
|
||||
_Last updated: 2026-06-01_
|
||||
|
||||
## Repo purpose
|
||||
|
||||
@@ -77,84 +77,111 @@ Sister repos (separate gitea repos, deployed by playbooks here):
|
||||
resulting path against the HOST filesystem. Always pass `-e VAR=/abs/path`
|
||||
to the docker run invocation for any relative-default config dir.**
|
||||
|
||||
- **`scripts/elway` sudo handling** — elway prompts for the sudo password
|
||||
ONCE via `getpass` before the first `sudo: true` step. That prompt is
|
||||
interactive → elway can't run unattended from a non-TTY tool if any step
|
||||
needs sudo. For sudo-free playbooks (no `sudo: true` steps) it runs fully
|
||||
non-interactive over key SSH. To create root-owned dirs WITHOUT host sudo,
|
||||
use the docker-daemon-root trick: `docker run --rm -v /worktank:/mnt alpine
|
||||
sh -c 'mkdir -p /mnt/<x> && chown -R 1000:1000 /mnt/<x>'`.
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-05-31:_
|
||||
_As of 2026-06-01:_
|
||||
|
||||
- **Dia2 LIVE as two instances on irv-ml1; zonos + csm still down.**
|
||||
`dia2-2b` (:8200, Dia2-2B) + `dia2-1b` (:8202, Dia2-1B) running +
|
||||
healthy on GPU 0 from `local/dia:v2`; catalog `status: ready` (both
|
||||
exercised). Legacy Dia 1.6B retired. **zonos**: REST adapter built
|
||||
(`stacks/zonos/adapter/`, `local/zonos-api`) but the stack is DOWN —
|
||||
brought down to free 3090 VRAM for Dia2; catalog entry `status: down`,
|
||||
adapter port 8203. **csm**: still PARKED (`status: down`, gated — needs
|
||||
`CSM_HF_TOKEN` before first start; non-reproducible → `experimental`
|
||||
when up). ComfyUI `:8188` still a catalog-deferred UI **link** (open
|
||||
hand-off to asset-engine-dev, no form-field API).
|
||||
|
||||
- **Dia2 voices staged — re-stage if host rebuilt.** The 43 curated
|
||||
voices baked at `/app/voices` aren't reachable from `/tts`'s clone path,
|
||||
so they're copied into `/worktank/dia/reference_audio` (restic-included).
|
||||
This is a **one-time manual host step** (`docker exec dia2-2b sh -c 'cp
|
||||
-n /app/voices/* /app/reference_audio/'`), documented in
|
||||
`stacks/dia/README.md`, NOT auto-run on recreate. Catalog voice default
|
||||
is `voice_mode: clone` + `clone_reference_filename: Abigail.wav`. Open
|
||||
offer (not done): make staging auto-run via container entrypoint.
|
||||
|
||||
- **asset-engine catalog fully synced + pushed.** `docs/asset-engine/
|
||||
services.yaml` (canonical) and `vh/asset-engine`'s vendored
|
||||
`data/services.yaml` (re-synced via `sync_catalog.py`, commit
|
||||
`58dadfe`) both on origin; catalog v1, 18 services; schema regenerated.
|
||||
asset-engine deploy-on-push triggered. asset-engine-dev notified on
|
||||
althing thread `01KSZM5MNQYP2DZ25Y2F2DXDBY` of the breaking dia→dia2
|
||||
swap (drop dia tile; add dia2-2b/dia2-1b/zonos), the `/tts` field
|
||||
shapes, and a UI-hardening ask (don't serialize empty optional selects
|
||||
as the string `"undefined"`; enforce clone_reference_filename
|
||||
required-when-clone).
|
||||
|
||||
- **ttyd fleet driver seat operational on nh3-dev** (browser/iPad/Vision
|
||||
Pro into the zellij `Claude` session): RO `:7681` / drive `:7682` via
|
||||
a Caddy front, network-gated (no auth). Lives in `~/.config` on
|
||||
nh3-dev (out of this repo). Full wiring + gotchas → auto-memory
|
||||
`reference_ttyd_fleet_seat`.
|
||||
|
||||
- **Worldtree all instances healthy on v0.29.13** (last-known, not
|
||||
re-checked this session). Demo (:8080), personal (:8081), pinned
|
||||
(:8082). v0.29.x = saga→echo refactor + selene-judgment validator +
|
||||
canonical `*.yaml.example` configs.
|
||||
|
||||
- **Skaldsong on v0.32.2** (ana-docker:8300). Kokoro TTS streaming via
|
||||
SSE working end-to-end; default voice `af_heart`. Last-known.
|
||||
|
||||
- **artemis-31b-v1i live on llama-swap + worldtree personal.** BeaverAI
|
||||
Gemma 4 31B Q6_K (~28.6GB, 128K ctx), reasoning via
|
||||
`--reasoning-format deepseek`.
|
||||
|
||||
- **All commits pushed (operator-authorized 2026-05-31).** The prior
|
||||
pending-push backlog + this session's TTS/catalog work are all on
|
||||
origin (eshpfi-management through `5c47843`; asset-engine through
|
||||
`58dadfe`). No local-only backlog. (Standing rule still holds: never
|
||||
push without an explicit greenlight — this push was granted.)
|
||||
|
||||
- **VibeVoice catalogued for skaldsong #73 contract** (TTS alongside
|
||||
Kokoro). Endpoint `http://10.100.79.3:8194`, OpenAI-compat. NO
|
||||
per-word timings (diffusion); read-along is a v2 forced-alignment
|
||||
upgrade (Parakeet irv-ml1:8765 or Whisper).
|
||||
|
||||
- **Disclosed-keys hygiene queue** — rotate at convenience:
|
||||
`/tmp/wt-personal-skaldsong-prod.key` on nh3-dev (shred after verify);
|
||||
mead-hall's prior Worldtree bearer (superseded by `a360822d`);
|
||||
Worldtree `Z_AI_API_KEY`/`ZAI_API_KEY` (2026-05-12 outage); chamber
|
||||
`forseti`/`agent_runner` api_keys (superseded by `50d85460`); Gitea
|
||||
runner registration token (`a1135753...`).
|
||||
|
||||
- **Still open from prior sessions:** rotate `MINIFLUX_PASSWORD`
|
||||
(leaked twice); clean up legacy `news-digest` detritus on ana-docker;
|
||||
watch nh3-nas `/volume1` (was 65%; recheck before ~80%); the
|
||||
`docker push 60s client-side ceiling` mystery remains uninstrumented.
|
||||
- **TTS fleet on irv-ml1 reworked this session; asset-engine catalog now
|
||||
17 services** (CSM removed). Canonical `docs/asset-engine/services.yaml`
|
||||
+ vendored `vh/asset-engine` both pushed (eshpfi `38d9e3b`; asset-engine
|
||||
`0b552bd`). Per-engine live state:
|
||||
- **dia2-2b (:8200) + dia2-1b (:8202)** — LIVE + healthy (restarted this
|
||||
session; `local/dia:v2`, voices + cache intact). `status: ready`.
|
||||
- **Chatterbox (:8196)** — LIVE. Catalog **v2**: routes to `/tts`, exposes
|
||||
emotion levers (exaggeration 0.5, cfg_weight 0.5, temperature 0.8), 28
|
||||
built-in predefined voices (default `Emily.wav`) + clone. Fast + competent
|
||||
cloner (it audibly cloned Imogen). Plain `Imogen.wav` uploaded to its clone
|
||||
dropdown (`/get_reference_files` now: Imogen.wav + glados_25s.wav).
|
||||
- **Fish-S2 (:8195)** — LIVE but **HEAVY**: ~240s torch.compile warmup on
|
||||
cold start, ~24 GB on the A6000; first call after any restart eats the
|
||||
warmup. Catalog **v6**: full `/tts` ServeTTSRequest surface
|
||||
(temperature/top_p/repetition_penalty/latency/seed/format) + `reference_id`
|
||||
is a 32-voice static "Voice" dropdown (default Emily) + advanced base64
|
||||
clone. NOT OpenAI-compat (only `/v1/tts`, no `/openapi.json`, no `/voices`
|
||||
API). Research/non-commercial license.
|
||||
- **zonos (:8203)** down (adapter built, not deployed). **ComfyUI :8188**
|
||||
catalog-deferred link.
|
||||
- **BLOCKED on asset-engine-dev — P1 "undefined" select bug** (default voice
|
||||
picks send the literal string `"undefined"` → Chatterbox 404s, Fish falls to
|
||||
default voice). ROOT-CAUSED: the form's Kokoro voice-blend widget
|
||||
(`templates/partials/_fields.html`) runs `recompute()` at page load, reading
|
||||
the Shoelace `<sl-select>.value` BEFORE hydration → writes "undefined" into
|
||||
the submit field. Escalated with full root cause + 3-part fix (althing msg
|
||||
`01KT2K2SY9N7AY69R9V0B4RXSW` → asset-engine-dev). **Workaround until fixed:
|
||||
click the dropdown (fires change) or hit the API directly.** My half
|
||||
(`blendable: false` catalog flag) is queued — see Recent decisions.
|
||||
- **Fish cloning verdict UNRESOLVED — re-test parked.** My in-session "Fish is
|
||||
a weak cloner" conclusion was confounded (bogus reference transcript + maybe
|
||||
`reference_id: "undefined"` from the form bug + weak resemblyzer encoder) —
|
||||
see Tried and abandoned. Literature says Fish/OpenAudio is SOTA cloning.
|
||||
Whisper has the real transcript ("If the red of a second bow…"). Clean
|
||||
re-test (correct transcript + ECAPA scoring) ready when wanted. Future voice
|
||||
clones are operator-handled (pitch-shift deepening abandoned).
|
||||
- **On-host consented voice library** — ~992 real-person clips cached in the
|
||||
kyutai tts-voices repo (`/worktank/kyutai-tts/.../snapshots/.../`): VCTK
|
||||
(CC BY 4.0, accent-tagged speaker IDs), Unmute voice-donations (CC0), EARS +
|
||||
expresso (CC BY-NC). Source for future clone voices. British-female Southern
|
||||
England speakers p225/p228/p229 staged into Fish as Imogen/Eleanor/Beatrice.
|
||||
- **Parakeet ASR (:8765) is DOWN** — no container, but catalog says
|
||||
`status: ready` (drift). Use Whisper-in-a-container (`faster-whisper`) for
|
||||
transcripts instead; it's self-contained (CTranslate2, no torch).
|
||||
- **Worldtree healthy v0.29.13** (last-known); **Skaldsong v0.32.2**
|
||||
(ana-docker:8300, Kokoro SSE streaming); **artemis-31b-v1i** live on
|
||||
llama-swap + worldtree personal; **ttyd fleet driver seat** on nh3-dev
|
||||
(RO :7681 / drive :7682 via Caddy) — all out of this repo, last-known.
|
||||
- **Disclosed-keys hygiene queue** — rotate at convenience: HF token `hf_HBl…`
|
||||
(lkraven's HF account) leaked into BuildKit logs during the CSM build attempt
|
||||
(logs shredded, never committed — low urgency);
|
||||
`/tmp/wt-personal-skaldsong-prod.key` on nh3-dev; mead-hall's prior Worldtree
|
||||
bearer (superseded by `a360822d`); Worldtree `Z_AI_API_KEY`/`ZAI_API_KEY`;
|
||||
chamber `forseti`/`agent_runner` api_keys (superseded by `50d85460`); Gitea
|
||||
runner registration token (`a1135753…`).
|
||||
- **Still open from prior sessions:** rotate `MINIFLUX_PASSWORD` (leaked
|
||||
twice); clean up legacy `news-digest` detritus on ana-docker; watch nh3-nas
|
||||
`/volume1` (was 65%; recheck before ~80%); the `docker push 60s client-side
|
||||
ceiling` mystery remains uninstrumented.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-06-01]` **CSM (Sesame csm-1b) torn down entirely** — removed from
|
||||
catalog, `stacks/csm/`, `playbooks/deploy-csm.yaml`, and host
|
||||
(`c54ab13`). Two reasons: (1) deep-research verdict — the acclaimed
|
||||
Maya/Miles demo is a fine-tuned, larger variant Sesame never open-sourced;
|
||||
the open 1B base ships no usable voices, can't generate text, English-only,
|
||||
no real-time streaming, absent from leaderboards, dominated by
|
||||
Kokoro/Dia2/Fish/IndexTTS. (2) The upstream image is rotted (see Tried and
|
||||
abandoned). asset-engine re-synced (`306aae9`).
|
||||
|
||||
- `[2026-06-01]` **Chatterbox catalog v1→v2** (`93cedd6`) — re-point to the
|
||||
devnen wrapper's `/tts` (was the thin OpenAI `/v1/audio/speech` exposing
|
||||
none of the emotion knobs → poor output). Default `exaggeration: 0.5` per
|
||||
Resemble's README ("works well for most"); the server SHIPS demo-tuned 1.3
|
||||
which over-acts — deliberately NOT adopted. 28 predefined voices via
|
||||
`/get_predefined_voices` (default Emily.wav). Corrected seedable false→true.
|
||||
|
||||
- `[2026-06-01]` **Fish-S2 catalog v1→v6** (`f3c260e`..`38d9e3b`) — was a
|
||||
blank 2-field form; now exposes the full fish-speech ServeTTSRequest surface
|
||||
+ a 32-voice `reference_id` "Voice" dropdown (STATIC list — Fish has no
|
||||
/voices API). `reference_id`-by-name is the WORKING clone path; the
|
||||
path-form `references` shape 500s (see Tried and abandoned). Corrected vram
|
||||
4→24 GB, cold_start 8→240s, seedable false→true. Added 3 consenting
|
||||
British-female VCTK voices (Imogen=p225/Eleanor=p228/Beatrice=p229).
|
||||
|
||||
- `[2026-06-01]` **`blendable: false` catalog flag — DEFERRED**, pending
|
||||
asset-engine-dev honoring it in the form template. Clone-voice selects
|
||||
(chatterbox/fish/dia2) must render as plain dropdowns, not the Kokoro blend
|
||||
widget (blending reference clones is nonsensical). I ship the catalog flag +
|
||||
schema field once they read it. Tracked: althing msg
|
||||
`01KT2K2SY9N7AY69R9V0B4RXSW` to asset-engine-dev.
|
||||
|
||||
- `[2026-05-31]` Dia2 deployed as **two fixed-model instances**
|
||||
(`dia2-2b` :8200, `dia2-1b` :8202) from `local/dia:v2`, retiring
|
||||
legacy Dia 1.6B; catalog `dia` entry **removed → dia2-2b + dia2-1b**
|
||||
@@ -185,171 +212,100 @@ _As of 2026-05-31:_
|
||||
Pydantic already supported both — schema file was just behind.)
|
||||
|
||||
- `[2026-05-31]` TTS bench expanded with `stacks/{dia,zonos,csm}`
|
||||
(`666f7f3` dia+zonos, `a4b8c2a` csm): dia = Nari dialogue TTS via
|
||||
devnen/Dia-TTS-Server (OpenAI-compat, :8200), zonos = Zyphra Gradio
|
||||
audition surface (:8199, no API), csm = Sesame conversational
|
||||
(gated model, :8201). dia + csm catalogued in
|
||||
`docs/asset-engine/services.yaml` `status: down` + repro audits
|
||||
(`8eac39b`); all three deployed PARKED to irv-ml1 (no GPU spin).
|
||||
Zonos + ComfyUI = catalog-deferred (no form-field API) → asset-engine
|
||||
UI **links**, handed to asset-engine-dev. The bench already had Fish
|
||||
(`666f7f3` dia+zonos, `a4b8c2a` csm). The bench already had Fish
|
||||
S2-Pro / Chatterbox-Turbo / IndexTTS-2 / CosyVoice3 / Kokoro /
|
||||
VibeVoice / Qwen3-TTS / Kyutai, so the research-agent's top-2 picks
|
||||
were already deployed.
|
||||
VibeVoice / Qwen3-TTS / Kyutai. (csm since removed 2026-06-01.)
|
||||
|
||||
- `[2026-05-31]` Remote browser/iPad/Vision-Pro driver seat for the
|
||||
agent-fleet zellij session `Claude` stood up on nh3-dev (ttyd behind
|
||||
Caddy, network-gated). Brokkr-smithy-dev request; no Tailscale (WG at
|
||||
firewall). Out of this repo — full architecture + the
|
||||
HTTP2/OSC52/Safari-auth gotchas in auto-memory
|
||||
`reference_ttyd_fleet_seat`.
|
||||
Caddy, network-gated). Out of this repo — full architecture + the
|
||||
HTTP2/OSC52/Safari-auth gotchas in auto-memory `reference_ttyd_fleet_seat`.
|
||||
|
||||
- `[2026-05-30]` esh-docker-vm NFS boot-ordering fix:
|
||||
`playbooks/fix-esh-nfs-boot-ordering.yaml` (`c0458d9`, + `53157b1`
|
||||
drop-in filename-collision fix) adds `_netdev,nofail` to the four
|
||||
10.0.50.50 NFS mounts + a docker `After=remote-fs.target` drop-in —
|
||||
resolves paperless `Exited(255)` on every reboot. traefik also gained
|
||||
`restart: unless-stopped` (was Docker-default `no` → never
|
||||
auto-started after a reboot; "homepage broken" was collateral via the
|
||||
traefik route). Full incident → auto-memory
|
||||
resolves paperless `Exited(255)` on reboot. traefik also gained
|
||||
`restart: unless-stopped`. Full incident → auto-memory
|
||||
`incident_esh_docker_nfs_boot_race`.
|
||||
|
||||
- `[2026-05-27]` Worldtree v0.29.x landed full saga→echo refactor +
|
||||
config-validator hardening. Net image:
|
||||
- v0.29.10 — `create_provider` checks model_catalog `family:`
|
||||
before regex auto-detect (artemis routing fix)
|
||||
- v0.29.11 — collect-then-raise echo startup validators (UX fix
|
||||
for cascading one-at-a-time crash cycle that bit demo + personal
|
||||
during the v0.29.0 → v0.29.9 deploy)
|
||||
- v0.29.12 — ships `providers.yaml.example` + `defaults.yaml.example`
|
||||
canonical configs (REQUIRED/OPTIONAL annotated, guaranteed-pass-
|
||||
validator). Replaces reverse-engineer-from-validator-errors as
|
||||
the operator's path.
|
||||
- v0.29.13 — `reasoning_content` extraction in base provider +
|
||||
catalog `family:` lookup fix (artemis reasoning route through
|
||||
worldtree GemmaProvider).
|
||||
Operator-asked, worldtree-dev-shipped, all auto-deployed via CI.
|
||||
|
||||
- `[2026-05-27]` artemis-31b-v1i added to llama-swap + worldtree
|
||||
personal providers.yaml. BeaverAI Gemma 4 31B Q6_K (~28.6GB)
|
||||
pulled via `pull-hf-repo.yaml`, llama-swap config mirrors
|
||||
gemma4-31b-dense settings at 128K ctx, `--reasoning-format deepseek`
|
||||
(gemma format not supported in deployed llama.cpp). Worldtree
|
||||
catalog entry has `family: gemma` so GemmaProvider routes reasoning
|
||||
tokens correctly. Pinned locally — push pending operator greenlight.
|
||||
|
||||
- `[2026-05-27]` Skaldsong streaming TTS v0.32.0 → v0.32.1 → v0.32.2.
|
||||
v0.32.0: chunked-batch via SSE on `/api/stories/{id}/audio?voice=X`,
|
||||
one Kokoro POST per paragraph. v0.32.1: defensive catch-all in
|
||||
event_stream after silent JSONDecodeError (made the real
|
||||
exception surface in an SSE error frame). v0.32.2: NDJSON parsing
|
||||
— Kokoro `/dev/captioned_speech` returns multi-line JSON for
|
||||
large inputs (one object per chunk); was using `resp.json()` which
|
||||
only parsed the first object. Verified end-to-end clean.
|
||||
|
||||
_22 older entries archived to archival-memory.md._
|
||||
_25 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-05-31]` Building the dia2-capable image surfaced THREE upstream
|
||||
packaging quirks: (1) `pip install -e nari-labs/dia2` fails — its
|
||||
pyproject build backend lacks the PEP 660 `build_editable` hook; (2)
|
||||
plain `pip install` builds an empty `UNKNOWN-0.0.0` wheel because the
|
||||
base image's setuptools (59.6) is older than dia2's required `>=70`, so
|
||||
it ignores the `[project]` metadata; (3) `--no-deps` (which the wrapper's
|
||||
own auto-installer uses) leaves `transformers`/`sphn`/`whisper-timestamped`
|
||||
missing. Fix that worked (`local/dia:v2`, `stacks/dia/dia2-image/`): copy
|
||||
the pure-python `dia2/` package straight into site-packages + install
|
||||
ONLY those 3 deps; torch 2.12 / numpy 2.2 in the base already satisfy
|
||||
Dia2, so no risky CUDA/torch churn.
|
||||
- `[2026-06-01]` **CSM bring-up** — upstream `phildougherty/sesame_csm_openai`
|
||||
Dockerfile pins NO `huggingface_hub` version, so it now resolves to 1.17.0
|
||||
where `huggingface-cli` is fully removed (replaced by `hf`) → the build dies
|
||||
at `huggingface-cli login`/`download`. Passing the HF token as a build-arg
|
||||
got the token in (the model downloads at BUILD time, not runtime — our
|
||||
compose only wired it as runtime env, the original failure) but then hit the
|
||||
rotted CLI. Fixing would need vendoring + patching the Dockerfile; not worth
|
||||
it for a weak model → CSM removed.
|
||||
|
||||
- `[2026-05-31]` Dia2 predefined voices (43, baked at `/app/voices`) are
|
||||
NOT reachable from the `/tts` clone path — it resolves
|
||||
`clone_reference_filename` against the reference_audio dir ONLY
|
||||
(`engine.py`). The OpenAI `/v1/audio/speech` `voice` param DOES
|
||||
auto-resolve them by name (separate code path), which masked the gap in
|
||||
early testing. Fix: stage `/app/voices/*` into
|
||||
`/worktank/dia/reference_audio` so `/tts` clone can use them. (So a
|
||||
"verify on the endpoint the catalog actually targets" lesson — testing
|
||||
the OpenAI endpoint nearly shipped a catalog that 404s on `/tts`.)
|
||||
- `[2026-06-01]` **Fish path-form references** (`{audio:"/app/references/x.wav"}`)
|
||||
→ HTTP 500 on this build, despite the stack README showing that shape. The
|
||||
working clone paths are `reference_id` (by name → reads `<name>.wav` +
|
||||
optional `<name>.txt`) or inline base64 `references`. The catalog now uses
|
||||
`reference_id`.
|
||||
|
||||
- `[2026-06-01]` **Fish reference transcript as a provenance note** (not the
|
||||
actual spoken words) — Fish/fish-speech uses the reference transcript to
|
||||
disambiguate phonemes, and a misaligned transcript hurts cloning badly. I
|
||||
staged VCTK voices with provenance-note `.txt` sidecars, which CONFOUNDED my
|
||||
"Fish is a weak cloner" verdict (that verdict is retracted/unproven). Lesson:
|
||||
stage the REAL transcript (VCTK ships ground-truth; or ASR via Whisper) for
|
||||
any clone reference.
|
||||
|
||||
- `[2026-06-01]` **Pitch-shift register control** (rubberband, to deepen Imogen
|
||||
to contralto/mezzo) — Fish ignores small reference shifts and overshoots
|
||||
large ones; `-3st` WITHOUT formant preservation → "baritone through a tuba"
|
||||
(formants collapsed); `-2/-3st` WITH `--formant` → too subtle / still bad.
|
||||
Abandoned at every depth; all variants deleted. Finer independent
|
||||
pitch/formant control needs praat (not installed). Future clones = operator's.
|
||||
|
||||
- `[2026-06-01]` **resemblyzer for cloning-fidelity scoring** — weak/dated
|
||||
encoder (2019 LSTM, English-biased) + a synthetic-vs-natural domain gap that
|
||||
depresses cosine regardless of true fidelity → conclusions were soft. Use
|
||||
ECAPA-TDNN (speechbrain `spkrec-ecapa-voxceleb`) for rigorous
|
||||
speaker-verification scoring.
|
||||
|
||||
- `[2026-05-31]` Building the dia2-capable image surfaced THREE upstream
|
||||
packaging quirks: (1) `pip install -e nari-labs/dia2` fails — no PEP 660
|
||||
`build_editable` hook; (2) plain `pip install` builds an empty
|
||||
`UNKNOWN-0.0.0` wheel (base setuptools 59.6 < dia2's required ≥70); (3)
|
||||
`--no-deps` leaves `transformers`/`sphn`/`whisper-timestamped` missing. Fix
|
||||
(`local/dia:v2`): copy the pure-python `dia2/` package into site-packages +
|
||||
install ONLY those 3 deps; base torch/numpy already satisfy Dia2.
|
||||
|
||||
- `[2026-05-31]` Dia2 predefined voices (43, baked at `/app/voices`) are NOT
|
||||
reachable from the `/tts` clone path — it resolves `clone_reference_filename`
|
||||
against the reference_audio dir ONLY. The OpenAI `/v1/audio/speech` `voice`
|
||||
param auto-resolves them (separate code path), which masked the gap. Fix:
|
||||
stage `/app/voices/*` into `/worktank/dia/reference_audio`. Lesson: verify on
|
||||
the endpoint the catalog ACTUALLY targets.
|
||||
|
||||
- `[2026-05-31]` `voice_mode=clone` with an empty `clone_reference_filename`
|
||||
→ asset-engine serializes it as the literal string `"undefined"` →
|
||||
`/tts` 404 `Reference audio file not found: undefined`. Reproduced
|
||||
directly. Worked around in the catalog (default the field to a real
|
||||
voice); proper fix is asset-engine-side (don't send `"undefined"` for
|
||||
empty optionals; enforce conditional-required) — flagged to
|
||||
asset-engine-dev.
|
||||
→ asset-engine serializes it as the literal string `"undefined"` → `/tts`
|
||||
404. First observed on dia2; worked around in the catalog (default the field
|
||||
to a real voice). **[2026-06-01] root cause found** — the Kokoro voice-blend
|
||||
widget reading Shoelace `.value` before hydration (see Current state); the
|
||||
real fix is asset-engine-side and is escalated.
|
||||
|
||||
- `[2026-05-31]` asset-engine `docs/asset-engine/services.schema.json`
|
||||
is a DERIVED artifact (regen from the Pydantic model via
|
||||
`dump_schema.py`) and had DRIFTED: it rejected the `lifecycle` field
|
||||
that 12/14 live services already use (13 pre-existing validation
|
||||
errors against the committed catalog).
|
||||
**[2026-05-31] RESOLVED:** schema regenerated with `CatalogLifecycle`
|
||||
+ `reproducibility.seed_field` (`b7b2130`); catalog now validates clean.
|
||||
Standing lesson: the schema is generated from asset_engine's
|
||||
`catalog.py` — hand-editing `services.yaml` shape without regenerating
|
||||
re-introduces drift.
|
||||
- `[2026-05-31]` asset-engine `services.schema.json` is DERIVED (regen from the
|
||||
Pydantic model via `dump_schema.py`) and had DRIFTED — rejected the
|
||||
`lifecycle` field 12/14 services use. RESOLVED: regenerated with
|
||||
`CatalogLifecycle` + `reproducibility.seed_field` (`b7b2130`). Lesson:
|
||||
hand-editing `services.yaml` shape without regenerating re-introduces drift.
|
||||
|
||||
- `[2026-05-30]` esh-docker-vm: `hard` NFS mounts from 10.0.50.50 froze
|
||||
a container worker in UNKILLABLE D-state when the NAS stalled —
|
||||
`docker restart` fails with "tried to kill container, but did not
|
||||
receive an exit event"; only a host reboot clears it. Separately,
|
||||
`fstab defaults` (no `_netdev`) made NFS-bind containers `Exited(255)`
|
||||
on reboot. Diagnosis + recovery steps → auto-memory
|
||||
- `[2026-05-31]` ttyd-over-TLS forces HTTP/2 (kills ttyd's terminal WebSocket
|
||||
→ blank screen); Safari/WebKit never sends HTTP basic-auth on WS upgrades.
|
||||
Both solved for the fleet seat (Caddy forces HTTP/1.1; auth → network-gating)
|
||||
— detail in auto-memory `reference_ttyd_fleet_seat`.
|
||||
|
||||
- `[2026-05-30]` esh-docker-vm: `hard` NFS mounts from 10.0.50.50 froze a
|
||||
container worker in UNKILLABLE D-state when the NAS stalled — only a host
|
||||
reboot clears it. Separately, `fstab defaults` (no `_netdev`) made NFS-bind
|
||||
containers `Exited(255)` on reboot. → auto-memory
|
||||
`incident_esh_docker_nfs_boot_race`.
|
||||
|
||||
- `[2026-05-31]` ttyd-over-TLS forces HTTP/2 (no flag disables it) which
|
||||
kills ttyd's terminal WebSocket (blank screen); Safari/WebKit never
|
||||
sends HTTP basic-auth on WS upgrades (→ ttyd "User code denied"). Both
|
||||
solved for the fleet seat (Caddy forces HTTP/1.1; auth dropped to
|
||||
network-gating) — detail in auto-memory `reference_ttyd_fleet_seat`.
|
||||
|
||||
- `[2026-05-27]` `docker compose up -d` from within the `docker:cli`
|
||||
sandbox: `${VAR:-./config}` defaults resolve `./config` to the
|
||||
sandbox CWD (e.g. `/wt/config`). Docker daemon interprets the
|
||||
resulting path against the HOST filesystem, not the sandbox's.
|
||||
Auto-creates an empty dir on the host → container saw an empty
|
||||
bind target → entrypoint shim seeded image-baked defaults
|
||||
(lost all the host-side providers.yaml patches I'd applied earlier
|
||||
in the day). Root cause for ratatoskr-dev's Tier 3 sessions 500.
|
||||
Fix: pass `-e WORLDTREE_CONFIG_DIR=/opt/worldtree-personal/config`
|
||||
to the docker run invocation. Captured into the docker-as-root
|
||||
pattern note in Tools and conventions.
|
||||
|
||||
- `[2026-05-27]` `:latest`-pinned compose + private gitea registry +
|
||||
sandboxed pull = recreate on ancient cached `:latest`. Local
|
||||
`:latest` tag never advanced because deploy.yml pulls by SHA
|
||||
(not by tag); sandbox can't `docker pull` (no creds for private
|
||||
registry); `compose up -d --pull never` used the 2-week-old
|
||||
cached `:latest`. Symptom: personal worldtree's matrix container
|
||||
crash-looped because the ancient image didn't understand the new
|
||||
config schema. Fix: retag SHA → `:latest` on host first
|
||||
(`docker tag <registry>/<repo>:<sha> <registry>/<repo>:latest`),
|
||||
then recreate with `--pull never`. Better long-term: pin SHA
|
||||
explicitly in `.env`, advance in CI.
|
||||
|
||||
- `[2026-05-27]` Container recreate during in-flight skaldsong gen
|
||||
kills the runner. Pattern: my llama-swap restart at 23:57Z + a
|
||||
v0.31.2 `compose up -d` at 02:47Z both took out active dramatist
|
||||
passes. Confirmed by skaldsong-dev INV-006 ("partial state only
|
||||
flips on host restart"). With deploys landing every ~10min during
|
||||
active dev and stories taking >5min, runner-killed-by-recreate is
|
||||
structural, not incidental. Roadmap (skaldsong-dev): pre-shutdown
|
||||
signal handler (cleanup row state to failed/aborted), per-scene
|
||||
resume-from-checkpoint, `/api/admin/quiesce` endpoint. None shipped
|
||||
yet; operator's call which path.
|
||||
|
||||
- `[2026-05-27]` `--reasoning-format gemma` on artemis-31b-v1i —
|
||||
not supported in the deployed llama.cpp build. Accepts only
|
||||
`none | deepseek | deepseek-legacy`. `deepseek` populates the
|
||||
`reasoning_content` SSE delta field which Worldtree GemmaProvider's
|
||||
fallback path checks. Was load-bearing for getting reasoning
|
||||
tokens through to ratatoskr's wire.
|
||||
|
||||
- `[2026-05-27]` `head -c N` piped after a streaming curl SIGPIPEs
|
||||
the curl, killing the request early. Bit me twice while probing
|
||||
SSE-shape samples. Use file-write + separate read.
|
||||
|
||||
_24 older entries archived to archival-memory.md._
|
||||
_29 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user