diff --git a/docs/asset-engine/services.yaml b/docs/asset-engine/services.yaml index 1d22695..c4e7459 100644 --- a/docs/asset-engine/services.yaml +++ b/docs/asset-engine/services.yaml @@ -749,10 +749,15 @@ services: Eleanor (p228), Beatrice (p229) — consenting VCTK volunteers (CC BY 4.0), NOT modeled on or representing any public figure. Other female: Abigail, Alice, Cora, Elena, Emily, Gianna, Jade, Layla, Olivia, - glados. Resolves to .wav + optional .txt transcript; - blank = model default/random speaker. To add: drop a clean 5–15s WAV - into the references dir + add the name here. (No /voices API → static - list; a list-endpoint is the durable fix — see notes.) + glados. Resolves against the DIRECTORY references// (audio + + a same-basename .lab transcript inside it) — NOT a flat + references/.wav, which fish-speech ignores (empty/flat-only + dirs silently collapse every voice to the model default; root-caused + + fixed 2026-06-01). Blank = model default/random speaker. To add: + create references// with .wav (clean 5–15s) + .lab + (transcript) and add the name here; the deploy playbook's normalize + step also mirrors any flat .wav into this layout. (No /voices + API → static list; a list-endpoint is the durable fix — see notes.) - name: references type: json label: Custom clone (inline base64) diff --git a/persistent-memory.md b/persistent-memory.md index b26ceac..9865316 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -118,16 +118,30 @@ _As of 2026-06-01:_ `01KT2K2SY9N7AY69R9V0B4RXSW` → asset-engine-dev). **Workaround until fixed: click the dropdown (fires change) or hit the API directly.** My half (`blendable: false` catalog flag) is queued — see Recent decisions. -- **Fish cloning RESOLVED — it clones competently; "not British" was the - undefined bug.** ECAPA-TDNN re-test (2026-06-01): an Imogen-referenced Fish - clone scores **~0.79 cosine vs the real `Imogen.wav`** (firmly same-speaker) - and only **~0.10 vs Fish's no-reference default voice**. So Fish faithfully - clones Imogen WHEN it receives the reference. The operator's "Imogen doesn't - sound British at all" was the `"undefined"` select bug feeding Fish its - DEFAULT voice (≈ a different speaker), NOT a Fish cloning failure. Earlier - resemblyzer-based "weak cloner" verdict RETRACTED. No Fish-side fix needed — - the blocker is entirely the undefined bug (asset-engine-dev). Future voice - clones operator-handled (pitch-shift deepening abandoned). +- **Fish "not British" had TWO independent root causes — BOTH now fixed.** + The ECAPA-TDNN re-test (2026-06-01) showed Fish's cloning *engine* works: an + Imogen reference scores **~0.79 cosine vs real `Imogen.wav`** vs **~0.10 vs + the no-reference default** — but that test fed the reference via the inline + base64 `references` path, which masked the real defect. **`reference_id`-by- + name (the path the catalog + form actually use) was silently a NO-OP.** Proof + (2026-06-01): `reference_id=Abigail`, `=Imogen`, and no-reference all returned + **byte-identical** audio (md5 `40b6ac73…`, same text+seed) → reference_id was + being ignored entirely, so every dropdown voice collapsed to Fish's default + speaker. Cause: fish-speech resolves `reference_id=` against the + **directory** `references//` (audio + same-basename `.lab`), but the + voices were staged as **flat** `references/.wav` + `.txt` with the + per-name dirs left **empty** → nothing resolved. **FIX (2026-06-01):** + populated `references//.wav` + `.lab` for all 32 voices; + re-test → Imogen/Eleanor/Beatrice/Abigail/no-ref all 5 **distinct**. + reference_id now works; British accents come through. Hardened: the + deploy playbook gained a normalize-layout step + an A/B smoke gate + (`reference_id` MUST change output) so the empty-dir regression can't ship + silently. So the "not British" saga was (A) the `"undefined"` select bug + (asset-engine, still belt-and-suspenders) AND (B) this reference_id no-op + (server-side, FIXED) — B alone would have kept every voice identical even + with A fixed. Earlier "reference_id-by-name is THE working path, verified" + claim RETRACTED. Pitch-shift deepening abandoned; future clones operator- + handled. - **On-host consented voice library** — ~992 real-person clips cached in the kyutai tts-voices repo (`/worktank/kyutai-tts/.../snapshots/.../`): VCTK (CC BY 4.0, accent-tagged speaker IDs), Unmute voice-donations (CC0), EARS + @@ -182,8 +196,11 @@ _As of 2026-06-01:_ - `[2026-06-01]` **Fish-S2 catalog v1→v6** (`f3c260e`..`38d9e3b`) — was a blank 2-field form; now exposes the full fish-speech ServeTTSRequest surface + a 32-voice `reference_id` "Voice" dropdown (STATIC list — Fish has no - /voices API). `reference_id`-by-name is the WORKING clone path; the - path-form `references` shape 500s (see Tried and abandoned). Corrected vram + /voices API). `reference_id`-by-name works ONLY when each voice is staged as + a `references//` dir with `.wav` + `.lab` — was silently a + no-op until that layout fix on 2026-06-01 (see the two-root-causes entry + above); the inline-path-as-filesystem-path `references` shape 500s (see Tried + and abandoned). Corrected vram 4→24 GB, cold_start 8→240s, seedable false→true. Added 3 consenting British-female VCTK voices (Imogen=p225/Eleanor=p228/Beatrice=p229). @@ -256,9 +273,10 @@ _25 older entries archived to archival-memory.md._ - `[2026-06-01]` **Fish path-form references** (`{audio:"/app/references/x.wav"}`) → HTTP 500 on this build, despite the stack README showing that shape. The - working clone paths are `reference_id` (by name → reads `.wav` + - optional `.txt`) or inline base64 `references`. The catalog now uses - `reference_id`. + working clone paths are `reference_id` (by name → resolves the DIRECTORY + `references//` containing `.wav` + `.lab`; the flat + `references/.wav` is ignored — see the two-root-causes entry above) or + inline base64 `references`. The catalog uses `reference_id`. - `[2026-06-01]` **Reference transcript barely affects Fish clone IDENTITY** (disproving my mid-session theory). I'd blamed a bogus provenance-note `.txt` diff --git a/playbooks/deploy-fish-s2.yaml b/playbooks/deploy-fish-s2.yaml index b4c4746..24ec438 100644 --- a/playbooks/deploy-fish-s2.yaml +++ b/playbooks/deploy-fish-s2.yaml @@ -36,6 +36,27 @@ steps: shell: mkdir -p {{ references_dir }} creates: "{{ references_dir }}" + - name: Normalize reference layout (flat .wav → /.wav + .lab) + # fish-speech resolves reference_id= against the DIRECTORY + # references// (audio + a same-basename .lab transcript), NOT a + # flat references/.wav. A flat-only layout makes every + # reference_id silently resolve to nothing → Fish falls back to its + # default speaker, so every voice in the dropdown sounds identical + # (this was the 2026-06-01 "no accent" root cause). External voice + # imports drop flat .wav + .txt here; this step mirrors + # each into references//.wav + /.lab. `cp -u` + # makes it idempotent; the `when:` skips it once every flat wav has a + # matching nested wav (audio-only voices like glados need no .lab). + shell: | + cd {{ references_dir }} || exit 1 + for w in *.wav; do + [ -e "$w" ] || continue + n="${w%.wav}"; mkdir -p "$n" + cp -u "$w" "$n/$n.wav" + [ -e "$n.txt" ] && cp -u "$n.txt" "$n/$n.lab" || true + done + when: "[ \"$(cd {{ references_dir }} 2>/dev/null && ls *.wav 2>/dev/null | wc -l)\" -ne \"$(cd {{ references_dir }} 2>/dev/null && ls */*.wav 2>/dev/null | wc -l)\" ]" + - name: Ensure checkpoints dir exists shell: mkdir -p {{ checkpoints_dir }} creates: "{{ checkpoints_dir }}" @@ -106,9 +127,10 @@ verify: - name: /v1/tts returns a real WAV (POST with text body) # Fish's API is NOT OpenAI-compatible — there's no /v1/audio/speech # and no /v1/audio/voices. The single TTS endpoint is POST /v1/tts - # with at minimum {"text":"..."} returning audio/wav. Voice cloning - # is via reference= field in the body (paths under /app/references). - # Verify by POST + asserting the response is a real RIFF WAV. + # with at minimum {"text":"..."} returning audio/wav. Named-voice + # cloning is via the reference_id field, which resolves against the + # references// directory (audio + .lab) — see the A/B gate + # below. Verify by POST + asserting the response is a real RIFF WAV. shell: | out=$(mktemp --suffix=.wav) curl -sf -X POST http://localhost:{{ host_port }}/v1/tts \ @@ -119,6 +141,23 @@ verify: rm -f "$out" changed_when: "false" + - name: reference_id actually changes the voice (A/B — guards the empty-dir regression) + # Two different reference_ids with identical text+seed MUST yield + # different audio. If byte-identical, reference_id is being ignored + # (empty references// dirs) and every voice has silently + # collapsed to Fish's default speaker — the 2026-06-01 root cause. + # Hard-fail the deploy so the regression can never ship silently. + shell: | + a=$(mktemp); b=$(mktemp) + curl -sf -X POST http://localhost:{{ host_port }}/v1/tts -H 'Content-Type: application/json' \ + -d '{"text":"The quick brown fox.","seed":42,"reference_id":"Abigail"}' -o "$a" --max-time 60 + curl -sf -X POST http://localhost:{{ host_port }}/v1/tts -H 'Content-Type: application/json' \ + -d '{"text":"The quick brown fox.","seed":42,"reference_id":"Imogen"}' -o "$b" --max-time 60 + rc=0; [ "$(md5sum < "$a")" = "$(md5sum < "$b")" ] && rc=1 + rm -f "$a" "$b" + exit $rc + changed_when: "false" + - name: Container is running shell: docker inspect fish-s2 --format '{{.State.Status}}' | grep -q running changed_when: "false" diff --git a/stacks/fish-s2/README.md b/stacks/fish-s2/README.md index 3ef74bb..8b9fea9 100644 --- a/stacks/fish-s2/README.md +++ b/stacks/fish-s2/README.md @@ -94,10 +94,22 @@ root path; there's no `/openapi.json` endpoint exposed directly. ## Voice library -Drop reference WAV / MP3 / FLAC into -`/worktank/fish-s2/references/` on the host. The wrapper scans on -request — no restart needed. Use clean ~5-15 s clips, single -speaker, ideally with diverse intonation samples. +Named voices are selected via the `reference_id` field, which Fish +resolves against the **directory** `references//` — NOT a flat +`references/.wav`. Stage each voice as: + +``` +/worktank/fish-s2/references//.wav # clean ~5-15 s, single speaker +/worktank/fish-s2/references//.lab # transcript of that clip +``` + +A flat `references/.wav` (or an empty `references//` dir) +is **silently ignored** — `reference_id` resolves to nothing and Fish +falls back to its default speaker, so every voice sounds identical. +This was the 2026-06-01 "no accent" root cause; the deploy playbook now +carries a normalize-layout step + an A/B smoke gate (`reference_id` +MUST change the output) to keep it from regressing. The wrapper scans +on request — no restart needed after adding a voice. ## Deploy