The 2026-09-06 headscale cutover retired irv-ml1's wg0 tunnel IP 10.100.79.3
(now 10.6.110.50). Repointed all LIVE canonical refs to the DNS NAME so the next
move can't re-break them: homepage.href/siteMonitor labels across 25 stack
composes, load-bearing env defaults (asset-engine INFERENCE_HOST, open-webui
AUDIO_TTS_OPENAI_API_BASE_URL, skaldsong SKALDSONG_TTS_BASE_URL, zonos-gateway
ZONOS_URL, dia), homepage services.yaml manual cards (Voice Design Studio,
IRV-ML1), and servers/irv-ml1/ssh-target. Updated the stale 'WG tunnel' comment
to the mesh reality.
Left as-is: README curl-examples and .env.example comments (docs), and historical
mentions in CLAUDE.md/persistent-memory. NOTE: applying the label repoints to the
RUNNING irv-ml1 containers needs a recreate per service (labels read at creation);
deployed .env values are separate from these canonical defaults.
Move the ~22-service flat "AI Systems" group off the Main tab into a new
four-tab layout (Main / AI / Infrastructure / Toolchain). The AI tab sorts
the inference fleet by function into seven groups:
AI - Inference gen, char-rp, char-rp-reasoning, Granite summarizer
AI - Eval & Retrieval Selene, Skywork Reward, Qwen3 rerank/embed, image-bench
AI - Gateways & Chat LiteLLM, Asset Engine, Gateway Chat, Open WebUI, ...
AI - Speech (TTS) Chatterbox Fast, Kokoro, mOrpheus
AI - Audio Tools Parakeet ASR, YT Voice Clipper
AI - Image & Media ComfyUI, Arbo
AI - Dormant stopped rollback seats + retired auditions
Relabel each stack's homepage.group so canonical stacks/ matches the live
containers on ana-ml2, ana-docker, and irv-ml1. Dormant stacks were refreshed
with `docker compose up --no-start` so they carry the new label while staying
stopped (compose-start rollback preserved). settings.yaml drives tab/order/
columns; services.yaml and README updated to the new scheme.
qwen3-tts: deploy was using the -Base checkpoint, which sounds like
the right one ("supports voice cloning") but the upstream wrapper's
only synthesis path goes through generate_custom_voice. The -Base
variant doesn't expose that, so every request — including ones with
the wrapper's listed built-in voices like Ryan/Vivian — errored with
"does not support generate_custom_voice". The -CustomVoice variant
exposes both the cloning machinery and the preset voices, and is
what the wrapper actually needs.
The .env.example comments had the variant labels backward; fixed in
this commit. Live host already updated to -CustomVoice via direct
.env edit (model downloaded on container restart).
chatterbox README listed [whisper] and [breath] as supported tags —
those are in the base Chatterbox tag set but NOT in the Turbo set
that's actually loaded. Replaced with the canonical 9-tag list
verified against /api/model-info: laugh, chuckle, sigh, gasp, cough,
clear throat, sniff, groan, shush.
Both reported (unhealthy) in docker ps. Two distinct root causes:
* news-digest-web: switched from nginx:alpine to python:3.12-alpine
(uvicorn) but kept the wget healthcheck against `localhost`. Alpine's
/etc/hosts maps localhost to BOTH ::1 and 127.0.0.1; busybox wget
tries IPv6 first, hits "connection refused" because uvicorn binds
IPv4-only, and doesn't fall back. Pinned to 127.0.0.1.
* chatterbox: devnen's image is built from a python:3.10 base and
doesn't ship curl, so `curl -fsS http://localhost:8004/api/model-info`
failed with `/bin/sh: 1: curl: not found`. Replaced with a python
urllib one-liner that fetches + asserts `b'"loaded":true' in body`,
also pinned to 127.0.0.1 to dodge the same IPv4/IPv6 race.
Both YAML extractions tested directly inside the running containers
(via `sh < script`) — chatterbox python check returns 0 when the model
is loaded.
devnen/Chatterbox-TTS-Server doesn't expose /health — neither in code
nor OpenAPI. The deploy hung on the playbook's `Wait for /health to
respond` loop indefinitely (each curl -> 404, retry forever) even
though the container was up and the model loaded clean to CUDA at
22:52:21 (~42s after start).
/api/model-info returns `{"loaded":true,...}` only after the model
finishes loading, so it doubles as liveness + readiness. Updated:
* compose.yaml healthcheck — grep for `"loaded":true` from
/api/model-info.
* playbook wait step — same probe instead of /health.
* verify /health → verify /api/model-info reports loaded.
* verify /v1/audio/voices — switched from greping for `voice|alloy|echo`
literals to parsing JSON and asserting the actual response shape:
`{"status":"ok","voices":[...]}` (devnen's shape — note this is NOT
the OpenAI list-format vibevoice uses).
Both deploys failed against irv-ml1 today with upstream-changed-on-us
errors:
* vibevoice: VIBEVOICE_SHA=7614c469a145 (12-char short) made docker
buildx report "repository does not contain ref 7614c469a145" — same
commit IS still HEAD of main, but buildx's git source resolver
doesn't accept short hashes even when unambiguous. Now full 40-char.
* chatterbox: dockerfile: docker/Dockerfile.gpu — devnen restructured
the repo to put Dockerfiles at root, renamed by CUDA version
(Dockerfile.cu128, .cpu, .rocm). Switched to Dockerfile.cu128 (GPU
build for CUDA 12.8 toolkit; works on irv-ml1's 595.58.03 driver).
Also pinned CHATTERBOX_SHA to a full 40-char SHA instead of `main`
so future upstream churn doesn't break the deploy without warning.
Live host .env files patched directly (the playbook only seeds .env
when absent, so canonical edits don't propagate to existing installs).
Three TTS additions to round out coverage on irv-ml1, each filling a
distinct niche the existing slate doesn't own.
Final coverage matrix (all on irv-ml1):
Kokoro — low-latency English, fixed voice library, ~300ms TTFA
Chatterbox Turbo — low-latency English w/ voice cloning + paralinguistic tags
IndexTTS-2 — English voice cloning + emotion vector / text control
Qwen3-TTS-1.7B-Base — high-quality English voice cloning
CosyVoice 3 — multilingual (Chinese-leaning)
VibeVoice 1.5B — long-form / multi-speaker dialogue
stacks/kokoro:
- port 8193, GPU device 0 (3090)
- pulls ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.4-master (no Dockerfile,
no first-run model download — models baked in)
- 60+ built-in voices, OpenAI-compat with stream=true over chunked HTTP
- Apache-2.0 weights + code, ~1 GB VRAM
stacks/vibevoice:
- port 8194, GPU device 1 (A6000 — for 7B headroom)
- builds groxaxo/VibeVoice-FastAPI1 (more current fork of ncoder-ai)
pinned to 7614c469a145
- default model microsoft/VibeVoice-1.5B (~7 GB bf16 VRAM); env var
swap to rsxdalv/VibeVoice-Large (7B) or FabioSarracino/VibeVoice-Large-Q8
- multi-speaker dialogue via /v1/vibevoice/generate with Speaker N: format
- long-form niche only — not low-latency
stacks/chatterbox:
- port 8196, GPU device 0 (3090)
- builds devnen/Chatterbox-TTS-Server (most active Turbo-supporting wrapper)
- default model ResembleAI/chatterbox-turbo (~2.5 GB fp16, ~75ms latency)
- paralinguistic tags inline ([laugh] [whisper] etc) — different shape
from IndexTTS-2's emotion vector; fills the speed+cloning niche
Kokoro/IndexTTS don't cover together
- mandatory PerTh watermark on outputs (Resemble policy)
Three matching playbooks under playbooks/deploy-{kokoro,vibevoice,
chatterbox}.yaml. All idempotent, creates-/when-gated.
Cold-deploy disk on /worktank/: ~7 GB Kokoro + ~19 GB VibeVoice 1.5B
+ ~12 GB Chatterbox = ~38 GB total. VRAM concurrent: ~10-11 GB across
both GPUs.
Skipped from the original four-stack proposal: VibeVoice Realtime
(overlaps Kokoro's niche; Kokoro wins on latency, license, and not
needing a build).
vh