Same shape as task-board: build-on-host from vh/vor, bind-mounted
persistence for sessions/ and responses/ (the user-published markdown
files), exposed at port 7879 (adjacent to task-board's 7878 since both
are claude-tooling sidecars).
Workflow template assumes the same DEPLOY_SSH_KEY + MGMT_REPO_TOKEN
secrets at user scope; nothing new to provision. Playbook accepts SHA
or branch refs (same fix as deploy-task-board.yaml) so manual runs
and CI runs share the same code path.
Centralized vs upstream-local: README documents the trade. Claude
fetches response markdown via /api/sessions/{id} JSON instead of a
local file read — the only API-flow change from the upstream README.
Runner is now re-registered with `:docker://node:20-bookworm-slim`
schema in its labels, so workflows targeting `pfi-fleet` get that
image automatically. Saves a few lines per workflow and gives us one
place (the runner config) to bump the default image when a new
node/debian release lands.
CI passes --var ref=<github.sha> (a full SHA), but the playbook
hardcoded `git reset --hard origin/{{ ref }}` which only works for
branch names — `origin/<sha>` is invalid syntax. Resolve ref via
git rev-parse with `^{commit}` (try origin/<ref> first for branch
names, fall back to bare <ref> for SHAs/tags) so manual runs (ref=main)
and CI runs (ref=<sha>) both work.
Same fix applied to the changed_when comparison so no-op reruns still
report ok instead of changed.
Runner's .runner registration cached :host mode at first start; env-var
label updates aren't sticky once the runner is registered. Until we
re-register with docker-schema labels, workflows must declare their
own container. node:20-bookworm-slim has node (for actions/checkout)
and apt (for python3-yaml + openssh-client install).
debian:bookworm-slim lacks node, so actions/checkout@v4 (a JS action
running dist/index.js) fails with `exec: "node": executable file not
found in $PATH`. Dropping the explicit `container:` directive lets
the runner use its label-default — node:20-bookworm-slim has node +
git out of the box. Install step shrinks to python3 + pyyaml +
openssh-client.
Central runner on ana-docker (gitea is local; existing fleet tooling
already SSHes from there). Playbook is parameterized so future
site-local runners (nh3-docker, esh-docker-vm) drop in via --var
overrides instead of copy-paste.
Includes a workflow template for vh/task-board that calls the existing
deploy-task-board.yaml playbook — keeps the playbook as the single
source of truth for "how task-board is deployed", manual or automated.
Labels embed `:docker://node:20-bookworm-slim` schema; without it,
act_runner v0.6+ silently falls back to host-mode and runs job steps
inside the Alpine runner container (no apt/python/node), breaking any
real workflow. node:20-bookworm-slim is small + has git + node so
actions/checkout works out of the box.
The applet outgrew "stack alongside the infra-management workspace" —
it has its own pyproject, multi-tenant deploy story, separate
release cadence, and isn't actually about managing infrastructure.
Lives at https://gitea.phasefinal.com/vh/nevermore now, with
provenance noted in its initial commit.
This commit removes:
stacks/news-digest/ (full stack tree)
playbooks/deploy-news-digest.yaml
scripts/add-digest-user.sh
The existing ana-docker deployment continues running on its baked
local/news-digest:v5 image — nothing changes for the live install
until you choose to redeploy from the new repo. Migration steps
(rename data dir, redeploy, retire old compose dir) are in
nevermore's README.
Updated:
README.md — Current stacks listing now points at the new repo
STATUS.md — milestones entry for the extraction
Replaced the 9-month-stale 'Last Updated: 2025-07-14' line with a
status disclaimer — this is KB-derived advisory material, not
something we maintain in lockstep with code. Tracer-date claim was
implying a freshness contract we don't fulfill. Found via
/tend-docs.
Same anti-pattern as the deleted model-list.md — a hardcoded snapshot
of mutable config that's guaranteed to drift. Replaced the 15-row
table with a one-liner pointing at stacks/llama-swap/conf/config.yaml
(the canonical source) and the live /v1/models HTTP endpoint. Found
via /tend-docs.
The compose-side default was still pinning qwen3.5-35-a3b — broken
on launch since its GGUF stopped working months ago. Real .env on
ana-docker overrides to granite-4-small so live deploys are unaffected,
but the default was misleading for anyone forking the stack. Found
via /tend-docs.
The original default model in .env.example was changed to
granite-4-small months ago when qwen3.5-35-a3b's GGUF file started
exiting on launch, but the README still named the old one as
"current". Also bumped the summarization-style description from
"one sentence" to "2-3 sentences" to match the post-trafilatura
prompt rewrite. Found via /tend-docs.
Bumped "Last updated" to 2026-04-29 and added a milestones section
for the audio-gen + news-digest applet + task-board + tooling work
that landed since 2026-04-24. Found via /tend-docs.
These four ana-docker stacks were missing from the discoverability
index — readers landing on the README couldn't find them without
walking stacks/. Found via /tend-docs.
Self-reported "Synchronized with llama-swap config.yaml on 2025-07-18"
— 9 months stale. Canonical model truth is stacks/llama-swap/conf/config.yaml;
the static snapshot was guaranteed to drift. No historical value (not
an ADR, post-mortem, or migration runbook).
Found via /tend-docs.
Stock neosmemo/memos:stable, port 5230, SQLite at
/opt/docker/conf/memos/data/. Joins traefik-net and ships homepage
labels (group=Notes) so it auto-appears on the dashboard via docker
discovery — no edit to configs/homepage/services.yaml needed.
First-run bootstrap is via the UI: visit http://10.250.50.70:5230
and create the Host account through the sign-up form.
Playbook idiom note: docker compose pull lines need the literal
block scalar (|) when the grep pattern contains colons — bare-string
shell value made YAML parse the colon as a mapping separator and
elway choked on first try.
Mirrors the reusable tooling (scripts/, generic playbook templates,
.gitignore, conventions section of CLAUDE.md) into a new directory
and strips everything fleet-specific: servers/, stacks/, configs/,
fleet-named playbooks (deploy-*, decouple-*), runbooks, status,
host-pinned scripts.
Output is an empty conventionally-organized workspace with fresh git
history, ready to populate with a different fleet. Skeleton
CLAUDE.md / README.md / STATUS.md are written with the new fleet
name baked in but no server table or placement rules pre-populated.
Deliberately does NOT create a remote or push — the user picks the
namespace + name explicitly via tea / git remote add.
scripts/fork-fleet.sh ~/development/acme-prod-management
scripts/fork-fleet.sh /tmp/test-fork test-fleet
The task-board v0.1.11 orange-while-waiting treatment depends on the
assistant pushing the pending-shells list — hooks can't enumerate
Claude Code's background tasks externally (confirmed against the
hook docs: no payload field, no state file, no lifecycle event, no
JSON form of /tasks). So the discipline lives in CLAUDE.md.
Adds a "Customizing the run schedule" section (DIGEST_CRON_AM/PM env
vars, edit-and-recreate flow) and a "Multi-tenant: one instance per
teammate" section covering scripts/add-digest-user.sh end to end:
what it does, the per-user file layout on ana-docker, idempotent
schedule/password updates, and the teardown path.
Updated the stale "two editions per day" intro line to note the
schedule is now configurable.
Hardcoded crontab → render at container start from
DIGEST_CRON_AM + DIGEST_CRON_PM. Defaults match the original
0800 / 2000 so existing deploys are no-ops.
scripts/add-digest-user.sh learns --am and --pm flags so each
teammate's stack can fire on their hours:
scripts/add-digest-user.sh bob --am "0 6 * * *" --pm "0 17 * * *"
scripts/add-digest-user.sh carol --pm "30 18 * * 1-5" # weekdays only
Standard 5-field cron syntax; busybox crond honors the container's
\$TZ. Removed the now-unused stacks/news-digest/crontab file and
the matching COPY in the Dockerfile.
Two pieces:
1) Multi-tenant onboarding via scripts/add-digest-user.sh
Shared miniflux + per-user digest stack. Onboarding a teammate
takes one command (plus a one-time sudo for dir creation):
scripts/add-digest-user.sh <username>
What the script does:
- Reads miniflux admin creds from ana-docker
- Allocates next free port (scans existing digest-*/.env)
- Generates a random password (or accepts one as 2nd arg)
- Creates the miniflux user via the admin API
- Materializes a per-user .env at /opt/docker/compose/digest-<user>/
(inherits NEWS_DIGEST_TAG from the canonical stack so all
tenants run the same image)
- Brings up `docker compose -p digest-<user> up -d`
- Seeds default world/local feeds in the new user's miniflux
- Triggers a first digest run
compose.yaml now uses ${DIGEST_PROJECT:-news-digest} to namespace
container_name + homepage labels. Default keeps backward-compat
for the singleton install — existing stacks unaffected.
2) Masthead overlap on phone widths
Desktop CSS pinned .masthead-edition to grid-row 1, which collided
with .masthead-brand once the mobile media query collapsed both
to grid-column 1. Result: "MORNING EDITION" badge stacked on top
of the "DAILY DIGEST" hero. Reset grid-row to `auto` for all
three masthead children in the ≤720 px breakpoint so they
auto-flow vertically.
Three things were broken on phones:
1. The collapse button I added to .desk-head had no grid placement,
so it auto-flowed into the desk-sub row and looked like a floating
chevron. Made the desk-head grid 4 columns explicit (num | title |
count | collapse) and pinned the button to col 4 row 1.
2. The 720px breakpoint was the only one — everything inherited
tablet rules at iPhone widths. Added a true-phone tier at
≤480 px that hides the section number badge and the rail
gutter, floats chips inline above the title, makes the jumpnav
horizontally scrollable for narrow widths, drops the edition
number, and bumps touch targets.
3. Long URLs / unbroken tokens could push horizontal overflow.
Added overflow-wrap: anywhere on titles + tldrs and overflow-x:
hidden on body as a belt-and-suspenders catch.
Two upgrades to make the digest actually readable:
1) Article-grounded 2-3 sentence summaries (everywhere)
The old prompt got just the title + miniflux's content excerpt,
which for HN/Lobsters/wire feeds is barely more than the title
itself — so summaries paraphrased the title and added nothing.
Now every URL gets fetched and main-content-extracted via
trafilatura on a parallel pre-pass (10 workers, ~15s for ~50
URLs). Extracted text caches to /output/.article-cache.json with
a 7-day TTL so repeat runs in the same window don't re-pull.
Headlines also get summarized now — one batched LLM call per
category (world / local). Rendered as a paragraph below the
title with source + time on the right rail.
Prompt rewrites tell the model to pull names/numbers/places
from the body and explicitly forbid restating the title.
Result: real specifics ("71% saw no pay increase globally",
"third time in less than two weeks", "Islamabad and Moscow
intermediaries") instead of title paraphrase.
2) Per-desk collapse buttons
Chevron next to .desk-count toggles a .is-collapsed class.
Collapsed state is per-device (localStorage by section id) since
collapse is a viewing preference, not content state.
Browsers were serving stale frontend assets after rebuilds, which hid
the new world/local headline desks: the OLD app.js's refreshCounts()
only counted .item children (not .headline), so the new headline desks
came up with visibleItems=0 and got the .is-empty class which is
display:none. Hard refresh fixed it but only for the user who knew
to do that.
Append ?v=<generated_at strftime> to both link/script tags in
digest.html.j2 and archive.html.j2 so every digest run produces a new
asset URL. Works with the existing entrypoint.sh static-asset sync —
no other infra needed.
Two new dense headline rails above the existing reddit/tech cards.
Designed for high-volume "what happened" coverage where the title
is the deliverable — no LLM summarization, ~15 items per section,
6-column-collapsing grid (title / source / time).
Digest pipeline:
* fetch_miniflux_headlines(category) — flat list per category, dedup
by lowercased title (different feeds syndicate the same wire stories)
* 8h look-back window (vs 12h for tech/reddit) since headlines move
faster
* cap of 15 per section (DIGEST_MINIFLUX_HEADLINES_MAX)
Frontend:
* .headline element parallels .item for the hide-button machinery
(both have data-id, both honored by app.js)
* dense 3-col layout collapses to 1-col on narrow screens
* jumpnav now numbers world=01, local=02, reddit=03, tech=04
Setup:
* seed-headlines.py — one-shot script (lives in the image at
/app/seed-headlines.py). Creates the World + Local categories in
miniflux, subscribes a curated feed list, and renames each feed
to a short display title (BBC vs "BBC News", "LA Times" vs "California").
Idempotent — reruns only add new feeds.
* Default world: BBC, NPR, Al Jazeera. Default local: LA Times Local,
LA Times CA, Voice of OC. (OC Register blocks miniflux; left out.)
* entrypoint.sh now syncs templates/{style.css,app.js,favicon.svg}
to /output on container start so frontend asset updates land
without a manual copy after rebuild.
Three upstream gaps surfaced once /generate was actually exercised:
1. infer-api.py builds an 18-arg positional tuple but the pipeline
expects 24 — first missing arg is `format`, so audio_duration
shifts into format's slot and the pipeline calls len() on an
int. Ship a patched copy of infer-api.py and COPY over upstream's
in the Dockerfile. Also handle empty lora_name_or_path -> "none"
(empty string trips HF Hub's repo-id validator).
2. torchcodec + ffmpeg are required by the WAV save path but neither
is in upstream requirements.txt. Without them every /generate
runs to completion and then 500s at write-time.
3. ACE-Step caches checkpoints at /root/.cache/ace-step/checkpoints
(HARDCODED, not honored by HF_HOME). Mount our persistent dir
there so the ~7 GB model survives container recreates.
Bench on A6000 (cached model, lo-fi hip hop, 60-step euler/apg):
10s @ 27 steps -> 9.4s (0.94x)
30s @ 60 steps -> 11.2s (0.37x, ~2.7x realtime)
60s @ 60 steps -> 14.8s (0.24x, ~4x realtime)
Two new audio-generation stacks alongside the TTS slate:
ace-step :8210 — Apache 2.0 music generation foundation model
(hybrid diffusion + LLM). Lyric-aware multi-minute songs. ~10-12 GB
VRAM during inference, A6000-pinned. Custom Dockerfile patches
upstream's torch/cu126 resolution bug (--extra-index-url cu126 was
falling back to pypi-default cu13 wheels, mismatching torchvision).
stable-audio-open :8211 — Stability AI 1.21B latent-diffusion SFX +
ambience. Up to 47s clips at 44.1 kHz. ~6 GB VRAM in fp16,
A6000-pinned. Custom FastAPI shim around diffusers' StableAudioPipeline
(no upstream HTTP server). Dockerfile pins torchsde explicitly —
diffusers doesn't pull it as a hard dep but
CosineDPMSolverMultistepScheduler needs it.
Three deploy iterations + four backend attempts (subprocess CUDA,
resident-server CUDA, Vulkan rebuild) all failed to deliver speedup
over fish-s2:
* CUDA path: ggml_cuda_init succeeded, weights loaded onto GPU per
s2's logs, but nvidia-smi showed 0% utilization during synthesis.
Wall time 20s/long phrase vs fish-s2's 7.5s. The "CUDA get_rows
unsupported for type q6_K" warning hints at incomplete op coverage
in s2.cpp's alpha CUDA backend for fish-speech architecture.
* Vulkan path: vk::IncompatibleDriverError on container init. NVIDIA
Vulkan ICD not accessible inside the container despite
NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics. Would need
host-side nvidia-utils-vulkan installation or manual ICD bind
mount. Didn't pursue.
Both are fixable — CUDA needs op coverage upstream (author actively
working on it; "selective embedding dequant" commit landed 16 days
ago), Vulkan needs host-side ICD setup. Neither is a config-flip,
both are real work for marginal-or-zero return. Better to delete the
stack and revisit when s2.cpp matures or when we tackle FP8
quantization on ana-ml2's RTX 6000 Ada (sm_89, native FP8 hardware).
Local image rmi'd, /opt/docker/compose/fish-cpp removed on irv-ml1.
/worktank/fish-cpp left for user-side sudo cleanup.
Future Fish acceleration paths (in order of decreasing certainty):
1. Wait for s2.cpp CUDA op coverage to mature (track upstream commits).
2. Quantize Fish BF16 → FP8 via TransformerEngine, deploy on
ana-ml2's RTX 6000 Ada (Ada has native FP8 tensor cores, A6000
doesn't). ~2x speedup if it works.
3. vLLM port of Fish (no upstream support today).
CUDA backend confirmed broken for fish-speech ops on s2.cpp v0.x — alpha,
incomplete op coverage, GPU stays at 0% during generation despite
ggml_cuda_init succeeding. Vulkan was the original README example
(`-v 0`), so likely the more battle-tested path.
Build the image with BOTH backends so we can flip via env without
rebuilding:
* libvulkan-dev + glslc in the build stage (GGML's Vulkan backend
compiles its shaders with glslc at build time; without it the
cmake configure silently disables Vulkan).
* libvulkan1 + the libggml-vulkan.so copy in the runtime stage.
* compose env NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics —
default nvidia-container-toolkit only mounts compute libs; Vulkan
needs the graphics ICD (libGLX_nvidia + nvidia_icd.json) too.
* entrypoint reads FISH_CPP_BACKEND (cuda/vulkan/cpu) and selects
the appropriate -c/-v/no-flag invocation.
* Default backend = vulkan.
Subprocess-per-request architecture forced CUDA + model load on every
/v1/tts call (~10-20s init, then 5-15s generation). Even though CUDA
is now actually being used (`-c 0` fix landed), 32s for "Verify."
proved per-request init was the bottleneck.
s2.cpp ships a built-in HTTP server (`--server -H -P`) that keeps the
model resident on the GPU. Refactor:
* entrypoint.sh — backgrounds `s2 --server -P 3030 -c 0 -m ... -t ...`,
waits for it to bind 3030, then foregrounds uvicorn. tini supervises
via `wait -n` so either child dying takes down the container.
* server.py — drops subprocess.run; instead httpx-POSTs Fish-shaped
/v1/tts JSON to s2's localhost:3030/generate (multipart form: text
+ optional prompt_text/prompt_audio for cloning). Model load + CUDA
init now happen once at container start, not per-request.
* Dockerfile — added httpx (shim dep), curl (entrypoint readiness
probe), and the entrypoint.sh COPY+chmod. CMD now invokes
entrypoint.sh instead of uvicorn directly.
* deploy-fish-cpp.yaml — uploads entrypoint.sh alongside server.py.
s2.cpp's README example uses `-v 0` which is `--vulkan 0` (Vulkan
device 0), easy to misread as "voice 0". The shim copied that
verbatim, so even after fixing the libcuda.so build problem AND the
libgomp.so runtime dep, every synthesis ran on CPU because the wrong
backend was selected.
Direct verification: `[Model] NPU not compiled, falling back to CPU`
in stderr; nvidia-smi showed no s2 process; bench timed out at 60s
on phrases that fish-s2 (HF, GPU) does in 7s.
s2.cpp's CLI:
-v <id> = --vulkan <device>
-c <id> = --cuda <device>
-M = --metal (Apple Silicon)
Switched the shim to `-c 0`. The CUDA backend IS in the build (-DS2_CUDA=ON
worked, libggml-cuda.so links fine per ldd, libcuda.so.1 mounts at
runtime via NVIDIA container runtime) — just wasn't being told to use it.
Build succeeded after the libcuda.so symlink fix, but the first
/v1/tts request returned HTTP 500 with:
s2 binary failed (rc=127): /usr/local/bin/s2: error while loading
shared libraries: libgomp.so.1: cannot open shared object file
CMake auto-enabled OpenMP during the build (gcc's -fopenmp flag), so
the s2 binary dynamically links libgomp.so.1. The build-stage devel
image had it; the slim cuda:runtime base doesn't ship it by default.
Adding libgomp1 to the runtime image's apt install resolves it.
Second attempt's CMAKE_LIBRARY_PATH + LIBRARY_PATH didn't get picked
up by ggml's nested CMake — same linker errors as the first run.
Robust fix: symlink the stub at /usr/local/cuda/lib64/stubs/libcuda.so
into /usr/local/lib (which ld searches unconditionally) and provide
both libcuda.so AND libcuda.so.1 (the SONAME ggml-cuda's
libggml-cuda.so links against). ldconfig refreshes the cache.
The symlinks live only in the build stage. The runtime image inherits
the real driver-provided libcuda.so.1 via NVIDIA's container runtime
mount, so the stubs never get used at execution time.
Two issues from the first deploy attempt:
1) Build failure (real): linker errors on s2.cpp's CUDA build —
undefined references to cuMemSetAccess, cuDeviceGet, etc. These
are CUDA Driver API symbols (in libcuda.so), not Runtime API
(libcudart.so). The driver lib is provided by NVIDIA's container
runtime at RUN time, not BUILD time.
Fix: nvidia/cuda:devel images ship a stubs library at
/usr/local/cuda/lib64/stubs/libcuda.so that provides the symbols
for linking but is non-runnable. Adding that path via
LIBRARY_PATH + CMAKE_LIBRARY_PATH lets the linker resolve while
leaving runtime unchanged (real libcuda.so comes from the
driver mount).
2) Verify false positive: the /v1/tts verify step's last command was
`rm -f "$out"` — which always exits 0. This made the shell's
final exit code 0 regardless of whether curl/file/grep succeeded,
so verify reported OK even when nothing was running on host_port.
Fix: `set -e` at top + trap-based cleanup. Failures now propagate;
the rm still runs on either path via EXIT trap.
New stack scaffolding for the Fish quantized-realtime experiment. Not
deployed yet — this commit lands the canonical files; deploy follows.
Architecture decisions made in Phase 1:
* CUDA backend, NOT Vulkan. s2.cpp's CMakeLists exposes both
-DS2_VULKAN and -DS2_CUDA; the most recent upstream commit
(2026-04-12) was specifically about CUDA improvements, and CUDA
on the A6000 will be substantially faster than Vulkan for ML
matmul. -DS2_CUDA=ON in the Dockerfile build args.
* Pinned to s2.cpp commit e48ce8e02d8335bd9a0ba94679f605724b31d12
(2026-04-12 HEAD of main). Repo is alpha software per README;
pin tightly so future churn doesn't break our build. Bump
deliberately when wanting upstream improvements.
* Multi-stage Dockerfile: nvidia/cuda:12.6.0-devel for build (needs
CMake + ninja + git + the CUDA toolchain) → nvidia/cuda:12.6.0-runtime
for serve (slimmer; just the s2 binary + GGML libs + a small Python
shim). Cuts image size by ~50% vs single-stage devel.
* FastAPI shim (server.py) wraps s2.cpp CLI in Fish's `/v1/tts`
contract so the same bench harness + clients work against fish-cpp
with no changes. Per-request flow: decode optional reference WAV
from base64 → write to temp → subprocess.run the s2 binary → stream
resulting WAV back. Adds ~50-100ms per-request fork+exec overhead;
negligible vs the multi-second generation cost.
* `streaming: true` accepted in request body but IGNORED — s2.cpp
writes a complete WAV before returning, so chunked output isn't
available. Unlike fish-s2 (HF wrapper) where streaming drops TTFB
to 26ms, fish-cpp's TTFB ≈ total wall time. Speed depends entirely
on raw generation throughput.
* q6_k as default quant — sweet spot per typical GGUF guidance:
near-bf16 quality at ~5GB. Other variants (q4_k_m, q5_k_m, q8_0,
f16) selectable via FISH_CPP_MODEL env.
* Pinned to GPU 1 (A6000) by default to share with fish-s2 for
direct A/B benching. q6_k weights ~5GB + runtime ~3GB ≈ 8GB —
comfortable on either GPU.
* Port 8199 (next free in the irv-ml1 TTS slate).
Phase 2 (next) is the actual deploy + first build. Reserved 30-45 min
for cold-cache build + weights pull.
Voxtral final fix (8th iteration):
* The bundled voxtral_tts.yaml hardcodes gpu_memory_utilization: 0.8
on the language_model stage — overrides the CLI flag. Mounted a
patched copy (0.4) at /etc/voxtral/voxtral_tts.yaml and pointed
--stage-configs-path there.
* With Kyutai stopped to free 5 GB on the 3090, both stages fit
(target 9.4 + 2.4 GB ≈ 11.8 GB; 17 GB free post-kyutai-stop).
* Voxtral now healthy on GPU 0 — bench: 1.9-2.7 s TTFB, real WAV.
Fish s2-pro optimization (per-request sweep, no model swap):
* `streaming: true` in request body drops TTFB from 7.7 s → 0.026 s
(300×). Total time goes up ~1 s (chunked HTTP overhead) but
perceived latency = TTFB. Use stream:true for any interactive use.
* `latency: "balanced"` actually slower than default — bad name; skip.
* `use_memory_cache: "on"` no measurable benefit.
* `chunk_length: 100` (default 200) no TTFB benefit non-streaming.
* Server-side `--half` (fp16 inference) added via compose `command`
override — passes through start_server.sh's $@ unchanged into
api_server.py. Should reduce total time too. Validation pending
the post-restart bench.
Kyutai stopped to free GPU 0 budget — the bench numbers earlier
(3.4 s avg) were unimpressive vs Voxtral's 2.3 s in the same
multilingual slot. Kept the stack files for future re-deploy if
needed; just the running container is gone.
Fourth attempt finally found the right invocation. Voxtral is a
two-stage TTS pipeline (language_model → acoustic_transformer →
audio output), not a flat MistralForCausalLM. Standard `vllm serve`
errored with "no module named 'acoustic_transformer'" because it
loads the model as a vanilla Mistral causal LM.
Pattern from /workspace/vllm-omni/examples/online_serving/
qwen3_tts/run_server.sh (closest in-image analog):
vllm-omni serve <MODEL> \
--stage-configs-path vllm_omni/model_executor/stage_configs/voxtral_tts.yaml \
--host 0.0.0.0 --port 8000 \
--gpu-memory-utilization 0.45 \
--trust-remote-code --omni
Key differences from previous attempt:
* `vllm-omni` binary, not `vllm`
* `--omni` flag activates multi-stage pipeline
* `--stage-configs-path` points at the bundled YAML that maps
stages to GPU + scheduler + worker classes
* Dropped --load-format/--tokenizer-mode/--config-format=mistral
flags — the stage config handles tokenizer_mode internally
* --trust-remote-code is required for the acoustic_transformer
custom code path
Default .env.example now: GPU 0 (3090) with util 0.45 (~10.6 GB
target on 24 GB GPU). The A6000 is fully booked by Fish s2-pro.
Third voxtral attempt: image pulled clean (3 min, v0.18.0), entrypoint
parsed correctly, vLLM started, but engine init failed two ways:
1. HF rate-limited the irv-ml1 IP (38.120.94.3) during the metadata
fetch — 429 Too Many Requests from too many large unauthenticated
pulls today (heretic, 27b, fish-s2, fish-s1-mini, voxtral). Added
HF_TOKEN env passthrough; user generates a token at
https://huggingface.co/settings/tokens and sets VOXTRAL_HF_TOKEN
in .env.
2. Voxtral uses Mistral's native model format (params.json +
tekken.json tokenizer + consolidated.safetensors single file),
NOT HF transformers format (config.json + tokenizer.json + sharded
.safetensors). vLLM errored with "ensure presence of params.json
for Mistral models." Fix: pass --load-format=mistral
--tokenizer-mode=mistral --config-format=mistral to vllm serve.
Confirmed by inspecting the Voxtral-4B-TTS-2603 HF tree:
25 files, ships params.json + tekken.json + consolidated.safetensors.
Both fixes baked into compose. User needs to drop their HF_TOKEN into
.env once and recreate.
Side note discovered while debugging: fish-s2 s1-mini variant uses
the tiktoken tokenizer format; the wrapper can't load it (errors with
"NoneType has no attribute encode" on warmup). So s1-mini isn't a
drop-in optimization for s2-pro — different code path needed. Fish
back on s2-pro for now.
Second voxtral attempt got past the image pull (v0.18.0 published,
~3 min download) but container init failed:
unable to start container process: error during container init:
exec: "--model=mistralai/Voxtral-4B-TTS-2603": stat ...: no such file
vllm/vllm-omni:v0.18.0 has Entrypoint=null AND Cmd=null — there's no
default executable. The compose's `command:` array becomes the full
exec invocation, with --model=... interpreted as the binary name.
Standard vLLM serving CLI is `vllm serve <model> [flags]`. The
binary's at /usr/local/bin/vllm. Set entrypoint: ["vllm", "serve"]
and pass the model as a positional arg.
While we're here: HF cache was empty too (Voxtral 4B BF16 ~8 GB
download on first start) — vLLM auto-downloads from HF on model
load, so no separate pre-pull step needed.
Three fixes from the second-wave deploy attempts:
* voxtral: vllm/vllm-omni doesn't publish a `latest` tag — pull
failed with "manifest unknown". Pinned VOXTRAL_VLLM_TAG to v0.18.0
(released 2026-03-29, the day after the Voxtral 4B TTS release —
first cut with Voxtral support).
* kyutai-tts: NillPointer wrapper exposes ONLY /health (root) and
POST /v1/audio/speech. No /v1/models, no /v1/audio/voices —
those return 404. Verified by /openapi.json against the live
container. Compose healthcheck + playbook wait + verify steps
all repointed at the actual paths. POST /v1/audio/speech is now
smoke-tested with a RIFF WAV assertion (same pattern as fish-s2).
* fish-s2: added FISH_S2_MODEL env var so the model variant is
swappable via .env without rebuilding. Both s2-pro (default) and
s1-mini are pre-pulled into the bind-mount; LLAMA_CHECKPOINT_PATH
+ DECODER_CHECKPOINT_PATH now use ${FISH_S2_MODEL:-s2-pro}.
s1-mini was originally gated on fishaudio's HF org (401), but
niobures/OpenAudio-S1 mirrors the same files openly — pulled
from there via a one-shot snapshot_download.
After getting fish-s2 finally healthy on attempt #5, the playbook's
verify still failed because /v1/audio/voices doesn't exist. Discovery:
the Fish wrapper has a custom API surface, not OpenAI-compatible.
Real endpoints:
POST /v1/tts — synthesis (text body, optional `references`
field for voice cloning, returns audio/wav)
GET /v1/health — liveness (used by Docker healthcheck)
GET /heartbeat — alternate liveness signal
GET / — Swagger Editor UI for the OpenAPI spec
No /v1/audio/speech, /v1/audio/voices, /v1/models — those return 404.
Updated:
* Playbook verify — replaced the JSON-shape /v1/audio/voices check
with a POST /v1/tts smoke that asserts a real RIFF WAV comes back.
* README API section — replaced the OpenAI-compat examples with
Fish's actual {"text":"...","references":[...]} body shape.
* README disk footprint — corrected ~9 GB → ~11 GB (codec.pth was
larger than I estimated; 1.9 GB + 9 GB safetensors).
* README Lessons learned section — recorded the 5-iteration deploy
story so the next time we touch a Fish-style upstream we don't
re-walk the dockerfile / target / pre-pull / API-shape traps.
Fourth fish-s2 attempt got past build + checkpoints, then container
crashlooped silently again. Diagnosis: the upstream docker/Dockerfile
is multi-stage with `webui` and `server` targets; without specifying
a target, docker builds the LAST stage (webui — gradio-only, no
start_server.sh, no API server). start_server.sh is the entrypoint
script that lives only in the `server` stage.
Confirmed by `cat /app/start_server.sh` inside the built image:
"No such file or directory."
Upstream's compose.yml uses target: server on its server service —
doing the same here.
Third deploy attempt got past the build but crashlooped at container
start: Fish's start_server.sh validates checkpoints/s2-pro/ exists
and exits cleanly (rc=0) if missing — no auto-download, no helpful
message. /worktank/fish-s2/checkpoints/ was empty, so the container
exited every ~52s under restart policy.
Added an idempotent pre-pull step using the same one-shot
python:3.12-slim + huggingface_hub.snapshot_download + hf_transfer
pattern we used for the Qwen 3.6 GGUFs earlier today. Pulls the 9
relevant files (~11 GB total: codec.pth + 2 safetensors shards +
config + tokenizer/template) directly into the bind-mount at
/worktank/fish-s2/checkpoints/s2-pro/ — gated by `creates:` on
codec.pth so the pre-pull step is a no-op on reruns.
~83 s wall-clock for the 11 GB pull on first deploy.
Wrapped .masthead-brand in <a href="index.html"> in both digest.html.j2
and archive.html.j2 so the hero is a clickable shortcut to the latest
edition. Useful when reading an archived edition and you want to jump
back to the freshest one without going through the archive list.
CSS: color: inherit + text-decoration: none keeps the visual
identical; hover drops opacity to 0.85 for affordance; focus-visible
gets an accent outline so keyboard nav is discoverable.
Second deploy attempt failed at build time:
failed to fetch anonymous token: ... ghcr.io/fishaudio/fish-speech ... 403 Forbidden
Root cause: dockerfile.dev is a thin two-line wrapper around
`FROM ghcr.io/fishaudio/fish-speech:${VERSION}`, which is a private
GHCR base image. Anonymous pulls 403, and we'd need GHCR auth to use
that path. The dev variant is meant for upstream's CI / fish-speech
contributors, not external consumers.
The REAL production path (from upstream's compose.base.yml) is to
build from `docker/Dockerfile` with build args BACKEND=cuda,
CUDA_VER=12.9.0, UV_EXTRA=cu129, UV_VERSION=0.8.15. That builds
everything from source — slower (15-20 min cold), but fully self-
contained.
irv-ml1's driver (595.58.03, CUDA 13.2 capable) is forward-compatible
with the 12.9 PyTorch wheels.
Took three iterations to find the right Dockerfile because:
1. First try: dockerfile (lowercase) — doesn't exist
2. Second try: dockerfile.dev — exists but pulls a private base
3. Third try: docker/Dockerfile — actual production path
First fish-s2 deploy attempt failed in step 9/11:
failed to read dockerfile: open dockerfile: no such file or directory
Upstream fishaudio/fish-speech ships:
* dockerfile.dev (lowercase, dev/test image)
* compose.yml + compose.base.yml (intended deploy path:
`docker compose --profile server up`)
There is no standalone production Dockerfile. The dockerfile.dev
image is what their own compose.yml builds from anyway, so building
against it directly is functionally equivalent to using their compose
profile — we just keep our own restart-policy / labels / bind-mount
conventions on the outer compose.
Comment in the build block now documents this so future-Claude doesn't
re-walk the path.
Adds the three premier 2026 TTS releases we missed during the original
fleet build-out (early April), all licensed for self-host:
* Fish Audio S2-Pro (port 8195, GPU 1 / A6000) — released 2026-03-09.
4B dual-AR (Slow + Fast) trained on 10M+ hours / 80+ languages.
Headline: 15,000+ paralinguistic / emotion tags via natural language
([laugh] [whispers] [super happy] etc.) — a step-function over
Chatterbox Turbo's 9 fixed tags. 91.61% paralinguistic win rate on
EmergentTTS-Eval. ~150 ms streaming TTFB, voice cloning, MIT-style
open. ~17 GB VRAM.
* Voxtral TTS (port 8197, GPU 1 / A6000) — Mistral, released 2026-03-28.
4B open-weight, 70 ms model latency, 9.7× realtime. 68.4% blind A/B
win rate vs ElevenLabs Flash v2.5 in cloning. 8 languages
(EN/FR/DE/ES/IT/PT/NL/HI). Served via vLLM-Omni (Mistral's partner
serving stack) — published Docker image, no local build. ~16 GB VRAM.
CC BY-NC license — personal/research use only; flagged in README.
* Kyutai TTS (port 8198, GPU 0 / 3090) — kyutai/tts-1.6b-en_fr.
Trained on 2.5M hours from the Moshi/Mimi team. Claimed 220 ms in
solo setup, 32 simultaneous streams under 350 ms on L40. Kyutai's
official deploy is Rust + websockets only; using NillPointer's
community OpenAI-compat wrapper to bridge to /v1/audio/speech so
it slots into the same bench harness. ~4-6 GB VRAM.
Each stack: compose.yaml (build context, env, volumes, healthcheck,
homepage label), .env.example (all tunables documented), README.md
(why it exists, headline numbers, API, deploy + hardware notes).
Playbooks at playbooks/deploy-{fish-s2,voxtral,kyutai-tts}.yaml are
idempotent in the same shape as the existing deploy-vibevoice /
deploy-chatterbox playbooks.
Port allocations on irv-ml1 after this lands: 8188 ComfyUI, 8190
CosyVoice, 8191 Qwen3-TTS, 8192 IndexTTS-2, 8193 Kokoro, 8194
VibeVoice, 8195 Fish, 8196 Chatterbox, 8197 Voxtral, 8198 Kyutai,
8765 Parakeet ASR.
Investigation of the slow (8-12s) qwen3-tts TTFB found the upstream
wrapper has 5 backend options. The advertised path to fast TTFB is
TTS_BACKEND=optimized (torch.compile + CUDA graphs + real-time
streaming). It loads cleanly but crashes the container during its
hardcoded warmup phase — silent exit (ExitCode 0, no traceback,
no OOM kill), repeats every ~22s under restart policy.
TTS_WARMUP_ON_START=false suppresses the factory-level warmup but
the optimized backend has its own internal warmup that fires
regardless and triggers the crash.
Updated the .env.example block to enumerate all 5 backend options
with their actual current behavior so future-Claude doesn't re-walk
this path. official is staying as the default.
The wrapper's `optimized` backend (torch.compile + CUDA graphs +
real-time streaming) reads its model registry from a YAML config:
default path is ~/qwen3-tts/config.yaml inside the container, which
doesn't exist. Without TTS_CONFIG set, the backend boots with an
empty registry and every synthesis request fails with
"Unknown model key: '<name>'. Available: []".
The repo ships /app/config.yaml with all 4 model variants defined.
Pointing TTS_CONFIG at it lets the optimized backend load cleanly.
This is a prerequisite for benching the optimized backend properly
— it's the path to the upstream's claimed 97 ms streaming TTFB. The
default `official` backend uses naive HF transformers autoregressive
generation that pegged GPU at only 27% utilization and gave us 8-12 s
TTFB on bench (no recompile theory needed — same phrase repeated 4x
plateaued at 8.5 s, ruling out shape-specific recompilation).
qwen3-tts: deploy was using the -Base checkpoint, which sounds like
the right one ("supports voice cloning") but the upstream wrapper's
only synthesis path goes through generate_custom_voice. The -Base
variant doesn't expose that, so every request — including ones with
the wrapper's listed built-in voices like Ryan/Vivian — errored with
"does not support generate_custom_voice". The -CustomVoice variant
exposes both the cloning machinery and the preset voices, and is
what the wrapper actually needs.
The .env.example comments had the variant labels backward; fixed in
this commit. Live host already updated to -CustomVoice via direct
.env edit (model downloaded on container restart).
chatterbox README listed [whisper] and [breath] as supported tags —
those are in the base Chatterbox tag set but NOT in the Turbo set
that's actually loaded. Replaced with the canonical 9-tag list
verified against /api/model-info: laugh, chuckle, sigh, gasp, cough,
clear throat, sniff, groan, shush.
New runbook captures the three-phase process:
Phase 1 — Drop --append-only via DSM Container Manager web UI
Phase 2 — sudo resticprofile forget --prune --verbose on each of
nh3-docker, nh3-dev, irv-ml1 (interactive sudo per host)
Phase 3 — Restore --append-only via DSM
Why each phase looks the way it does, what to expect (largely no-op
runs for the first 6 months while no snapshots have aged out of the
keep window), how to verify each phase non-destructively (curl 401
on the rest-server root proves the container's up + serving), what
to do if Phase 2 fails with `repository is configured as append-only`
(skipped Phase 1 / DSM didn't apply), and the path to future
automation (find docker bin path on DSM, NOPASSWD-lock syncuser to
the specific recreate command).
Includes a "last run history" table seeded with today's first
post-pipeline run (no-op, irv-ml1 only had 3 snapshots due to the
04-25→27 CUDA stall).
Cross-referenced from docs/README.md (runbook tree), docs/
orientation.md (where-to-look table), and STATUS.md item 9 (which
now points at the runbook + records the next-round date 2026-07-27).
Both reported (unhealthy) in docker ps. Two distinct root causes:
* news-digest-web: switched from nginx:alpine to python:3.12-alpine
(uvicorn) but kept the wget healthcheck against `localhost`. Alpine's
/etc/hosts maps localhost to BOTH ::1 and 127.0.0.1; busybox wget
tries IPv6 first, hits "connection refused" because uvicorn binds
IPv4-only, and doesn't fall back. Pinned to 127.0.0.1.
* chatterbox: devnen's image is built from a python:3.10 base and
doesn't ship curl, so `curl -fsS http://localhost:8004/api/model-info`
failed with `/bin/sh: 1: curl: not found`. Replaced with a python
urllib one-liner that fetches + asserts `b'"loaded":true' in body`,
also pinned to 127.0.0.1 to dodge the same IPv4/IPv6 race.
Both YAML extractions tested directly inside the running containers
(via `sh < script`) — chatterbox python check returns 0 when the model
is loaded.
Big update for 2026-04-27. Sections added:
* Marked the "🟥 Blocked — irv-ml1 stalled" header as RECOVERED with
resolution notes (driver 595.58.03 / CUDA 13.2 IS working, both GPUs
detected; the original "stall" must have been a one-shot
post-install hiccup that resolved on a later boot).
* New "Session milestones — 2026-04-27" section covering:
- irv-ml1 unstall + 5 pre-existing GPU stacks restored
- Kokoro GPU variant deployed (irv-ml1:8193) with the .env.example
default flipped to gpu now that the driver works
- VibeVoice 1.5B deployed (irv-ml1:8194) after fixing two bugs:
full 40-char SHA required by buildx + verify regex didn't match
the OpenAI list-format response shape
- Chatterbox Turbo deployed (irv-ml1:8196) after fixing three:
upstream moved Dockerfile path (docker/Dockerfile.gpu →
Dockerfile.cu128 at root), pinned to current SHA instead of `main`,
/health doesn't exist (switched all probes to /api/model-info
which is the wrapper's own ready-after-loaded signal)
- llama-swap qwen3.6 ttl removal across non-pinned variants;
qwen3.6-35-a3b unpinned (was OOM'ing other loads via the pinned
group's persistent: true flag); granite-4-small added to the
pinned group to stop it swapping with qwen3.6-27b
- Backup verification: all three layers green (per-host restic,
PBS-ANA, PBS-NH3 mirror — 2026-04-27 snapshots everywhere). Noted
that backrest's empty dashboard is expected (no plans configured;
the actual orchestration is the per-host resticprofile timers).
Symptom: granite-4-small and qwen3.6-27b were evicting each other
when called in alternation. granite is the news-digest curator (fires
twice daily on cron) — being evicted means a cold reload (~5s) on
every digest tick, plus visible churn whenever the user uses 27b
concurrently.
Added granite-4-small to the `pinned` group as a persistent member.
~5-6 GB at Q4_K_M + 120K KV ≈ comfortable inside the existing pin
budget (qwen3.5-9b ~6 GB → ~12 GB total persistent). Single RTX 6000
Ada is 48 GB, leaves ~36 GB headroom for whichever non-pinned model
the user invokes (qwen3.6-27b at ~30 GB fits cleanly).
Updated the pinned group's docstring to capture the current member set
+ VRAM math + the historical context (qwen3.6-35-a3b was here, was
too heavy, got removed yesterday). Marked the granite ttl: 0 with the
matching "pinned — never unloads" comment as the other group members.
Symptom: qwen3.6-35-a3b refused to deload when other models needed
the VRAM, even with the model itself at ttl: 0. The pinning came from
the `pinned` group's `persistent: true` flag, which exempts members
from eviction by the scheduler regardless of memory pressure. The
model's ttl: 0 only governs idle-timeout, NOT scheduler eviction —
those are separate concerns.
Removed qwen3.6-35-a3b from the group's members. Kept ttl: 0 on the
model itself: still no idle-unload, but the scheduler CAN now evict
it when another non-coexistent model is requested. qwen3.5-9b stays
pinned (~6 GB at Q4 — cheap to hold).
Updated the inline comment + the group-header docstring to reflect
the new semantics so future-Claude doesn't undo this.
The base qwen3.6-35-a3b is already ttl: 0 via the `pinned` group.
The three other Qwen 3.6 variants (abliterated, heretic, 27b) had
ttl: 600 → llama-swap auto-unloaded them after 10 min idle, costing
the next request a full reload (~5-15s). Removed so they stay loaded
once warm. Still get evicted by the normal swap when another
non-pinned model is requested — these aren't joining the pinned group,
just losing their idle-unload timer.
devnen/Chatterbox-TTS-Server doesn't expose /health — neither in code
nor OpenAPI. The deploy hung on the playbook's `Wait for /health to
respond` loop indefinitely (each curl -> 404, retry forever) even
though the container was up and the model loaded clean to CUDA at
22:52:21 (~42s after start).
/api/model-info returns `{"loaded":true,...}` only after the model
finishes loading, so it doubles as liveness + readiness. Updated:
* compose.yaml healthcheck — grep for `"loaded":true` from
/api/model-info.
* playbook wait step — same probe instead of /health.
* verify /health → verify /api/model-info reports loaded.
* verify /v1/audio/voices — switched from greping for `voice|alloy|echo`
literals to parsing JSON and asserting the actual response shape:
`{"status":"ok","voices":[...]}` (devnen's shape — note this is NOT
the OpenAI list-format vibevoice uses).
Build + container + /health all came up clean on the re-run; only the
voices-endpoint verify failed. The check greped the response body for
"voices"/"voice"/alloy/Carter — but VibeVoice's actual response shape
is OpenAI list-format `{"object":"list","data":[...]}`, which contains
none of those substrings. On a fresh install the data array is also
empty (voices live at /worktank/vibevoice/voices/ and the user seeds
them).
Switched the check to parse the JSON and assert the shape (object="list",
data is a list). Robust against empty voices, robust against future
schema additions.
Both deploys failed against irv-ml1 today with upstream-changed-on-us
errors:
* vibevoice: VIBEVOICE_SHA=7614c469a145 (12-char short) made docker
buildx report "repository does not contain ref 7614c469a145" — same
commit IS still HEAD of main, but buildx's git source resolver
doesn't accept short hashes even when unambiguous. Now full 40-char.
* chatterbox: dockerfile: docker/Dockerfile.gpu — devnen restructured
the repo to put Dockerfiles at root, renamed by CUDA version
(Dockerfile.cu128, .cpu, .rocm). Switched to Dockerfile.cu128 (GPU
build for CUDA 12.8 toolkit; works on irv-ml1's 595.58.03 driver).
Also pinned CHATTERBOX_SHA to a full 40-char SHA instead of `main`
so future upstream churn doesn't break the deploy without warning.
Live host .env files patched directly (the playbook only seeds .env
when absent, so canonical edits don't propagate to existing installs).
irv-ml1's driver upgrade to 595.58.03 (kernel 6.1.0-37, CUDA 13.2) is
working — both GPUs detected, modules loaded. The gpu variant of the
Kokoro-FastAPI image (which requires CUDA >= 12.9) is now the right
default for new deploys. Flipping KOKORO_VARIANT=gpu, KOKORO_USE_GPU=true,
KOKORO_GPU_DEVICES=0 (pins to the RTX 3090 — Kokoro is ~1 GB VRAM and
doesn't need the A6000).
Driver bump survived after all (595.58.03, kernel 6.1.0-37, both GPUs
detected and modules loaded). 5 GPU stacks back up clean (comfyui,
cosyvoice, qwen3-tts, index-tts, parakeet — all healthy). Homepage
discovery can resume polling 10.100.79.3:2375 over the WG tunnel.
Two new sections:
* "llama-swap — added two vision-capable Qwen 3.6 entries" documents
the heretic + 27b additions, their pre-pull into HF_HOME=/hfcache
via the one-shot python:3.12-slim + hf_transfer recipe (4:10 and
3:46 wall-clock for 29 GB and 26.5 GB respectively), and the fact
that llama-server's -hf flag auto-loads mmproj when present.
* "Stack tree convention (canonical vs mirror) — clarified" captures
the deploy-stack.sh-was-reading-from-the-wrong-tree bug and the
resolution: stacks/<stack>/ is canonical/intent (deploy source),
stacks-mirror/<host>/<stack>/ is gitignored snapshot for drift
detection only. CLAUDE.md and memory updated separately in the
prior commit.
Decision recorded in CLAUDE.md ("Stack tree convention") and memory
(convention_stacks_vs_mirror.md):
stacks/<stack>/ canonical / intent. git-tracked.
deploy-stack.sh reads from here.
stacks-mirror/<host>/<stack>/ snapshot / reality. gitignored.
sync-stacks.sh writes here. Used
for drift inspection only — never
a deploy source.
Bug this fixes: deploy-stack.sh was reading from the mirror, so edits
to stacks/llama-swap/config.yaml never reached ana-ml2. Today's
two new model entries (qwen3.6-35-a3b-heretic + qwen3.6-27b) lived
in the canonical for hours but the deploy reported "in sync" because
the script only diffed mirror vs server.
Changes:
* deploy-stack.sh: source switched from MIRROR_DIR/$HOST/$STACK to
STACKS_DIR/$STACK. Header comment + error message updated.
* sync-stacks.sh: header explicitly identifies its role as drift
detection; documents the diff command for comparing canonical vs
mirror.
* stacks/llama-swap/{config.yaml → conf/config.yaml}: matches the
deploy mapping (conf/ in canonical → /opt/docker/conf/ on host).
* CLAUDE.md: "Stack mirror (pull / push)" section rewritten as
"Stack tree convention (canonical vs mirror)" with the role table
+ workflow rules + diff recipe. Layout diagram updated.
Both models pre-pulled into /tank/aimodels/huggingface (HF_HOME=/hfcache
inside the container) via huggingface_hub.snapshot_download with
hf_transfer for parallel chunked download — heretic's 29 GB landed in
~4 min, unsloth's 26.5 GB in ~3:46 (~118 MB/s each).
heretic: llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF:Q6_K
27b: unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL
Both repos include mmproj-BF16.gguf alongside the main GGUF, and
llama-server's -hf flag auto-loads the mmproj when present in the same
repo (-hf docs: "mmproj is also downloaded automatically if available").
So both entries get vision (image-text-to-text) without needing an
explicit --mmproj path. ttl: 600 (10-min idle unload), matching the
existing abliterated entry's style.
The server-rendered .source-count / .desk-count badges were correct
at render time but went stale the moment the user hid anything —
"r/HOMELAB (4)" stayed at 4 even after all 4 items were hidden.
Worse, the entire source header still rendered with a (0) badge
once every item underneath was gone.
app.js gains a refreshCounts() pass that walks every .source and
.desk, recomputes the visible (non-.is-hidden) child count, updates
the badge text, and toggles an .is-empty class. CSS rule for
.source.is-empty and .desk.is-empty sets display:none so empty
groups collapse out entirely. Hooked into hideItem, restoreItem,
and the initial-paint hidden-set application.
New section documenting the architecture change (news-digest-web
moved from nginx:alpine to a FastAPI app on uvicorn built from the
worker's same Dockerfile), the three new endpoints
(GET /api/hidden, POST /api/hide, POST /api/restore), the item-id
scheme (12-char sha1 of reddit:<post_id> or miniflux:<entry_id>
so hide-once = hide-forever-for-that-article), and the playbook
changes (dropped DOCKER_BUILDKIT=0 now that ana-docker is on
docker-ce 29, added round-trip API verify steps).
Adds a small × on each item that hides it from the page. State is
server-side at /output/hidden.json so the same hidden set follows
the user across devices (home, ipad, laptop, work). A "Hidden (N)"
tray at the bottom shows what's hidden on the current page with a
restore button per row; older hidden ids that aren't on this page
sit silently and continue to filter future editions that include
the same article.
Architecture change: news-digest-web swaps from nginx:alpine to a
FastAPI app on uvicorn, built from the same Dockerfile as the
worker. Same image, different command (`uvicorn web:app` overrides
the worker's cron entrypoint via compose). Drops one image dependency,
adds /api/{hidden,hide,restore}.
Item ids are stable 12-char sha1 prefixes (`reddit:<post_id>` /
`miniflux:<entry_id>`) computed in digest.py at render time and
emitted as `data-id` on each .item. The frontend reads /api/hidden
once on load, applies `is-hidden` to matching items, and POSTs
hide/restore on user interaction (optimistic, with rollback on
network error).
Storage: single JSON array at /output/hidden.json, atomic writes
via tempfile + rename, threading.Lock around the read-modify-write
inside the single uvicorn worker. No auth — the digest itself is
unauthenticated on LAN; same trust boundary applies.
Playbook also drops the DOCKER_BUILDKIT=0 fallback now that
ana-docker is on docker-ce 29, and adds three verify steps
(/api/hidden returns a JSON array, app.js is reachable, full
hide/restore round-trip with a synthetic id).
autorestic removal completed on both esh-docker-vm and esh-vm-db
after two playbook fixes (YAML plain-scalar folding ate a backslash
continuation; YAML tag indicator stripped a leading `!`). Both
documented inline.
seafile seahub race resolved by adding a healthcheck to mariadb
(bundled healthcheck.sh --connect --innodb_initialized) and
converting seafile's depends_on to long-form with
condition: service_healthy on db. Compose now waits for InnoDB
to initialize before starting seahub, so the daemon-restart race
that wedged the python frontend can't recur. Verified: seahub log
clean post-recreate, traefik 502 rate dropped to zero on
seafile@docker. Compose change lives on the server (the mirror is
gitignored by design).
YAML treats a leading `!` as a tag indicator, so the unquoted
`shell: ! command -v autorestic >/dev/null` was parsed as a tagged
scalar with the `!` stripped. The verify ended up running just
`command -v autorestic >/dev/null` — which exits non-zero when
autorestic is absent, the OPPOSITE of what the assertion needed.
Quoted version `"! command -v autorestic >/dev/null"` survives
parsing and gives the intended bash negation.
The previous version listed four unit paths separated by `\` + newline.
That looks fine in source but YAML plain-scalar folding collapses the
sequence to a literal `\ ` — the backslash + space no longer functions
as a shell line continuation, and only the first path actually gets
passed to rm. End result on esh-docker-vm's first run: backup.service
removed; backup.timer + prune.service + prune.timer survived; verify
correctly caught the partial state.
Switched to `rm -f /etc/systemd/system/autorestic-*.{service,timer}`
form — single string, no folding hazard, and idempotent on hosts where
some or all of the files are already gone. Re-running on esh-docker-vm
will mop up the leftovers cleanly.
Migration complete:
* ana-docker on docker-ce 29.4.1, all 29 containers back up. Traefik
routing live (verified 200s on matrix.phasefinal.com presence +
seafile.phasefinal.com syncs).
* traefik-postboot.service installed + enabled on both traefik hosts
(esh-docker-vm, ana-docker) — one-shot systemd unit that restarts
traefik 60s after every boot, fixing the long-standing routing-races-
after-reboot symptom.
New playbook: remove-autorestic. Triggered by a typo (`D:escription`
in autorestic-backup.timer line 2) flagged by systemd-analyze during
the traefik-postboot install on esh-docker-vm. Rather than fix it,
remove autorestic — it's redundant with the PBS + structured-restic
two-layer pipeline that's been operational since 2026-04-22. Detected
on two ESH-side hosts: esh-docker-vm and esh-vm-db. Playbook removes
the four unit files + the /usr/local/bin/autorestic binary; leaves
/srv/backups/autorestic/.autorestic.yml (archival) and
/mnt/backup/restic/repo/esh (historical snapshots) for separate
disposition.
Sub-finding from ana-docker upgrade: seafile's seahub (the Python
frontend at port 8000 inside the container) failed to start because
mysql wasn't ready when seafile booted, and a single restart didn't
recover it. Traefik routes return 502 on seafile dynamic endpoints
until seahub is up. Needs separate triage of seafile's depends_on
wiring or seahub's retry behavior — not a docker-ce regression.
Traefik often misses backends after a reboot or daemon swap because
(a) its docker provider debounces / drops events when 30+ containers
start in a burst, and (b) backends can be `Created` on the docker
socket but not yet attached to traefik-net when traefik scans. The
empirical workaround is `docker restart traefik` once the topology
settles — this unit bakes that in.
Type=oneshot, After=docker.service, ExecStartPre=/bin/sleep 60,
ExecStart=docker restart traefik. Runs once per boot. delay_seconds
and container name are tunable via --var.
Verify phase: file mode, enabled state, ExecStart references the
right container, container actually exists on the host, and
systemd-analyze parses the unit cleanly (lint without executing —
avoids needlessly bouncing traefik on healthy hosts).
In scope: esh-docker-vm, ana-docker (the two hosts that run traefik).
docker-ce 29 ships docker-compose-plugin renumbered to v5.x (was v2.x
with docker-ce 26-28). Same Compose v2 codebase under the hood —
Docker just realigned the major number. The verify regex was hardcoded
to `v2\.[0-9]+\.[0-9]+`, so a successful migration on esh-docker-vm
(29.4.1, 16/16 stacks back up clean) reported FAILED on the verify
phase. Switched to `docker compose version --short` parsed for major,
gated `>= 2` — works across future plugin renumbers too.
STATUS.md: mark esh-docker-vm done. ana-docker is the last host.
After nh3-docker's swap, two systemd unit gotchas surfaced that the
playbook now handles automatically:
* The docker.io-era /etc/systemd/system/docker.service.d/override.conf
hardcoded ExecStart=/usr/sbin/dockerd; docker-ce installs at
/usr/bin/dockerd → daemon failed status=203/EXEC.
* The shipped docker-ce unit's ExecStart=dockerd -H fd:// conflicts
with daemon.json hosts: (defined for the 0.0.0.0:2375 homepage
discovery binding) → "conflicting host options".
The "Rewrite docker.service drop-in" step now backs up any existing
override, probes daemon.json for a hosts: setting, and installs an
override that strips -H from ExecStart when needed. Also added an
explicit systemctl reset-failed step to clear the start-rate-limit
state that 3 failed install-time starts leave behind.
configs/homepage/docker.yaml: comment out irv-ml1-docker provider —
20s-per-poll ETIMEDOUTs from the stalled host were drowning homepage's
logs and apparently blocking ana-pfi-docker discovery (the Miniflux
card in the News group wouldn't render until removal). Re-enable when
irv-ml1 is back.
STATUS.md: new "Active migration" section tracking the docker-ce
rollout — nh3-docker done; esh-docker-vm + ana-docker queued.
nh3-docker's daemon kept failing post-package-swap with status=203
even after daemon-reload. Root cause: a stale
/etc/systemd/system/docker.service.d/override.conf from the docker.io
era hardcoding ExecStart=/usr/sbin/dockerd. The override (a) points
at the no-longer-existing path, AND (b) typically also adds
-H tcp://... which now duplicates the hosts: setting in
/etc/docker/daemon.json — dockerd refuses to start when both define
hosts ('conflicting host options').
Daemon.json is the modern way to expose the TCP socket. The
override is redundant and wrong. Move it aside (preserve a
.pre-upgrade copy for forensics), then daemon-reload, then start.
Should let esh-docker-vm and ana-docker upgrades go through cleanly
without the manual debug loop nh3-docker required.
Editorial-briefing favicon: 32×32 SVG, Australis palette. Cyan
masthead-rule across the top echoes the page's aurora-rule, four
descending text-line indicators below evoke a newspaper column.
Reads cleanly at 16×16 (the typical browser tab size). Static
markup only — no script, no animation — so all browsers honor
it for tab + bookmark icons.
Linked from both digest.html.j2 and archive.html.j2 with the
proper type="image/svg+xml" attribute. Served by nginx from
the bind-mounted /output dir alongside index.html and style.css.
Deploy playbook also updated to copy the favicon into /output at
deploy-time so a fresh deploy doesn't 404 on the icon before the
first cron fire.
Docker's official package installs dockerd at /usr/bin/dockerd; the
Debian docker.io package put it at /usr/sbin/dockerd. After the apt
swap, the new docker.service unit file is on disk with the right
path, but systemd's cached unit still has the OLD ExecStart pointing
at /usr/sbin/dockerd. Daemon start fails with:
status=203/EXEC "No such file or directory"
Fix is systemctl daemon-reload between install and start. nh3-docker
hit this; adding the step so esh-docker-vm and ana-docker don't.
Two requested polish items:
1. Reddit items now show TWO distinct action chips in the footer:
[↗ SOURCE huggingface.co] [⌥ REDDIT THREAD] 3h · u/foo
Previously the external link was a tiny dot-separated text link
that visually competed with the post metadata. Now: explicit
labeled buttons, distinct colors (cyan for source, blue for
thread), with hover states that match the Australis accent
palette. Non-Reddit items get just the SOURCE chip.
2. Archive page at /archive.html — lists every edition-*.html on
disk, newest-first, sorted PM-before-AM within a day. Each row
is the date in editorial serif + an AM/PM chip color-coded
(yellow morning / cyan evening) + a hover-affordance arrow.
Header link from the main edition reads "ARCHIVE →".
Generation: digest.py walks OUTPUT_DIR for the edition-*.html
filename pattern on every run, sorts, renders archive.html.j2,
writes atomically. Cheap (~1ms even with hundreds of editions).
No retention cap — twice-daily for a year is ~700 small files,
well under any reasonable concern.
CSS additions: .action chip styles (with per-action color variants),
.archive-link in masthead, full .archive-row + .archive-edition
treatment.
Multi-line shell with backslash-continued URL had the continuation
line starting at column 0, which breaks YAML's | literal block
('could not find expected :'). Stash the URL into a shell variable
and emit on one logical line.
Three docker hosts on the fleet still run docker.io 20.10.24 (the
Debian bookworm package) which:
* sticks at API 1.41 — newer compose clients (1.52+) refuse to talk
to it without DOCKER_BUILDKIT=0 fallback (caught during the
news-digest deploy on ana-docker today)
* is functionally EOL — docker.io's upstream no longer ships to it
* is missing modern buildx driver versions
This playbook handles a single-host migration: snapshot existing
docker package versions for rollback reference, stop every running
compose stack, apt-remove (NOT purge — preserves /var/lib/docker)
docker.io + plugins, add Docker's signed APT repo for Debian, install
docker-ce + docker-compose-plugin + containerd.io + buildx-plugin,
restart the daemon, bring stacks back up.
Volumes / images / containers survive the swap because:
* /var/lib/docker is preserved by `apt remove` (vs purge)
* both packages default to the overlay2 storage driver
Recommended host order (least → most blast radius):
1. nh3-docker (NH3 site, fewer services)
2. esh-docker-vm (home lab; many services but single-consumer)
3. ana-docker (production-ish; vaultwarden, gitea, synapse,
task-board, miniflux, news-digest, paperless-ng)
Run as `scripts/elway <host> --playbook playbooks/upgrade-docker-ce.yaml`
per host. Verify between hosts via `docker version` + spot-check a
few containers.
Rollback if a daemon won't start or a container errors:
ssh <host> 'sudo apt install --allow-downgrades \$(cat /tmp/docker-pre-upgrade.txt | tr "\n" " ")'
Three iterations to get end-to-end:
1. Dockerfile missed COPY run-digest.sh — cron's exec target wasn't
in the image, every fire failed. Added COPY + chmod.
2. Jinja template used {{ list|sum(attribute='items') }} which
sum()s lists with start=0 → TypeError int+list. Switched to
computing reddit_total / tech_total in Python and passing as
template args.
3. LLM defaulted to qwen3.5-35-a3b which (a) is broken in
llama-swap (model process exits on launch), (b) when working,
defaults to extended-thinking mode that eats the entire token
budget without producing any visible content. Same pattern with
qwen3.6-35-a3b. Switched default to granite-4-small — small (4B),
fast (~1s/call), no thinking-mode pathology, returns clean JSON.
Whole pipeline now runs in ~35s total across 8 sources.
Also hardened the LLM response parser to fall back to
reasoning_content when content is empty — catches the thinking-mode
case if anyone ever points the digest at one of those models. Plus
the deploy playbook gained DOCKER_BUILDKIT=0 because ana-docker is
on docker 20.10 which doesn't carry the buildx driver versions our
newer client expects ("client version 1.52 is too new"). Real fix is
upgrading docker on the fleet — separate workstream.
The Miniflux inbox got noisy after a few subreddits + HN + Lobste.rs.
This stack distills a single static page twice a day — at 0800 and
2000 local — that surfaces only what cleared score + ratio filters,
each item tldr'd by qwen3.5-35-a3b on llama-swap.
Pipeline (digest.py, ~330 lines):
1. Discover subreddits from Miniflux feeds (any reddit.com/r/<sub>/
URL — single source of truth, no duplicated config).
2. Reddit JSON top-of-day per sub. Filter: score >= 50,
upvote_ratio >= 0.85. Cap 8 items per sub.
3. Miniflux /v1/entries for the 'Tech aggregators' category
(HN, Lobste.rs) — last 12 hours.
4. Batched per-source summarization via llama-swap
/v1/chat/completions. Each post gets a one-sentence tldr +
one-word tag (news / tutorial / release / discussion /
question / showcase / drama / meme).
5. Render Jinja2 template. Atomic write to /output/index.html
(.tmp + rename) so partial pages never get served. Per-edition
archive at /output/edition-YYYY-MM-DD-{am,pm}.html.
Two containers:
news-digest-worker python:3.12-alpine + busybox crond
news-digest-web nginx:alpine, port 8181, homepage card via
docker labels (group=News, fits next to Miniflux)
Both bind-mount /opt/docker/data/news-digest as /output and
/usr/share/nginx/html respectively.
Aesthetic — operations-center chrome (Australis cool-mono palette,
JetBrains Mono UPPERCASE eyebrows, mdi-glyph anchor) wrapping
editorial-serif news content (Fraunces variable serif w/ optical
sizes). Two type families that wouldn't normally meet, intentionally
combined: chrome says 'filed at 0800 from the bridge'; headlines say
'this is news, read it like news.' Sticky aurora-glow rule under the
masthead is the only sanctioned Australis gradient.
Edition stamp (AM/PM in big mono Australis-yellow) is the signature
piece — establishes the twice-daily rhythm at a glance.
All filtering + LLM + scheduling knobs in .env. Subreddit list is
implicit (read from Miniflux), so adding a sub = subscribing in
Miniflux, no config edit on this stack.
Subscribed live via Miniflux API; mirroring back to the canonical
OPML so a fresh deploy elsewhere starts with the same feed set.
All three land in the existing 'Subreddits — selfhosting' category
(retitled to mention image-gen + LLM + homelab to reflect scope).
Initial deploy failed with 'Container cannot be connected to network
endpoints: miniflux-net, traefik-net' — the docker engine balks at
joining a brand-new internal network and an existing external
network in one create step.
Flattened both containers onto traefik-net only. The DB password
still protects miniflux-db, and traefik-net is internal-LAN-only,
so co-locating them is fine. Verify step updated to check for
traefik-net membership instead of the (now-gone) miniflux-net.
Adds Miniflux on ana-docker as the unified inbox for tech blogs,
Hacker News, lobste.rs, and selected subreddits. Reddit serves clean
RSS for any sub at https://reddit.com/r/<sub>/.rss, so subreddit
follows fold into the same inbox as everything else — no Reddit
account needed, no manual polling.
Stack:
stacks/miniflux/
compose.yaml — miniflux + bundled postgres:16
.env.example — placeholders for DB password + admin user
starter-feeds.opml — initial subscriptions (HN, Lobste.rs,
r/selfhosted, r/homelab, r/LocalLLaMA, r/nba)
README.md — deploy / OPML import / r/nba spoiler
block-list / backup / update flow
Postgres bundled with the stack (not pfi-postgres) — single-user RSS
DB is tiny and the bundle keeps the dependency graph flat.
Homepage gets a new 'News' group at the TOP of the Main tab (above
Monitoring) so the Miniflux card sits prominently. The card itself
auto-discovers via the homepage.* labels on the miniflux container.
Per-feed block-list rule for r/nba documented in README — Reddit's
RSS titles for game threads include scores ("Lakers 108 - Warriors
102 [Final]") which spoil the game; a regex catches the score
patterns and skips those entries while keeping discussion/highlights.
Deploy:
scripts/elway ana-docker --playbook playbooks/deploy-miniflux.yaml
Then edit /opt/docker/compose/miniflux/.env on the host to fill in
the two CHANGE_ME passwords and `docker compose up -d` again.
Driver bump 570.124.06 → 595.58.03 attempted via
scripts/upgrade-irv-ml1-cuda.sh on 2026-04-25; host did not come up
cleanly after the post-install reboot.
Captures, in one place, everything currently in-flight against irv-ml1
so it can be revisited as a unit once the machine is healthy:
* Recovery options ordered by invasiveness (console diagnose →
rollback script → manual driver pin → try a different driver
series).
* Three deferred TTS deploys (Kokoro GPU flip, VibeVoice 1.5B,
Chatterbox Turbo) with the exact playbook commands to re-run.
* Smoke-test reminder for the 5 existing GPU stacks (comfyui,
cosyvoice, qwen3-tts, index-tts, parakeet) which were healthy
before the upgrade but unverified after.
* Explicit 'don't touch irv-ml1' guard rail until resolved.
Sits in 🟥 Blocked — surfaces above the existing 'Quick wins'
section so future sessions see it first.
Phase 1 silently aborted right after the 'cuda-drivers candidate' step
— before snapshotting, stopping containers, installing the new driver,
or rebooting. Symptom: phase2 showed driver still at 570.124.06 and
containers with multi-hour uptimes (proof they were never stopped).
Cause: `apt-cache policy cuda-drivers | head -10`. head reads its 10
lines and closes the pipe early; that delivers SIGPIPE to apt-cache,
which under `set -o pipefail` propagates as a non-zero exit, and
`set -e` immediately terminates the remote shell. The same hazard
existed in phase2's `docker ps ... | head -15`.
Fix is one-line per call site: drop the head. apt-cache policy output
for cuda-drivers is short (~10 lines), and docker ps's container list
on irv-ml1 is small enough to print fully.
Phase 1 was unconditionally adding /etc/apt/sources.list.d/nvidia-cuda.list
pointing at /etc/apt/keyrings/nvidia-cuda.gpg. But irv-ml1 already had
the upstream-style /etc/apt/sources.list.d/cuda-debian12-x86_64.list
with /usr/share/keyrings/cuda-archive-keyring.gpg — that's how the
existing 570 driver got installed in the first place. APT then refused
both: 'Conflicting values set for option Signed-By regarding source ...
debian12/x86_64/'.
Now the script:
* removes any orphan nvidia-cuda.{list,gpg} files from a previous
failed run
* greps existing /etc/apt/sources.list.d/*.list for a debian12 cuda
repo entry; if found, trusts and uses it
* only writes a new repo entry if no existing one is found, and
when it does, uses the same upstream paths (cuda-archive-keyring.gpg,
cuda-debian12-x86_64.list) so reruns are stable
Re-run safe — phase1 can be invoked any number of times without
accumulating conflicting configs.
The previous version piped a heredoc to `ssh -t host bash -s` —
which can't work because `-t` requires a real TTY and heredoc puts
stdin in pipe mode. They're mutually exclusive (orientation.md
gotcha, tripped over once already). The remote sudo therefore aborted
with 'a terminal is required to read the password' before doing
anything.
Restructured: the remote half of each phase is built as a tempfile
locally, scp'd to /tmp on irv-ml1, made executable, and invoked via
`ssh -t host /tmp/upgrade-cuda-remote.sh phase1`. Now stdin is the
real TTY (no pipe), sudo prompts work, and the rest of the script
runs as before.
Same three phases, same rollback behavior, same idempotency story.
Just a transport fix for the local→remote half.
Two-phase script for bumping irv-ml1's NVIDIA driver to support
CUDA 12.9 — currently blocked at 12.8 per driver 570.124.06, which
is why the Kokoro GPU image (built against CUDA 12.9) wouldn't
start.
Phase 1 (interactive — sudo prompted once via ssh -t):
* snapshot current nvidia packages to /tmp/nvidia-pre-upgrade.txt
(rollback reference)
* add NVIDIA's CUDA APT repo for Debian 12 with signed key
* stop GPU containers (comfyui, cosyvoice, qwen3-tts, index-tts,
parakeet, kokoro)
* apt install cuda-drivers (currently pulls 580 series)
* reboot
Phase 2 (re-run after host comes back):
* poll for SSH availability
* verify new driver + CUDA version
* restart GPU containers via docker compose up -d
* print next steps (flip Kokoro .env to gpu variant + redeploy)
Includes a rollback subcommand that reinstalls the snapshotted
packages with --allow-downgrades and reboots — for the case where
the new driver regresses on this kernel.
Driver source switched from Debian's nvidia-driver metapackage to
NVIDIA's official cuda-drivers metapackage; bookworm doesn't carry
575+ in its main or backports repos as of 2026-04-25.
Two fixes from the failed first deploy on irv-ml1:
1. CPU/GPU variant. Kokoro's GPU image needs CUDA >= 12.9; irv-ml1's
driver 570.124.06 caps at 12.8 so the gpu variant fails with
"nvidia-container-cli: requirement error: unsatisfied condition:
cuda>=12.9". Make the variant a knob:
KOKORO_VARIANT=cpu (default — works anywhere)
KOKORO_VARIANT=gpu (after driver bump)
KOKORO_USE_GPU=false|true (matches the variant)
Kokoro is tiny (82M params) so CPU is workable: TTFA ~1s vs ~300ms
on GPU. Acceptable while the driver bump gets scheduled. compose.yaml
no longer hard-codes `runtime: nvidia` — relies on the daemon's
default-runtime + NVIDIA_VISIBLE_DEVICES gating, same as how the
wrapper's USE_GPU flag selects the inference path inside the
container. Toggling between variants is now a `.env` edit + restart.
2. Tighter pull-log filter. --quiet on `docker compose pull` only
suppresses the pull command's stdout; the docker daemon still
emits per-layer extraction events on stderr ("ffbfd7a09415
Extracting 64.06MB" repeated dozens of times per layer). Drop those
too via grep on the SHA-prefixed pattern. set -o pipefail keeps a
real pull failure visible.
For existing deployments: removing /opt/docker/compose/kokoro/.env
on the host and rerunning the playbook re-seeds with the new schema.
Profiling the index-tts deploy log (2057 lines) showed ~25% was just
pip's per-package Downloading / Collecting / Requirement-already /
progress-bar spam — useless for ops, hard to scan when something
actually breaks.
Three changes across the four TTS deploy playbooks:
1. Pulls (Kokoro): add --quiet. 6.5 GB pull no longer floods the log
with per-layer progress redraws. Final "X Pulled" still prints.
2. Builds (VibeVoice, Chatterbox, IndexTTS-2): add --progress=plain
to stop the BuildKit TUI from littering the captured log with
carriage-return overdraws, then pipe through a grep filter that
drops pip's noisy lines but keeps:
- buildkit step transitions (#NN [stage])
- DONE / CACHED / ERROR markers
- apt + build-stage messages
set -o pipefail keeps a real build failure from being swallowed
by the grep's exit code.
Net effect: ~25% smaller logs, much more scannable; full visibility
into step progress and errors preserved.
"docker compose pull (first run: ~6.5 GB from GHCR)" had an unquoted
colon-space inside a plain scalar value, which YAML parses as a
nested mapping — elway aborted on load. Single-line fix: wrap the
value in double quotes.
Three TTS additions to round out coverage on irv-ml1, each filling a
distinct niche the existing slate doesn't own.
Final coverage matrix (all on irv-ml1):
Kokoro — low-latency English, fixed voice library, ~300ms TTFA
Chatterbox Turbo — low-latency English w/ voice cloning + paralinguistic tags
IndexTTS-2 — English voice cloning + emotion vector / text control
Qwen3-TTS-1.7B-Base — high-quality English voice cloning
CosyVoice 3 — multilingual (Chinese-leaning)
VibeVoice 1.5B — long-form / multi-speaker dialogue
stacks/kokoro:
- port 8193, GPU device 0 (3090)
- pulls ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.4-master (no Dockerfile,
no first-run model download — models baked in)
- 60+ built-in voices, OpenAI-compat with stream=true over chunked HTTP
- Apache-2.0 weights + code, ~1 GB VRAM
stacks/vibevoice:
- port 8194, GPU device 1 (A6000 — for 7B headroom)
- builds groxaxo/VibeVoice-FastAPI1 (more current fork of ncoder-ai)
pinned to 7614c469a145
- default model microsoft/VibeVoice-1.5B (~7 GB bf16 VRAM); env var
swap to rsxdalv/VibeVoice-Large (7B) or FabioSarracino/VibeVoice-Large-Q8
- multi-speaker dialogue via /v1/vibevoice/generate with Speaker N: format
- long-form niche only — not low-latency
stacks/chatterbox:
- port 8196, GPU device 0 (3090)
- builds devnen/Chatterbox-TTS-Server (most active Turbo-supporting wrapper)
- default model ResembleAI/chatterbox-turbo (~2.5 GB fp16, ~75ms latency)
- paralinguistic tags inline ([laugh] [whisper] etc) — different shape
from IndexTTS-2's emotion vector; fills the speed+cloning niche
Kokoro/IndexTTS don't cover together
- mandatory PerTh watermark on outputs (Resemble policy)
Three matching playbooks under playbooks/deploy-{kokoro,vibevoice,
chatterbox}.yaml. All idempotent, creates-/when-gated.
Cold-deploy disk on /worktank/: ~7 GB Kokoro + ~19 GB VibeVoice 1.5B
+ ~12 GB Chatterbox = ~38 GB total. VRAM concurrent: ~10-11 GB across
both GPUs.
Skipped from the original four-stack proposal: VibeVoice Realtime
(overlaps Kokoro's niche; Kokoro wins on latency, license, and not
needing a build).
IndexTTS-2's tts.infer(stream_return=True) is a generator that yields
audio chunks per text segment as they finish, plus inter-segment
silence. Expose this via the existing POST /v1/audio/speech with a new
"stream": true field on the request body.
Wire-up:
- 44-byte WAV header emitted up front with placeholder data length
(0xFFFFFFFF) so chunks can be written before total samples are
known. Players that read until EOF (mpv, ffplay, aplay, sox,
browsers via <audio>) handle this fine.
- Each yielded chunk goes through _chunk_to_pcm_bytes(), which
handles torch tensors / numpy arrays in either int16 or float
(-1..1) form.
- 22050 Hz mono int16 — IndexTTS-2's hardcoded output shape.
Time-to-first-audio drops from full-file latency to ~one-segment
latency. Single-sentence inputs barely benefit; long passages /
multi-paragraph reads benefit a lot. Strict metadata parsers may
balk at the placeholder size — request without stream for a
closed-length WAV in that case.
INDEX_TTS_TAG bumped to v2 to force a rebuild.
budget hazard + media-CDN workaround
Tried adding git-lfs install + git lfs pull to the build to get
real example WAVs into the image — failed with:
Error downloading object: examples/emo_hate.wav: Smudge error:
batch response: This repository exceeded its LFS budget. The
account responsible for the budget should increase it to
restore access.
The index-tts org's LFS bandwidth quota is exhausted upstream and
out of our control. Reverting the Dockerfile change. The examples
aren't needed for the wrapper to work; emotion_text and
emotion_vector are sufficient for end-to-end testing without any
WAV file at all.
For users who want the bundled example clips as starter audio,
README now documents the media-CDN URL trick — same LFS objects
served via a different code path that doesn't count against the
LFS API budget. INDEX_TTS_TAG stays at v1.
The IndexTTS-2 repo stores examples/emo_*.wav and examples/voice_*.wav
as Git LFS objects. v1 of our image cloned the repo without an LFS
pull, leaving those paths as ~130-byte pointer text files — unusable
for `docker cp` into /worktank/index-tts/{voices,emotions}/ as starter
references. (Caught when an emotion_voice="hate" call returned audio
that was actually the pointer text round-tripped through file IO.)
v2 adds git-lfs to the apt list, calls `git lfs install --system`
once, and `git lfs pull` after the checkout. Adds ~1-2 MB to the
image (the examples are small audio clips). INDEX_TTS_TAG bumped to
v2 to force a clean rebuild.
Adds a third TTS to the irv-ml1 fleet. IndexTTS-2 is Bilibili's
emotion-controllable zero-shot TTS (paper 2506.21619). Distinguishing
capability vs the existing two: timbre and emotion are disentangled —
clone a voice's timbre from one reference and the emotion from a
different reference, OR set emotion via 8-vector, OR derive it from a
text description. Neither CosyVoice 3 nor Qwen3-TTS-1.7B-Base does
this cleanly in English.
Wrapper is owned end-to-end (~150 lines in app.py) — the only existing
FastAPI fork (csllpr/index-tts-fastapi) targets v1 and is a dormant
single-commit repo. Upstream IndexTTS-2 ships only a Gradio webui.
Layout follows the qwen3-tts pattern:
stacks/index-tts/
Dockerfile — CUDA 12.8 base, IndexTTS pinned to a SHA
app.py — FastAPI: POST /v1/audio/speech + /v1/voices
entrypoint.sh — one-time HF snapshot_download of the weights
compose.yaml — env-driven, GPU pinning support, bind mounts
.env.example — port 8192, fp16, paths
README.md — API examples + comparison vs the other TTS
playbooks/deploy-index-tts.yaml — elway playbook for irv-ml1
Voice and emotion libraries are flat host dirs of WAVs, bind-mounted.
Drop a new <name>.wav and /v1/voices picks it up immediately.
License caveat: IndexTTS-2 weights ship under a custom Bilibili
license (free at our scale, not OSI-open). README documents it.
Toggles --append-only off on the rest-server via a temporary
docker-compose.override.yaml (canonical compose untouched), runs
resticprofile forget --prune --verbose on each client of that
rest-server, then restores --append-only. The restore is wrapped in
a trap so a partial-failure prune still leaves the rest-server in
its safe configuration.
ANA side is fully automated against ana-docker (5 clients:
ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas, esh-vm-db).
NH3 side currently prints a manual DSM ceremony — Synology Container
Manager doesn't expose docker on the expected paths and syncuser
sudo isn't NOPASSWD, so the toggle isn't safely scriptable from
this workstation. The instructions cover the same flow in DSM web
UI + interactive ssh on each NH3 client (nh3-docker, nh3-dev,
irv-ml1).
Usage:
scripts/restic-prune.sh ana # ANA only (auto)
scripts/restic-prune.sh nh3 # NH3 instructions
scripts/restic-prune.sh all # both
scripts/restic-prune.sh -h # help
scripts/restic-prune.sh --dry-run ana # show every command
vh