Both models pre-pulled into /tank/aimodels/huggingface (HF_HOME=/hfcache
inside the container) via huggingface_hub.snapshot_download with
hf_transfer for parallel chunked download — heretic's 29 GB landed in
~4 min, unsloth's 26.5 GB in ~3:46 (~118 MB/s each).
heretic: llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF:Q6_K
27b: unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL
Both repos include mmproj-BF16.gguf alongside the main GGUF, and
llama-server's -hf flag auto-loads the mmproj when present in the same
repo (-hf docs: "mmproj is also downloaded automatically if available").
So both entries get vision (image-text-to-text) without needing an
explicit --mmproj path. ttl: 600 (10-min idle unload), matching the
existing abliterated entry's style.
The server-rendered .source-count / .desk-count badges were correct
at render time but went stale the moment the user hid anything —
"r/HOMELAB (4)" stayed at 4 even after all 4 items were hidden.
Worse, the entire source header still rendered with a (0) badge
once every item underneath was gone.
app.js gains a refreshCounts() pass that walks every .source and
.desk, recomputes the visible (non-.is-hidden) child count, updates
the badge text, and toggles an .is-empty class. CSS rule for
.source.is-empty and .desk.is-empty sets display:none so empty
groups collapse out entirely. Hooked into hideItem, restoreItem,
and the initial-paint hidden-set application.
New section documenting the architecture change (news-digest-web
moved from nginx:alpine to a FastAPI app on uvicorn built from the
worker's same Dockerfile), the three new endpoints
(GET /api/hidden, POST /api/hide, POST /api/restore), the item-id
scheme (12-char sha1 of reddit:<post_id> or miniflux:<entry_id>
so hide-once = hide-forever-for-that-article), and the playbook
changes (dropped DOCKER_BUILDKIT=0 now that ana-docker is on
docker-ce 29, added round-trip API verify steps).
Adds a small × on each item that hides it from the page. State is
server-side at /output/hidden.json so the same hidden set follows
the user across devices (home, ipad, laptop, work). A "Hidden (N)"
tray at the bottom shows what's hidden on the current page with a
restore button per row; older hidden ids that aren't on this page
sit silently and continue to filter future editions that include
the same article.
Architecture change: news-digest-web swaps from nginx:alpine to a
FastAPI app on uvicorn, built from the same Dockerfile as the
worker. Same image, different command (`uvicorn web:app` overrides
the worker's cron entrypoint via compose). Drops one image dependency,
adds /api/{hidden,hide,restore}.
Item ids are stable 12-char sha1 prefixes (`reddit:<post_id>` /
`miniflux:<entry_id>`) computed in digest.py at render time and
emitted as `data-id` on each .item. The frontend reads /api/hidden
once on load, applies `is-hidden` to matching items, and POSTs
hide/restore on user interaction (optimistic, with rollback on
network error).
Storage: single JSON array at /output/hidden.json, atomic writes
via tempfile + rename, threading.Lock around the read-modify-write
inside the single uvicorn worker. No auth — the digest itself is
unauthenticated on LAN; same trust boundary applies.
Playbook also drops the DOCKER_BUILDKIT=0 fallback now that
ana-docker is on docker-ce 29, and adds three verify steps
(/api/hidden returns a JSON array, app.js is reachable, full
hide/restore round-trip with a synthetic id).
autorestic removal completed on both esh-docker-vm and esh-vm-db
after two playbook fixes (YAML plain-scalar folding ate a backslash
continuation; YAML tag indicator stripped a leading `!`). Both
documented inline.
seafile seahub race resolved by adding a healthcheck to mariadb
(bundled healthcheck.sh --connect --innodb_initialized) and
converting seafile's depends_on to long-form with
condition: service_healthy on db. Compose now waits for InnoDB
to initialize before starting seahub, so the daemon-restart race
that wedged the python frontend can't recur. Verified: seahub log
clean post-recreate, traefik 502 rate dropped to zero on
seafile@docker. Compose change lives on the server (the mirror is
gitignored by design).
YAML treats a leading `!` as a tag indicator, so the unquoted
`shell: ! command -v autorestic >/dev/null` was parsed as a tagged
scalar with the `!` stripped. The verify ended up running just
`command -v autorestic >/dev/null` — which exits non-zero when
autorestic is absent, the OPPOSITE of what the assertion needed.
Quoted version `"! command -v autorestic >/dev/null"` survives
parsing and gives the intended bash negation.
The previous version listed four unit paths separated by `\` + newline.
That looks fine in source but YAML plain-scalar folding collapses the
sequence to a literal `\ ` — the backslash + space no longer functions
as a shell line continuation, and only the first path actually gets
passed to rm. End result on esh-docker-vm's first run: backup.service
removed; backup.timer + prune.service + prune.timer survived; verify
correctly caught the partial state.
Switched to `rm -f /etc/systemd/system/autorestic-*.{service,timer}`
form — single string, no folding hazard, and idempotent on hosts where
some or all of the files are already gone. Re-running on esh-docker-vm
will mop up the leftovers cleanly.
Migration complete:
* ana-docker on docker-ce 29.4.1, all 29 containers back up. Traefik
routing live (verified 200s on matrix.phasefinal.com presence +
seafile.phasefinal.com syncs).
* traefik-postboot.service installed + enabled on both traefik hosts
(esh-docker-vm, ana-docker) — one-shot systemd unit that restarts
traefik 60s after every boot, fixing the long-standing routing-races-
after-reboot symptom.
New playbook: remove-autorestic. Triggered by a typo (`D:escription`
in autorestic-backup.timer line 2) flagged by systemd-analyze during
the traefik-postboot install on esh-docker-vm. Rather than fix it,
remove autorestic — it's redundant with the PBS + structured-restic
two-layer pipeline that's been operational since 2026-04-22. Detected
on two ESH-side hosts: esh-docker-vm and esh-vm-db. Playbook removes
the four unit files + the /usr/local/bin/autorestic binary; leaves
/srv/backups/autorestic/.autorestic.yml (archival) and
/mnt/backup/restic/repo/esh (historical snapshots) for separate
disposition.
Sub-finding from ana-docker upgrade: seafile's seahub (the Python
frontend at port 8000 inside the container) failed to start because
mysql wasn't ready when seafile booted, and a single restart didn't
recover it. Traefik routes return 502 on seafile dynamic endpoints
until seahub is up. Needs separate triage of seafile's depends_on
wiring or seahub's retry behavior — not a docker-ce regression.
Traefik often misses backends after a reboot or daemon swap because
(a) its docker provider debounces / drops events when 30+ containers
start in a burst, and (b) backends can be `Created` on the docker
socket but not yet attached to traefik-net when traefik scans. The
empirical workaround is `docker restart traefik` once the topology
settles — this unit bakes that in.
Type=oneshot, After=docker.service, ExecStartPre=/bin/sleep 60,
ExecStart=docker restart traefik. Runs once per boot. delay_seconds
and container name are tunable via --var.
Verify phase: file mode, enabled state, ExecStart references the
right container, container actually exists on the host, and
systemd-analyze parses the unit cleanly (lint without executing —
avoids needlessly bouncing traefik on healthy hosts).
In scope: esh-docker-vm, ana-docker (the two hosts that run traefik).
docker-ce 29 ships docker-compose-plugin renumbered to v5.x (was v2.x
with docker-ce 26-28). Same Compose v2 codebase under the hood —
Docker just realigned the major number. The verify regex was hardcoded
to `v2\.[0-9]+\.[0-9]+`, so a successful migration on esh-docker-vm
(29.4.1, 16/16 stacks back up clean) reported FAILED on the verify
phase. Switched to `docker compose version --short` parsed for major,
gated `>= 2` — works across future plugin renumbers too.
STATUS.md: mark esh-docker-vm done. ana-docker is the last host.
After nh3-docker's swap, two systemd unit gotchas surfaced that the
playbook now handles automatically:
* The docker.io-era /etc/systemd/system/docker.service.d/override.conf
hardcoded ExecStart=/usr/sbin/dockerd; docker-ce installs at
/usr/bin/dockerd → daemon failed status=203/EXEC.
* The shipped docker-ce unit's ExecStart=dockerd -H fd:// conflicts
with daemon.json hosts: (defined for the 0.0.0.0:2375 homepage
discovery binding) → "conflicting host options".
The "Rewrite docker.service drop-in" step now backs up any existing
override, probes daemon.json for a hosts: setting, and installs an
override that strips -H from ExecStart when needed. Also added an
explicit systemctl reset-failed step to clear the start-rate-limit
state that 3 failed install-time starts leave behind.
configs/homepage/docker.yaml: comment out irv-ml1-docker provider —
20s-per-poll ETIMEDOUTs from the stalled host were drowning homepage's
logs and apparently blocking ana-pfi-docker discovery (the Miniflux
card in the News group wouldn't render until removal). Re-enable when
irv-ml1 is back.
STATUS.md: new "Active migration" section tracking the docker-ce
rollout — nh3-docker done; esh-docker-vm + ana-docker queued.
nh3-docker's daemon kept failing post-package-swap with status=203
even after daemon-reload. Root cause: a stale
/etc/systemd/system/docker.service.d/override.conf from the docker.io
era hardcoding ExecStart=/usr/sbin/dockerd. The override (a) points
at the no-longer-existing path, AND (b) typically also adds
-H tcp://... which now duplicates the hosts: setting in
/etc/docker/daemon.json — dockerd refuses to start when both define
hosts ('conflicting host options').
Daemon.json is the modern way to expose the TCP socket. The
override is redundant and wrong. Move it aside (preserve a
.pre-upgrade copy for forensics), then daemon-reload, then start.
Should let esh-docker-vm and ana-docker upgrades go through cleanly
without the manual debug loop nh3-docker required.
Editorial-briefing favicon: 32×32 SVG, Australis palette. Cyan
masthead-rule across the top echoes the page's aurora-rule, four
descending text-line indicators below evoke a newspaper column.
Reads cleanly at 16×16 (the typical browser tab size). Static
markup only — no script, no animation — so all browsers honor
it for tab + bookmark icons.
Linked from both digest.html.j2 and archive.html.j2 with the
proper type="image/svg+xml" attribute. Served by nginx from
the bind-mounted /output dir alongside index.html and style.css.
Deploy playbook also updated to copy the favicon into /output at
deploy-time so a fresh deploy doesn't 404 on the icon before the
first cron fire.
Docker's official package installs dockerd at /usr/bin/dockerd; the
Debian docker.io package put it at /usr/sbin/dockerd. After the apt
swap, the new docker.service unit file is on disk with the right
path, but systemd's cached unit still has the OLD ExecStart pointing
at /usr/sbin/dockerd. Daemon start fails with:
status=203/EXEC "No such file or directory"
Fix is systemctl daemon-reload between install and start. nh3-docker
hit this; adding the step so esh-docker-vm and ana-docker don't.
Two requested polish items:
1. Reddit items now show TWO distinct action chips in the footer:
[↗ SOURCE huggingface.co] [⌥ REDDIT THREAD] 3h · u/foo
Previously the external link was a tiny dot-separated text link
that visually competed with the post metadata. Now: explicit
labeled buttons, distinct colors (cyan for source, blue for
thread), with hover states that match the Australis accent
palette. Non-Reddit items get just the SOURCE chip.
2. Archive page at /archive.html — lists every edition-*.html on
disk, newest-first, sorted PM-before-AM within a day. Each row
is the date in editorial serif + an AM/PM chip color-coded
(yellow morning / cyan evening) + a hover-affordance arrow.
Header link from the main edition reads "ARCHIVE →".
Generation: digest.py walks OUTPUT_DIR for the edition-*.html
filename pattern on every run, sorts, renders archive.html.j2,
writes atomically. Cheap (~1ms even with hundreds of editions).
No retention cap — twice-daily for a year is ~700 small files,
well under any reasonable concern.
CSS additions: .action chip styles (with per-action color variants),
.archive-link in masthead, full .archive-row + .archive-edition
treatment.
Multi-line shell with backslash-continued URL had the continuation
line starting at column 0, which breaks YAML's | literal block
('could not find expected :'). Stash the URL into a shell variable
and emit on one logical line.
Three docker hosts on the fleet still run docker.io 20.10.24 (the
Debian bookworm package) which:
* sticks at API 1.41 — newer compose clients (1.52+) refuse to talk
to it without DOCKER_BUILDKIT=0 fallback (caught during the
news-digest deploy on ana-docker today)
* is functionally EOL — docker.io's upstream no longer ships to it
* is missing modern buildx driver versions
This playbook handles a single-host migration: snapshot existing
docker package versions for rollback reference, stop every running
compose stack, apt-remove (NOT purge — preserves /var/lib/docker)
docker.io + plugins, add Docker's signed APT repo for Debian, install
docker-ce + docker-compose-plugin + containerd.io + buildx-plugin,
restart the daemon, bring stacks back up.
Volumes / images / containers survive the swap because:
* /var/lib/docker is preserved by `apt remove` (vs purge)
* both packages default to the overlay2 storage driver
Recommended host order (least → most blast radius):
1. nh3-docker (NH3 site, fewer services)
2. esh-docker-vm (home lab; many services but single-consumer)
3. ana-docker (production-ish; vaultwarden, gitea, synapse,
task-board, miniflux, news-digest, paperless-ng)
Run as `scripts/elway <host> --playbook playbooks/upgrade-docker-ce.yaml`
per host. Verify between hosts via `docker version` + spot-check a
few containers.
Rollback if a daemon won't start or a container errors:
ssh <host> 'sudo apt install --allow-downgrades \$(cat /tmp/docker-pre-upgrade.txt | tr "\n" " ")'
Three iterations to get end-to-end:
1. Dockerfile missed COPY run-digest.sh — cron's exec target wasn't
in the image, every fire failed. Added COPY + chmod.
2. Jinja template used {{ list|sum(attribute='items') }} which
sum()s lists with start=0 → TypeError int+list. Switched to
computing reddit_total / tech_total in Python and passing as
template args.
3. LLM defaulted to qwen3.5-35-a3b which (a) is broken in
llama-swap (model process exits on launch), (b) when working,
defaults to extended-thinking mode that eats the entire token
budget without producing any visible content. Same pattern with
qwen3.6-35-a3b. Switched default to granite-4-small — small (4B),
fast (~1s/call), no thinking-mode pathology, returns clean JSON.
Whole pipeline now runs in ~35s total across 8 sources.
Also hardened the LLM response parser to fall back to
reasoning_content when content is empty — catches the thinking-mode
case if anyone ever points the digest at one of those models. Plus
the deploy playbook gained DOCKER_BUILDKIT=0 because ana-docker is
on docker 20.10 which doesn't carry the buildx driver versions our
newer client expects ("client version 1.52 is too new"). Real fix is
upgrading docker on the fleet — separate workstream.
The Miniflux inbox got noisy after a few subreddits + HN + Lobste.rs.
This stack distills a single static page twice a day — at 0800 and
2000 local — that surfaces only what cleared score + ratio filters,
each item tldr'd by qwen3.5-35-a3b on llama-swap.
Pipeline (digest.py, ~330 lines):
1. Discover subreddits from Miniflux feeds (any reddit.com/r/<sub>/
URL — single source of truth, no duplicated config).
2. Reddit JSON top-of-day per sub. Filter: score >= 50,
upvote_ratio >= 0.85. Cap 8 items per sub.
3. Miniflux /v1/entries for the 'Tech aggregators' category
(HN, Lobste.rs) — last 12 hours.
4. Batched per-source summarization via llama-swap
/v1/chat/completions. Each post gets a one-sentence tldr +
one-word tag (news / tutorial / release / discussion /
question / showcase / drama / meme).
5. Render Jinja2 template. Atomic write to /output/index.html
(.tmp + rename) so partial pages never get served. Per-edition
archive at /output/edition-YYYY-MM-DD-{am,pm}.html.
Two containers:
news-digest-worker python:3.12-alpine + busybox crond
news-digest-web nginx:alpine, port 8181, homepage card via
docker labels (group=News, fits next to Miniflux)
Both bind-mount /opt/docker/data/news-digest as /output and
/usr/share/nginx/html respectively.
Aesthetic — operations-center chrome (Australis cool-mono palette,
JetBrains Mono UPPERCASE eyebrows, mdi-glyph anchor) wrapping
editorial-serif news content (Fraunces variable serif w/ optical
sizes). Two type families that wouldn't normally meet, intentionally
combined: chrome says 'filed at 0800 from the bridge'; headlines say
'this is news, read it like news.' Sticky aurora-glow rule under the
masthead is the only sanctioned Australis gradient.
Edition stamp (AM/PM in big mono Australis-yellow) is the signature
piece — establishes the twice-daily rhythm at a glance.
All filtering + LLM + scheduling knobs in .env. Subreddit list is
implicit (read from Miniflux), so adding a sub = subscribing in
Miniflux, no config edit on this stack.
Subscribed live via Miniflux API; mirroring back to the canonical
OPML so a fresh deploy elsewhere starts with the same feed set.
All three land in the existing 'Subreddits — selfhosting' category
(retitled to mention image-gen + LLM + homelab to reflect scope).
Initial deploy failed with 'Container cannot be connected to network
endpoints: miniflux-net, traefik-net' — the docker engine balks at
joining a brand-new internal network and an existing external
network in one create step.
Flattened both containers onto traefik-net only. The DB password
still protects miniflux-db, and traefik-net is internal-LAN-only,
so co-locating them is fine. Verify step updated to check for
traefik-net membership instead of the (now-gone) miniflux-net.
Adds Miniflux on ana-docker as the unified inbox for tech blogs,
Hacker News, lobste.rs, and selected subreddits. Reddit serves clean
RSS for any sub at https://reddit.com/r/<sub>/.rss, so subreddit
follows fold into the same inbox as everything else — no Reddit
account needed, no manual polling.
Stack:
stacks/miniflux/
compose.yaml — miniflux + bundled postgres:16
.env.example — placeholders for DB password + admin user
starter-feeds.opml — initial subscriptions (HN, Lobste.rs,
r/selfhosted, r/homelab, r/LocalLLaMA, r/nba)
README.md — deploy / OPML import / r/nba spoiler
block-list / backup / update flow
Postgres bundled with the stack (not pfi-postgres) — single-user RSS
DB is tiny and the bundle keeps the dependency graph flat.
Homepage gets a new 'News' group at the TOP of the Main tab (above
Monitoring) so the Miniflux card sits prominently. The card itself
auto-discovers via the homepage.* labels on the miniflux container.
Per-feed block-list rule for r/nba documented in README — Reddit's
RSS titles for game threads include scores ("Lakers 108 - Warriors
102 [Final]") which spoil the game; a regex catches the score
patterns and skips those entries while keeping discussion/highlights.
Deploy:
scripts/elway ana-docker --playbook playbooks/deploy-miniflux.yaml
Then edit /opt/docker/compose/miniflux/.env on the host to fill in
the two CHANGE_ME passwords and `docker compose up -d` again.
Driver bump 570.124.06 → 595.58.03 attempted via
scripts/upgrade-irv-ml1-cuda.sh on 2026-04-25; host did not come up
cleanly after the post-install reboot.
Captures, in one place, everything currently in-flight against irv-ml1
so it can be revisited as a unit once the machine is healthy:
* Recovery options ordered by invasiveness (console diagnose →
rollback script → manual driver pin → try a different driver
series).
* Three deferred TTS deploys (Kokoro GPU flip, VibeVoice 1.5B,
Chatterbox Turbo) with the exact playbook commands to re-run.
* Smoke-test reminder for the 5 existing GPU stacks (comfyui,
cosyvoice, qwen3-tts, index-tts, parakeet) which were healthy
before the upgrade but unverified after.
* Explicit 'don't touch irv-ml1' guard rail until resolved.
Sits in 🟥 Blocked — surfaces above the existing 'Quick wins'
section so future sessions see it first.
Phase 1 silently aborted right after the 'cuda-drivers candidate' step
— before snapshotting, stopping containers, installing the new driver,
or rebooting. Symptom: phase2 showed driver still at 570.124.06 and
containers with multi-hour uptimes (proof they were never stopped).
Cause: `apt-cache policy cuda-drivers | head -10`. head reads its 10
lines and closes the pipe early; that delivers SIGPIPE to apt-cache,
which under `set -o pipefail` propagates as a non-zero exit, and
`set -e` immediately terminates the remote shell. The same hazard
existed in phase2's `docker ps ... | head -15`.
Fix is one-line per call site: drop the head. apt-cache policy output
for cuda-drivers is short (~10 lines), and docker ps's container list
on irv-ml1 is small enough to print fully.
Phase 1 was unconditionally adding /etc/apt/sources.list.d/nvidia-cuda.list
pointing at /etc/apt/keyrings/nvidia-cuda.gpg. But irv-ml1 already had
the upstream-style /etc/apt/sources.list.d/cuda-debian12-x86_64.list
with /usr/share/keyrings/cuda-archive-keyring.gpg — that's how the
existing 570 driver got installed in the first place. APT then refused
both: 'Conflicting values set for option Signed-By regarding source ...
debian12/x86_64/'.
Now the script:
* removes any orphan nvidia-cuda.{list,gpg} files from a previous
failed run
* greps existing /etc/apt/sources.list.d/*.list for a debian12 cuda
repo entry; if found, trusts and uses it
* only writes a new repo entry if no existing one is found, and
when it does, uses the same upstream paths (cuda-archive-keyring.gpg,
cuda-debian12-x86_64.list) so reruns are stable
Re-run safe — phase1 can be invoked any number of times without
accumulating conflicting configs.
The previous version piped a heredoc to `ssh -t host bash -s` —
which can't work because `-t` requires a real TTY and heredoc puts
stdin in pipe mode. They're mutually exclusive (orientation.md
gotcha, tripped over once already). The remote sudo therefore aborted
with 'a terminal is required to read the password' before doing
anything.
Restructured: the remote half of each phase is built as a tempfile
locally, scp'd to /tmp on irv-ml1, made executable, and invoked via
`ssh -t host /tmp/upgrade-cuda-remote.sh phase1`. Now stdin is the
real TTY (no pipe), sudo prompts work, and the rest of the script
runs as before.
Same three phases, same rollback behavior, same idempotency story.
Just a transport fix for the local→remote half.
Two-phase script for bumping irv-ml1's NVIDIA driver to support
CUDA 12.9 — currently blocked at 12.8 per driver 570.124.06, which
is why the Kokoro GPU image (built against CUDA 12.9) wouldn't
start.
Phase 1 (interactive — sudo prompted once via ssh -t):
* snapshot current nvidia packages to /tmp/nvidia-pre-upgrade.txt
(rollback reference)
* add NVIDIA's CUDA APT repo for Debian 12 with signed key
* stop GPU containers (comfyui, cosyvoice, qwen3-tts, index-tts,
parakeet, kokoro)
* apt install cuda-drivers (currently pulls 580 series)
* reboot
Phase 2 (re-run after host comes back):
* poll for SSH availability
* verify new driver + CUDA version
* restart GPU containers via docker compose up -d
* print next steps (flip Kokoro .env to gpu variant + redeploy)
Includes a rollback subcommand that reinstalls the snapshotted
packages with --allow-downgrades and reboots — for the case where
the new driver regresses on this kernel.
Driver source switched from Debian's nvidia-driver metapackage to
NVIDIA's official cuda-drivers metapackage; bookworm doesn't carry
575+ in its main or backports repos as of 2026-04-25.
Two fixes from the failed first deploy on irv-ml1:
1. CPU/GPU variant. Kokoro's GPU image needs CUDA >= 12.9; irv-ml1's
driver 570.124.06 caps at 12.8 so the gpu variant fails with
"nvidia-container-cli: requirement error: unsatisfied condition:
cuda>=12.9". Make the variant a knob:
KOKORO_VARIANT=cpu (default — works anywhere)
KOKORO_VARIANT=gpu (after driver bump)
KOKORO_USE_GPU=false|true (matches the variant)
Kokoro is tiny (82M params) so CPU is workable: TTFA ~1s vs ~300ms
on GPU. Acceptable while the driver bump gets scheduled. compose.yaml
no longer hard-codes `runtime: nvidia` — relies on the daemon's
default-runtime + NVIDIA_VISIBLE_DEVICES gating, same as how the
wrapper's USE_GPU flag selects the inference path inside the
container. Toggling between variants is now a `.env` edit + restart.
2. Tighter pull-log filter. --quiet on `docker compose pull` only
suppresses the pull command's stdout; the docker daemon still
emits per-layer extraction events on stderr ("ffbfd7a09415
Extracting 64.06MB" repeated dozens of times per layer). Drop those
too via grep on the SHA-prefixed pattern. set -o pipefail keeps a
real pull failure visible.
For existing deployments: removing /opt/docker/compose/kokoro/.env
on the host and rerunning the playbook re-seeds with the new schema.
Profiling the index-tts deploy log (2057 lines) showed ~25% was just
pip's per-package Downloading / Collecting / Requirement-already /
progress-bar spam — useless for ops, hard to scan when something
actually breaks.
Three changes across the four TTS deploy playbooks:
1. Pulls (Kokoro): add --quiet. 6.5 GB pull no longer floods the log
with per-layer progress redraws. Final "X Pulled" still prints.
2. Builds (VibeVoice, Chatterbox, IndexTTS-2): add --progress=plain
to stop the BuildKit TUI from littering the captured log with
carriage-return overdraws, then pipe through a grep filter that
drops pip's noisy lines but keeps:
- buildkit step transitions (#NN [stage])
- DONE / CACHED / ERROR markers
- apt + build-stage messages
set -o pipefail keeps a real build failure from being swallowed
by the grep's exit code.
Net effect: ~25% smaller logs, much more scannable; full visibility
into step progress and errors preserved.
"docker compose pull (first run: ~6.5 GB from GHCR)" had an unquoted
colon-space inside a plain scalar value, which YAML parses as a
nested mapping — elway aborted on load. Single-line fix: wrap the
value in double quotes.
Three TTS additions to round out coverage on irv-ml1, each filling a
distinct niche the existing slate doesn't own.
Final coverage matrix (all on irv-ml1):
Kokoro — low-latency English, fixed voice library, ~300ms TTFA
Chatterbox Turbo — low-latency English w/ voice cloning + paralinguistic tags
IndexTTS-2 — English voice cloning + emotion vector / text control
Qwen3-TTS-1.7B-Base — high-quality English voice cloning
CosyVoice 3 — multilingual (Chinese-leaning)
VibeVoice 1.5B — long-form / multi-speaker dialogue
stacks/kokoro:
- port 8193, GPU device 0 (3090)
- pulls ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.4-master (no Dockerfile,
no first-run model download — models baked in)
- 60+ built-in voices, OpenAI-compat with stream=true over chunked HTTP
- Apache-2.0 weights + code, ~1 GB VRAM
stacks/vibevoice:
- port 8194, GPU device 1 (A6000 — for 7B headroom)
- builds groxaxo/VibeVoice-FastAPI1 (more current fork of ncoder-ai)
pinned to 7614c469a145
- default model microsoft/VibeVoice-1.5B (~7 GB bf16 VRAM); env var
swap to rsxdalv/VibeVoice-Large (7B) or FabioSarracino/VibeVoice-Large-Q8
- multi-speaker dialogue via /v1/vibevoice/generate with Speaker N: format
- long-form niche only — not low-latency
stacks/chatterbox:
- port 8196, GPU device 0 (3090)
- builds devnen/Chatterbox-TTS-Server (most active Turbo-supporting wrapper)
- default model ResembleAI/chatterbox-turbo (~2.5 GB fp16, ~75ms latency)
- paralinguistic tags inline ([laugh] [whisper] etc) — different shape
from IndexTTS-2's emotion vector; fills the speed+cloning niche
Kokoro/IndexTTS don't cover together
- mandatory PerTh watermark on outputs (Resemble policy)
Three matching playbooks under playbooks/deploy-{kokoro,vibevoice,
chatterbox}.yaml. All idempotent, creates-/when-gated.
Cold-deploy disk on /worktank/: ~7 GB Kokoro + ~19 GB VibeVoice 1.5B
+ ~12 GB Chatterbox = ~38 GB total. VRAM concurrent: ~10-11 GB across
both GPUs.
Skipped from the original four-stack proposal: VibeVoice Realtime
(overlaps Kokoro's niche; Kokoro wins on latency, license, and not
needing a build).
IndexTTS-2's tts.infer(stream_return=True) is a generator that yields
audio chunks per text segment as they finish, plus inter-segment
silence. Expose this via the existing POST /v1/audio/speech with a new
"stream": true field on the request body.
Wire-up:
- 44-byte WAV header emitted up front with placeholder data length
(0xFFFFFFFF) so chunks can be written before total samples are
known. Players that read until EOF (mpv, ffplay, aplay, sox,
browsers via <audio>) handle this fine.
- Each yielded chunk goes through _chunk_to_pcm_bytes(), which
handles torch tensors / numpy arrays in either int16 or float
(-1..1) form.
- 22050 Hz mono int16 — IndexTTS-2's hardcoded output shape.
Time-to-first-audio drops from full-file latency to ~one-segment
latency. Single-sentence inputs barely benefit; long passages /
multi-paragraph reads benefit a lot. Strict metadata parsers may
balk at the placeholder size — request without stream for a
closed-length WAV in that case.
INDEX_TTS_TAG bumped to v2 to force a rebuild.
budget hazard + media-CDN workaround
Tried adding git-lfs install + git lfs pull to the build to get
real example WAVs into the image — failed with:
Error downloading object: examples/emo_hate.wav: Smudge error:
batch response: This repository exceeded its LFS budget. The
account responsible for the budget should increase it to
restore access.
The index-tts org's LFS bandwidth quota is exhausted upstream and
out of our control. Reverting the Dockerfile change. The examples
aren't needed for the wrapper to work; emotion_text and
emotion_vector are sufficient for end-to-end testing without any
WAV file at all.
For users who want the bundled example clips as starter audio,
README now documents the media-CDN URL trick — same LFS objects
served via a different code path that doesn't count against the
LFS API budget. INDEX_TTS_TAG stays at v1.
The IndexTTS-2 repo stores examples/emo_*.wav and examples/voice_*.wav
as Git LFS objects. v1 of our image cloned the repo without an LFS
pull, leaving those paths as ~130-byte pointer text files — unusable
for `docker cp` into /worktank/index-tts/{voices,emotions}/ as starter
references. (Caught when an emotion_voice="hate" call returned audio
that was actually the pointer text round-tripped through file IO.)
v2 adds git-lfs to the apt list, calls `git lfs install --system`
once, and `git lfs pull` after the checkout. Adds ~1-2 MB to the
image (the examples are small audio clips). INDEX_TTS_TAG bumped to
v2 to force a clean rebuild.
Adds a third TTS to the irv-ml1 fleet. IndexTTS-2 is Bilibili's
emotion-controllable zero-shot TTS (paper 2506.21619). Distinguishing
capability vs the existing two: timbre and emotion are disentangled —
clone a voice's timbre from one reference and the emotion from a
different reference, OR set emotion via 8-vector, OR derive it from a
text description. Neither CosyVoice 3 nor Qwen3-TTS-1.7B-Base does
this cleanly in English.
Wrapper is owned end-to-end (~150 lines in app.py) — the only existing
FastAPI fork (csllpr/index-tts-fastapi) targets v1 and is a dormant
single-commit repo. Upstream IndexTTS-2 ships only a Gradio webui.
Layout follows the qwen3-tts pattern:
stacks/index-tts/
Dockerfile — CUDA 12.8 base, IndexTTS pinned to a SHA
app.py — FastAPI: POST /v1/audio/speech + /v1/voices
entrypoint.sh — one-time HF snapshot_download of the weights
compose.yaml — env-driven, GPU pinning support, bind mounts
.env.example — port 8192, fp16, paths
README.md — API examples + comparison vs the other TTS
playbooks/deploy-index-tts.yaml — elway playbook for irv-ml1
Voice and emotion libraries are flat host dirs of WAVs, bind-mounted.
Drop a new <name>.wav and /v1/voices picks it up immediately.
License caveat: IndexTTS-2 weights ship under a custom Bilibili
license (free at our scale, not OSI-open). README documents it.
Toggles --append-only off on the rest-server via a temporary
docker-compose.override.yaml (canonical compose untouched), runs
resticprofile forget --prune --verbose on each client of that
rest-server, then restores --append-only. The restore is wrapped in
a trap so a partial-failure prune still leaves the rest-server in
its safe configuration.
ANA side is fully automated against ana-docker (5 clients:
ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas, esh-vm-db).
NH3 side currently prints a manual DSM ceremony — Synology Container
Manager doesn't expose docker on the expected paths and syncuser
sudo isn't NOPASSWD, so the toggle isn't safely scriptable from
this workstation. The instructions cover the same flow in DSM web
UI + interactive ssh on each NH3 client (nh3-docker, nh3-dev,
irv-ml1).
Usage:
scripts/restic-prune.sh ana # ANA only (auto)
scripts/restic-prune.sh nh3 # NH3 instructions
scripts/restic-prune.sh all # both
scripts/restic-prune.sh -h # help
scripts/restic-prune.sh --dry-run ana # show every command
- STATUS.md: marks items 11 (mattermost dir cleanup verified gone),
13 (UniFi UDM cards added to homepage), 15 (docs first + second
pass — README + chromadb-setup deletion + VM-102 frontmatter
strip), and 16 (drift discipline rule, no auto-regen) as done.
New session-milestones section captures the four task-board
iterations (v0.1.1 session disambiguation + dormant-timer fix,
v0.1.2 favicon + 1s tick rate, v0.1.3 case-insensitive session
names with real ALTER migration), the parakeet healthcheck fix,
the AIPA-MCP session relabel + DB rename to Architect, the
homepage Toolchain dedup, and the qwen3-tts variant flip.
- .gitignore: ignore .claude/scheduled_tasks.lock and any other
.claude/*.lock — runtime per-machine state, not interesting to
cross-checkout.
The task-board compose carries homepage.group=Toolchain. With no
matching entry in settings.yaml's layout: map, homepage placed it
on the default tab (Main) AND it appeared under the Toolchain tab,
producing duplicate cards. Declare the Toolchain group explicitly
with tab: Toolchain so it renders in exactly one place.
Second pass on docs/ cleanup (item #15 in STATUS.md):
- pfi/chromadb-setup.md: deleted. References configs/pfi-ana/... and
scripts/setup-chromadb.sh, neither of which exist in this repo
(artifacts of an earlier project layout). ChromaDB is already live
per docker-stack.md; the operational truth lives there.
- pfi/docker-stack.md: removed the cross-link to the deleted file and
pulled the bit of useful content from it (auth-token generation +
client Settings example) into the inline ChromaDB section.
- pfi/vm-102-matrix-{synapse,appservice}.md: stripped the YAML
frontmatter. The `path:` values pointed at docs/pfi-ana/... which
doesn't exist in this repo; no toolchain consumed the metadata.
- README.md: tree updated to reflect the deletion.
VM-102 docs kept separate by design — each is right-sized; merging
would push past the ~500-line guideline.
- parakeet/compose.yaml: healthcheck was using curl which isn't in the
image (only wget is, via apt). 2,190 failing checks — switched to
`wget -q -O /dev/null`, container went healthy on recreate.
- qwen3-tts/.env.example: variant annotation was reversed. The upstream
wrapper's runtime error is unambiguous: voice cloning requires the
-Base variant, not -CustomVoice. Corrected the comment block and
flipped the default to Qwen/Qwen3-TTS-12Hz-1.7B-Base.
- qwen3-tts/README.md: 0.6B switch snippet now suffixes -Base too,
since plain `Qwen/Qwen3-TTS-12Hz-0.6B` isn't published on HF.
Bundles the inventory expansion since 2026-04-22:
- New host dirs (READMEs + ssh-target where dir name doesn't resolve):
ana-nas, ana-wg, esh-vm-db, nh3-nas, pbs-ana, pbs-nh3.
- New PFI VM snapshots (registered + key-installed 2026-04-23):
ana-filebot, pfi-ana-webhost, pfi-postgres, pfi-pteradactyl,
pfi-tacticalrmm, sf-ana-container, sfsrv-ana (system + proxmox).
- servers/irv-ml1: ONBOARDING.md (the first-time setup notes from when
the host was brought into the fleet) + ssh-target (10.100.79.3 over
the WG tunnel — name doesn't DNS-resolve from this workstation).
- servers/{ana-ml2,pfi-pve,sf-r630}/README.md: updates to capture BMC
IPs, the iDRAC vs OS hostname distinction (sf-r630 hardware =
sfsrv-ana OS), and the ana-ml2 Supermicro BMC (10.250.250.50,
distinct from the Dell R750xs iDRAC).
- configs/homepage/docker.yaml: irv-ml1-docker provider added so
homepage auto-discovers irv-ml1's stacks over the WG tunnel.
- docs/orientation.md: narrative fleet overview written for fresh
Claude sessions — sites, backup architecture, governing principles,
gotchas, where-to-look guide. Pointed at from CLAUDE.md.
Bundles the post-2026-04-21 work that built out the two-layer backup
architecture (PBS for VM images + restic for file/DB), plus the cross-
site mirror and the disaster-recovery runbook.
- configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete
*_offen_backup_data exclude (offen sidecars retired fleet-wide
2026-04-23; restic now covers the equivalent scope directly).
- configs/restic/esh-vm-db/: new profile for the dedicated DB VM
(10.0.50.60), with pre-backup pg_dumpall + mongodump hooks.
- configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven)
and nh3-nas → ana-nas (05:00 daily, runs as root because DSM
rest-server-nh3 writes mode-400 files only root can read).
- docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook,
refined during the 2026-04-22 deployment with per-hypervisor
namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the
Synology syno_acl flatten step.
- docs/runbooks/disaster-recovery.md: blast-radius runbook ordered
Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty);
references incident memory + recovery-step playbooks per consumer.
- services.yaml: PFI-UDMSE (10.100.0.1, NH3 edge) under Infra - NH3,
ESH-UDMPM (10.0.0.1) under Infra - ESH. PFI-UDMSE replaces the
retired Fortigate 101F at NH3 — comment updated. Both use si-ubiquiti.
Diff also folds in the previously-deployed-but-uncommitted PBS-ANA /
PBS-NH3 / IRV section / retired NH3-SW1 cleanup, bringing the tracked
copy in sync with what's live on esh-docker-vm.
- docs/README.md: navigation map of the docs/ tree (orientation /
runbooks / pfi) with what-goes-where conventions.
- docs/runbooks/tea-0.14.0-linux-amd64.sha256: deleted (build artifact,
not a runbook).
- STATUS.md: items 11 / 13 / 15 marked done; 15 leaves Matrix and
chromadb consolidation candidates as future focused-session work.
Three fixes from the first deploy attempt on irv-ml1:
- build.target=production. Upstream Dockerfile is multistage; the last
stage `cpu-base` was selected by default, producing a CPU-only image
with no flash-attn and `torch ... whl/cpu`.
- user: "0:0". Upstream image declares USER appuser but writes runtime
state under /root (mode 0700). appuser cannot traverse /root, so
/v1/voices 500s on PermissionError. Run as root to sidestep.
- QWEN3_TTS_MODEL=Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice. The bare
`1.7B` id we had isn't a real HF identifier; upstream publishes
-CustomVoice / -Base variants of each size. Use -CustomVoice so
`voice="clone:<name>"` works.
Tag bumped to v2 to keep the v1 cpu image distinguishable in the local
registry.
After: all 5 verify steps pass, GPU synthesis ~5s for 3-4s of audio,
three contrasting English `instructions` produce three distinct
hashes — emotion steering actually works (unlike CosyVoice's English
path).
Avoids the "paste the full output" friction. Every elway run now
writes its full streamed output to two files in addition to the
terminal:
/tmp/elway-last.log
always overwritten — the easy "what just happened" target
~/.cache/elway/runs/<ts>-<host>-<playbook>.log
timestamped permanent record; accumulates across runs
Implementation: small _Tee class wraps sys.stdout for the duration
of main() so all `print(...)` calls fan out to the real terminal +
both file handles. Subprocess output already goes through print()
via _stream_process, so the build/healthz/etc. text is captured.
ANSI color codes are kept in the file so colors are preserved in
log viewers that handle them (less -R, modern tail). Strip with
`sed 's/\x1b\[[0-9;]*m//g'` for paste-elsewhere.
New flags:
--log <path> override path; replaces both default destinations
--no-log terminal-only, skip both files
Path of the permanent log is printed at the top of every run so
you know where it landed without remembering the timestamp pattern.
Alibaba's open-weight TTS (Apache 2.0, Jan 2026), deployed via
groxaxo/Qwen3-TTS-Openai-Fastapi wrapper. Built locally from a
pinned git SHA via docker buildx's git context — no source
vendored. 1.7B flagship model by default; 0.6B available via
QWEN3_TTS_MODEL env override.
Why we need a second TTS stack: cosyvoice 3 emits Chinese phonemes
for English content per upstream FunAudioLLM/CosyVoice#1790
(unfixed). Qwen3-TTS is from the same Alibaba team but with
English first-class in the checkpoint — 10 languages, 97 ms
streaming TTFB, instruction-driven emotion. Coexists with cosyvoice
on irv-ml1 (port 8191; cosyvoice keeps 8190).
Voice cloning shape DIFFERS from cosyvoice: profile-based, not
voice-id. Profiles live under voice_library/profiles/<name>/ and
are referenced as voice="clone:<name>".
Path layout: /worktank/qwen3-tts/{cache,voices}/, with cache excluded
from restic (regenerable from HF Hub) and voices included (cloned
profiles need original reference audio to recreate).
playbooks/deploy-qwen3-tts.yaml: 10 steps + 5 verify, idempotent;
the wait step polls /health for up to ~10 min to absorb first-run
model download.
Stack only — restic profile update for /worktank/qwen3-tts/voices/
to follow when this is empirically validated against the GLaDOS
voice (the "did Qwen inherit the Chinese-bias bug?" question).
Verified empirically (gitea #7): the neosun wrapper's \`instruct\`
field routes through CosyVoice's inference_instruct2 which expects
a Chinese template. English directives produce Chinese-phonemed
speech of the directive itself — the model is stuck in Chinese
context regardless of the directive language.
XML inline tags (<angry>, <sad>, etc.) bypass that path entirely
and work cleanly for English emotion control.
README updated:
- Added Gotcha section entry explaining the instruct trap with
the specific failure mode and the byte-identical-response
evidence
- Smoke-test recipes (steps 4 + 5) reworked to use <angry>...</angry>
in the input instead of an English instruct value
- "Emotion / style control" table now marks instruct as Chinese-
only with ⚠️ and XML tags as ✅
STATUS.md:
- Mark 4b done (both Postgres migration + NFS decoupling)
- Add arch decisions for gitea remote + prefer-elway policy
- Add tooling entries for elway + tea CLI
- Document 2026-04-24 session milestones (irv-ml1 AI stacks,
elway, task-board, 4b finish)
- Expand memory-pointer list with the files added this session
CLAUDE.md:
- Tell new sessions to use elway for SSH-driven work, point at
the smoke playbook template
- Document the task-board plugin + MCP-tool contract so assistant
sessions with the plugin enabled know the assistant should call
task_start / task_update / task_wait / task_complete at
meaningful checkpoints
.claude/settings.json:
- Project-level env: TASK_BOARD_SESSION=Infra so every Claude Code
session opened here labels its task-board cards "Infra"
playbooks/decouple-pfi-postgres-from-ana-nas.yaml:
- Finishes the DB-off-NFS migration on pfi-postgres. Already ran
against prod today; fstab clean, unmounted, no systemd mnt-db
unit. Verify 3 was mis-expressed on first run (`grep -q active`
matched "inactive") — fixed to invert systemctl exit code
directly.
Previously the startup logic ran `probe_sudo()` if any step in the
playbook declared `sudo: true`, regardless of whether that step's
creates:/when: gates would actually let it fire. The result on the
task-board deploy rerun was a spurious password prompt followed by
six SKIPPED lines — the prompt served no purpose.
New flow:
- Remove the upfront probe in main().
- SSHContext.sudo_password defaults to None; new sudo_probed flag
tracks whether we've already prompted this session.
- run_shell_step + run_upload_step call ensure_sudo(ctx) only at
the point a sudo step is actually executing — i.e. after its
skip conditions have been evaluated and passed. Idempotent:
probes at most once per playbook run.
Tradeoff accepted: if the user fat-fingers the password, they see it
mid-run on the first sudo step rather than upfront. `stop_on_fail`
(default true) halts cleanly; they rerun. Lower friction for the
common idempotent-rerun case, same recoverability.
Verified against playbooks/deploy-task-board.yaml — prior run
prompted + completed in 1.7s; new run completes in 1.7s with no
prompt because every sudo step skip-gated.
stacks/task-board/compose.yaml + .env.example describe the runtime —
image tag is task-board:local (built on the host), 7878 on host maps
to 7878 in container, SQLite lives at /opt/docker/conf/task-board/data/
(bind mount, uid 1000 friendly), homepage auto-card labels under
Toolchain group, on traefik-net like the rest of the fleet.
playbooks/deploy-task-board.yaml is the first real elway playbook —
exercises everything we built tier-1 + tier-2 idempotency for:
- `creates:` on mkdir + first-time clone + compose dir + data dir
- `when:` to chown /opt/docker/build only if it came up root-owned
- `when:` to seed .env only if one doesn't already exist (never
clobbers user edits on rerun)
- `changed_when:` on the `git reset --hard` step so repeat runs
against the same ref report `ok` instead of `changed`
- `changed_when: "false"` on every verify step (they attest, not
change)
- `upload:` with mode for compose.yaml + .env
Post-up the playbook polls /api/health for 30s before handing off to
the verify phase, so verification doesn't race the healthcheck's
start_period. Verify covers: /api/health 200, /api/tasks shape, /mcp
reachable, container on traefik-net.
Prereqs documented in the playbook header: Docker + compose plugin,
traefik-net network, git SSH access to gitea from the target host.
vh