Editorial-briefing favicon: 32×32 SVG, Australis palette. Cyan
masthead-rule across the top echoes the page's aurora-rule, four
descending text-line indicators below evoke a newspaper column.
Reads cleanly at 16×16 (the typical browser tab size). Static
markup only — no script, no animation — so all browsers honor
it for tab + bookmark icons.
Linked from both digest.html.j2 and archive.html.j2 with the
proper type="image/svg+xml" attribute. Served by nginx from
the bind-mounted /output dir alongside index.html and style.css.
Deploy playbook also updated to copy the favicon into /output at
deploy-time so a fresh deploy doesn't 404 on the icon before the
first cron fire.
Docker's official package installs dockerd at /usr/bin/dockerd; the
Debian docker.io package put it at /usr/sbin/dockerd. After the apt
swap, the new docker.service unit file is on disk with the right
path, but systemd's cached unit still has the OLD ExecStart pointing
at /usr/sbin/dockerd. Daemon start fails with:
status=203/EXEC "No such file or directory"
Fix is systemctl daemon-reload between install and start. nh3-docker
hit this; adding the step so esh-docker-vm and ana-docker don't.
Multi-line shell with backslash-continued URL had the continuation
line starting at column 0, which breaks YAML's | literal block
('could not find expected :'). Stash the URL into a shell variable
and emit on one logical line.
Three docker hosts on the fleet still run docker.io 20.10.24 (the
Debian bookworm package) which:
* sticks at API 1.41 — newer compose clients (1.52+) refuse to talk
to it without DOCKER_BUILDKIT=0 fallback (caught during the
news-digest deploy on ana-docker today)
* is functionally EOL — docker.io's upstream no longer ships to it
* is missing modern buildx driver versions
This playbook handles a single-host migration: snapshot existing
docker package versions for rollback reference, stop every running
compose stack, apt-remove (NOT purge — preserves /var/lib/docker)
docker.io + plugins, add Docker's signed APT repo for Debian, install
docker-ce + docker-compose-plugin + containerd.io + buildx-plugin,
restart the daemon, bring stacks back up.
Volumes / images / containers survive the swap because:
* /var/lib/docker is preserved by `apt remove` (vs purge)
* both packages default to the overlay2 storage driver
Recommended host order (least → most blast radius):
1. nh3-docker (NH3 site, fewer services)
2. esh-docker-vm (home lab; many services but single-consumer)
3. ana-docker (production-ish; vaultwarden, gitea, synapse,
task-board, miniflux, news-digest, paperless-ng)
Run as `scripts/elway <host> --playbook playbooks/upgrade-docker-ce.yaml`
per host. Verify between hosts via `docker version` + spot-check a
few containers.
Rollback if a daemon won't start or a container errors:
ssh <host> 'sudo apt install --allow-downgrades \$(cat /tmp/docker-pre-upgrade.txt | tr "\n" " ")'
Three iterations to get end-to-end:
1. Dockerfile missed COPY run-digest.sh — cron's exec target wasn't
in the image, every fire failed. Added COPY + chmod.
2. Jinja template used {{ list|sum(attribute='items') }} which
sum()s lists with start=0 → TypeError int+list. Switched to
computing reddit_total / tech_total in Python and passing as
template args.
3. LLM defaulted to qwen3.5-35-a3b which (a) is broken in
llama-swap (model process exits on launch), (b) when working,
defaults to extended-thinking mode that eats the entire token
budget without producing any visible content. Same pattern with
qwen3.6-35-a3b. Switched default to granite-4-small — small (4B),
fast (~1s/call), no thinking-mode pathology, returns clean JSON.
Whole pipeline now runs in ~35s total across 8 sources.
Also hardened the LLM response parser to fall back to
reasoning_content when content is empty — catches the thinking-mode
case if anyone ever points the digest at one of those models. Plus
the deploy playbook gained DOCKER_BUILDKIT=0 because ana-docker is
on docker 20.10 which doesn't carry the buildx driver versions our
newer client expects ("client version 1.52 is too new"). Real fix is
upgrading docker on the fleet — separate workstream.
The Miniflux inbox got noisy after a few subreddits + HN + Lobste.rs.
This stack distills a single static page twice a day — at 0800 and
2000 local — that surfaces only what cleared score + ratio filters,
each item tldr'd by qwen3.5-35-a3b on llama-swap.
Pipeline (digest.py, ~330 lines):
1. Discover subreddits from Miniflux feeds (any reddit.com/r/<sub>/
URL — single source of truth, no duplicated config).
2. Reddit JSON top-of-day per sub. Filter: score >= 50,
upvote_ratio >= 0.85. Cap 8 items per sub.
3. Miniflux /v1/entries for the 'Tech aggregators' category
(HN, Lobste.rs) — last 12 hours.
4. Batched per-source summarization via llama-swap
/v1/chat/completions. Each post gets a one-sentence tldr +
one-word tag (news / tutorial / release / discussion /
question / showcase / drama / meme).
5. Render Jinja2 template. Atomic write to /output/index.html
(.tmp + rename) so partial pages never get served. Per-edition
archive at /output/edition-YYYY-MM-DD-{am,pm}.html.
Two containers:
news-digest-worker python:3.12-alpine + busybox crond
news-digest-web nginx:alpine, port 8181, homepage card via
docker labels (group=News, fits next to Miniflux)
Both bind-mount /opt/docker/data/news-digest as /output and
/usr/share/nginx/html respectively.
Aesthetic — operations-center chrome (Australis cool-mono palette,
JetBrains Mono UPPERCASE eyebrows, mdi-glyph anchor) wrapping
editorial-serif news content (Fraunces variable serif w/ optical
sizes). Two type families that wouldn't normally meet, intentionally
combined: chrome says 'filed at 0800 from the bridge'; headlines say
'this is news, read it like news.' Sticky aurora-glow rule under the
masthead is the only sanctioned Australis gradient.
Edition stamp (AM/PM in big mono Australis-yellow) is the signature
piece — establishes the twice-daily rhythm at a glance.
All filtering + LLM + scheduling knobs in .env. Subreddit list is
implicit (read from Miniflux), so adding a sub = subscribing in
Miniflux, no config edit on this stack.
Initial deploy failed with 'Container cannot be connected to network
endpoints: miniflux-net, traefik-net' — the docker engine balks at
joining a brand-new internal network and an existing external
network in one create step.
Flattened both containers onto traefik-net only. The DB password
still protects miniflux-db, and traefik-net is internal-LAN-only,
so co-locating them is fine. Verify step updated to check for
traefik-net membership instead of the (now-gone) miniflux-net.
Adds Miniflux on ana-docker as the unified inbox for tech blogs,
Hacker News, lobste.rs, and selected subreddits. Reddit serves clean
RSS for any sub at https://reddit.com/r/<sub>/.rss, so subreddit
follows fold into the same inbox as everything else — no Reddit
account needed, no manual polling.
Stack:
stacks/miniflux/
compose.yaml — miniflux + bundled postgres:16
.env.example — placeholders for DB password + admin user
starter-feeds.opml — initial subscriptions (HN, Lobste.rs,
r/selfhosted, r/homelab, r/LocalLLaMA, r/nba)
README.md — deploy / OPML import / r/nba spoiler
block-list / backup / update flow
Postgres bundled with the stack (not pfi-postgres) — single-user RSS
DB is tiny and the bundle keeps the dependency graph flat.
Homepage gets a new 'News' group at the TOP of the Main tab (above
Monitoring) so the Miniflux card sits prominently. The card itself
auto-discovers via the homepage.* labels on the miniflux container.
Per-feed block-list rule for r/nba documented in README — Reddit's
RSS titles for game threads include scores ("Lakers 108 - Warriors
102 [Final]") which spoil the game; a regex catches the score
patterns and skips those entries while keeping discussion/highlights.
Deploy:
scripts/elway ana-docker --playbook playbooks/deploy-miniflux.yaml
Then edit /opt/docker/compose/miniflux/.env on the host to fill in
the two CHANGE_ME passwords and `docker compose up -d` again.
Two fixes from the failed first deploy on irv-ml1:
1. CPU/GPU variant. Kokoro's GPU image needs CUDA >= 12.9; irv-ml1's
driver 570.124.06 caps at 12.8 so the gpu variant fails with
"nvidia-container-cli: requirement error: unsatisfied condition:
cuda>=12.9". Make the variant a knob:
KOKORO_VARIANT=cpu (default — works anywhere)
KOKORO_VARIANT=gpu (after driver bump)
KOKORO_USE_GPU=false|true (matches the variant)
Kokoro is tiny (82M params) so CPU is workable: TTFA ~1s vs ~300ms
on GPU. Acceptable while the driver bump gets scheduled. compose.yaml
no longer hard-codes `runtime: nvidia` — relies on the daemon's
default-runtime + NVIDIA_VISIBLE_DEVICES gating, same as how the
wrapper's USE_GPU flag selects the inference path inside the
container. Toggling between variants is now a `.env` edit + restart.
2. Tighter pull-log filter. --quiet on `docker compose pull` only
suppresses the pull command's stdout; the docker daemon still
emits per-layer extraction events on stderr ("ffbfd7a09415
Extracting 64.06MB" repeated dozens of times per layer). Drop those
too via grep on the SHA-prefixed pattern. set -o pipefail keeps a
real pull failure visible.
For existing deployments: removing /opt/docker/compose/kokoro/.env
on the host and rerunning the playbook re-seeds with the new schema.
Profiling the index-tts deploy log (2057 lines) showed ~25% was just
pip's per-package Downloading / Collecting / Requirement-already /
progress-bar spam — useless for ops, hard to scan when something
actually breaks.
Three changes across the four TTS deploy playbooks:
1. Pulls (Kokoro): add --quiet. 6.5 GB pull no longer floods the log
with per-layer progress redraws. Final "X Pulled" still prints.
2. Builds (VibeVoice, Chatterbox, IndexTTS-2): add --progress=plain
to stop the BuildKit TUI from littering the captured log with
carriage-return overdraws, then pipe through a grep filter that
drops pip's noisy lines but keeps:
- buildkit step transitions (#NN [stage])
- DONE / CACHED / ERROR markers
- apt + build-stage messages
set -o pipefail keeps a real build failure from being swallowed
by the grep's exit code.
Net effect: ~25% smaller logs, much more scannable; full visibility
into step progress and errors preserved.
"docker compose pull (first run: ~6.5 GB from GHCR)" had an unquoted
colon-space inside a plain scalar value, which YAML parses as a
nested mapping — elway aborted on load. Single-line fix: wrap the
value in double quotes.
Three TTS additions to round out coverage on irv-ml1, each filling a
distinct niche the existing slate doesn't own.
Final coverage matrix (all on irv-ml1):
Kokoro — low-latency English, fixed voice library, ~300ms TTFA
Chatterbox Turbo — low-latency English w/ voice cloning + paralinguistic tags
IndexTTS-2 — English voice cloning + emotion vector / text control
Qwen3-TTS-1.7B-Base — high-quality English voice cloning
CosyVoice 3 — multilingual (Chinese-leaning)
VibeVoice 1.5B — long-form / multi-speaker dialogue
stacks/kokoro:
- port 8193, GPU device 0 (3090)
- pulls ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.4-master (no Dockerfile,
no first-run model download — models baked in)
- 60+ built-in voices, OpenAI-compat with stream=true over chunked HTTP
- Apache-2.0 weights + code, ~1 GB VRAM
stacks/vibevoice:
- port 8194, GPU device 1 (A6000 — for 7B headroom)
- builds groxaxo/VibeVoice-FastAPI1 (more current fork of ncoder-ai)
pinned to 7614c469a145
- default model microsoft/VibeVoice-1.5B (~7 GB bf16 VRAM); env var
swap to rsxdalv/VibeVoice-Large (7B) or FabioSarracino/VibeVoice-Large-Q8
- multi-speaker dialogue via /v1/vibevoice/generate with Speaker N: format
- long-form niche only — not low-latency
stacks/chatterbox:
- port 8196, GPU device 0 (3090)
- builds devnen/Chatterbox-TTS-Server (most active Turbo-supporting wrapper)
- default model ResembleAI/chatterbox-turbo (~2.5 GB fp16, ~75ms latency)
- paralinguistic tags inline ([laugh] [whisper] etc) — different shape
from IndexTTS-2's emotion vector; fills the speed+cloning niche
Kokoro/IndexTTS don't cover together
- mandatory PerTh watermark on outputs (Resemble policy)
Three matching playbooks under playbooks/deploy-{kokoro,vibevoice,
chatterbox}.yaml. All idempotent, creates-/when-gated.
Cold-deploy disk on /worktank/: ~7 GB Kokoro + ~19 GB VibeVoice 1.5B
+ ~12 GB Chatterbox = ~38 GB total. VRAM concurrent: ~10-11 GB across
both GPUs.
Skipped from the original four-stack proposal: VibeVoice Realtime
(overlaps Kokoro's niche; Kokoro wins on latency, license, and not
needing a build).
Adds a third TTS to the irv-ml1 fleet. IndexTTS-2 is Bilibili's
emotion-controllable zero-shot TTS (paper 2506.21619). Distinguishing
capability vs the existing two: timbre and emotion are disentangled —
clone a voice's timbre from one reference and the emotion from a
different reference, OR set emotion via 8-vector, OR derive it from a
text description. Neither CosyVoice 3 nor Qwen3-TTS-1.7B-Base does
this cleanly in English.
Wrapper is owned end-to-end (~150 lines in app.py) — the only existing
FastAPI fork (csllpr/index-tts-fastapi) targets v1 and is a dormant
single-commit repo. Upstream IndexTTS-2 ships only a Gradio webui.
Layout follows the qwen3-tts pattern:
stacks/index-tts/
Dockerfile — CUDA 12.8 base, IndexTTS pinned to a SHA
app.py — FastAPI: POST /v1/audio/speech + /v1/voices
entrypoint.sh — one-time HF snapshot_download of the weights
compose.yaml — env-driven, GPU pinning support, bind mounts
.env.example — port 8192, fp16, paths
README.md — API examples + comparison vs the other TTS
playbooks/deploy-index-tts.yaml — elway playbook for irv-ml1
Voice and emotion libraries are flat host dirs of WAVs, bind-mounted.
Drop a new <name>.wav and /v1/voices picks it up immediately.
License caveat: IndexTTS-2 weights ship under a custom Bilibili
license (free at our scale, not OSI-open). README documents it.
Alibaba's open-weight TTS (Apache 2.0, Jan 2026), deployed via
groxaxo/Qwen3-TTS-Openai-Fastapi wrapper. Built locally from a
pinned git SHA via docker buildx's git context — no source
vendored. 1.7B flagship model by default; 0.6B available via
QWEN3_TTS_MODEL env override.
Why we need a second TTS stack: cosyvoice 3 emits Chinese phonemes
for English content per upstream FunAudioLLM/CosyVoice#1790
(unfixed). Qwen3-TTS is from the same Alibaba team but with
English first-class in the checkpoint — 10 languages, 97 ms
streaming TTFB, instruction-driven emotion. Coexists with cosyvoice
on irv-ml1 (port 8191; cosyvoice keeps 8190).
Voice cloning shape DIFFERS from cosyvoice: profile-based, not
voice-id. Profiles live under voice_library/profiles/<name>/ and
are referenced as voice="clone:<name>".
Path layout: /worktank/qwen3-tts/{cache,voices}/, with cache excluded
from restic (regenerable from HF Hub) and voices included (cloned
profiles need original reference audio to recreate).
playbooks/deploy-qwen3-tts.yaml: 10 steps + 5 verify, idempotent;
the wait step polls /health for up to ~10 min to absorb first-run
model download.
Stack only — restic profile update for /worktank/qwen3-tts/voices/
to follow when this is empirically validated against the GLaDOS
voice (the "did Qwen inherit the Chinese-bias bug?" question).
STATUS.md:
- Mark 4b done (both Postgres migration + NFS decoupling)
- Add arch decisions for gitea remote + prefer-elway policy
- Add tooling entries for elway + tea CLI
- Document 2026-04-24 session milestones (irv-ml1 AI stacks,
elway, task-board, 4b finish)
- Expand memory-pointer list with the files added this session
CLAUDE.md:
- Tell new sessions to use elway for SSH-driven work, point at
the smoke playbook template
- Document the task-board plugin + MCP-tool contract so assistant
sessions with the plugin enabled know the assistant should call
task_start / task_update / task_wait / task_complete at
meaningful checkpoints
.claude/settings.json:
- Project-level env: TASK_BOARD_SESSION=Infra so every Claude Code
session opened here labels its task-board cards "Infra"
playbooks/decouple-pfi-postgres-from-ana-nas.yaml:
- Finishes the DB-off-NFS migration on pfi-postgres. Already ran
against prod today; fstab clean, unmounted, no systemd mnt-db
unit. Verify 3 was mis-expressed on first run (`grep -q active`
matched "inactive") — fixed to invert systemctl exit code
directly.
stacks/task-board/compose.yaml + .env.example describe the runtime —
image tag is task-board:local (built on the host), 7878 on host maps
to 7878 in container, SQLite lives at /opt/docker/conf/task-board/data/
(bind mount, uid 1000 friendly), homepage auto-card labels under
Toolchain group, on traefik-net like the rest of the fleet.
playbooks/deploy-task-board.yaml is the first real elway playbook —
exercises everything we built tier-1 + tier-2 idempotency for:
- `creates:` on mkdir + first-time clone + compose dir + data dir
- `when:` to chown /opt/docker/build only if it came up root-owned
- `when:` to seed .env only if one doesn't already exist (never
clobbers user edits on rerun)
- `changed_when:` on the `git reset --hard` step so repeat runs
against the same ref report `ok` instead of `changed`
- `changed_when: "false"` on every verify step (they attest, not
change)
- `upload:` with mode for compose.yaml + .env
Post-up the playbook polls /api/health for 30s before handing off to
the verify phase, so verification doesn't race the healthcheck's
start_period. Verify covers: /api/health 200, /api/tasks shape, /mcp
reachable, container on traefik-net.
Prereqs documented in the playbook header: Docker + compose plugin,
traefik-net network, git SSH access to gitea from the target host.
Tier 1 — pre-step skip conditions:
when: <remote shell expr> skip unless expr exits 0
creates: <remote path> skip if path already exists
removes: <remote path> skip if path is already absent
Any of the three saying "skip" marks the step `skipped` and moves on.
Evaluated under bash -c on the remote so `!`, `[[`, pipes etc. behave
consistently regardless of the default remote shell.
Tier 2 — post-step change detection:
changed_when: <remote shell expr>
Evaluated after a successful step. Exit 0 → step counts as `changed`
(default). Exit != 0 → `ok` (ran, nothing actually different).
Without this field, successful steps default to `changed`, matching
Ansible's shell/command defaults. Useful on verify steps:
`changed_when: "false"` reports them as `ok` since they only attest.
Status model moved from pass/fail to four states:
ok / changed / failed / skipped
Summary reports each count; overall outcome is CHANGED if any step
changed, OK if none did, FAILED on any non-skipped failure.
Rerunnable smoke: playbooks/elway-smoke.yaml now proves it. On a
clean target the cold run reports 4 changed, 3 ok. Rerunning with
the same vars reports 2 skipped / 2 changed (upload + log-record
have no idempotency hooks and are always `changed`). Overriding
--var greeting=... re-runs the gated step exactly as intended.
Doc block at the top of the script updated with the new schema
fields and state machine.
`scripts/elway` is a ~600-line Python tool (stdlib + python3-yaml) for
driving one-off ssh commands, ad-hoc file uploads, and YAML playbooks
against a single host. Fills the gap between "single ssh one-liner"
and "reach for Ansible."
Highlights:
- Three invocation modes: --shell, --upload (LOCAL:REMOTE[:MODE]),
and --playbook <path>
- Playbook schema: inline vars, list of steps, optional verify block.
Template via {{ var }}; CLI --var overrides inline defaults
- stop_on_fail global (default on), per-step override. Verify phase
always runs, even after a halt — you see end-state regardless
- Sudo handled once: probes NOPASSWD; if not, prompts locally via
getpass, validates up-front, then feeds via `sudo -S` per step.
Password never written to disk/logs. Upload-with-sudo stages to
/tmp then sudo-mv + sudo-chmod
- SSH connection reuse via ControlMaster (60s persist) keeps
multi-step playbooks responsive (~30ms/step reuse vs ~550ms cold)
- Live interleaved stdout/stderr with per-step prefix and colored
pass/fail summary. --dry-run prints the plan without executing
- Shebang pinned to /usr/bin/python3 to bypass venv-shadowing
when python3-yaml lives in the system site-packages
Smoke test (playbooks/elway-smoke.yaml) covers vars + upload + verify;
drove out a YAML-scalar-coercion bug before first commit (`shell: false`
parsed to Python bool, crashed the templater — now coerced to string
at load time with a clear error on nulls).
vh