New runbook captures the three-phase process:
Phase 1 — Drop --append-only via DSM Container Manager web UI
Phase 2 — sudo resticprofile forget --prune --verbose on each of
nh3-docker, nh3-dev, irv-ml1 (interactive sudo per host)
Phase 3 — Restore --append-only via DSM
Why each phase looks the way it does, what to expect (largely no-op
runs for the first 6 months while no snapshots have aged out of the
keep window), how to verify each phase non-destructively (curl 401
on the rest-server root proves the container's up + serving), what
to do if Phase 2 fails with `repository is configured as append-only`
(skipped Phase 1 / DSM didn't apply), and the path to future
automation (find docker bin path on DSM, NOPASSWD-lock syncuser to
the specific recreate command).
Includes a "last run history" table seeded with today's first
post-pipeline run (no-op, irv-ml1 only had 3 snapshots due to the
04-25→27 CUDA stall).
Cross-referenced from docs/README.md (runbook tree), docs/
orientation.md (where-to-look table), and STATUS.md item 9 (which
now points at the runbook + records the next-round date 2026-07-27).
Both reported (unhealthy) in docker ps. Two distinct root causes:
* news-digest-web: switched from nginx:alpine to python:3.12-alpine
(uvicorn) but kept the wget healthcheck against `localhost`. Alpine's
/etc/hosts maps localhost to BOTH ::1 and 127.0.0.1; busybox wget
tries IPv6 first, hits "connection refused" because uvicorn binds
IPv4-only, and doesn't fall back. Pinned to 127.0.0.1.
* chatterbox: devnen's image is built from a python:3.10 base and
doesn't ship curl, so `curl -fsS http://localhost:8004/api/model-info`
failed with `/bin/sh: 1: curl: not found`. Replaced with a python
urllib one-liner that fetches + asserts `b'"loaded":true' in body`,
also pinned to 127.0.0.1 to dodge the same IPv4/IPv6 race.
Both YAML extractions tested directly inside the running containers
(via `sh < script`) — chatterbox python check returns 0 when the model
is loaded.
Big update for 2026-04-27. Sections added:
* Marked the "🟥 Blocked — irv-ml1 stalled" header as RECOVERED with
resolution notes (driver 595.58.03 / CUDA 13.2 IS working, both GPUs
detected; the original "stall" must have been a one-shot
post-install hiccup that resolved on a later boot).
* New "Session milestones — 2026-04-27" section covering:
- irv-ml1 unstall + 5 pre-existing GPU stacks restored
- Kokoro GPU variant deployed (irv-ml1:8193) with the .env.example
default flipped to gpu now that the driver works
- VibeVoice 1.5B deployed (irv-ml1:8194) after fixing two bugs:
full 40-char SHA required by buildx + verify regex didn't match
the OpenAI list-format response shape
- Chatterbox Turbo deployed (irv-ml1:8196) after fixing three:
upstream moved Dockerfile path (docker/Dockerfile.gpu →
Dockerfile.cu128 at root), pinned to current SHA instead of `main`,
/health doesn't exist (switched all probes to /api/model-info
which is the wrapper's own ready-after-loaded signal)
- llama-swap qwen3.6 ttl removal across non-pinned variants;
qwen3.6-35-a3b unpinned (was OOM'ing other loads via the pinned
group's persistent: true flag); granite-4-small added to the
pinned group to stop it swapping with qwen3.6-27b
- Backup verification: all three layers green (per-host restic,
PBS-ANA, PBS-NH3 mirror — 2026-04-27 snapshots everywhere). Noted
that backrest's empty dashboard is expected (no plans configured;
the actual orchestration is the per-host resticprofile timers).
Symptom: granite-4-small and qwen3.6-27b were evicting each other
when called in alternation. granite is the news-digest curator (fires
twice daily on cron) — being evicted means a cold reload (~5s) on
every digest tick, plus visible churn whenever the user uses 27b
concurrently.
Added granite-4-small to the `pinned` group as a persistent member.
~5-6 GB at Q4_K_M + 120K KV ≈ comfortable inside the existing pin
budget (qwen3.5-9b ~6 GB → ~12 GB total persistent). Single RTX 6000
Ada is 48 GB, leaves ~36 GB headroom for whichever non-pinned model
the user invokes (qwen3.6-27b at ~30 GB fits cleanly).
Updated the pinned group's docstring to capture the current member set
+ VRAM math + the historical context (qwen3.6-35-a3b was here, was
too heavy, got removed yesterday). Marked the granite ttl: 0 with the
matching "pinned — never unloads" comment as the other group members.
Symptom: qwen3.6-35-a3b refused to deload when other models needed
the VRAM, even with the model itself at ttl: 0. The pinning came from
the `pinned` group's `persistent: true` flag, which exempts members
from eviction by the scheduler regardless of memory pressure. The
model's ttl: 0 only governs idle-timeout, NOT scheduler eviction —
those are separate concerns.
Removed qwen3.6-35-a3b from the group's members. Kept ttl: 0 on the
model itself: still no idle-unload, but the scheduler CAN now evict
it when another non-coexistent model is requested. qwen3.5-9b stays
pinned (~6 GB at Q4 — cheap to hold).
Updated the inline comment + the group-header docstring to reflect
the new semantics so future-Claude doesn't undo this.
The base qwen3.6-35-a3b is already ttl: 0 via the `pinned` group.
The three other Qwen 3.6 variants (abliterated, heretic, 27b) had
ttl: 600 → llama-swap auto-unloaded them after 10 min idle, costing
the next request a full reload (~5-15s). Removed so they stay loaded
once warm. Still get evicted by the normal swap when another
non-pinned model is requested — these aren't joining the pinned group,
just losing their idle-unload timer.
devnen/Chatterbox-TTS-Server doesn't expose /health — neither in code
nor OpenAPI. The deploy hung on the playbook's `Wait for /health to
respond` loop indefinitely (each curl -> 404, retry forever) even
though the container was up and the model loaded clean to CUDA at
22:52:21 (~42s after start).
/api/model-info returns `{"loaded":true,...}` only after the model
finishes loading, so it doubles as liveness + readiness. Updated:
* compose.yaml healthcheck — grep for `"loaded":true` from
/api/model-info.
* playbook wait step — same probe instead of /health.
* verify /health → verify /api/model-info reports loaded.
* verify /v1/audio/voices — switched from greping for `voice|alloy|echo`
literals to parsing JSON and asserting the actual response shape:
`{"status":"ok","voices":[...]}` (devnen's shape — note this is NOT
the OpenAI list-format vibevoice uses).
Build + container + /health all came up clean on the re-run; only the
voices-endpoint verify failed. The check greped the response body for
"voices"/"voice"/alloy/Carter — but VibeVoice's actual response shape
is OpenAI list-format `{"object":"list","data":[...]}`, which contains
none of those substrings. On a fresh install the data array is also
empty (voices live at /worktank/vibevoice/voices/ and the user seeds
them).
Switched the check to parse the JSON and assert the shape (object="list",
data is a list). Robust against empty voices, robust against future
schema additions.
Both deploys failed against irv-ml1 today with upstream-changed-on-us
errors:
* vibevoice: VIBEVOICE_SHA=7614c469a145 (12-char short) made docker
buildx report "repository does not contain ref 7614c469a145" — same
commit IS still HEAD of main, but buildx's git source resolver
doesn't accept short hashes even when unambiguous. Now full 40-char.
* chatterbox: dockerfile: docker/Dockerfile.gpu — devnen restructured
the repo to put Dockerfiles at root, renamed by CUDA version
(Dockerfile.cu128, .cpu, .rocm). Switched to Dockerfile.cu128 (GPU
build for CUDA 12.8 toolkit; works on irv-ml1's 595.58.03 driver).
Also pinned CHATTERBOX_SHA to a full 40-char SHA instead of `main`
so future upstream churn doesn't break the deploy without warning.
Live host .env files patched directly (the playbook only seeds .env
when absent, so canonical edits don't propagate to existing installs).
irv-ml1's driver upgrade to 595.58.03 (kernel 6.1.0-37, CUDA 13.2) is
working — both GPUs detected, modules loaded. The gpu variant of the
Kokoro-FastAPI image (which requires CUDA >= 12.9) is now the right
default for new deploys. Flipping KOKORO_VARIANT=gpu, KOKORO_USE_GPU=true,
KOKORO_GPU_DEVICES=0 (pins to the RTX 3090 — Kokoro is ~1 GB VRAM and
doesn't need the A6000).
Driver bump survived after all (595.58.03, kernel 6.1.0-37, both GPUs
detected and modules loaded). 5 GPU stacks back up clean (comfyui,
cosyvoice, qwen3-tts, index-tts, parakeet — all healthy). Homepage
discovery can resume polling 10.100.79.3:2375 over the WG tunnel.
Two new sections:
* "llama-swap — added two vision-capable Qwen 3.6 entries" documents
the heretic + 27b additions, their pre-pull into HF_HOME=/hfcache
via the one-shot python:3.12-slim + hf_transfer recipe (4:10 and
3:46 wall-clock for 29 GB and 26.5 GB respectively), and the fact
that llama-server's -hf flag auto-loads mmproj when present.
* "Stack tree convention (canonical vs mirror) — clarified" captures
the deploy-stack.sh-was-reading-from-the-wrong-tree bug and the
resolution: stacks/<stack>/ is canonical/intent (deploy source),
stacks-mirror/<host>/<stack>/ is gitignored snapshot for drift
detection only. CLAUDE.md and memory updated separately in the
prior commit.
Decision recorded in CLAUDE.md ("Stack tree convention") and memory
(convention_stacks_vs_mirror.md):
stacks/<stack>/ canonical / intent. git-tracked.
deploy-stack.sh reads from here.
stacks-mirror/<host>/<stack>/ snapshot / reality. gitignored.
sync-stacks.sh writes here. Used
for drift inspection only — never
a deploy source.
Bug this fixes: deploy-stack.sh was reading from the mirror, so edits
to stacks/llama-swap/config.yaml never reached ana-ml2. Today's
two new model entries (qwen3.6-35-a3b-heretic + qwen3.6-27b) lived
in the canonical for hours but the deploy reported "in sync" because
the script only diffed mirror vs server.
Changes:
* deploy-stack.sh: source switched from MIRROR_DIR/$HOST/$STACK to
STACKS_DIR/$STACK. Header comment + error message updated.
* sync-stacks.sh: header explicitly identifies its role as drift
detection; documents the diff command for comparing canonical vs
mirror.
* stacks/llama-swap/{config.yaml → conf/config.yaml}: matches the
deploy mapping (conf/ in canonical → /opt/docker/conf/ on host).
* CLAUDE.md: "Stack mirror (pull / push)" section rewritten as
"Stack tree convention (canonical vs mirror)" with the role table
+ workflow rules + diff recipe. Layout diagram updated.
Both models pre-pulled into /tank/aimodels/huggingface (HF_HOME=/hfcache
inside the container) via huggingface_hub.snapshot_download with
hf_transfer for parallel chunked download — heretic's 29 GB landed in
~4 min, unsloth's 26.5 GB in ~3:46 (~118 MB/s each).
heretic: llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF:Q6_K
27b: unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL
Both repos include mmproj-BF16.gguf alongside the main GGUF, and
llama-server's -hf flag auto-loads the mmproj when present in the same
repo (-hf docs: "mmproj is also downloaded automatically if available").
So both entries get vision (image-text-to-text) without needing an
explicit --mmproj path. ttl: 600 (10-min idle unload), matching the
existing abliterated entry's style.
The server-rendered .source-count / .desk-count badges were correct
at render time but went stale the moment the user hid anything —
"r/HOMELAB (4)" stayed at 4 even after all 4 items were hidden.
Worse, the entire source header still rendered with a (0) badge
once every item underneath was gone.
app.js gains a refreshCounts() pass that walks every .source and
.desk, recomputes the visible (non-.is-hidden) child count, updates
the badge text, and toggles an .is-empty class. CSS rule for
.source.is-empty and .desk.is-empty sets display:none so empty
groups collapse out entirely. Hooked into hideItem, restoreItem,
and the initial-paint hidden-set application.
New section documenting the architecture change (news-digest-web
moved from nginx:alpine to a FastAPI app on uvicorn built from the
worker's same Dockerfile), the three new endpoints
(GET /api/hidden, POST /api/hide, POST /api/restore), the item-id
scheme (12-char sha1 of reddit:<post_id> or miniflux:<entry_id>
so hide-once = hide-forever-for-that-article), and the playbook
changes (dropped DOCKER_BUILDKIT=0 now that ana-docker is on
docker-ce 29, added round-trip API verify steps).
Adds a small × on each item that hides it from the page. State is
server-side at /output/hidden.json so the same hidden set follows
the user across devices (home, ipad, laptop, work). A "Hidden (N)"
tray at the bottom shows what's hidden on the current page with a
restore button per row; older hidden ids that aren't on this page
sit silently and continue to filter future editions that include
the same article.
Architecture change: news-digest-web swaps from nginx:alpine to a
FastAPI app on uvicorn, built from the same Dockerfile as the
worker. Same image, different command (`uvicorn web:app` overrides
the worker's cron entrypoint via compose). Drops one image dependency,
adds /api/{hidden,hide,restore}.
Item ids are stable 12-char sha1 prefixes (`reddit:<post_id>` /
`miniflux:<entry_id>`) computed in digest.py at render time and
emitted as `data-id` on each .item. The frontend reads /api/hidden
once on load, applies `is-hidden` to matching items, and POSTs
hide/restore on user interaction (optimistic, with rollback on
network error).
Storage: single JSON array at /output/hidden.json, atomic writes
via tempfile + rename, threading.Lock around the read-modify-write
inside the single uvicorn worker. No auth — the digest itself is
unauthenticated on LAN; same trust boundary applies.
Playbook also drops the DOCKER_BUILDKIT=0 fallback now that
ana-docker is on docker-ce 29, and adds three verify steps
(/api/hidden returns a JSON array, app.js is reachable, full
hide/restore round-trip with a synthetic id).
autorestic removal completed on both esh-docker-vm and esh-vm-db
after two playbook fixes (YAML plain-scalar folding ate a backslash
continuation; YAML tag indicator stripped a leading `!`). Both
documented inline.
seafile seahub race resolved by adding a healthcheck to mariadb
(bundled healthcheck.sh --connect --innodb_initialized) and
converting seafile's depends_on to long-form with
condition: service_healthy on db. Compose now waits for InnoDB
to initialize before starting seahub, so the daemon-restart race
that wedged the python frontend can't recur. Verified: seahub log
clean post-recreate, traefik 502 rate dropped to zero on
seafile@docker. Compose change lives on the server (the mirror is
gitignored by design).
YAML treats a leading `!` as a tag indicator, so the unquoted
`shell: ! command -v autorestic >/dev/null` was parsed as a tagged
scalar with the `!` stripped. The verify ended up running just
`command -v autorestic >/dev/null` — which exits non-zero when
autorestic is absent, the OPPOSITE of what the assertion needed.
Quoted version `"! command -v autorestic >/dev/null"` survives
parsing and gives the intended bash negation.
The previous version listed four unit paths separated by `\` + newline.
That looks fine in source but YAML plain-scalar folding collapses the
sequence to a literal `\ ` — the backslash + space no longer functions
as a shell line continuation, and only the first path actually gets
passed to rm. End result on esh-docker-vm's first run: backup.service
removed; backup.timer + prune.service + prune.timer survived; verify
correctly caught the partial state.
Switched to `rm -f /etc/systemd/system/autorestic-*.{service,timer}`
form — single string, no folding hazard, and idempotent on hosts where
some or all of the files are already gone. Re-running on esh-docker-vm
will mop up the leftovers cleanly.
Migration complete:
* ana-docker on docker-ce 29.4.1, all 29 containers back up. Traefik
routing live (verified 200s on matrix.phasefinal.com presence +
seafile.phasefinal.com syncs).
* traefik-postboot.service installed + enabled on both traefik hosts
(esh-docker-vm, ana-docker) — one-shot systemd unit that restarts
traefik 60s after every boot, fixing the long-standing routing-races-
after-reboot symptom.
New playbook: remove-autorestic. Triggered by a typo (`D:escription`
in autorestic-backup.timer line 2) flagged by systemd-analyze during
the traefik-postboot install on esh-docker-vm. Rather than fix it,
remove autorestic — it's redundant with the PBS + structured-restic
two-layer pipeline that's been operational since 2026-04-22. Detected
on two ESH-side hosts: esh-docker-vm and esh-vm-db. Playbook removes
the four unit files + the /usr/local/bin/autorestic binary; leaves
/srv/backups/autorestic/.autorestic.yml (archival) and
/mnt/backup/restic/repo/esh (historical snapshots) for separate
disposition.
Sub-finding from ana-docker upgrade: seafile's seahub (the Python
frontend at port 8000 inside the container) failed to start because
mysql wasn't ready when seafile booted, and a single restart didn't
recover it. Traefik routes return 502 on seafile dynamic endpoints
until seahub is up. Needs separate triage of seafile's depends_on
wiring or seahub's retry behavior — not a docker-ce regression.
Traefik often misses backends after a reboot or daemon swap because
(a) its docker provider debounces / drops events when 30+ containers
start in a burst, and (b) backends can be `Created` on the docker
socket but not yet attached to traefik-net when traefik scans. The
empirical workaround is `docker restart traefik` once the topology
settles — this unit bakes that in.
Type=oneshot, After=docker.service, ExecStartPre=/bin/sleep 60,
ExecStart=docker restart traefik. Runs once per boot. delay_seconds
and container name are tunable via --var.
Verify phase: file mode, enabled state, ExecStart references the
right container, container actually exists on the host, and
systemd-analyze parses the unit cleanly (lint without executing —
avoids needlessly bouncing traefik on healthy hosts).
In scope: esh-docker-vm, ana-docker (the two hosts that run traefik).
docker-ce 29 ships docker-compose-plugin renumbered to v5.x (was v2.x
with docker-ce 26-28). Same Compose v2 codebase under the hood —
Docker just realigned the major number. The verify regex was hardcoded
to `v2\.[0-9]+\.[0-9]+`, so a successful migration on esh-docker-vm
(29.4.1, 16/16 stacks back up clean) reported FAILED on the verify
phase. Switched to `docker compose version --short` parsed for major,
gated `>= 2` — works across future plugin renumbers too.
STATUS.md: mark esh-docker-vm done. ana-docker is the last host.
After nh3-docker's swap, two systemd unit gotchas surfaced that the
playbook now handles automatically:
* The docker.io-era /etc/systemd/system/docker.service.d/override.conf
hardcoded ExecStart=/usr/sbin/dockerd; docker-ce installs at
/usr/bin/dockerd → daemon failed status=203/EXEC.
* The shipped docker-ce unit's ExecStart=dockerd -H fd:// conflicts
with daemon.json hosts: (defined for the 0.0.0.0:2375 homepage
discovery binding) → "conflicting host options".
The "Rewrite docker.service drop-in" step now backs up any existing
override, probes daemon.json for a hosts: setting, and installs an
override that strips -H from ExecStart when needed. Also added an
explicit systemctl reset-failed step to clear the start-rate-limit
state that 3 failed install-time starts leave behind.
configs/homepage/docker.yaml: comment out irv-ml1-docker provider —
20s-per-poll ETIMEDOUTs from the stalled host were drowning homepage's
logs and apparently blocking ana-pfi-docker discovery (the Miniflux
card in the News group wouldn't render until removal). Re-enable when
irv-ml1 is back.
STATUS.md: new "Active migration" section tracking the docker-ce
rollout — nh3-docker done; esh-docker-vm + ana-docker queued.
nh3-docker's daemon kept failing post-package-swap with status=203
even after daemon-reload. Root cause: a stale
/etc/systemd/system/docker.service.d/override.conf from the docker.io
era hardcoding ExecStart=/usr/sbin/dockerd. The override (a) points
at the no-longer-existing path, AND (b) typically also adds
-H tcp://... which now duplicates the hosts: setting in
/etc/docker/daemon.json — dockerd refuses to start when both define
hosts ('conflicting host options').
Daemon.json is the modern way to expose the TCP socket. The
override is redundant and wrong. Move it aside (preserve a
.pre-upgrade copy for forensics), then daemon-reload, then start.
Should let esh-docker-vm and ana-docker upgrades go through cleanly
without the manual debug loop nh3-docker required.
Editorial-briefing favicon: 32×32 SVG, Australis palette. Cyan
masthead-rule across the top echoes the page's aurora-rule, four
descending text-line indicators below evoke a newspaper column.
Reads cleanly at 16×16 (the typical browser tab size). Static
markup only — no script, no animation — so all browsers honor
it for tab + bookmark icons.
Linked from both digest.html.j2 and archive.html.j2 with the
proper type="image/svg+xml" attribute. Served by nginx from
the bind-mounted /output dir alongside index.html and style.css.
Deploy playbook also updated to copy the favicon into /output at
deploy-time so a fresh deploy doesn't 404 on the icon before the
first cron fire.
Docker's official package installs dockerd at /usr/bin/dockerd; the
Debian docker.io package put it at /usr/sbin/dockerd. After the apt
swap, the new docker.service unit file is on disk with the right
path, but systemd's cached unit still has the OLD ExecStart pointing
at /usr/sbin/dockerd. Daemon start fails with:
status=203/EXEC "No such file or directory"
Fix is systemctl daemon-reload between install and start. nh3-docker
hit this; adding the step so esh-docker-vm and ana-docker don't.
Two requested polish items:
1. Reddit items now show TWO distinct action chips in the footer:
[↗ SOURCE huggingface.co] [⌥ REDDIT THREAD] 3h · u/foo
Previously the external link was a tiny dot-separated text link
that visually competed with the post metadata. Now: explicit
labeled buttons, distinct colors (cyan for source, blue for
thread), with hover states that match the Australis accent
palette. Non-Reddit items get just the SOURCE chip.
2. Archive page at /archive.html — lists every edition-*.html on
disk, newest-first, sorted PM-before-AM within a day. Each row
is the date in editorial serif + an AM/PM chip color-coded
(yellow morning / cyan evening) + a hover-affordance arrow.
Header link from the main edition reads "ARCHIVE →".
Generation: digest.py walks OUTPUT_DIR for the edition-*.html
filename pattern on every run, sorts, renders archive.html.j2,
writes atomically. Cheap (~1ms even with hundreds of editions).
No retention cap — twice-daily for a year is ~700 small files,
well under any reasonable concern.
CSS additions: .action chip styles (with per-action color variants),
.archive-link in masthead, full .archive-row + .archive-edition
treatment.
Multi-line shell with backslash-continued URL had the continuation
line starting at column 0, which breaks YAML's | literal block
('could not find expected :'). Stash the URL into a shell variable
and emit on one logical line.
Three docker hosts on the fleet still run docker.io 20.10.24 (the
Debian bookworm package) which:
* sticks at API 1.41 — newer compose clients (1.52+) refuse to talk
to it without DOCKER_BUILDKIT=0 fallback (caught during the
news-digest deploy on ana-docker today)
* is functionally EOL — docker.io's upstream no longer ships to it
* is missing modern buildx driver versions
This playbook handles a single-host migration: snapshot existing
docker package versions for rollback reference, stop every running
compose stack, apt-remove (NOT purge — preserves /var/lib/docker)
docker.io + plugins, add Docker's signed APT repo for Debian, install
docker-ce + docker-compose-plugin + containerd.io + buildx-plugin,
restart the daemon, bring stacks back up.
Volumes / images / containers survive the swap because:
* /var/lib/docker is preserved by `apt remove` (vs purge)
* both packages default to the overlay2 storage driver
Recommended host order (least → most blast radius):
1. nh3-docker (NH3 site, fewer services)
2. esh-docker-vm (home lab; many services but single-consumer)
3. ana-docker (production-ish; vaultwarden, gitea, synapse,
task-board, miniflux, news-digest, paperless-ng)
Run as `scripts/elway <host> --playbook playbooks/upgrade-docker-ce.yaml`
per host. Verify between hosts via `docker version` + spot-check a
few containers.
Rollback if a daemon won't start or a container errors:
ssh <host> 'sudo apt install --allow-downgrades \$(cat /tmp/docker-pre-upgrade.txt | tr "\n" " ")'
Three iterations to get end-to-end:
1. Dockerfile missed COPY run-digest.sh — cron's exec target wasn't
in the image, every fire failed. Added COPY + chmod.
2. Jinja template used {{ list|sum(attribute='items') }} which
sum()s lists with start=0 → TypeError int+list. Switched to
computing reddit_total / tech_total in Python and passing as
template args.
3. LLM defaulted to qwen3.5-35-a3b which (a) is broken in
llama-swap (model process exits on launch), (b) when working,
defaults to extended-thinking mode that eats the entire token
budget without producing any visible content. Same pattern with
qwen3.6-35-a3b. Switched default to granite-4-small — small (4B),
fast (~1s/call), no thinking-mode pathology, returns clean JSON.
Whole pipeline now runs in ~35s total across 8 sources.
Also hardened the LLM response parser to fall back to
reasoning_content when content is empty — catches the thinking-mode
case if anyone ever points the digest at one of those models. Plus
the deploy playbook gained DOCKER_BUILDKIT=0 because ana-docker is
on docker 20.10 which doesn't carry the buildx driver versions our
newer client expects ("client version 1.52 is too new"). Real fix is
upgrading docker on the fleet — separate workstream.
The Miniflux inbox got noisy after a few subreddits + HN + Lobste.rs.
This stack distills a single static page twice a day — at 0800 and
2000 local — that surfaces only what cleared score + ratio filters,
each item tldr'd by qwen3.5-35-a3b on llama-swap.
Pipeline (digest.py, ~330 lines):
1. Discover subreddits from Miniflux feeds (any reddit.com/r/<sub>/
URL — single source of truth, no duplicated config).
2. Reddit JSON top-of-day per sub. Filter: score >= 50,
upvote_ratio >= 0.85. Cap 8 items per sub.
3. Miniflux /v1/entries for the 'Tech aggregators' category
(HN, Lobste.rs) — last 12 hours.
4. Batched per-source summarization via llama-swap
/v1/chat/completions. Each post gets a one-sentence tldr +
one-word tag (news / tutorial / release / discussion /
question / showcase / drama / meme).
5. Render Jinja2 template. Atomic write to /output/index.html
(.tmp + rename) so partial pages never get served. Per-edition
archive at /output/edition-YYYY-MM-DD-{am,pm}.html.
Two containers:
news-digest-worker python:3.12-alpine + busybox crond
news-digest-web nginx:alpine, port 8181, homepage card via
docker labels (group=News, fits next to Miniflux)
Both bind-mount /opt/docker/data/news-digest as /output and
/usr/share/nginx/html respectively.
Aesthetic — operations-center chrome (Australis cool-mono palette,
JetBrains Mono UPPERCASE eyebrows, mdi-glyph anchor) wrapping
editorial-serif news content (Fraunces variable serif w/ optical
sizes). Two type families that wouldn't normally meet, intentionally
combined: chrome says 'filed at 0800 from the bridge'; headlines say
'this is news, read it like news.' Sticky aurora-glow rule under the
masthead is the only sanctioned Australis gradient.
Edition stamp (AM/PM in big mono Australis-yellow) is the signature
piece — establishes the twice-daily rhythm at a glance.
All filtering + LLM + scheduling knobs in .env. Subreddit list is
implicit (read from Miniflux), so adding a sub = subscribing in
Miniflux, no config edit on this stack.
Subscribed live via Miniflux API; mirroring back to the canonical
OPML so a fresh deploy elsewhere starts with the same feed set.
All three land in the existing 'Subreddits — selfhosting' category
(retitled to mention image-gen + LLM + homelab to reflect scope).
Initial deploy failed with 'Container cannot be connected to network
endpoints: miniflux-net, traefik-net' — the docker engine balks at
joining a brand-new internal network and an existing external
network in one create step.
Flattened both containers onto traefik-net only. The DB password
still protects miniflux-db, and traefik-net is internal-LAN-only,
so co-locating them is fine. Verify step updated to check for
traefik-net membership instead of the (now-gone) miniflux-net.
Adds Miniflux on ana-docker as the unified inbox for tech blogs,
Hacker News, lobste.rs, and selected subreddits. Reddit serves clean
RSS for any sub at https://reddit.com/r/<sub>/.rss, so subreddit
follows fold into the same inbox as everything else — no Reddit
account needed, no manual polling.
Stack:
stacks/miniflux/
compose.yaml — miniflux + bundled postgres:16
.env.example — placeholders for DB password + admin user
starter-feeds.opml — initial subscriptions (HN, Lobste.rs,
r/selfhosted, r/homelab, r/LocalLLaMA, r/nba)
README.md — deploy / OPML import / r/nba spoiler
block-list / backup / update flow
Postgres bundled with the stack (not pfi-postgres) — single-user RSS
DB is tiny and the bundle keeps the dependency graph flat.
Homepage gets a new 'News' group at the TOP of the Main tab (above
Monitoring) so the Miniflux card sits prominently. The card itself
auto-discovers via the homepage.* labels on the miniflux container.
Per-feed block-list rule for r/nba documented in README — Reddit's
RSS titles for game threads include scores ("Lakers 108 - Warriors
102 [Final]") which spoil the game; a regex catches the score
patterns and skips those entries while keeping discussion/highlights.
Deploy:
scripts/elway ana-docker --playbook playbooks/deploy-miniflux.yaml
Then edit /opt/docker/compose/miniflux/.env on the host to fill in
the two CHANGE_ME passwords and `docker compose up -d` again.
Driver bump 570.124.06 → 595.58.03 attempted via
scripts/upgrade-irv-ml1-cuda.sh on 2026-04-25; host did not come up
cleanly after the post-install reboot.
Captures, in one place, everything currently in-flight against irv-ml1
so it can be revisited as a unit once the machine is healthy:
* Recovery options ordered by invasiveness (console diagnose →
rollback script → manual driver pin → try a different driver
series).
* Three deferred TTS deploys (Kokoro GPU flip, VibeVoice 1.5B,
Chatterbox Turbo) with the exact playbook commands to re-run.
* Smoke-test reminder for the 5 existing GPU stacks (comfyui,
cosyvoice, qwen3-tts, index-tts, parakeet) which were healthy
before the upgrade but unverified after.
* Explicit 'don't touch irv-ml1' guard rail until resolved.
Sits in 🟥 Blocked — surfaces above the existing 'Quick wins'
section so future sessions see it first.
Phase 1 silently aborted right after the 'cuda-drivers candidate' step
— before snapshotting, stopping containers, installing the new driver,
or rebooting. Symptom: phase2 showed driver still at 570.124.06 and
containers with multi-hour uptimes (proof they were never stopped).
Cause: `apt-cache policy cuda-drivers | head -10`. head reads its 10
lines and closes the pipe early; that delivers SIGPIPE to apt-cache,
which under `set -o pipefail` propagates as a non-zero exit, and
`set -e` immediately terminates the remote shell. The same hazard
existed in phase2's `docker ps ... | head -15`.
Fix is one-line per call site: drop the head. apt-cache policy output
for cuda-drivers is short (~10 lines), and docker ps's container list
on irv-ml1 is small enough to print fully.
Phase 1 was unconditionally adding /etc/apt/sources.list.d/nvidia-cuda.list
pointing at /etc/apt/keyrings/nvidia-cuda.gpg. But irv-ml1 already had
the upstream-style /etc/apt/sources.list.d/cuda-debian12-x86_64.list
with /usr/share/keyrings/cuda-archive-keyring.gpg — that's how the
existing 570 driver got installed in the first place. APT then refused
both: 'Conflicting values set for option Signed-By regarding source ...
debian12/x86_64/'.
Now the script:
* removes any orphan nvidia-cuda.{list,gpg} files from a previous
failed run
* greps existing /etc/apt/sources.list.d/*.list for a debian12 cuda
repo entry; if found, trusts and uses it
* only writes a new repo entry if no existing one is found, and
when it does, uses the same upstream paths (cuda-archive-keyring.gpg,
cuda-debian12-x86_64.list) so reruns are stable
Re-run safe — phase1 can be invoked any number of times without
accumulating conflicting configs.
The previous version piped a heredoc to `ssh -t host bash -s` —
which can't work because `-t` requires a real TTY and heredoc puts
stdin in pipe mode. They're mutually exclusive (orientation.md
gotcha, tripped over once already). The remote sudo therefore aborted
with 'a terminal is required to read the password' before doing
anything.
Restructured: the remote half of each phase is built as a tempfile
locally, scp'd to /tmp on irv-ml1, made executable, and invoked via
`ssh -t host /tmp/upgrade-cuda-remote.sh phase1`. Now stdin is the
real TTY (no pipe), sudo prompts work, and the rest of the script
runs as before.
Same three phases, same rollback behavior, same idempotency story.
Just a transport fix for the local→remote half.
Two-phase script for bumping irv-ml1's NVIDIA driver to support
CUDA 12.9 — currently blocked at 12.8 per driver 570.124.06, which
is why the Kokoro GPU image (built against CUDA 12.9) wouldn't
start.
Phase 1 (interactive — sudo prompted once via ssh -t):
* snapshot current nvidia packages to /tmp/nvidia-pre-upgrade.txt
(rollback reference)
* add NVIDIA's CUDA APT repo for Debian 12 with signed key
* stop GPU containers (comfyui, cosyvoice, qwen3-tts, index-tts,
parakeet, kokoro)
* apt install cuda-drivers (currently pulls 580 series)
* reboot
Phase 2 (re-run after host comes back):
* poll for SSH availability
* verify new driver + CUDA version
* restart GPU containers via docker compose up -d
* print next steps (flip Kokoro .env to gpu variant + redeploy)
Includes a rollback subcommand that reinstalls the snapshotted
packages with --allow-downgrades and reboots — for the case where
the new driver regresses on this kernel.
Driver source switched from Debian's nvidia-driver metapackage to
NVIDIA's official cuda-drivers metapackage; bookworm doesn't carry
575+ in its main or backports repos as of 2026-04-25.
Two fixes from the failed first deploy on irv-ml1:
1. CPU/GPU variant. Kokoro's GPU image needs CUDA >= 12.9; irv-ml1's
driver 570.124.06 caps at 12.8 so the gpu variant fails with
"nvidia-container-cli: requirement error: unsatisfied condition:
cuda>=12.9". Make the variant a knob:
KOKORO_VARIANT=cpu (default — works anywhere)
KOKORO_VARIANT=gpu (after driver bump)
KOKORO_USE_GPU=false|true (matches the variant)
Kokoro is tiny (82M params) so CPU is workable: TTFA ~1s vs ~300ms
on GPU. Acceptable while the driver bump gets scheduled. compose.yaml
no longer hard-codes `runtime: nvidia` — relies on the daemon's
default-runtime + NVIDIA_VISIBLE_DEVICES gating, same as how the
wrapper's USE_GPU flag selects the inference path inside the
container. Toggling between variants is now a `.env` edit + restart.
2. Tighter pull-log filter. --quiet on `docker compose pull` only
suppresses the pull command's stdout; the docker daemon still
emits per-layer extraction events on stderr ("ffbfd7a09415
Extracting 64.06MB" repeated dozens of times per layer). Drop those
too via grep on the SHA-prefixed pattern. set -o pipefail keeps a
real pull failure visible.
For existing deployments: removing /opt/docker/compose/kokoro/.env
on the host and rerunning the playbook re-seeds with the new schema.
Profiling the index-tts deploy log (2057 lines) showed ~25% was just
pip's per-package Downloading / Collecting / Requirement-already /
progress-bar spam — useless for ops, hard to scan when something
actually breaks.
Three changes across the four TTS deploy playbooks:
1. Pulls (Kokoro): add --quiet. 6.5 GB pull no longer floods the log
with per-layer progress redraws. Final "X Pulled" still prints.
2. Builds (VibeVoice, Chatterbox, IndexTTS-2): add --progress=plain
to stop the BuildKit TUI from littering the captured log with
carriage-return overdraws, then pipe through a grep filter that
drops pip's noisy lines but keeps:
- buildkit step transitions (#NN [stage])
- DONE / CACHED / ERROR markers
- apt + build-stage messages
set -o pipefail keeps a real build failure from being swallowed
by the grep's exit code.
Net effect: ~25% smaller logs, much more scannable; full visibility
into step progress and errors preserved.
"docker compose pull (first run: ~6.5 GB from GHCR)" had an unquoted
colon-space inside a plain scalar value, which YAML parses as a
nested mapping — elway aborted on load. Single-line fix: wrap the
value in double quotes.
Three TTS additions to round out coverage on irv-ml1, each filling a
distinct niche the existing slate doesn't own.
Final coverage matrix (all on irv-ml1):
Kokoro — low-latency English, fixed voice library, ~300ms TTFA
Chatterbox Turbo — low-latency English w/ voice cloning + paralinguistic tags
IndexTTS-2 — English voice cloning + emotion vector / text control
Qwen3-TTS-1.7B-Base — high-quality English voice cloning
CosyVoice 3 — multilingual (Chinese-leaning)
VibeVoice 1.5B — long-form / multi-speaker dialogue
stacks/kokoro:
- port 8193, GPU device 0 (3090)
- pulls ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.4-master (no Dockerfile,
no first-run model download — models baked in)
- 60+ built-in voices, OpenAI-compat with stream=true over chunked HTTP
- Apache-2.0 weights + code, ~1 GB VRAM
stacks/vibevoice:
- port 8194, GPU device 1 (A6000 — for 7B headroom)
- builds groxaxo/VibeVoice-FastAPI1 (more current fork of ncoder-ai)
pinned to 7614c469a145
- default model microsoft/VibeVoice-1.5B (~7 GB bf16 VRAM); env var
swap to rsxdalv/VibeVoice-Large (7B) or FabioSarracino/VibeVoice-Large-Q8
- multi-speaker dialogue via /v1/vibevoice/generate with Speaker N: format
- long-form niche only — not low-latency
stacks/chatterbox:
- port 8196, GPU device 0 (3090)
- builds devnen/Chatterbox-TTS-Server (most active Turbo-supporting wrapper)
- default model ResembleAI/chatterbox-turbo (~2.5 GB fp16, ~75ms latency)
- paralinguistic tags inline ([laugh] [whisper] etc) — different shape
from IndexTTS-2's emotion vector; fills the speed+cloning niche
Kokoro/IndexTTS don't cover together
- mandatory PerTh watermark on outputs (Resemble policy)
Three matching playbooks under playbooks/deploy-{kokoro,vibevoice,
chatterbox}.yaml. All idempotent, creates-/when-gated.
Cold-deploy disk on /worktank/: ~7 GB Kokoro + ~19 GB VibeVoice 1.5B
+ ~12 GB Chatterbox = ~38 GB total. VRAM concurrent: ~10-11 GB across
both GPUs.
Skipped from the original four-stack proposal: VibeVoice Realtime
(overlaps Kokoro's niche; Kokoro wins on latency, license, and not
needing a build).
IndexTTS-2's tts.infer(stream_return=True) is a generator that yields
audio chunks per text segment as they finish, plus inter-segment
silence. Expose this via the existing POST /v1/audio/speech with a new
"stream": true field on the request body.
Wire-up:
- 44-byte WAV header emitted up front with placeholder data length
(0xFFFFFFFF) so chunks can be written before total samples are
known. Players that read until EOF (mpv, ffplay, aplay, sox,
browsers via <audio>) handle this fine.
- Each yielded chunk goes through _chunk_to_pcm_bytes(), which
handles torch tensors / numpy arrays in either int16 or float
(-1..1) form.
- 22050 Hz mono int16 — IndexTTS-2's hardcoded output shape.
Time-to-first-audio drops from full-file latency to ~one-segment
latency. Single-sentence inputs barely benefit; long passages /
multi-paragraph reads benefit a lot. Strict metadata parsers may
balk at the placeholder size — request without stream for a
closed-length WAV in that case.
INDEX_TTS_TAG bumped to v2 to force a rebuild.
budget hazard + media-CDN workaround
Tried adding git-lfs install + git lfs pull to the build to get
real example WAVs into the image — failed with:
Error downloading object: examples/emo_hate.wav: Smudge error:
batch response: This repository exceeded its LFS budget. The
account responsible for the budget should increase it to
restore access.
The index-tts org's LFS bandwidth quota is exhausted upstream and
out of our control. Reverting the Dockerfile change. The examples
aren't needed for the wrapper to work; emotion_text and
emotion_vector are sufficient for end-to-end testing without any
WAV file at all.
For users who want the bundled example clips as starter audio,
README now documents the media-CDN URL trick — same LFS objects
served via a different code path that doesn't count against the
LFS API budget. INDEX_TTS_TAG stays at v1.
The IndexTTS-2 repo stores examples/emo_*.wav and examples/voice_*.wav
as Git LFS objects. v1 of our image cloned the repo without an LFS
pull, leaving those paths as ~130-byte pointer text files — unusable
for `docker cp` into /worktank/index-tts/{voices,emotions}/ as starter
references. (Caught when an emotion_voice="hate" call returned audio
that was actually the pointer text round-tripped through file IO.)
v2 adds git-lfs to the apt list, calls `git lfs install --system`
once, and `git lfs pull` after the checkout. Adds ~1-2 MB to the
image (the examples are small audio clips). INDEX_TTS_TAG bumped to
v2 to force a clean rebuild.
Adds a third TTS to the irv-ml1 fleet. IndexTTS-2 is Bilibili's
emotion-controllable zero-shot TTS (paper 2506.21619). Distinguishing
capability vs the existing two: timbre and emotion are disentangled —
clone a voice's timbre from one reference and the emotion from a
different reference, OR set emotion via 8-vector, OR derive it from a
text description. Neither CosyVoice 3 nor Qwen3-TTS-1.7B-Base does
this cleanly in English.
Wrapper is owned end-to-end (~150 lines in app.py) — the only existing
FastAPI fork (csllpr/index-tts-fastapi) targets v1 and is a dormant
single-commit repo. Upstream IndexTTS-2 ships only a Gradio webui.
Layout follows the qwen3-tts pattern:
stacks/index-tts/
Dockerfile — CUDA 12.8 base, IndexTTS pinned to a SHA
app.py — FastAPI: POST /v1/audio/speech + /v1/voices
entrypoint.sh — one-time HF snapshot_download of the weights
compose.yaml — env-driven, GPU pinning support, bind mounts
.env.example — port 8192, fp16, paths
README.md — API examples + comparison vs the other TTS
playbooks/deploy-index-tts.yaml — elway playbook for irv-ml1
Voice and emotion libraries are flat host dirs of WAVs, bind-mounted.
Drop a new <name>.wav and /v1/voices picks it up immediately.
License caveat: IndexTTS-2 weights ship under a custom Bilibili
license (free at our scale, not OSI-open). README documents it.
Toggles --append-only off on the rest-server via a temporary
docker-compose.override.yaml (canonical compose untouched), runs
resticprofile forget --prune --verbose on each client of that
rest-server, then restores --append-only. The restore is wrapped in
a trap so a partial-failure prune still leaves the rest-server in
its safe configuration.
ANA side is fully automated against ana-docker (5 clients:
ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas, esh-vm-db).
NH3 side currently prints a manual DSM ceremony — Synology Container
Manager doesn't expose docker on the expected paths and syncuser
sudo isn't NOPASSWD, so the toggle isn't safely scriptable from
this workstation. The instructions cover the same flow in DSM web
UI + interactive ssh on each NH3 client (nh3-docker, nh3-dev,
irv-ml1).
Usage:
scripts/restic-prune.sh ana # ANA only (auto)
scripts/restic-prune.sh nh3 # NH3 instructions
scripts/restic-prune.sh all # both
scripts/restic-prune.sh -h # help
scripts/restic-prune.sh --dry-run ana # show every command
- STATUS.md: marks items 11 (mattermost dir cleanup verified gone),
13 (UniFi UDM cards added to homepage), 15 (docs first + second
pass — README + chromadb-setup deletion + VM-102 frontmatter
strip), and 16 (drift discipline rule, no auto-regen) as done.
New session-milestones section captures the four task-board
iterations (v0.1.1 session disambiguation + dormant-timer fix,
v0.1.2 favicon + 1s tick rate, v0.1.3 case-insensitive session
names with real ALTER migration), the parakeet healthcheck fix,
the AIPA-MCP session relabel + DB rename to Architect, the
homepage Toolchain dedup, and the qwen3-tts variant flip.
- .gitignore: ignore .claude/scheduled_tasks.lock and any other
.claude/*.lock — runtime per-machine state, not interesting to
cross-checkout.
vh