New runbook captures the three-phase process:
Phase 1 — Drop --append-only via DSM Container Manager web UI
Phase 2 — sudo resticprofile forget --prune --verbose on each of
nh3-docker, nh3-dev, irv-ml1 (interactive sudo per host)
Phase 3 — Restore --append-only via DSM
Why each phase looks the way it does, what to expect (largely no-op
runs for the first 6 months while no snapshots have aged out of the
keep window), how to verify each phase non-destructively (curl 401
on the rest-server root proves the container's up + serving), what
to do if Phase 2 fails with `repository is configured as append-only`
(skipped Phase 1 / DSM didn't apply), and the path to future
automation (find docker bin path on DSM, NOPASSWD-lock syncuser to
the specific recreate command).
Includes a "last run history" table seeded with today's first
post-pipeline run (no-op, irv-ml1 only had 3 snapshots due to the
04-25→27 CUDA stall).
Cross-referenced from docs/README.md (runbook tree), docs/
orientation.md (where-to-look table), and STATUS.md item 9 (which
now points at the runbook + records the next-round date 2026-07-27).
Big update for 2026-04-27. Sections added:
* Marked the "🟥 Blocked — irv-ml1 stalled" header as RECOVERED with
resolution notes (driver 595.58.03 / CUDA 13.2 IS working, both GPUs
detected; the original "stall" must have been a one-shot
post-install hiccup that resolved on a later boot).
* New "Session milestones — 2026-04-27" section covering:
- irv-ml1 unstall + 5 pre-existing GPU stacks restored
- Kokoro GPU variant deployed (irv-ml1:8193) with the .env.example
default flipped to gpu now that the driver works
- VibeVoice 1.5B deployed (irv-ml1:8194) after fixing two bugs:
full 40-char SHA required by buildx + verify regex didn't match
the OpenAI list-format response shape
- Chatterbox Turbo deployed (irv-ml1:8196) after fixing three:
upstream moved Dockerfile path (docker/Dockerfile.gpu →
Dockerfile.cu128 at root), pinned to current SHA instead of `main`,
/health doesn't exist (switched all probes to /api/model-info
which is the wrapper's own ready-after-loaded signal)
- llama-swap qwen3.6 ttl removal across non-pinned variants;
qwen3.6-35-a3b unpinned (was OOM'ing other loads via the pinned
group's persistent: true flag); granite-4-small added to the
pinned group to stop it swapping with qwen3.6-27b
- Backup verification: all three layers green (per-host restic,
PBS-ANA, PBS-NH3 mirror — 2026-04-27 snapshots everywhere). Noted
that backrest's empty dashboard is expected (no plans configured;
the actual orchestration is the per-host resticprofile timers).
Two new sections:
* "llama-swap — added two vision-capable Qwen 3.6 entries" documents
the heretic + 27b additions, their pre-pull into HF_HOME=/hfcache
via the one-shot python:3.12-slim + hf_transfer recipe (4:10 and
3:46 wall-clock for 29 GB and 26.5 GB respectively), and the fact
that llama-server's -hf flag auto-loads mmproj when present.
* "Stack tree convention (canonical vs mirror) — clarified" captures
the deploy-stack.sh-was-reading-from-the-wrong-tree bug and the
resolution: stacks/<stack>/ is canonical/intent (deploy source),
stacks-mirror/<host>/<stack>/ is gitignored snapshot for drift
detection only. CLAUDE.md and memory updated separately in the
prior commit.
New section documenting the architecture change (news-digest-web
moved from nginx:alpine to a FastAPI app on uvicorn built from the
worker's same Dockerfile), the three new endpoints
(GET /api/hidden, POST /api/hide, POST /api/restore), the item-id
scheme (12-char sha1 of reddit:<post_id> or miniflux:<entry_id>
so hide-once = hide-forever-for-that-article), and the playbook
changes (dropped DOCKER_BUILDKIT=0 now that ana-docker is on
docker-ce 29, added round-trip API verify steps).
autorestic removal completed on both esh-docker-vm and esh-vm-db
after two playbook fixes (YAML plain-scalar folding ate a backslash
continuation; YAML tag indicator stripped a leading `!`). Both
documented inline.
seafile seahub race resolved by adding a healthcheck to mariadb
(bundled healthcheck.sh --connect --innodb_initialized) and
converting seafile's depends_on to long-form with
condition: service_healthy on db. Compose now waits for InnoDB
to initialize before starting seahub, so the daemon-restart race
that wedged the python frontend can't recur. Verified: seahub log
clean post-recreate, traefik 502 rate dropped to zero on
seafile@docker. Compose change lives on the server (the mirror is
gitignored by design).
Migration complete:
* ana-docker on docker-ce 29.4.1, all 29 containers back up. Traefik
routing live (verified 200s on matrix.phasefinal.com presence +
seafile.phasefinal.com syncs).
* traefik-postboot.service installed + enabled on both traefik hosts
(esh-docker-vm, ana-docker) — one-shot systemd unit that restarts
traefik 60s after every boot, fixing the long-standing routing-races-
after-reboot symptom.
New playbook: remove-autorestic. Triggered by a typo (`D:escription`
in autorestic-backup.timer line 2) flagged by systemd-analyze during
the traefik-postboot install on esh-docker-vm. Rather than fix it,
remove autorestic — it's redundant with the PBS + structured-restic
two-layer pipeline that's been operational since 2026-04-22. Detected
on two ESH-side hosts: esh-docker-vm and esh-vm-db. Playbook removes
the four unit files + the /usr/local/bin/autorestic binary; leaves
/srv/backups/autorestic/.autorestic.yml (archival) and
/mnt/backup/restic/repo/esh (historical snapshots) for separate
disposition.
Sub-finding from ana-docker upgrade: seafile's seahub (the Python
frontend at port 8000 inside the container) failed to start because
mysql wasn't ready when seafile booted, and a single restart didn't
recover it. Traefik routes return 502 on seafile dynamic endpoints
until seahub is up. Needs separate triage of seafile's depends_on
wiring or seahub's retry behavior — not a docker-ce regression.
docker-ce 29 ships docker-compose-plugin renumbered to v5.x (was v2.x
with docker-ce 26-28). Same Compose v2 codebase under the hood —
Docker just realigned the major number. The verify regex was hardcoded
to `v2\.[0-9]+\.[0-9]+`, so a successful migration on esh-docker-vm
(29.4.1, 16/16 stacks back up clean) reported FAILED on the verify
phase. Switched to `docker compose version --short` parsed for major,
gated `>= 2` — works across future plugin renumbers too.
STATUS.md: mark esh-docker-vm done. ana-docker is the last host.
After nh3-docker's swap, two systemd unit gotchas surfaced that the
playbook now handles automatically:
* The docker.io-era /etc/systemd/system/docker.service.d/override.conf
hardcoded ExecStart=/usr/sbin/dockerd; docker-ce installs at
/usr/bin/dockerd → daemon failed status=203/EXEC.
* The shipped docker-ce unit's ExecStart=dockerd -H fd:// conflicts
with daemon.json hosts: (defined for the 0.0.0.0:2375 homepage
discovery binding) → "conflicting host options".
The "Rewrite docker.service drop-in" step now backs up any existing
override, probes daemon.json for a hosts: setting, and installs an
override that strips -H from ExecStart when needed. Also added an
explicit systemctl reset-failed step to clear the start-rate-limit
state that 3 failed install-time starts leave behind.
configs/homepage/docker.yaml: comment out irv-ml1-docker provider —
20s-per-poll ETIMEDOUTs from the stalled host were drowning homepage's
logs and apparently blocking ana-pfi-docker discovery (the Miniflux
card in the News group wouldn't render until removal). Re-enable when
irv-ml1 is back.
STATUS.md: new "Active migration" section tracking the docker-ce
rollout — nh3-docker done; esh-docker-vm + ana-docker queued.
Driver bump 570.124.06 → 595.58.03 attempted via
scripts/upgrade-irv-ml1-cuda.sh on 2026-04-25; host did not come up
cleanly after the post-install reboot.
Captures, in one place, everything currently in-flight against irv-ml1
so it can be revisited as a unit once the machine is healthy:
* Recovery options ordered by invasiveness (console diagnose →
rollback script → manual driver pin → try a different driver
series).
* Three deferred TTS deploys (Kokoro GPU flip, VibeVoice 1.5B,
Chatterbox Turbo) with the exact playbook commands to re-run.
* Smoke-test reminder for the 5 existing GPU stacks (comfyui,
cosyvoice, qwen3-tts, index-tts, parakeet) which were healthy
before the upgrade but unverified after.
* Explicit 'don't touch irv-ml1' guard rail until resolved.
Sits in 🟥 Blocked — surfaces above the existing 'Quick wins'
section so future sessions see it first.
Toggles --append-only off on the rest-server via a temporary
docker-compose.override.yaml (canonical compose untouched), runs
resticprofile forget --prune --verbose on each client of that
rest-server, then restores --append-only. The restore is wrapped in
a trap so a partial-failure prune still leaves the rest-server in
its safe configuration.
ANA side is fully automated against ana-docker (5 clients:
ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas, esh-vm-db).
NH3 side currently prints a manual DSM ceremony — Synology Container
Manager doesn't expose docker on the expected paths and syncuser
sudo isn't NOPASSWD, so the toggle isn't safely scriptable from
this workstation. The instructions cover the same flow in DSM web
UI + interactive ssh on each NH3 client (nh3-docker, nh3-dev,
irv-ml1).
Usage:
scripts/restic-prune.sh ana # ANA only (auto)
scripts/restic-prune.sh nh3 # NH3 instructions
scripts/restic-prune.sh all # both
scripts/restic-prune.sh -h # help
scripts/restic-prune.sh --dry-run ana # show every command
- STATUS.md: marks items 11 (mattermost dir cleanup verified gone),
13 (UniFi UDM cards added to homepage), 15 (docs first + second
pass — README + chromadb-setup deletion + VM-102 frontmatter
strip), and 16 (drift discipline rule, no auto-regen) as done.
New session-milestones section captures the four task-board
iterations (v0.1.1 session disambiguation + dormant-timer fix,
v0.1.2 favicon + 1s tick rate, v0.1.3 case-insensitive session
names with real ALTER migration), the parakeet healthcheck fix,
the AIPA-MCP session relabel + DB rename to Architect, the
homepage Toolchain dedup, and the qwen3-tts variant flip.
- .gitignore: ignore .claude/scheduled_tasks.lock and any other
.claude/*.lock — runtime per-machine state, not interesting to
cross-checkout.
- services.yaml: PFI-UDMSE (10.100.0.1, NH3 edge) under Infra - NH3,
ESH-UDMPM (10.0.0.1) under Infra - ESH. PFI-UDMSE replaces the
retired Fortigate 101F at NH3 — comment updated. Both use si-ubiquiti.
Diff also folds in the previously-deployed-but-uncommitted PBS-ANA /
PBS-NH3 / IRV section / retired NH3-SW1 cleanup, bringing the tracked
copy in sync with what's live on esh-docker-vm.
- docs/README.md: navigation map of the docs/ tree (orientation /
runbooks / pfi) with what-goes-where conventions.
- docs/runbooks/tea-0.14.0-linux-amd64.sha256: deleted (build artifact,
not a runbook).
- STATUS.md: items 11 / 13 / 15 marked done; 15 leaves Matrix and
chromadb consolidation candidates as future focused-session work.
STATUS.md:
- Mark 4b done (both Postgres migration + NFS decoupling)
- Add arch decisions for gitea remote + prefer-elway policy
- Add tooling entries for elway + tea CLI
- Document 2026-04-24 session milestones (irv-ml1 AI stacks,
elway, task-board, 4b finish)
- Expand memory-pointer list with the files added this session
CLAUDE.md:
- Tell new sessions to use elway for SSH-driven work, point at
the smoke playbook template
- Document the task-board plugin + MCP-tool contract so assistant
sessions with the plugin enabled know the assistant should call
task_start / task_update / task_wait / task_complete at
meaningful checkpoints
.claude/settings.json:
- Project-level env: TASK_BOARD_SESSION=Infra so every Claude Code
session opened here labels its task-board cards "Infra"
playbooks/decouple-pfi-postgres-from-ana-nas.yaml:
- Finishes the DB-off-NFS migration on pfi-postgres. Already ran
against prod today; fstab clean, unmounted, no systemd mnt-db
unit. Verify 3 was mis-expressed on first run (`grep -q active`
matched "inactive") — fixed to invert systemctl exit code
directly.
End-to-end runbook for standing up Proxmox Backup Server across the
fleet. Path A architecture: single primary at ANA, one-way sync to NH3
for disaster recovery. All 5 hypervisors (pfi-pve, nh3-pve, esh-pve,
esh-pve-nas, sfsrv-ana) migrate from local-dump vzdump to PBS-ANA.
Key decisions captured in the runbook:
- PBS in a Debian VM (not LXC) for clean capability model.
- PBS-ANA on pfi-pve, datastore via NFS from 10.250.50.50 —
separates backup data from hypervisor boot disk.
- PBS-NH3 on nh3-pve with local storage (independent failure
domain from ANA).
- Dedicated fleet-vzdump API token; read-only sync token for
PBS-NH3's pull job.
- sfsrv-ana specifically goes from zero backup coverage to full
vzdump coverage in Phase 3.
9 phases, each self-contained with a done-state and rollback
posture. User can stop between phases without leaving the fleet in a
bad state.
STATUS.md: added item 6b tracking this deployment. Original item 6
(cross-site rsync) now scoped to restic-only since PBS handles the
VM-image cross-site redundancy directly.
Forget against an --append-only rest-server fails every night (delete
ops blocked). The resulting daily failure cluttered service status and
logs without ever actually retiring old snapshots. Schedule is now
removed from the forget block in all six profiles; the keep-daily /
keep-weekly / keep-monthly / keep-yearly policy remains so manual
invocations (during prune ceremonies, when --append-only is
temporarily off) honor the intended retention.
Files:
configs/restic/ana-docker/profiles.yaml
configs/restic/ana-ml2/profiles.yaml
configs/restic/nh3-docker/profiles.yaml
configs/restic/esh-docker-vm/profiles.yaml
configs/restic/vm-esh-nas/profiles.yaml
configs/restic/nh3-dev/profiles.yaml
Each file has an inline comment marking why the schedule was dropped
so a future reader doesn't re-add it thinking it was an oversight.
STATUS.md: removed the "install Backrest nightly-restart timer" line
item. User confirmed the UI timeout hits even at startup, so periodic
restart wouldn't actually help. Root cause remains deferred.
Snapshot of what's in place and what's outstanding as of end of
2026-04-20/21 session. Grouped by urgency (red/orange/yellow/green/blue)
so a glance tells you what's next regardless of who's picking it up.
Notable open items:
- Backrest esh-docker-vm URI mismatch (still pointed at NH3 Synology
instead of rest-server-ana)
- ssh-target verification on the 9 newly-added host entries
- Forget schedules need patching (fail nightly against --append-only)
- ~6 secrets captured in this session's transcripts need rotation
- SureFire tenant backup plan pending decision
Lists session milestones (homepage reorg, 6/6 restic coverage, discovery
scripts, CWA migration, 9 host registrations, etc.) and memory
pointers so future sessions have context without re-reading the full
chat log.
vh