playbooks/remove-autorestic + STATUS: decommission autorestic, mark docker-ce migration done

Migration complete:
* ana-docker on docker-ce 29.4.1, all 29 containers back up. Traefik
  routing live (verified 200s on matrix.phasefinal.com presence +
  seafile.phasefinal.com syncs).
* traefik-postboot.service installed + enabled on both traefik hosts
  (esh-docker-vm, ana-docker) — one-shot systemd unit that restarts
  traefik 60s after every boot, fixing the long-standing routing-races-
  after-reboot symptom.

New playbook: remove-autorestic. Triggered by a typo (`D:escription`
in autorestic-backup.timer line 2) flagged by systemd-analyze during
the traefik-postboot install on esh-docker-vm. Rather than fix it,
remove autorestic — it's redundant with the PBS + structured-restic
two-layer pipeline that's been operational since 2026-04-22. Detected
on two ESH-side hosts: esh-docker-vm and esh-vm-db. Playbook removes
the four unit files + the /usr/local/bin/autorestic binary; leaves
/srv/backups/autorestic/.autorestic.yml (archival) and
/mnt/backup/restic/repo/esh (historical snapshots) for separate
disposition.

Sub-finding from ana-docker upgrade: seafile's seahub (the Python
frontend at port 8000 inside the container) failed to start because
mysql wasn't ready when seafile booted, and a single restart didn't
recover it. Traefik routes return 502 on seafile dynamic endpoints
until seahub is up. Needs separate triage of seafile's depends_on
wiring or seahub's retry behavior — not a docker-ce regression.
This commit is contained in:
vh
2026-04-26 14:36:56 -07:00
parent 6907d0bef5
commit ac282c5526
2 changed files with 133 additions and 3 deletions
+44 -3
View File
@@ -346,9 +346,15 @@ Order (least → most blast radius):
docker-ce 29 is renumbered v5.x (was v2.x with 26-28); same Compose
v2 codebase, just realigned major. Playbook's compose-version verify
was hardcoded to `v2.X.Y` and false-failed — now accepts any major ≥ 2.
3. **ana-docker** — pending. Largest blast radius (homepage, miniflux,
news-digest, synapse, seafile, vaultwarden, gitea, openwebui, …).
Verify each card / service after.
3. ~~**ana-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, all 29
containers back up clean. Traefik routing live (200s for
matrix.phasefinal.com presence + seafile.phasefinal.com syncs
verified post-restart). One sub-finding: seafile's internal
`seahub` (the python web frontend at port 8000) failed on first
start because mysql wasn't accepting connections yet, and one
`docker restart seafile` didn't recover it — needs a separate
triage pass on seafile's compose `depends_on` wiring or seahub's
retry behavior. Other 28 containers fine.
Also during the nh3-docker debug: `configs/homepage/docker.yaml`
commented out the `irv-ml1-docker` provider — the 20s-per-poll
@@ -356,6 +362,41 @@ ETIMEDOUTs from the stalled host were drowning homepage's logs and
appeared to block ana-pfi-docker discovery (News group's Miniflux card
wouldn't render until removal). Re-enable when irv-ml1 is back.
## traefik-postboot — host-level recovery for daemon-restart routing races
Installed 2026-04-24 on both hosts that run traefik (esh-docker-vm,
ana-docker). One-shot systemd unit (`traefik-postboot.service`,
After=docker.service, Type=oneshot, sleep 60 → `docker restart
traefik`) that fires once per boot. Forces a clean cold scan after
the topology has settled, fixing the long-standing "traefik misses
backends after host reboot" symptom (events-stream debounce + network-
attach race when 30+ containers start in a burst).
Playbook: `playbooks/install-traefik-postboot.yaml`. Tunables:
`--var delay_seconds=N` (default 60), `--var container=NAME`
(default `traefik`). Runs `systemd-analyze verify` non-disruptively
so installing on a healthy host doesn't blip traefik.
## autorestic decommission — redundant with the new two-layer pipeline
Hosts with autorestic detected 2026-04-24: `esh-docker-vm`, `esh-vm-db`.
Both are pre-pipeline ESH manual installs at `/usr/local/bin/autorestic`
+ four systemd units backing up to `/mnt/backup/restic/repo/esh`.
The new two-layer pipeline (PBS for VM images, structured restic
profiles via `configs/restic/<host>/profiles.yaml`) covers everything
autorestic was doing. Trigger: a typo on esh-docker-vm's
`autorestic-backup.timer` line 2 (`D:escription` instead of
`Description`) surfaced during the traefik-postboot install verify.
Playbook: `playbooks/remove-autorestic.yaml`. Pending user run:
`scripts/elway esh-docker-vm --playbook playbooks/remove-autorestic.yaml`
`scripts/elway esh-vm-db --playbook playbooks/remove-autorestic.yaml`
Deliberately untouched (decide separately):
- `/srv/backups/autorestic/.autorestic.yml` — config; archival value
- `/mnt/backup/restic/repo/esh` — historical autorestic snapshots;
prune via `restic forget` once comfortable with new pipeline coverage.
## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm)
Morning / early afternoon — the original tooling day:
+89
View File
@@ -0,0 +1,89 @@
# remove-autorestic — fully decommission autorestic on a host.
#
# Why: autorestic was the original ESH home-lab backup tool (manual install
# at /usr/local/bin/autorestic, daily + monthly systemd timers). The new
# two-layer pipeline (PBS for VM images via the hypervisor, structured
# restic profiles via configs/restic/<host>/profiles.yaml) covers
# everything autorestic was doing, so it's redundant — and the typo
# `D:escription` in autorestic-backup.timer caught during the
# traefik-postboot install on esh-docker-vm prompted ripping it out
# rather than fixing it.
#
# Hosts in scope (autorestic detected 2026-04-26):
# esh-docker-vm, esh-vm-db
# Skips on hosts without it (every step is `creates:`-/`when:`-gated).
#
# What this DOES remove:
# - /etc/systemd/system/autorestic-{backup,prune}.{service,timer}
# - /usr/local/bin/autorestic
#
# What this DOES NOT touch (decide separately):
# - /srv/backups/autorestic/.autorestic.yml — the config; archival
# value (records what was being backed up + the restic key)
# - /mnt/backup/restic/repo/esh — historical autorestic snapshots
# (real backup data; user can prune with `restic forget` once they're
# comfortable the new pipeline has equivalent coverage)
#
# Usage:
# scripts/elway esh-docker-vm --playbook playbooks/remove-autorestic.yaml
# scripts/elway esh-vm-db --playbook playbooks/remove-autorestic.yaml
steps:
- name: Stop + disable autorestic-backup.timer
shell: systemctl disable --now autorestic-backup.timer
sudo: true
when: "systemctl list-unit-files autorestic-backup.timer --no-legend 2>/dev/null | grep -q ."
- name: Stop + disable autorestic-prune.timer
shell: systemctl disable --now autorestic-prune.timer
sudo: true
when: "systemctl list-unit-files autorestic-prune.timer --no-legend 2>/dev/null | grep -q ."
- name: Stop autorestic-backup.service (if running)
# Should be a no-op after disable --now above, but a long-running
# backup invocation triggered manually could still be live. Safe
# to call on a stopped unit.
shell: systemctl stop autorestic-backup.service 2>/dev/null || true
sudo: true
- name: Stop autorestic-prune.service (if running)
shell: systemctl stop autorestic-prune.service 2>/dev/null || true
sudo: true
- name: Remove autorestic systemd unit files
shell: rm -f /etc/systemd/system/autorestic-backup.service \
/etc/systemd/system/autorestic-backup.timer \
/etc/systemd/system/autorestic-prune.service \
/etc/systemd/system/autorestic-prune.timer
sudo: true
removes: /etc/systemd/system/autorestic-backup.timer
- name: systemctl daemon-reload (drop the removed units from systemd's view)
shell: systemctl daemon-reload
sudo: true
- name: systemctl reset-failed (clear any leftover failed-state for autorestic-*)
shell: systemctl reset-failed 'autorestic-*' 2>/dev/null || true
sudo: true
- name: Remove the autorestic binary
shell: rm -f /usr/local/bin/autorestic
sudo: true
removes: /usr/local/bin/autorestic
verify:
- name: No autorestic systemd units remain
shell: |
n=$(ls /etc/systemd/system/autorestic-* 2>/dev/null | wc -l)
[ "$n" = "0" ]
changed_when: "false"
- name: No autorestic binary on PATH
shell: ! command -v autorestic >/dev/null
changed_when: "false"
- name: systemd no longer knows about autorestic units
shell: |
n=$(systemctl list-unit-files 'autorestic-*' --no-legend 2>/dev/null | wc -l)
[ "$n" = "0" ]
changed_when: "false"