playbooks/remove-autorestic + STATUS: decommission autorestic, mark docker-ce migration done
Migration complete: * ana-docker on docker-ce 29.4.1, all 29 containers back up. Traefik routing live (verified 200s on matrix.phasefinal.com presence + seafile.phasefinal.com syncs). * traefik-postboot.service installed + enabled on both traefik hosts (esh-docker-vm, ana-docker) — one-shot systemd unit that restarts traefik 60s after every boot, fixing the long-standing routing-races- after-reboot symptom. New playbook: remove-autorestic. Triggered by a typo (`D:escription` in autorestic-backup.timer line 2) flagged by systemd-analyze during the traefik-postboot install on esh-docker-vm. Rather than fix it, remove autorestic — it's redundant with the PBS + structured-restic two-layer pipeline that's been operational since 2026-04-22. Detected on two ESH-side hosts: esh-docker-vm and esh-vm-db. Playbook removes the four unit files + the /usr/local/bin/autorestic binary; leaves /srv/backups/autorestic/.autorestic.yml (archival) and /mnt/backup/restic/repo/esh (historical snapshots) for separate disposition. Sub-finding from ana-docker upgrade: seafile's seahub (the Python frontend at port 8000 inside the container) failed to start because mysql wasn't ready when seafile booted, and a single restart didn't recover it. Traefik routes return 502 on seafile dynamic endpoints until seahub is up. Needs separate triage of seafile's depends_on wiring or seahub's retry behavior — not a docker-ce regression.
This commit is contained in:
@@ -346,9 +346,15 @@ Order (least → most blast radius):
|
||||
docker-ce 29 is renumbered v5.x (was v2.x with 26-28); same Compose
|
||||
v2 codebase, just realigned major. Playbook's compose-version verify
|
||||
was hardcoded to `v2.X.Y` and false-failed — now accepts any major ≥ 2.
|
||||
3. **ana-docker** — pending. Largest blast radius (homepage, miniflux,
|
||||
news-digest, synapse, seafile, vaultwarden, gitea, openwebui, …).
|
||||
Verify each card / service after.
|
||||
3. ~~**ana-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, all 29
|
||||
containers back up clean. Traefik routing live (200s for
|
||||
matrix.phasefinal.com presence + seafile.phasefinal.com syncs
|
||||
verified post-restart). One sub-finding: seafile's internal
|
||||
`seahub` (the python web frontend at port 8000) failed on first
|
||||
start because mysql wasn't accepting connections yet, and one
|
||||
`docker restart seafile` didn't recover it — needs a separate
|
||||
triage pass on seafile's compose `depends_on` wiring or seahub's
|
||||
retry behavior. Other 28 containers fine.
|
||||
|
||||
Also during the nh3-docker debug: `configs/homepage/docker.yaml`
|
||||
commented out the `irv-ml1-docker` provider — the 20s-per-poll
|
||||
@@ -356,6 +362,41 @@ ETIMEDOUTs from the stalled host were drowning homepage's logs and
|
||||
appeared to block ana-pfi-docker discovery (News group's Miniflux card
|
||||
wouldn't render until removal). Re-enable when irv-ml1 is back.
|
||||
|
||||
## traefik-postboot — host-level recovery for daemon-restart routing races
|
||||
|
||||
Installed 2026-04-24 on both hosts that run traefik (esh-docker-vm,
|
||||
ana-docker). One-shot systemd unit (`traefik-postboot.service`,
|
||||
After=docker.service, Type=oneshot, sleep 60 → `docker restart
|
||||
traefik`) that fires once per boot. Forces a clean cold scan after
|
||||
the topology has settled, fixing the long-standing "traefik misses
|
||||
backends after host reboot" symptom (events-stream debounce + network-
|
||||
attach race when 30+ containers start in a burst).
|
||||
|
||||
Playbook: `playbooks/install-traefik-postboot.yaml`. Tunables:
|
||||
`--var delay_seconds=N` (default 60), `--var container=NAME`
|
||||
(default `traefik`). Runs `systemd-analyze verify` non-disruptively
|
||||
so installing on a healthy host doesn't blip traefik.
|
||||
|
||||
## autorestic decommission — redundant with the new two-layer pipeline
|
||||
|
||||
Hosts with autorestic detected 2026-04-24: `esh-docker-vm`, `esh-vm-db`.
|
||||
Both are pre-pipeline ESH manual installs at `/usr/local/bin/autorestic`
|
||||
+ four systemd units backing up to `/mnt/backup/restic/repo/esh`.
|
||||
The new two-layer pipeline (PBS for VM images, structured restic
|
||||
profiles via `configs/restic/<host>/profiles.yaml`) covers everything
|
||||
autorestic was doing. Trigger: a typo on esh-docker-vm's
|
||||
`autorestic-backup.timer` line 2 (`D:escription` instead of
|
||||
`Description`) surfaced during the traefik-postboot install verify.
|
||||
|
||||
Playbook: `playbooks/remove-autorestic.yaml`. Pending user run:
|
||||
`scripts/elway esh-docker-vm --playbook playbooks/remove-autorestic.yaml`
|
||||
`scripts/elway esh-vm-db --playbook playbooks/remove-autorestic.yaml`
|
||||
|
||||
Deliberately untouched (decide separately):
|
||||
- `/srv/backups/autorestic/.autorestic.yml` — config; archival value
|
||||
- `/mnt/backup/restic/repo/esh` — historical autorestic snapshots;
|
||||
prune via `restic forget` once comfortable with new pipeline coverage.
|
||||
|
||||
## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm)
|
||||
|
||||
Morning / early afternoon — the original tooling day:
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# remove-autorestic — fully decommission autorestic on a host.
|
||||
#
|
||||
# Why: autorestic was the original ESH home-lab backup tool (manual install
|
||||
# at /usr/local/bin/autorestic, daily + monthly systemd timers). The new
|
||||
# two-layer pipeline (PBS for VM images via the hypervisor, structured
|
||||
# restic profiles via configs/restic/<host>/profiles.yaml) covers
|
||||
# everything autorestic was doing, so it's redundant — and the typo
|
||||
# `D:escription` in autorestic-backup.timer caught during the
|
||||
# traefik-postboot install on esh-docker-vm prompted ripping it out
|
||||
# rather than fixing it.
|
||||
#
|
||||
# Hosts in scope (autorestic detected 2026-04-26):
|
||||
# esh-docker-vm, esh-vm-db
|
||||
# Skips on hosts without it (every step is `creates:`-/`when:`-gated).
|
||||
#
|
||||
# What this DOES remove:
|
||||
# - /etc/systemd/system/autorestic-{backup,prune}.{service,timer}
|
||||
# - /usr/local/bin/autorestic
|
||||
#
|
||||
# What this DOES NOT touch (decide separately):
|
||||
# - /srv/backups/autorestic/.autorestic.yml — the config; archival
|
||||
# value (records what was being backed up + the restic key)
|
||||
# - /mnt/backup/restic/repo/esh — historical autorestic snapshots
|
||||
# (real backup data; user can prune with `restic forget` once they're
|
||||
# comfortable the new pipeline has equivalent coverage)
|
||||
#
|
||||
# Usage:
|
||||
# scripts/elway esh-docker-vm --playbook playbooks/remove-autorestic.yaml
|
||||
# scripts/elway esh-vm-db --playbook playbooks/remove-autorestic.yaml
|
||||
|
||||
steps:
|
||||
- name: Stop + disable autorestic-backup.timer
|
||||
shell: systemctl disable --now autorestic-backup.timer
|
||||
sudo: true
|
||||
when: "systemctl list-unit-files autorestic-backup.timer --no-legend 2>/dev/null | grep -q ."
|
||||
|
||||
- name: Stop + disable autorestic-prune.timer
|
||||
shell: systemctl disable --now autorestic-prune.timer
|
||||
sudo: true
|
||||
when: "systemctl list-unit-files autorestic-prune.timer --no-legend 2>/dev/null | grep -q ."
|
||||
|
||||
- name: Stop autorestic-backup.service (if running)
|
||||
# Should be a no-op after disable --now above, but a long-running
|
||||
# backup invocation triggered manually could still be live. Safe
|
||||
# to call on a stopped unit.
|
||||
shell: systemctl stop autorestic-backup.service 2>/dev/null || true
|
||||
sudo: true
|
||||
|
||||
- name: Stop autorestic-prune.service (if running)
|
||||
shell: systemctl stop autorestic-prune.service 2>/dev/null || true
|
||||
sudo: true
|
||||
|
||||
- name: Remove autorestic systemd unit files
|
||||
shell: rm -f /etc/systemd/system/autorestic-backup.service \
|
||||
/etc/systemd/system/autorestic-backup.timer \
|
||||
/etc/systemd/system/autorestic-prune.service \
|
||||
/etc/systemd/system/autorestic-prune.timer
|
||||
sudo: true
|
||||
removes: /etc/systemd/system/autorestic-backup.timer
|
||||
|
||||
- name: systemctl daemon-reload (drop the removed units from systemd's view)
|
||||
shell: systemctl daemon-reload
|
||||
sudo: true
|
||||
|
||||
- name: systemctl reset-failed (clear any leftover failed-state for autorestic-*)
|
||||
shell: systemctl reset-failed 'autorestic-*' 2>/dev/null || true
|
||||
sudo: true
|
||||
|
||||
- name: Remove the autorestic binary
|
||||
shell: rm -f /usr/local/bin/autorestic
|
||||
sudo: true
|
||||
removes: /usr/local/bin/autorestic
|
||||
|
||||
verify:
|
||||
- name: No autorestic systemd units remain
|
||||
shell: |
|
||||
n=$(ls /etc/systemd/system/autorestic-* 2>/dev/null | wc -l)
|
||||
[ "$n" = "0" ]
|
||||
changed_when: "false"
|
||||
|
||||
- name: No autorestic binary on PATH
|
||||
shell: ! command -v autorestic >/dev/null
|
||||
changed_when: "false"
|
||||
|
||||
- name: systemd no longer knows about autorestic units
|
||||
shell: |
|
||||
n=$(systemctl list-unit-files 'autorestic-*' --no-legend 2>/dev/null | wc -l)
|
||||
[ "$n" = "0" ]
|
||||
changed_when: "false"
|
||||
Reference in New Issue
Block a user