diff --git a/STATUS.md b/STATUS.md index 6458d05..b8db394 100644 --- a/STATUS.md +++ b/STATUS.md @@ -346,9 +346,15 @@ Order (least → most blast radius): docker-ce 29 is renumbered v5.x (was v2.x with 26-28); same Compose v2 codebase, just realigned major. Playbook's compose-version verify was hardcoded to `v2.X.Y` and false-failed — now accepts any major ≥ 2. -3. **ana-docker** — pending. Largest blast radius (homepage, miniflux, - news-digest, synapse, seafile, vaultwarden, gitea, openwebui, …). - Verify each card / service after. +3. ~~**ana-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, all 29 + containers back up clean. Traefik routing live (200s for + matrix.phasefinal.com presence + seafile.phasefinal.com syncs + verified post-restart). One sub-finding: seafile's internal + `seahub` (the python web frontend at port 8000) failed on first + start because mysql wasn't accepting connections yet, and one + `docker restart seafile` didn't recover it — needs a separate + triage pass on seafile's compose `depends_on` wiring or seahub's + retry behavior. Other 28 containers fine. Also during the nh3-docker debug: `configs/homepage/docker.yaml` commented out the `irv-ml1-docker` provider — the 20s-per-poll @@ -356,6 +362,41 @@ ETIMEDOUTs from the stalled host were drowning homepage's logs and appeared to block ana-pfi-docker discovery (News group's Miniflux card wouldn't render until removal). Re-enable when irv-ml1 is back. +## traefik-postboot — host-level recovery for daemon-restart routing races + +Installed 2026-04-24 on both hosts that run traefik (esh-docker-vm, +ana-docker). One-shot systemd unit (`traefik-postboot.service`, +After=docker.service, Type=oneshot, sleep 60 → `docker restart +traefik`) that fires once per boot. Forces a clean cold scan after +the topology has settled, fixing the long-standing "traefik misses +backends after host reboot" symptom (events-stream debounce + network- +attach race when 30+ containers start in a burst). + +Playbook: `playbooks/install-traefik-postboot.yaml`. Tunables: +`--var delay_seconds=N` (default 60), `--var container=NAME` +(default `traefik`). Runs `systemd-analyze verify` non-disruptively +so installing on a healthy host doesn't blip traefik. + +## autorestic decommission — redundant with the new two-layer pipeline + +Hosts with autorestic detected 2026-04-24: `esh-docker-vm`, `esh-vm-db`. +Both are pre-pipeline ESH manual installs at `/usr/local/bin/autorestic` ++ four systemd units backing up to `/mnt/backup/restic/repo/esh`. +The new two-layer pipeline (PBS for VM images, structured restic +profiles via `configs/restic//profiles.yaml`) covers everything +autorestic was doing. Trigger: a typo on esh-docker-vm's +`autorestic-backup.timer` line 2 (`D:escription` instead of +`Description`) surfaced during the traefik-postboot install verify. + +Playbook: `playbooks/remove-autorestic.yaml`. Pending user run: +`scripts/elway esh-docker-vm --playbook playbooks/remove-autorestic.yaml` +`scripts/elway esh-vm-db --playbook playbooks/remove-autorestic.yaml` + +Deliberately untouched (decide separately): +- `/srv/backups/autorestic/.autorestic.yml` — config; archival value +- `/mnt/backup/restic/repo/esh` — historical autorestic snapshots; + prune via `restic forget` once comfortable with new pipeline coverage. + ## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm) Morning / early afternoon — the original tooling day: diff --git a/playbooks/remove-autorestic.yaml b/playbooks/remove-autorestic.yaml new file mode 100644 index 0000000..fc613e9 --- /dev/null +++ b/playbooks/remove-autorestic.yaml @@ -0,0 +1,89 @@ +# remove-autorestic — fully decommission autorestic on a host. +# +# Why: autorestic was the original ESH home-lab backup tool (manual install +# at /usr/local/bin/autorestic, daily + monthly systemd timers). The new +# two-layer pipeline (PBS for VM images via the hypervisor, structured +# restic profiles via configs/restic//profiles.yaml) covers +# everything autorestic was doing, so it's redundant — and the typo +# `D:escription` in autorestic-backup.timer caught during the +# traefik-postboot install on esh-docker-vm prompted ripping it out +# rather than fixing it. +# +# Hosts in scope (autorestic detected 2026-04-26): +# esh-docker-vm, esh-vm-db +# Skips on hosts without it (every step is `creates:`-/`when:`-gated). +# +# What this DOES remove: +# - /etc/systemd/system/autorestic-{backup,prune}.{service,timer} +# - /usr/local/bin/autorestic +# +# What this DOES NOT touch (decide separately): +# - /srv/backups/autorestic/.autorestic.yml — the config; archival +# value (records what was being backed up + the restic key) +# - /mnt/backup/restic/repo/esh — historical autorestic snapshots +# (real backup data; user can prune with `restic forget` once they're +# comfortable the new pipeline has equivalent coverage) +# +# Usage: +# scripts/elway esh-docker-vm --playbook playbooks/remove-autorestic.yaml +# scripts/elway esh-vm-db --playbook playbooks/remove-autorestic.yaml + +steps: + - name: Stop + disable autorestic-backup.timer + shell: systemctl disable --now autorestic-backup.timer + sudo: true + when: "systemctl list-unit-files autorestic-backup.timer --no-legend 2>/dev/null | grep -q ." + + - name: Stop + disable autorestic-prune.timer + shell: systemctl disable --now autorestic-prune.timer + sudo: true + when: "systemctl list-unit-files autorestic-prune.timer --no-legend 2>/dev/null | grep -q ." + + - name: Stop autorestic-backup.service (if running) + # Should be a no-op after disable --now above, but a long-running + # backup invocation triggered manually could still be live. Safe + # to call on a stopped unit. + shell: systemctl stop autorestic-backup.service 2>/dev/null || true + sudo: true + + - name: Stop autorestic-prune.service (if running) + shell: systemctl stop autorestic-prune.service 2>/dev/null || true + sudo: true + + - name: Remove autorestic systemd unit files + shell: rm -f /etc/systemd/system/autorestic-backup.service \ + /etc/systemd/system/autorestic-backup.timer \ + /etc/systemd/system/autorestic-prune.service \ + /etc/systemd/system/autorestic-prune.timer + sudo: true + removes: /etc/systemd/system/autorestic-backup.timer + + - name: systemctl daemon-reload (drop the removed units from systemd's view) + shell: systemctl daemon-reload + sudo: true + + - name: systemctl reset-failed (clear any leftover failed-state for autorestic-*) + shell: systemctl reset-failed 'autorestic-*' 2>/dev/null || true + sudo: true + + - name: Remove the autorestic binary + shell: rm -f /usr/local/bin/autorestic + sudo: true + removes: /usr/local/bin/autorestic + +verify: + - name: No autorestic systemd units remain + shell: | + n=$(ls /etc/systemd/system/autorestic-* 2>/dev/null | wc -l) + [ "$n" = "0" ] + changed_when: "false" + + - name: No autorestic binary on PATH + shell: ! command -v autorestic >/dev/null + changed_when: "false" + + - name: systemd no longer knows about autorestic units + shell: | + n=$(systemctl list-unit-files 'autorestic-*' --no-legend 2>/dev/null | wc -l) + [ "$n" = "0" ] + changed_when: "false"