playbooks/remove-autorestic + STATUS: decommission autorestic, mark docker-ce migration done
Migration complete: * ana-docker on docker-ce 29.4.1, all 29 containers back up. Traefik routing live (verified 200s on matrix.phasefinal.com presence + seafile.phasefinal.com syncs). * traefik-postboot.service installed + enabled on both traefik hosts (esh-docker-vm, ana-docker) — one-shot systemd unit that restarts traefik 60s after every boot, fixing the long-standing routing-races- after-reboot symptom. New playbook: remove-autorestic. Triggered by a typo (`D:escription` in autorestic-backup.timer line 2) flagged by systemd-analyze during the traefik-postboot install on esh-docker-vm. Rather than fix it, remove autorestic — it's redundant with the PBS + structured-restic two-layer pipeline that's been operational since 2026-04-22. Detected on two ESH-side hosts: esh-docker-vm and esh-vm-db. Playbook removes the four unit files + the /usr/local/bin/autorestic binary; leaves /srv/backups/autorestic/.autorestic.yml (archival) and /mnt/backup/restic/repo/esh (historical snapshots) for separate disposition. Sub-finding from ana-docker upgrade: seafile's seahub (the Python frontend at port 8000 inside the container) failed to start because mysql wasn't ready when seafile booted, and a single restart didn't recover it. Traefik routes return 502 on seafile dynamic endpoints until seahub is up. Needs separate triage of seafile's depends_on wiring or seahub's retry behavior — not a docker-ce regression.
This commit is contained in:
@@ -346,9 +346,15 @@ Order (least → most blast radius):
|
||||
docker-ce 29 is renumbered v5.x (was v2.x with 26-28); same Compose
|
||||
v2 codebase, just realigned major. Playbook's compose-version verify
|
||||
was hardcoded to `v2.X.Y` and false-failed — now accepts any major ≥ 2.
|
||||
3. **ana-docker** — pending. Largest blast radius (homepage, miniflux,
|
||||
news-digest, synapse, seafile, vaultwarden, gitea, openwebui, …).
|
||||
Verify each card / service after.
|
||||
3. ~~**ana-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, all 29
|
||||
containers back up clean. Traefik routing live (200s for
|
||||
matrix.phasefinal.com presence + seafile.phasefinal.com syncs
|
||||
verified post-restart). One sub-finding: seafile's internal
|
||||
`seahub` (the python web frontend at port 8000) failed on first
|
||||
start because mysql wasn't accepting connections yet, and one
|
||||
`docker restart seafile` didn't recover it — needs a separate
|
||||
triage pass on seafile's compose `depends_on` wiring or seahub's
|
||||
retry behavior. Other 28 containers fine.
|
||||
|
||||
Also during the nh3-docker debug: `configs/homepage/docker.yaml`
|
||||
commented out the `irv-ml1-docker` provider — the 20s-per-poll
|
||||
@@ -356,6 +362,41 @@ ETIMEDOUTs from the stalled host were drowning homepage's logs and
|
||||
appeared to block ana-pfi-docker discovery (News group's Miniflux card
|
||||
wouldn't render until removal). Re-enable when irv-ml1 is back.
|
||||
|
||||
## traefik-postboot — host-level recovery for daemon-restart routing races
|
||||
|
||||
Installed 2026-04-24 on both hosts that run traefik (esh-docker-vm,
|
||||
ana-docker). One-shot systemd unit (`traefik-postboot.service`,
|
||||
After=docker.service, Type=oneshot, sleep 60 → `docker restart
|
||||
traefik`) that fires once per boot. Forces a clean cold scan after
|
||||
the topology has settled, fixing the long-standing "traefik misses
|
||||
backends after host reboot" symptom (events-stream debounce + network-
|
||||
attach race when 30+ containers start in a burst).
|
||||
|
||||
Playbook: `playbooks/install-traefik-postboot.yaml`. Tunables:
|
||||
`--var delay_seconds=N` (default 60), `--var container=NAME`
|
||||
(default `traefik`). Runs `systemd-analyze verify` non-disruptively
|
||||
so installing on a healthy host doesn't blip traefik.
|
||||
|
||||
## autorestic decommission — redundant with the new two-layer pipeline
|
||||
|
||||
Hosts with autorestic detected 2026-04-24: `esh-docker-vm`, `esh-vm-db`.
|
||||
Both are pre-pipeline ESH manual installs at `/usr/local/bin/autorestic`
|
||||
+ four systemd units backing up to `/mnt/backup/restic/repo/esh`.
|
||||
The new two-layer pipeline (PBS for VM images, structured restic
|
||||
profiles via `configs/restic/<host>/profiles.yaml`) covers everything
|
||||
autorestic was doing. Trigger: a typo on esh-docker-vm's
|
||||
`autorestic-backup.timer` line 2 (`D:escription` instead of
|
||||
`Description`) surfaced during the traefik-postboot install verify.
|
||||
|
||||
Playbook: `playbooks/remove-autorestic.yaml`. Pending user run:
|
||||
`scripts/elway esh-docker-vm --playbook playbooks/remove-autorestic.yaml`
|
||||
`scripts/elway esh-vm-db --playbook playbooks/remove-autorestic.yaml`
|
||||
|
||||
Deliberately untouched (decide separately):
|
||||
- `/srv/backups/autorestic/.autorestic.yml` — config; archival value
|
||||
- `/mnt/backup/restic/repo/esh` — historical autorestic snapshots;
|
||||
prune via `restic forget` once comfortable with new pipeline coverage.
|
||||
|
||||
## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm)
|
||||
|
||||
Morning / early afternoon — the original tooling day:
|
||||
|
||||
Reference in New Issue
Block a user