STATUS: mark autorestic decommission done, seafile seahub race fixed

autorestic removal completed on both esh-docker-vm and esh-vm-db
after two playbook fixes (YAML plain-scalar folding ate a backslash
continuation; YAML tag indicator stripped a leading `!`). Both
documented inline.

seafile seahub race resolved by adding a healthcheck to mariadb
(bundled healthcheck.sh --connect --innodb_initialized) and
converting seafile's depends_on to long-form with
condition: service_healthy on db. Compose now waits for InnoDB
to initialize before starting seahub, so the daemon-restart race
that wedged the python frontend can't recur. Verified: seahub log
clean post-recreate, traefik 502 rate dropped to zero on
seafile@docker. Compose change lives on the server (the mirror is
gitignored by design).
This commit is contained in:
vh
2026-04-26 14:51:51 -07:00
parent 7a2f8a1954
commit c1427adc0f
+36 -9
View File
@@ -349,12 +349,8 @@ Order (least → most blast radius):
3. ~~**ana-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, all 29
containers back up clean. Traefik routing live (200s for
matrix.phasefinal.com presence + seafile.phasefinal.com syncs
verified post-restart). One sub-finding: seafile's internal
`seahub` (the python web frontend at port 8000) failed on first
start because mysql wasn't accepting connections yet, and one
`docker restart seafile` didn't recover it — needs a separate
triage pass on seafile's compose `depends_on` wiring or seahub's
retry behavior. Other 28 containers fine.
verified post-restart). Sub-finding (since fixed, see below):
seafile's internal `seahub` raced mysql on startup and wedged.
Also during the nh3-docker debug: `configs/homepage/docker.yaml`
commented out the `irv-ml1-docker` provider — the 20s-per-poll
@@ -388,15 +384,46 @@ autorestic was doing. Trigger: a typo on esh-docker-vm's
`autorestic-backup.timer` line 2 (`D:escription` instead of
`Description`) surfaced during the traefik-postboot install verify.
Playbook: `playbooks/remove-autorestic.yaml`. Pending user run:
`scripts/elway esh-docker-vm --playbook playbooks/remove-autorestic.yaml`
`scripts/elway esh-vm-db --playbook playbooks/remove-autorestic.yaml`
Playbook: `playbooks/remove-autorestic.yaml`. **Done 2026-04-24** on
both hosts after two playbook bug-fixes surfaced by the first run:
(a) `rm path1 \` + newline + `path2` was bitten by YAML plain-scalar
folding — the backslash-newline collapsed to a literal `\ `, so only
the first path got passed to rm. Fixed by switching to a glob.
(b) `shell: ! command -v autorestic >/dev/null` was parsed by YAML as
a tag indicator with the `!` stripped, inverting the verify's exit
code. Fixed by quoting the value.
Deliberately untouched (decide separately):
- `/srv/backups/autorestic/.autorestic.yml` — config; archival value
- `/mnt/backup/restic/repo/esh` — historical autorestic snapshots;
prune via `restic forget` once comfortable with new pipeline coverage.
## seafile seahub startup race — fixed 2026-04-24
When ana-docker rebooted (or any time the seafile container was
recreated alongside seafile-mysql), seahub (the python frontend at
port 8000 inside the seafile container) raced mariadb's startup.
Short-form `depends_on: [db, memcached]` in compose only waits for
the dependency container to be CREATED, not healthy — so seafile's
`start.py` would hit `mysql Connection refused` and give up,
leaving nginx serving seafile but proxying to a dead python backend.
Symptom in traefik logs: ~7 of 30 requests returning 502 on
seafile@docker dynamic endpoints; static assets fine.
Fix applied directly to ana-docker's `/opt/docker/compose/seafile/
compose.yaml` (and into `stacks-mirror/`, which is gitignored — so
not in the repo, but the file on the server is the source of truth):
- Added `healthcheck` to the `db` service using the bundled
`/usr/local/bin/healthcheck.sh --connect --innodb_initialized`
(5s interval, 30s start_period, 30 retries).
- Converted seafile's `depends_on` to long-form with
`condition: service_healthy` for db (and `service_started` for
memcached).
Verified post-recreate: seahub log shows `Seahub is started ... Done.`
without traceback; traefik `seafile@docker` dropped to 0 502s in
the next 20s window.
## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm)
Morning / early afternoon — the original tooling day: