STATUS: mark autorestic decommission done, seafile seahub race fixed
autorestic removal completed on both esh-docker-vm and esh-vm-db after two playbook fixes (YAML plain-scalar folding ate a backslash continuation; YAML tag indicator stripped a leading `!`). Both documented inline. seafile seahub race resolved by adding a healthcheck to mariadb (bundled healthcheck.sh --connect --innodb_initialized) and converting seafile's depends_on to long-form with condition: service_healthy on db. Compose now waits for InnoDB to initialize before starting seahub, so the daemon-restart race that wedged the python frontend can't recur. Verified: seahub log clean post-recreate, traefik 502 rate dropped to zero on seafile@docker. Compose change lives on the server (the mirror is gitignored by design).
This commit is contained in:
@@ -349,12 +349,8 @@ Order (least → most blast radius):
|
||||
3. ~~**ana-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, all 29
|
||||
containers back up clean. Traefik routing live (200s for
|
||||
matrix.phasefinal.com presence + seafile.phasefinal.com syncs
|
||||
verified post-restart). One sub-finding: seafile's internal
|
||||
`seahub` (the python web frontend at port 8000) failed on first
|
||||
start because mysql wasn't accepting connections yet, and one
|
||||
`docker restart seafile` didn't recover it — needs a separate
|
||||
triage pass on seafile's compose `depends_on` wiring or seahub's
|
||||
retry behavior. Other 28 containers fine.
|
||||
verified post-restart). Sub-finding (since fixed, see below):
|
||||
seafile's internal `seahub` raced mysql on startup and wedged.
|
||||
|
||||
Also during the nh3-docker debug: `configs/homepage/docker.yaml`
|
||||
commented out the `irv-ml1-docker` provider — the 20s-per-poll
|
||||
@@ -388,15 +384,46 @@ autorestic was doing. Trigger: a typo on esh-docker-vm's
|
||||
`autorestic-backup.timer` line 2 (`D:escription` instead of
|
||||
`Description`) surfaced during the traefik-postboot install verify.
|
||||
|
||||
Playbook: `playbooks/remove-autorestic.yaml`. Pending user run:
|
||||
`scripts/elway esh-docker-vm --playbook playbooks/remove-autorestic.yaml`
|
||||
`scripts/elway esh-vm-db --playbook playbooks/remove-autorestic.yaml`
|
||||
Playbook: `playbooks/remove-autorestic.yaml`. **Done 2026-04-24** on
|
||||
both hosts after two playbook bug-fixes surfaced by the first run:
|
||||
(a) `rm path1 \` + newline + `path2` was bitten by YAML plain-scalar
|
||||
folding — the backslash-newline collapsed to a literal `\ `, so only
|
||||
the first path got passed to rm. Fixed by switching to a glob.
|
||||
(b) `shell: ! command -v autorestic >/dev/null` was parsed by YAML as
|
||||
a tag indicator with the `!` stripped, inverting the verify's exit
|
||||
code. Fixed by quoting the value.
|
||||
|
||||
Deliberately untouched (decide separately):
|
||||
- `/srv/backups/autorestic/.autorestic.yml` — config; archival value
|
||||
- `/mnt/backup/restic/repo/esh` — historical autorestic snapshots;
|
||||
prune via `restic forget` once comfortable with new pipeline coverage.
|
||||
|
||||
## seafile seahub startup race — fixed 2026-04-24
|
||||
|
||||
When ana-docker rebooted (or any time the seafile container was
|
||||
recreated alongside seafile-mysql), seahub (the python frontend at
|
||||
port 8000 inside the seafile container) raced mariadb's startup.
|
||||
Short-form `depends_on: [db, memcached]` in compose only waits for
|
||||
the dependency container to be CREATED, not healthy — so seafile's
|
||||
`start.py` would hit `mysql Connection refused` and give up,
|
||||
leaving nginx serving seafile but proxying to a dead python backend.
|
||||
Symptom in traefik logs: ~7 of 30 requests returning 502 on
|
||||
seafile@docker dynamic endpoints; static assets fine.
|
||||
|
||||
Fix applied directly to ana-docker's `/opt/docker/compose/seafile/
|
||||
compose.yaml` (and into `stacks-mirror/`, which is gitignored — so
|
||||
not in the repo, but the file on the server is the source of truth):
|
||||
- Added `healthcheck` to the `db` service using the bundled
|
||||
`/usr/local/bin/healthcheck.sh --connect --innodb_initialized`
|
||||
(5s interval, 30s start_period, 30 retries).
|
||||
- Converted seafile's `depends_on` to long-form with
|
||||
`condition: service_healthy` for db (and `service_started` for
|
||||
memcached).
|
||||
|
||||
Verified post-recreate: seahub log shows `Seahub is started ... Done.`
|
||||
without traceback; traefik `seafile@docker` dropped to 0 502s in
|
||||
the next 20s window.
|
||||
|
||||
## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm)
|
||||
|
||||
Morning / early afternoon — the original tooling day:
|
||||
|
||||
Reference in New Issue
Block a user