From c1427adc0f8eda497d716fcc2b9fdbc39a506ada Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 26 Apr 2026 14:51:51 -0700 Subject: [PATCH] STATUS: mark autorestic decommission done, seafile seahub race fixed autorestic removal completed on both esh-docker-vm and esh-vm-db after two playbook fixes (YAML plain-scalar folding ate a backslash continuation; YAML tag indicator stripped a leading `!`). Both documented inline. seafile seahub race resolved by adding a healthcheck to mariadb (bundled healthcheck.sh --connect --innodb_initialized) and converting seafile's depends_on to long-form with condition: service_healthy on db. Compose now waits for InnoDB to initialize before starting seahub, so the daemon-restart race that wedged the python frontend can't recur. Verified: seahub log clean post-recreate, traefik 502 rate dropped to zero on seafile@docker. Compose change lives on the server (the mirror is gitignored by design). --- STATUS.md | 45 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/STATUS.md b/STATUS.md index b8db394..7dd6589 100644 --- a/STATUS.md +++ b/STATUS.md @@ -349,12 +349,8 @@ Order (least → most blast radius): 3. ~~**ana-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, all 29 containers back up clean. Traefik routing live (200s for matrix.phasefinal.com presence + seafile.phasefinal.com syncs - verified post-restart). One sub-finding: seafile's internal - `seahub` (the python web frontend at port 8000) failed on first - start because mysql wasn't accepting connections yet, and one - `docker restart seafile` didn't recover it — needs a separate - triage pass on seafile's compose `depends_on` wiring or seahub's - retry behavior. Other 28 containers fine. + verified post-restart). Sub-finding (since fixed, see below): + seafile's internal `seahub` raced mysql on startup and wedged. Also during the nh3-docker debug: `configs/homepage/docker.yaml` commented out the `irv-ml1-docker` provider — the 20s-per-poll @@ -388,15 +384,46 @@ autorestic was doing. Trigger: a typo on esh-docker-vm's `autorestic-backup.timer` line 2 (`D:escription` instead of `Description`) surfaced during the traefik-postboot install verify. -Playbook: `playbooks/remove-autorestic.yaml`. Pending user run: -`scripts/elway esh-docker-vm --playbook playbooks/remove-autorestic.yaml` -`scripts/elway esh-vm-db --playbook playbooks/remove-autorestic.yaml` +Playbook: `playbooks/remove-autorestic.yaml`. **Done 2026-04-24** on +both hosts after two playbook bug-fixes surfaced by the first run: +(a) `rm path1 \` + newline + `path2` was bitten by YAML plain-scalar +folding — the backslash-newline collapsed to a literal `\ `, so only +the first path got passed to rm. Fixed by switching to a glob. +(b) `shell: ! command -v autorestic >/dev/null` was parsed by YAML as +a tag indicator with the `!` stripped, inverting the verify's exit +code. Fixed by quoting the value. Deliberately untouched (decide separately): - `/srv/backups/autorestic/.autorestic.yml` — config; archival value - `/mnt/backup/restic/repo/esh` — historical autorestic snapshots; prune via `restic forget` once comfortable with new pipeline coverage. +## seafile seahub startup race — fixed 2026-04-24 + +When ana-docker rebooted (or any time the seafile container was +recreated alongside seafile-mysql), seahub (the python frontend at +port 8000 inside the seafile container) raced mariadb's startup. +Short-form `depends_on: [db, memcached]` in compose only waits for +the dependency container to be CREATED, not healthy — so seafile's +`start.py` would hit `mysql Connection refused` and give up, +leaving nginx serving seafile but proxying to a dead python backend. +Symptom in traefik logs: ~7 of 30 requests returning 502 on +seafile@docker dynamic endpoints; static assets fine. + +Fix applied directly to ana-docker's `/opt/docker/compose/seafile/ +compose.yaml` (and into `stacks-mirror/`, which is gitignored — so +not in the repo, but the file on the server is the source of truth): +- Added `healthcheck` to the `db` service using the bundled + `/usr/local/bin/healthcheck.sh --connect --innodb_initialized` + (5s interval, 30s start_period, 30 retries). +- Converted seafile's `depends_on` to long-form with + `condition: service_healthy` for db (and `service_started` for + memcached). + +Verified post-recreate: seahub log shows `Seahub is started ... Done.` +without traceback; traefik `seafile@docker` dropped to 0 502s in +the next 20s window. + ## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm) Morning / early afternoon — the original tooling day: