diff --git a/playbooks/upgrade-docker-ce.yaml b/playbooks/upgrade-docker-ce.yaml index fb2475f..3916c0d 100644 --- a/playbooks/upgrade-docker-ce.yaml +++ b/playbooks/upgrade-docker-ce.yaml @@ -204,7 +204,20 @@ steps: dir="/opt/docker/compose/$stack" if [ -d "$dir" ]; then echo " starting $stack" - (cd "$dir" && docker compose up -d) || echo " FAILED — investigate $stack" + out=$(cd "$dir" && docker compose up -d 2>&1); rc=$? + if [ $rc -ne 0 ]; then + # This loop runs as the deploy identity, not root, and a stack + # .env is allowed to be root-owned 0600 — unreadable to us, so + # compose bails before it does anything. Retry under sudo before + # calling it a failure. (nh3-dev 2026-09-14: beszel reported + # "FAILED — investigate beszel" on exactly this while + # restart=unless-stopped had already brought it back healthy. A + # false FAILED in automation output is worse than a quiet one — + # it trains readers to skim the failure lines.) + out=$(cd "$dir" && sudo -n docker compose up -d 2>&1); rc=$? + fi + printf '%s\n' "$out" + [ $rc -eq 0 ] || echo " FAILED — investigate $stack" fi done < /tmp/docker-pre-upgrade-stacks.txt true diff --git a/servers/nh3-dev/README.md b/servers/nh3-dev/README.md index 22ba73b..8e8a125 100644 --- a/servers/nh3-dev/README.md +++ b/servers/nh3-dev/README.md @@ -34,18 +34,25 @@ local Bash already executes here — no SSH-to-self needed for non-privileged wo user tables in every database**, so no data was lost. Lesson: capture `AutoRemove` and `RestartPolicy` together when snapshotting a container you are about to bounce. - - ⚠ The playbook's restart loop runs as `infra-ops` and cannot read a - root-owned `0600` stack `.env` (`/opt/docker/compose/beszel/.env`), so it - reports that stack as FAILED even when `restart: unless-stopped` brings it - back fine. Playbook-side fix pending. + - The playbook's restart loop runs as the deploy identity, not root, and a + stack `.env` may legitimately be root-owned `0600` + (`/opt/docker/compose/beszel/.env`) — compose bails before doing anything, + so the stack reported FAILED while `restart: unless-stopped` had already + brought it back healthy. **Fixed 2026-09-14**: the loop retries under + `sudo -n` before calling it a failure. Verified against beszel — plain + `rc=1 permission denied`, sudo retry `rc=0 Container beszel-agent Started`. + A false FAILED in automation output is worse than a quiet one; it trains + readers to skim the failure lines. - **`/opt/docker/compose` ownership — nh3-dev is the fleet outlier.** `root:root` here; `lkraven:lkraven` on irv-ml1, nh3-docker, ana-docker and esh-docker-vm. So "can a project session deploy its own stack" is false only on the box where sessions actually run. `/opt/docker/compose/talk` was created - `lkraven`-owned 2026-09-14 so tts-dev can migrate `talk` out of `~/talk` - (a convention violation that hides it from anything walking - `/opt/docker/compose/*/`). Normalising the parent is **unresolved — operator's - call**; `/opt/docker` itself is a separate three-way split (`755` root, `777` + `lkraven`-owned 2026-09-14 and **tts-dev migrated `talk` into it the same + day** — it had been at `~/talk`, a convention violation that hid it from + anything walking `/opt/docker/compose/*/`. Old path parked at + `~/talk.migrated-20260914`; `version: "3.8"` dropped from its compose file + (it existed only to satisfy the v1 CLI, which is now gone). Normalising the + parent directory is **unresolved — operator's call**; `/opt/docker` itself is a separate three-way split (`755` root, `777` root on two hosts, `755` lkraven). - **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`.