fix(upgrade-docker-ce): retry the stack restart under sudo before reporting FAILED
The restart loop runs as the deploy identity, not root, and a stack .env is allowed to be root-owned 0600. compose bails on the unreadable file before doing anything, so the stack was reported FAILED while restart=unless-stopped had already brought it back healthy — a false failure, which is worse than a quiet one because it trains readers to skim the failure lines. Retry under sudo -n before calling it a failure, and print compose's own output either way. Verified on nh3-dev against beszel: plain attempt rc=1 'open /opt/docker/compose/beszel/.env: permission denied', sudo retry rc=0 'Container beszel-agent Started', container back to healthy. The happy path is unchanged — the sudo attempt only fires after a failure. Also record that tts-dev migrated talk from ~/talk into /opt/docker/compose/talk, which removes the one stack on this host that was invisible to anything walking that path.
This commit is contained in:
@@ -204,7 +204,20 @@ steps:
|
||||
dir="/opt/docker/compose/$stack"
|
||||
if [ -d "$dir" ]; then
|
||||
echo " starting $stack"
|
||||
(cd "$dir" && docker compose up -d) || echo " FAILED — investigate $stack"
|
||||
out=$(cd "$dir" && docker compose up -d 2>&1); rc=$?
|
||||
if [ $rc -ne 0 ]; then
|
||||
# This loop runs as the deploy identity, not root, and a stack
|
||||
# .env is allowed to be root-owned 0600 — unreadable to us, so
|
||||
# compose bails before it does anything. Retry under sudo before
|
||||
# calling it a failure. (nh3-dev 2026-09-14: beszel reported
|
||||
# "FAILED — investigate beszel" on exactly this while
|
||||
# restart=unless-stopped had already brought it back healthy. A
|
||||
# false FAILED in automation output is worse than a quiet one —
|
||||
# it trains readers to skim the failure lines.)
|
||||
out=$(cd "$dir" && sudo -n docker compose up -d 2>&1); rc=$?
|
||||
fi
|
||||
printf '%s\n' "$out"
|
||||
[ $rc -eq 0 ] || echo " FAILED — investigate $stack"
|
||||
fi
|
||||
done < /tmp/docker-pre-upgrade-stacks.txt
|
||||
true
|
||||
|
||||
@@ -34,18 +34,25 @@ local Bash already executes here — no SSH-to-self needed for non-privileged wo
|
||||
user tables in every database**, so no data was lost. Lesson: capture
|
||||
`AutoRemove` and `RestartPolicy` together when snapshotting a container you
|
||||
are about to bounce.
|
||||
- ⚠ The playbook's restart loop runs as `infra-ops` and cannot read a
|
||||
root-owned `0600` stack `.env` (`/opt/docker/compose/beszel/.env`), so it
|
||||
reports that stack as FAILED even when `restart: unless-stopped` brings it
|
||||
back fine. Playbook-side fix pending.
|
||||
- The playbook's restart loop runs as the deploy identity, not root, and a
|
||||
stack `.env` may legitimately be root-owned `0600`
|
||||
(`/opt/docker/compose/beszel/.env`) — compose bails before doing anything,
|
||||
so the stack reported FAILED while `restart: unless-stopped` had already
|
||||
brought it back healthy. **Fixed 2026-09-14**: the loop retries under
|
||||
`sudo -n` before calling it a failure. Verified against beszel — plain
|
||||
`rc=1 permission denied`, sudo retry `rc=0 Container beszel-agent Started`.
|
||||
A false FAILED in automation output is worse than a quiet one; it trains
|
||||
readers to skim the failure lines.
|
||||
- **`/opt/docker/compose` ownership — nh3-dev is the fleet outlier.**
|
||||
`root:root` here; `lkraven:lkraven` on irv-ml1, nh3-docker, ana-docker and
|
||||
esh-docker-vm. So "can a project session deploy its own stack" is false only
|
||||
on the box where sessions actually run. `/opt/docker/compose/talk` was created
|
||||
`lkraven`-owned 2026-09-14 so tts-dev can migrate `talk` out of `~/talk`
|
||||
(a convention violation that hides it from anything walking
|
||||
`/opt/docker/compose/*/`). Normalising the parent is **unresolved — operator's
|
||||
call**; `/opt/docker` itself is a separate three-way split (`755` root, `777`
|
||||
`lkraven`-owned 2026-09-14 and **tts-dev migrated `talk` into it the same
|
||||
day** — it had been at `~/talk`, a convention violation that hid it from
|
||||
anything walking `/opt/docker/compose/*/`. Old path parked at
|
||||
`~/talk.migrated-20260914`; `version: "3.8"` dropped from its compose file
|
||||
(it existed only to satisfy the v1 CLI, which is now gone). Normalising the
|
||||
parent directory is **unresolved — operator's call**; `/opt/docker` itself is a separate three-way split (`755` root, `777`
|
||||
root on two hosts, `755` lkraven).
|
||||
- **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd
|
||||
behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`.
|
||||
|
||||
Reference in New Issue
Block a user