fix(upgrade-docker-ce): retry the stack restart under sudo before reporting FAILED
The restart loop runs as the deploy identity, not root, and a stack .env is allowed to be root-owned 0600. compose bails on the unreadable file before doing anything, so the stack was reported FAILED while restart=unless-stopped had already brought it back healthy — a false failure, which is worse than a quiet one because it trains readers to skim the failure lines. Retry under sudo -n before calling it a failure, and print compose's own output either way. Verified on nh3-dev against beszel: plain attempt rc=1 'open /opt/docker/compose/beszel/.env: permission denied', sudo retry rc=0 'Container beszel-agent Started', container back to healthy. The happy path is unchanged — the sudo attempt only fires after a failure. Also record that tts-dev migrated talk from ~/talk into /opt/docker/compose/talk, which removes the one stack on this host that was invisible to anything walking that path.
This commit is contained in:
@@ -204,7 +204,20 @@ steps:
|
||||
dir="/opt/docker/compose/$stack"
|
||||
if [ -d "$dir" ]; then
|
||||
echo " starting $stack"
|
||||
(cd "$dir" && docker compose up -d) || echo " FAILED — investigate $stack"
|
||||
out=$(cd "$dir" && docker compose up -d 2>&1); rc=$?
|
||||
if [ $rc -ne 0 ]; then
|
||||
# This loop runs as the deploy identity, not root, and a stack
|
||||
# .env is allowed to be root-owned 0600 — unreadable to us, so
|
||||
# compose bails before it does anything. Retry under sudo before
|
||||
# calling it a failure. (nh3-dev 2026-09-14: beszel reported
|
||||
# "FAILED — investigate beszel" on exactly this while
|
||||
# restart=unless-stopped had already brought it back healthy. A
|
||||
# false FAILED in automation output is worse than a quiet one —
|
||||
# it trains readers to skim the failure lines.)
|
||||
out=$(cd "$dir" && sudo -n docker compose up -d 2>&1); rc=$?
|
||||
fi
|
||||
printf '%s\n' "$out"
|
||||
[ $rc -eq 0 ] || echo " FAILED — investigate $stack"
|
||||
fi
|
||||
done < /tmp/docker-pre-upgrade-stacks.txt
|
||||
true
|
||||
|
||||
Reference in New Issue
Block a user