fix(upgrade-docker-ce): retry the stack restart under sudo before reporting FAILED

The restart loop runs as the deploy identity, not root, and a stack .env is
allowed to be root-owned 0600. compose bails on the unreadable file before
doing anything, so the stack was reported FAILED while restart=unless-stopped
had already brought it back healthy — a false failure, which is worse than a
quiet one because it trains readers to skim the failure lines.

Retry under sudo -n before calling it a failure, and print compose's own
output either way. Verified on nh3-dev against beszel: plain attempt rc=1
'open /opt/docker/compose/beszel/.env: permission denied', sudo retry rc=0
'Container beszel-agent Started', container back to healthy. The happy path
is unchanged — the sudo attempt only fires after a failure.

Also record that tts-dev migrated talk from ~/talk into
/opt/docker/compose/talk, which removes the one stack on this host that was
invisible to anything walking that path.
This commit is contained in:
vh
2026-09-14 12:50:26 -07:00
parent 92a4114b90
commit ccc0df6870
2 changed files with 29 additions and 9 deletions
+14 -1
View File
@@ -204,7 +204,20 @@ steps:
dir="/opt/docker/compose/$stack"
if [ -d "$dir" ]; then
echo " starting $stack"
(cd "$dir" && docker compose up -d) || echo " FAILED — investigate $stack"
out=$(cd "$dir" && docker compose up -d 2>&1); rc=$?
if [ $rc -ne 0 ]; then
# This loop runs as the deploy identity, not root, and a stack
# .env is allowed to be root-owned 0600 — unreadable to us, so
# compose bails before it does anything. Retry under sudo before
# calling it a failure. (nh3-dev 2026-09-14: beszel reported
# "FAILED — investigate beszel" on exactly this while
# restart=unless-stopped had already brought it back healthy. A
# false FAILED in automation output is worse than a quiet one —
# it trains readers to skim the failure lines.)
out=$(cd "$dir" && sudo -n docker compose up -d 2>&1); rc=$?
fi
printf '%s\n' "$out"
[ $rc -eq 0 ] || echo " FAILED — investigate $stack"
fi
done < /tmp/docker-pre-upgrade-stacks.txt
true