fix(upgrade-docker-ce): retry the stack restart under sudo before reporting FAILED

The restart loop runs as the deploy identity, not root, and a stack .env is
allowed to be root-owned 0600. compose bails on the unreadable file before
doing anything, so the stack was reported FAILED while restart=unless-stopped
had already brought it back healthy — a false failure, which is worse than a
quiet one because it trains readers to skim the failure lines.

Retry under sudo -n before calling it a failure, and print compose's own
output either way. Verified on nh3-dev against beszel: plain attempt rc=1
'open /opt/docker/compose/beszel/.env: permission denied', sudo retry rc=0
'Container beszel-agent Started', container back to healthy. The happy path
is unchanged — the sudo attempt only fires after a failure.

Also record that tts-dev migrated talk from ~/talk into
/opt/docker/compose/talk, which removes the one stack on this host that was
invisible to anything walking that path.
This commit is contained in:
2026-09-14 12:50:04 -07:00
parent 92a4114b90
commit ccc0df6870
2 changed files with 29 additions and 9 deletions
+14 -1
View File
@@ -204,7 +204,20 @@ steps:
dir="/opt/docker/compose/$stack"
if [ -d "$dir" ]; then
echo " starting $stack"
(cd "$dir" && docker compose up -d) || echo " FAILED — investigate $stack"
out=$(cd "$dir" && docker compose up -d 2>&1); rc=$?
if [ $rc -ne 0 ]; then
# This loop runs as the deploy identity, not root, and a stack
# .env is allowed to be root-owned 0600 — unreadable to us, so
# compose bails before it does anything. Retry under sudo before
# calling it a failure. (nh3-dev 2026-09-14: beszel reported
# "FAILED — investigate beszel" on exactly this while
# restart=unless-stopped had already brought it back healthy. A
# false FAILED in automation output is worse than a quiet one —
# it trains readers to skim the failure lines.)
out=$(cd "$dir" && sudo -n docker compose up -d 2>&1); rc=$?
fi
printf '%s\n' "$out"
[ $rc -eq 0 ] || echo " FAILED — investigate $stack"
fi
done < /tmp/docker-pre-upgrade-stacks.txt
true
+15 -8
View File
@@ -34,18 +34,25 @@ local Bash already executes here — no SSH-to-self needed for non-privileged wo
user tables in every database**, so no data was lost. Lesson: capture
`AutoRemove` and `RestartPolicy` together when snapshotting a container you
are about to bounce.
- The playbook's restart loop runs as `infra-ops` and cannot read a
root-owned `0600` stack `.env` (`/opt/docker/compose/beszel/.env`), so it
reports that stack as FAILED even when `restart: unless-stopped` brings it
back fine. Playbook-side fix pending.
- The playbook's restart loop runs as the deploy identity, not root, and a
stack `.env` may legitimately be root-owned `0600`
(`/opt/docker/compose/beszel/.env`) — compose bails before doing anything,
so the stack reported FAILED while `restart: unless-stopped` had already
brought it back healthy. **Fixed 2026-09-14**: the loop retries under
`sudo -n` before calling it a failure. Verified against beszel — plain
`rc=1 permission denied`, sudo retry `rc=0 Container beszel-agent Started`.
A false FAILED in automation output is worse than a quiet one; it trains
readers to skim the failure lines.
- **`/opt/docker/compose` ownership — nh3-dev is the fleet outlier.**
`root:root` here; `lkraven:lkraven` on irv-ml1, nh3-docker, ana-docker and
esh-docker-vm. So "can a project session deploy its own stack" is false only
on the box where sessions actually run. `/opt/docker/compose/talk` was created
`lkraven`-owned 2026-09-14 so tts-dev can migrate `talk` out of `~/talk`
(a convention violation that hides it from anything walking
`/opt/docker/compose/*/`). Normalising the parent is **unresolved — operator's
call**; `/opt/docker` itself is a separate three-way split (`755` root, `777`
`lkraven`-owned 2026-09-14 and **tts-dev migrated `talk` into it the same
day** — it had been at `~/talk`, a convention violation that hid it from
anything walking `/opt/docker/compose/*/`. Old path parked at
`~/talk.migrated-20260914`; `version: "3.8"` dropped from its compose file
(it existed only to satisfy the v1 CLI, which is now gone). Normalising the
parent directory is **unresolved — operator's call**; `/opt/docker` itself is a separate three-way split (`755` root, `777`
root on two hosts, `755` lkraven).
- **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd
behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`.