fix(upgrade-docker-ce): retry the stack restart under sudo before reporting FAILED
The restart loop runs as the deploy identity, not root, and a stack .env is allowed to be root-owned 0600. compose bails on the unreadable file before doing anything, so the stack was reported FAILED while restart=unless-stopped had already brought it back healthy — a false failure, which is worse than a quiet one because it trains readers to skim the failure lines. Retry under sudo -n before calling it a failure, and print compose's own output either way. Verified on nh3-dev against beszel: plain attempt rc=1 'open /opt/docker/compose/beszel/.env: permission denied', sudo retry rc=0 'Container beszel-agent Started', container back to healthy. The happy path is unchanged — the sudo attempt only fires after a failure. Also record that tts-dev migrated talk from ~/talk into /opt/docker/compose/talk, which removes the one stack on this host that was invisible to anything walking that path.
This commit is contained in:
@@ -204,7 +204,20 @@ steps:
|
|||||||
dir="/opt/docker/compose/$stack"
|
dir="/opt/docker/compose/$stack"
|
||||||
if [ -d "$dir" ]; then
|
if [ -d "$dir" ]; then
|
||||||
echo " starting $stack"
|
echo " starting $stack"
|
||||||
(cd "$dir" && docker compose up -d) || echo " FAILED — investigate $stack"
|
out=$(cd "$dir" && docker compose up -d 2>&1); rc=$?
|
||||||
|
if [ $rc -ne 0 ]; then
|
||||||
|
# This loop runs as the deploy identity, not root, and a stack
|
||||||
|
# .env is allowed to be root-owned 0600 — unreadable to us, so
|
||||||
|
# compose bails before it does anything. Retry under sudo before
|
||||||
|
# calling it a failure. (nh3-dev 2026-09-14: beszel reported
|
||||||
|
# "FAILED — investigate beszel" on exactly this while
|
||||||
|
# restart=unless-stopped had already brought it back healthy. A
|
||||||
|
# false FAILED in automation output is worse than a quiet one —
|
||||||
|
# it trains readers to skim the failure lines.)
|
||||||
|
out=$(cd "$dir" && sudo -n docker compose up -d 2>&1); rc=$?
|
||||||
|
fi
|
||||||
|
printf '%s\n' "$out"
|
||||||
|
[ $rc -eq 0 ] || echo " FAILED — investigate $stack"
|
||||||
fi
|
fi
|
||||||
done < /tmp/docker-pre-upgrade-stacks.txt
|
done < /tmp/docker-pre-upgrade-stacks.txt
|
||||||
true
|
true
|
||||||
|
|||||||
@@ -34,18 +34,25 @@ local Bash already executes here — no SSH-to-self needed for non-privileged wo
|
|||||||
user tables in every database**, so no data was lost. Lesson: capture
|
user tables in every database**, so no data was lost. Lesson: capture
|
||||||
`AutoRemove` and `RestartPolicy` together when snapshotting a container you
|
`AutoRemove` and `RestartPolicy` together when snapshotting a container you
|
||||||
are about to bounce.
|
are about to bounce.
|
||||||
- ⚠ The playbook's restart loop runs as `infra-ops` and cannot read a
|
- The playbook's restart loop runs as the deploy identity, not root, and a
|
||||||
root-owned `0600` stack `.env` (`/opt/docker/compose/beszel/.env`), so it
|
stack `.env` may legitimately be root-owned `0600`
|
||||||
reports that stack as FAILED even when `restart: unless-stopped` brings it
|
(`/opt/docker/compose/beszel/.env`) — compose bails before doing anything,
|
||||||
back fine. Playbook-side fix pending.
|
so the stack reported FAILED while `restart: unless-stopped` had already
|
||||||
|
brought it back healthy. **Fixed 2026-09-14**: the loop retries under
|
||||||
|
`sudo -n` before calling it a failure. Verified against beszel — plain
|
||||||
|
`rc=1 permission denied`, sudo retry `rc=0 Container beszel-agent Started`.
|
||||||
|
A false FAILED in automation output is worse than a quiet one; it trains
|
||||||
|
readers to skim the failure lines.
|
||||||
- **`/opt/docker/compose` ownership — nh3-dev is the fleet outlier.**
|
- **`/opt/docker/compose` ownership — nh3-dev is the fleet outlier.**
|
||||||
`root:root` here; `lkraven:lkraven` on irv-ml1, nh3-docker, ana-docker and
|
`root:root` here; `lkraven:lkraven` on irv-ml1, nh3-docker, ana-docker and
|
||||||
esh-docker-vm. So "can a project session deploy its own stack" is false only
|
esh-docker-vm. So "can a project session deploy its own stack" is false only
|
||||||
on the box where sessions actually run. `/opt/docker/compose/talk` was created
|
on the box where sessions actually run. `/opt/docker/compose/talk` was created
|
||||||
`lkraven`-owned 2026-09-14 so tts-dev can migrate `talk` out of `~/talk`
|
`lkraven`-owned 2026-09-14 and **tts-dev migrated `talk` into it the same
|
||||||
(a convention violation that hides it from anything walking
|
day** — it had been at `~/talk`, a convention violation that hid it from
|
||||||
`/opt/docker/compose/*/`). Normalising the parent is **unresolved — operator's
|
anything walking `/opt/docker/compose/*/`. Old path parked at
|
||||||
call**; `/opt/docker` itself is a separate three-way split (`755` root, `777`
|
`~/talk.migrated-20260914`; `version: "3.8"` dropped from its compose file
|
||||||
|
(it existed only to satisfy the v1 CLI, which is now gone). Normalising the
|
||||||
|
parent directory is **unresolved — operator's call**; `/opt/docker` itself is a separate three-way split (`755` root, `777`
|
||||||
root on two hosts, `755` lkraven).
|
root on two hosts, `755` lkraven).
|
||||||
- **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd
|
- **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd
|
||||||
behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`.
|
behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`.
|
||||||
|
|||||||
Reference in New Issue
Block a user