From be9109bc95df2ef11e1a3066f6f236f9a969f89e Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 26 Apr 2026 14:12:06 -0700 Subject: [PATCH] playbooks/upgrade-docker-ce: bake in the override.conf rewrite from nh3-docker debug MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit After nh3-docker's swap, two systemd unit gotchas surfaced that the playbook now handles automatically: * The docker.io-era /etc/systemd/system/docker.service.d/override.conf hardcoded ExecStart=/usr/sbin/dockerd; docker-ce installs at /usr/bin/dockerd → daemon failed status=203/EXEC. * The shipped docker-ce unit's ExecStart=dockerd -H fd:// conflicts with daemon.json hosts: (defined for the 0.0.0.0:2375 homepage discovery binding) → "conflicting host options". The "Rewrite docker.service drop-in" step now backs up any existing override, probes daemon.json for a hosts: setting, and installs an override that strips -H from ExecStart when needed. Also added an explicit systemctl reset-failed step to clear the start-rate-limit state that 3 failed install-time starts leave behind. configs/homepage/docker.yaml: comment out irv-ml1-docker provider — 20s-per-poll ETIMEDOUTs from the stalled host were drowning homepage's logs and apparently blocking ana-pfi-docker discovery (the Miniflux card in the News group wouldn't render until removal). Re-enable when irv-ml1 is back. STATUS.md: new "Active migration" section tracking the docker-ce rollout — nh3-docker done; esh-docker-vm + ana-docker queued. --- STATUS.md | 33 +++++++++++++++++ configs/homepage/docker.yaml | 13 +++++-- playbooks/upgrade-docker-ce.yaml | 62 ++++++++++++++++++++++++-------- 3 files changed, 91 insertions(+), 17 deletions(-) diff --git a/STATUS.md b/STATUS.md index 43e95ab..32c66cb 100644 --- a/STATUS.md +++ b/STATUS.md @@ -317,6 +317,39 @@ Once recovered, update this section with what fixed it and remove the rule: refresh STATUS.md at the end of any session where a 🟧 or 🟥 item closes, or three+ smaller items land. +## Active migration — docker.io 20.10 → docker-ce 29.x + +Bookworm's Debian-packaged `docker.io` is pinned at 20.10.24 with +client/daemon API 1.41. Newer compose clients (1.52+) refuse to talk +to it ("client version 1.52 is too new" during `docker compose +build`). The migration moves each host to Docker's official `docker-ce` +APT repo (29.x). Playbook: `playbooks/upgrade-docker-ce.yaml`. + +Order (least → most blast radius): + +1. ~~**nh3-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, two + stacks (portainer agent + portainer) restarted clean. Debug surfaced + two unit-file gotchas now baked into the playbook: + - `/etc/systemd/system/docker.service.d/override.conf` from the + docker.io era hardcoded `/usr/sbin/dockerd` (docker-ce installs + at `/usr/bin/dockerd`) → daemon failed status=203/EXEC. + - The shipped docker-ce unit's `ExecStart=dockerd -H fd:// …` + conflicts with `daemon.json` `hosts:` (defined for the 0.0.0.0:2375 + homepage-discovery binding) → "conflicting host options". + Playbook now backs up any existing override, probes `daemon.json` + for `hosts:`, and installs an override that strips `-H` when needed. +2. **esh-docker-vm** — pending. Run from your TTY: + `scripts/elway esh-docker-vm --playbook playbooks/upgrade-docker-ce.yaml` +3. **ana-docker** — pending. Largest blast radius (homepage, miniflux, + news-digest, synapse, seafile, vaultwarden, gitea, openwebui, …). + Verify each card / service after. + +Also during the nh3-docker debug: `configs/homepage/docker.yaml` +commented out the `irv-ml1-docker` provider — the 20s-per-poll +ETIMEDOUTs from the stalled host were drowning homepage's logs and +appeared to block ana-pfi-docker discovery (News group's Miniflux card +wouldn't render until removal). Re-enable when irv-ml1 is back. + ## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm) Morning / early afternoon — the original tooling day: diff --git a/configs/homepage/docker.yaml b/configs/homepage/docker.yaml index f47ca10..8aba4fe 100644 --- a/configs/homepage/docker.yaml +++ b/configs/homepage/docker.yaml @@ -24,9 +24,16 @@ ana-ml2-docker: # irv-ml1 — reachable ONLY via WireGuard tunnel, so docker auto-discovery # traffic traverses the tunnel. If WG drops, homepage will show irv-ml1 # services as offline until reconnect. -irv-ml1-docker: - host: 10.100.79.3 - port: 2375 +# +# DISABLED 2026-04-26 because the host is down post-CUDA-upgrade +# (see STATUS.md "🟥 Blocked — irv-ml1 stalled"). The 20s-per-poll +# ETIMEDOUTs were drowning homepage's logs and apparently blocking +# the ana-pfi-docker discovery from completing — News group's Miniflux +# card never surfaced until this was commented out. Re-enable once +# irv-ml1 is back on the network. +# irv-ml1-docker: +# host: 10.100.79.3 +# port: 2375 # Example TLS socket (if/when a host moves off plaintext 2375): # ana-pfi-docker: diff --git a/playbooks/upgrade-docker-ce.yaml b/playbooks/upgrade-docker-ce.yaml index 66a4aa6..38600b0 100644 --- a/playbooks/upgrade-docker-ce.yaml +++ b/playbooks/upgrade-docker-ce.yaml @@ -126,29 +126,63 @@ steps: shell: DEBIAN_FRONTEND=noninteractive apt-get install -y {{ pkgs_to_install }} sudo: true - - name: Move aside any stale docker.service drop-in override - # The docker.io era often left an /etc/systemd/system/docker.service.d/ - # override.conf adding `-H tcp://...` to ExecStart for remote docker - # discovery. After the swap, the override (a) hard-codes - # /usr/sbin/dockerd which no longer exists, and (b) duplicates the - # `hosts:` setting in /etc/docker/daemon.json — dockerd refuses to - # start when both are set ("conflicting host options"). daemon.json - # is the modern way; the override is redundant. + - name: Rewrite docker.service drop-in to match the new package layout + # Two related fixes for the systemd unit, learned the hard way on + # nh3-docker. The docker.io era left an + # /etc/systemd/system/docker.service.d/override.conf hardcoding + # ExecStart=/usr/sbin/dockerd (Debian package path) and usually + # adding `-H tcp://...` for remote discovery. + # + # After the swap to docker-ce two things are wrong: + # 1. /usr/sbin/dockerd doesn't exist — docker-ce installs at + # /usr/bin/dockerd. Daemon fails status=203/EXEC. + # 2. If daemon.json defines `hosts:` (typical here for the + # 0.0.0.0:2375 homepage-discovery binding), then ANY `-H` + # flag in ExecStart conflicts: dockerd refuses to start with + # "conflicting host options". + # + # The shipped docker-ce unit's ExecStart is `dockerd -H fd:// + # --containerd=/run/containerd/containerd.sock` — that `-H fd://` + # ALSO conflicts with daemon.json hosts:. So we always need an + # override that strips `-H` entirely when daemon.json defines + # hosts. + # + # Procedure: + # * Back up any existing override (preserves it as .pre-upgrade) + # * Probe /etc/docker/daemon.json for a `hosts:` setting + # * If hosts: is defined → install an override that clears the + # base unit's ExecStart and replaces it WITHOUT -H + # * Otherwise no override is needed (base unit's `-H fd://` is fine) shell: | - f=/etc/systemd/system/docker.service.d/override.conf + d=/etc/systemd/system/docker.service.d + f="$d/override.conf" + sudo install -d -m 0755 "$d" if [ -f "$f" ]; then - echo " moving aside $f" + echo " backing up existing $f → ${f}.pre-upgrade" sudo mv "$f" "${f}.pre-upgrade" fi + if [ -f /etc/docker/daemon.json ] && grep -q '"hosts"' /etc/docker/daemon.json; then + echo " daemon.json has hosts: — installing override that strips -H from ExecStart" + sudo tee "$f" >/dev/null <<'OVR' + [Service] + ExecStart= + ExecStart=/usr/bin/dockerd --containerd=/run/containerd/containerd.sock + OVR + else + echo " daemon.json has no hosts: — base unit's ExecStart is fine, no override needed" + fi sudo: true - - name: systemctl daemon-reload (CRITICAL — package swap put dockerd at - /usr/bin/dockerd; without reload, systemd keeps the old unit's - ExecStart pointing at /usr/sbin/dockerd from docker.io and fails - with status=203/EXEC "No such file or directory") + - name: systemctl daemon-reload (so the new unit + override are loaded) shell: systemctl daemon-reload sudo: true + - name: Reset the docker.service failed state (3 failed starts during + the install often leaves it in start-rate-limited state — reset + so the next start isn't immediately denied) + shell: systemctl reset-failed docker.service + sudo: true + - name: Enable + start the new daemon (idempotent) shell: systemctl enable --now docker sudo: true