playbooks/upgrade-docker-ce: bake in the override.conf rewrite from nh3-docker debug

After nh3-docker's swap, two systemd unit gotchas surfaced that the
playbook now handles automatically:

* The docker.io-era /etc/systemd/system/docker.service.d/override.conf
  hardcoded ExecStart=/usr/sbin/dockerd; docker-ce installs at
  /usr/bin/dockerd → daemon failed status=203/EXEC.
* The shipped docker-ce unit's ExecStart=dockerd -H fd:// conflicts
  with daemon.json hosts: (defined for the 0.0.0.0:2375 homepage
  discovery binding) → "conflicting host options".

The "Rewrite docker.service drop-in" step now backs up any existing
override, probes daemon.json for a hosts: setting, and installs an
override that strips -H from ExecStart when needed. Also added an
explicit systemctl reset-failed step to clear the start-rate-limit
state that 3 failed install-time starts leave behind.

configs/homepage/docker.yaml: comment out irv-ml1-docker provider —
20s-per-poll ETIMEDOUTs from the stalled host were drowning homepage's
logs and apparently blocking ana-pfi-docker discovery (the Miniflux
card in the News group wouldn't render until removal). Re-enable when
irv-ml1 is back.

STATUS.md: new "Active migration" section tracking the docker-ce
rollout — nh3-docker done; esh-docker-vm + ana-docker queued.

STATUS.md

@@ -317,6 +317,39 @@ Once recovered, update this section with what fixed it and remove the
     rule: refresh STATUS.md at the end of any session where a 🟧 or 🟥
     item closes, or three+ smaller items land.
 
+## Active migration — docker.io 20.10 → docker-ce 29.x
+
+Bookworm's Debian-packaged `docker.io` is pinned at 20.10.24 with
+client/daemon API 1.41. Newer compose clients (1.52+) refuse to talk
+to it ("client version 1.52 is too new" during `docker compose
+build`). The migration moves each host to Docker's official `docker-ce`
+APT repo (29.x). Playbook: `playbooks/upgrade-docker-ce.yaml`.
+
+Order (least → most blast radius):
+
+1. ~~**nh3-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, two
+   stacks (portainer agent + portainer) restarted clean. Debug surfaced
+   two unit-file gotchas now baked into the playbook:
+   - `/etc/systemd/system/docker.service.d/override.conf` from the
+     docker.io era hardcoded `/usr/sbin/dockerd` (docker-ce installs
+     at `/usr/bin/dockerd`) → daemon failed status=203/EXEC.
+   - The shipped docker-ce unit's `ExecStart=dockerd -H fd:// …`
+     conflicts with `daemon.json` `hosts:` (defined for the 0.0.0.0:2375
+     homepage-discovery binding) → "conflicting host options".
+   Playbook now backs up any existing override, probes `daemon.json`
+   for `hosts:`, and installs an override that strips `-H` when needed.
+2. **esh-docker-vm** — pending. Run from your TTY:
+   `scripts/elway esh-docker-vm --playbook playbooks/upgrade-docker-ce.yaml`
+3. **ana-docker** — pending. Largest blast radius (homepage, miniflux,
+   news-digest, synapse, seafile, vaultwarden, gitea, openwebui, …).
+   Verify each card / service after.
+
+Also during the nh3-docker debug: `configs/homepage/docker.yaml`
+commented out the `irv-ml1-docker` provider — the 20s-per-poll
+ETIMEDOUTs from the stalled host were drowning homepage's logs and
+appeared to block ana-pfi-docker discovery (News group's Miniflux card
+wouldn't render until removal). Re-enable when irv-ml1 is back.
+
 ## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm)
 
 Morning / early afternoon — the original tooling day:

configs/homepage/docker.yaml

@@ -24,9 +24,16 @@ ana-ml2-docker:
 # irv-ml1 — reachable ONLY via WireGuard tunnel, so docker auto-discovery
 # traffic traverses the tunnel. If WG drops, homepage will show irv-ml1
 # services as offline until reconnect.
-irv-ml1-docker:
+#
-  host: 10.100.79.3
+# DISABLED 2026-04-26 because the host is down post-CUDA-upgrade
-  port: 2375
+# (see STATUS.md "🟥 Blocked — irv-ml1 stalled"). The 20s-per-poll
+# ETIMEDOUTs were drowning homepage's logs and apparently blocking
+# the ana-pfi-docker discovery from completing — News group's Miniflux
+# card never surfaced until this was commented out. Re-enable once
+# irv-ml1 is back on the network.
+# irv-ml1-docker:
+#   host: 10.100.79.3
+#   port: 2375
 
 # Example TLS socket (if/when a host moves off plaintext 2375):
 # ana-pfi-docker:

playbooks/upgrade-docker-ce.yaml

@@ -126,29 +126,63 @@ steps:
     shell: DEBIAN_FRONTEND=noninteractive apt-get install -y {{ pkgs_to_install }}
     sudo: true
 
-  - name: Move aside any stale docker.service drop-in override
+  - name: Rewrite docker.service drop-in to match the new package layout
-    # The docker.io era often left an /etc/systemd/system/docker.service.d/
+    # Two related fixes for the systemd unit, learned the hard way on
-    # override.conf adding `-H tcp://...` to ExecStart for remote docker
+    # nh3-docker. The docker.io era left an
-    # discovery. After the swap, the override (a) hard-codes
+    # /etc/systemd/system/docker.service.d/override.conf hardcoding
-    # /usr/sbin/dockerd which no longer exists, and (b) duplicates the
+    # ExecStart=/usr/sbin/dockerd (Debian package path) and usually
-    # `hosts:` setting in /etc/docker/daemon.json — dockerd refuses to
+    # adding `-H tcp://...` for remote discovery.
-    # start when both are set ("conflicting host options"). daemon.json
+    #
-    # is the modern way; the override is redundant.
+    # After the swap to docker-ce two things are wrong:
+    #   1. /usr/sbin/dockerd doesn't exist — docker-ce installs at
+    #      /usr/bin/dockerd. Daemon fails status=203/EXEC.
+    #   2. If daemon.json defines `hosts:` (typical here for the
+    #      0.0.0.0:2375 homepage-discovery binding), then ANY `-H`
+    #      flag in ExecStart conflicts: dockerd refuses to start with
+    #      "conflicting host options".
+    #
+    # The shipped docker-ce unit's ExecStart is `dockerd -H fd://
+    # --containerd=/run/containerd/containerd.sock` — that `-H fd://`
+    # ALSO conflicts with daemon.json hosts:. So we always need an
+    # override that strips `-H` entirely when daemon.json defines
+    # hosts.
+    #
+    # Procedure:
+    #   * Back up any existing override (preserves it as .pre-upgrade)
+    #   * Probe /etc/docker/daemon.json for a `hosts:` setting
+    #   * If hosts: is defined → install an override that clears the
+    #     base unit's ExecStart and replaces it WITHOUT -H
+    #   * Otherwise no override is needed (base unit's `-H fd://` is fine)
     shell: |
-      f=/etc/systemd/system/docker.service.d/override.conf
+      d=/etc/systemd/system/docker.service.d
+      f="$d/override.conf"
+      sudo install -d -m 0755 "$d"
       if [ -f "$f" ]; then
-        echo "  moving aside $f"
+        echo "  backing up existing $f → ${f}.pre-upgrade"
         sudo mv "$f" "${f}.pre-upgrade"
       fi
+      if [ -f /etc/docker/daemon.json ] && grep -q '"hosts"' /etc/docker/daemon.json; then
+        echo "  daemon.json has hosts: — installing override that strips -H from ExecStart"
+        sudo tee "$f" >/dev/null <<'OVR'
+      [Service]
+      ExecStart=
+      ExecStart=/usr/bin/dockerd --containerd=/run/containerd/containerd.sock
+      OVR
+      else
+        echo "  daemon.json has no hosts: — base unit's ExecStart is fine, no override needed"
+      fi
     sudo: true
 
-  - name: systemctl daemon-reload (CRITICAL — package swap put dockerd at
+  - name: systemctl daemon-reload (so the new unit + override are loaded)
-      /usr/bin/dockerd; without reload, systemd keeps the old unit's
-      ExecStart pointing at /usr/sbin/dockerd from docker.io and fails
-      with status=203/EXEC "No such file or directory")
     shell: systemctl daemon-reload
     sudo: true
 
+  - name: Reset the docker.service failed state (3 failed starts during
+      the install often leaves it in start-rate-limited state — reset
+      so the next start isn't immediately denied)
+    shell: systemctl reset-failed docker.service
+    sudo: true
+
   - name: Enable + start the new daemon (idempotent)
     shell: systemctl enable --now docker
     sudo: true