From a3ab1a7b6ce01fd1e89ff2e1e9b458fdb2bb5138 Mon Sep 17 00:00:00 2001
From: Vuong Hoang <vh@phasefinal.com>
Date: Sun, 26 Apr 2026 14:03:06 -0700
Subject: [PATCH] playbooks/upgrade-docker-ce: detect + move aside stale unit
 override
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

nh3-docker's daemon kept failing post-package-swap with status=203
even after daemon-reload. Root cause: a stale
/etc/systemd/system/docker.service.d/override.conf from the docker.io
era hardcoding ExecStart=/usr/sbin/dockerd. The override (a) points
at the no-longer-existing path, AND (b) typically also adds
-H tcp://... which now duplicates the hosts: setting in
/etc/docker/daemon.json — dockerd refuses to start when both define
hosts ('conflicting host options').

Daemon.json is the modern way to expose the TCP socket. The
override is redundant and wrong. Move it aside (preserve a
.pre-upgrade copy for forensics), then daemon-reload, then start.

Should let esh-docker-vm and ana-docker upgrades go through cleanly
without the manual debug loop nh3-docker required.
---
 playbooks/upgrade-docker-ce.yaml | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/playbooks/upgrade-docker-ce.yaml b/playbooks/upgrade-docker-ce.yaml
index c1d4e25..66a4aa6 100644
--- a/playbooks/upgrade-docker-ce.yaml
+++ b/playbooks/upgrade-docker-ce.yaml
@@ -126,6 +126,22 @@ steps:
     shell: DEBIAN_FRONTEND=noninteractive apt-get install -y {{ pkgs_to_install }}
     sudo: true
 
+  - name: Move aside any stale docker.service drop-in override
+    # The docker.io era often left an /etc/systemd/system/docker.service.d/
+    # override.conf adding `-H tcp://...` to ExecStart for remote docker
+    # discovery. After the swap, the override (a) hard-codes
+    # /usr/sbin/dockerd which no longer exists, and (b) duplicates the
+    # `hosts:` setting in /etc/docker/daemon.json — dockerd refuses to
+    # start when both are set ("conflicting host options"). daemon.json
+    # is the modern way; the override is redundant.
+    shell: |
+      f=/etc/systemd/system/docker.service.d/override.conf
+      if [ -f "$f" ]; then
+        echo "  moving aside $f"
+        sudo mv "$f" "${f}.pre-upgrade"
+      fi
+    sudo: true
+
   - name: systemctl daemon-reload (CRITICAL — package swap put dockerd at
       /usr/bin/dockerd; without reload, systemd keeps the old unit's
       ExecStart pointing at /usr/sbin/dockerd from docker.io and fails