feat(fleet): normalize docker deploy trees to root:docker setgid

Operator ruling: root:docker, not a personal username and not a new admin
account. lkraven is one of three names he uses, so baking it into shared
infrastructure guarantees a stale owner later; a dedicated deploy account
buys nothing the existing docker group doesn't, since that group already
exists on every host holding exactly lkraven + infra-ops.

Applied to nh3-dev, nh3-docker, esh-docker-vm, irv-ml1, ana-docker. All
five now 2775 root:docker on /opt/docker and /opt/docker/compose. Clears
the 0777 on nh3-docker and ana-docker. 55 stack .env files normalized to
root:docker 0640, tightening 43 world-readable ones and opening 31 that
were readable by only one of the two deploy identities. No containers
bounced — inode metadata only, and .env is read at compose up.

Deliberately not a recursive chmod. Three acme.json files and an ssh
private key are mode 0600 and traefik/ssh refuse to start if that widens,
which would have been a delayed failure surfacing at the next restart
rather than now. Protection is both mode-based (0600/0400 untouched) and
name-based (acme.json, *.key, *.pem, *.pfx, id_*); modes are symbolic so
the 53 executable files in these trees keep their exec bit.

Two defects found and fixed mid-rollout. The name list was initially
reported but not enforced, so a .key already at 0644 on esh-docker-vm was
widened to 0664 — reverted, and the list is now enforced in the chgrp and
widening steps. And the exec-bit verify asserted every .sh is executable,
which was never true and false-FAILED irv-ml1; it now compares the
executable-file count against a recorded baseline.

playbooks/normalize-docker-tree.yaml

@@ -0,0 +1,210 @@
+# normalize-docker-tree — put every host's docker deploy tree on one
+# ownership scheme: root:docker, setgid, group-writable.
+#
+# Operator ruling 2026-09-14: root:docker, not a personal username and not a
+# new admin account. Rationale — `lkraven` is one of three names the operator
+# uses, so baking it into shared infrastructure guarantees a stale owner the
+# day he standardises on another; and a dedicated deploy account buys nothing
+# the existing `docker` group doesn't, since that group already exists on
+# every host and already contains exactly lkraven + infra-ops.
+#
+# ⚠ This does NOT create privilege separation. `docker` group membership is
+# root-equivalent (bind-mount / into a container). Both members already hold
+# NOPASSWD sudo, so today that costs nothing — but if a third party ever needs
+# to deploy one stack WITHOUT root, this scheme is not the tool for it. Stand
+# up a dedicated `deploy` group at that point.
+#
+# ⚠⚠ WHY THIS IS NOT A RECURSIVE CHMOD.
+# A blanket `chmod -R g+w` over these trees produces a DELAYED failure, which
+# is strictly worse than a loud one: the service keeps running on its current
+# file handles and dies at its next restart, weeks later, for a reason nobody
+# connects to this change. Measured on the fleet 2026-09-14:
+#   * 3 x acme.json (traefik-ana has two, traefik-esh one), mode 600.
+#     Traefik REFUSES TO START if acme.json is not 600.
+#   * /opt/docker/conf/asset-engine/ssh/id_ed25519, mode 600. ssh refuses a
+#     group-accessible private key.
+#   * 53 executable files (entrypoints, redeploy scripts, traefik.yml). A
+#     NUMERIC chmod strips +x; symbolic g+w does not. Modes here are symbolic
+#     for exactly that reason — do not "simplify" them to octal.
+# The rule that keeps this safe: only widen a file that is ALREADY
+# world-readable. If the world can read it, group-write leaks nothing new.
+# Anything at 0600/0400 is deliberate and is left alone.
+#
+# Usage:
+#   scripts/elway <host> --playbook playbooks/normalize-docker-tree.yaml --dry-run
+#   scripts/elway <host> --playbook playbooks/normalize-docker-tree.yaml
+#
+# Host order (ascending blast radius). ana-docker LAST — it is the only host
+# carrying traefik, the asset-engine ssh key and the gitea CI runner at once:
+#   nh3-dev(2) -> nh3-docker(7) -> esh-docker-vm(18) -> irv-ml1(36) -> ana-docker(38)
+#
+# Containers do NOT need bouncing. These are inode metadata changes; a running
+# process does not re-read them, and .env is consulted only at `compose up`.
+
+vars:
+  # Pruned from the walk entirely: vendored/CI/VCS trees where recursing is
+  # pointless churn and the files are not ours to re-own.
+  prune_dirs: ".venv .git host-workspace node_modules"
+  # Never touched by name, in ADDITION to the mode rule. The mode rule alone
+  # (leave 0600/0400) is not sufficient: a key file that is ALREADY 0644 would
+  # otherwise be picked up by the "already world-readable" widening, which is
+  # true but beside the point — we do not want to be the last writer on a
+  # credential either way. (Caught 2026-09-14 on esh-docker-vm:
+  # /opt/docker/conf/calibre-web/.key was 0644 and got widened to 0664 before
+  # this guard existed. Its world-readability is a pre-existing finding worth
+  # fixing on its own; it is not something this playbook should compound.)
+  protect_names: "acme.json *.key *.pem *.pfx id_*"
+
+steps:
+  - name: Preflight — docker group exists and holds the deploy identities
+    shell: |
+      getent group docker || { echo "NO docker GROUP — refusing"; exit 1; }
+      for u in lkraven infra-ops; do
+        id -nG "$u" 2>/dev/null | tr ' ' '\n' | grep -qx docker \
+          || echo "  WARN: $u is not in the docker group"
+      done
+    changed_when: "false"
+
+  - name: Snapshot current state (rollback reference + before/after evidence)
+    shell: |
+      out=/tmp/docker-tree-pre-normalize.txt
+      : | sudo tee "$out" >/dev/null
+      for d in /opt/docker /opt/docker/compose /opt/docker/conf; do
+        [ -e "$d" ] && sudo stat -c '%a %U:%G %n' "$d" | sudo tee -a "$out" >/dev/null
+      done
+      sudo find /opt/docker/compose /opt/docker/conf -maxdepth 2 -name '.env' \
+        -printf '%m %u:%g %p\n' 2>/dev/null | sudo tee -a "$out" >/dev/null
+      # Recorded so the exec-bit verify can compare before/after instead of
+      # asserting "every .sh is executable" — which was never true (plenty of
+      # .sh files here are sourced, not run) and produced a FALSE FAILED on
+      # irv-ml1 2026-09-14. A false failure in automation output is the exact
+      # defect this repo fixed in upgrade-docker-ce.yaml the same day.
+      if [ ! -f /tmp/docker-tree-execcount.txt ]; then
+        sudo find /opt/docker/compose /opt/docker/conf -type f -perm /0100 2>/dev/null \
+          | wc -l | sudo tee /tmp/docker-tree-execcount.txt >/dev/null
+      fi
+      echo "EXECCOUNT_BASELINE=$(cat /tmp/docker-tree-execcount.txt)"
+      cat "$out"
+    sudo: true
+    changed_when: "false"
+
+  - name: Record the permission-sensitive files this run must NOT touch
+    # Evidence, not action. If this list ever grows a new shape, the exclusion
+    # rules below need revisiting before the next host.
+    shell: |
+      sudo find /opt/docker/compose /opt/docker/conf \
+        \( -name .venv -o -name .git -o -name host-workspace -o -name node_modules \) -prune -o \
+        -type f \( -name 'acme.json' -o -name '*.key' -o -name '*.pem' -o -name '*.pfx' \
+                   -o -name 'id_*' -o -perm 0600 -o -perm 0400 \) \
+        -printf '  PROTECTED %m %u:%g %p\n' 2>/dev/null | head -40
+      echo "  (all of the above are left untouched by this playbook)"
+    sudo: true
+    changed_when: "false"
+
+  - name: Set group ownership to docker across the deploy tree
+    # chgrp only — no mode change. A 0600 file has no group bits, so this is a
+    # no-op for every protected file above while still making the tree
+    # collectively owned.
+    shell: |
+      for d in /opt/docker/compose /opt/docker/conf; do
+        [ -d "$d" ] || continue
+        sudo find "$d" \
+          \( -name .venv -o -name .git -o -name host-workspace -o -name node_modules \) -prune -o \
+          \( -name 'acme.json' -o -name '*.key' -o -name '*.pem' -o -name '*.pfx' -o -name 'id_*' \) -prune -o \
+          -print0 2>/dev/null | sudo xargs -0 -r chgrp docker
+      done
+      echo "  group set to docker"
+    sudo: true
+
+  - name: Set root ownership + setgid + group-write on DIRECTORIES
+    # setgid (g+s) is the whole point: every file created here afterwards
+    # inherits the docker group, so this fix does not decay.
+    shell: |
+      for d in /opt/docker/compose /opt/docker/conf; do
+        [ -d "$d" ] || continue
+        sudo find "$d" \
+          \( -name .venv -o -name .git -o -name host-workspace -o -name node_modules \) -prune -o \
+          -type d -print0 2>/dev/null | sudo xargs -0 -r chown root
+        sudo find "$d" \
+          \( -name .venv -o -name .git -o -name host-workspace -o -name node_modules \) -prune -o \
+          -type d -print0 2>/dev/null | sudo xargs -0 -r chmod g+ws
+      done
+      echo "  directories: root:docker, setgid, group-writable"
+    sudo: true
+
+  - name: Group-write ONLY files that are already world-readable
+    # SYMBOLIC mode — a numeric chmod would strip +x from the 53 executable
+    # files in these trees. -perm -o=r is the safety rule: widening a file the
+    # world can already read adds no exposure.
+    shell: |
+      for d in /opt/docker/compose /opt/docker/conf; do
+        [ -d "$d" ] || continue
+        sudo find "$d" \
+          \( -name .venv -o -name .git -o -name host-workspace -o -name node_modules \) -prune -o \
+          \( -name 'acme.json' -o -name '*.key' -o -name '*.pem' -o -name '*.pfx' -o -name 'id_*' \) -prune -o \
+          -type f -perm -o=r -print0 2>/dev/null | sudo xargs -0 -r chmod g+w
+      done
+      echo "  world-readable files: group-writable (exec bits preserved)"
+    sudo: true
+
+  - name: Normalise stack .env to root:docker 0640
+    # The file that started this. Fleet-wide these were split between 0600
+    # (readable by ONE of the two deploy identities, varying by file — this is
+    # what false-FAILED beszel during the docker-ce upgrade) and 0644
+    # (world-readable secrets). 0640 tightens the latter and opens the former
+    # to exactly the two deployers.
+    shell: |
+      n=0
+      for f in $(sudo find /opt/docker/compose -maxdepth 2 -name '.env' 2>/dev/null); do
+        sudo chown root:docker "$f" && sudo chmod 0640 "$f" && n=$((n+1))
+      done
+      echo "  normalised $n stack .env file(s) to root:docker 0640"
+    sudo: true
+
+  - name: Fix the /opt/docker parent (kills 0777 where present)
+    shell: |
+      before=$(stat -c '%a %U:%G' /opt/docker)
+      sudo chown root:docker /opt/docker
+      sudo chmod 2775 /opt/docker
+      echo "  /opt/docker: $before -> $(stat -c '%a %U:%G' /opt/docker)"
+    sudo: true
+
+verify:
+  - name: /opt/docker is root:docker, setgid, not world-writable
+    shell: |
+      stat -c '%a %U:%G' /opt/docker | grep -qE '^2775 root:docker$'
+
+  - name: compose tree directories are setgid + group-writable
+    shell: |
+      [ -d /opt/docker/compose ] || exit 0
+      bad=$(sudo find /opt/docker/compose \
+              \( -name .venv -o -name .git -o -name host-workspace -o -name node_modules \) -prune -o \
+              -type d ! -perm -g+w -print 2>/dev/null | head -3)
+      [ -z "$bad" ] || { echo "not group-writable: $bad"; exit 1; }
+    sudo: true
+
+  - name: every protected 0600 file kept its mode (traefik acme.json, ssh keys)
+    shell: |
+      bad=$(sudo find /opt/docker/conf -type f \
+              \( -name 'acme.json' -o -name 'id_ed25519' -o -name 'id_rsa' \) \
+              ! -perm 0600 -print 2>/dev/null | head -3)
+      [ -z "$bad" ] || { echo "MODE WIDENED on protected file: $bad"; exit 1; }
+    sudo: true
+
+  - name: no stack .env is world-readable any more
+    shell: |
+      bad=$(sudo find /opt/docker/compose -maxdepth 2 -name '.env' -perm -o=r -print 2>/dev/null | head -3)
+      [ -z "$bad" ] || { echo "world-readable .env: $bad"; exit 1; }
+    sudo: true
+
+  - name: executable-file count is unchanged (no exec bit was stripped)
+    # Compares against the baseline captured before the first run on this host.
+    # NOT "every .sh is executable" — that property was never true and asserting
+    # it manufactures a failure on a tree full of sourced scripts.
+    shell: |
+      [ -f /tmp/docker-tree-execcount.txt ] || exit 0
+      before=$(cat /tmp/docker-tree-execcount.txt)
+      after=$(sudo find /opt/docker/compose /opt/docker/conf -type f -perm /0100 2>/dev/null | wc -l)
+      [ "$before" = "$after" ] || { echo "exec count $before -> $after"; exit 1; }
+      echo "  exec-file count unchanged at $after"
+    sudo: true

servers/nh3-dev/README.md

@@ -43,10 +43,20 @@ local Bash already executes here — no SSH-to-self needed for non-privileged wo
     `rc=1 permission denied`, sudo retry `rc=0 Container beszel-agent Started`.
     A false FAILED in automation output is worse than a quiet one; it trains
     readers to skim the failure lines.
-- **`/opt/docker/compose` ownership — nh3-dev is the fleet outlier.**
+- **`/opt/docker` ownership — normalised fleet-wide 2026-09-14 to
-  `root:root` here; `lkraven:lkraven` on irv-ml1, nh3-docker, ana-docker and
+  `root:docker 2775`** (setgid) via `playbooks/normalize-docker-tree.yaml`,
-  esh-docker-vm. So "can a project session deploy its own stack" is false only
+  operator ruling. Was a three-way split: `root:root 755` here, `root:root 777`
-  on the box where sessions actually run. `/opt/docker/compose/talk` was created
+  on nh3-docker + ana-docker (world-writable, from a 2024 `chmod -R 777` to get
+  a git clone working), `lkraven 755` elsewhere. Not a personal username
+  (`lkraven` is one of three the operator uses) and not a new admin account —
+  the `docker` group already existed on every host holding exactly `lkraven` +
+  `infra-ops`. ⚠ This is **not** privilege separation: `docker` membership is
+  root-equivalent. A future non-root deployer needs a dedicated `deploy` group.
+  Stack `.env` files went to `root:docker 0640` — previously 31 of 74 were
+  `0600` readable by only ONE of the two deploy identities (varying by file,
+  which is what false-FAILED beszel during the docker-ce upgrade) and 43 were
+  world-readable `0644`. **No containers were bounced**; these are inode
+  metadata changes and `.env` is read only at `compose up`. `/opt/docker/compose/talk` was created
   `lkraven`-owned 2026-09-14 and **tts-dev migrated `talk` into it the same
   day** — it had been at `~/talk`, a convention violation that hid it from
   anything walking `/opt/docker/compose/*/`. Old path parked at