diff --git a/CLAUDE.md b/CLAUDE.md index 552d124..f47621e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -266,7 +266,7 @@ Observed and standardized across servers: - ESH home-lab workloads (`esteban.net`) → `esh-docker-vm` (general) or `vm-esh-nas` (needs direct NFS mounts from 10.0.50.50). Not part of the PFI colo topology, but shares monitoring/backup tooling. - Cross-site services (e.g. Beszel hub, Dozzle hub) live on `ana-docker` and pull from agents on the other hosts. - **SureFire (SF) client hosts** (`sf-*`, `sfsrv-ana`) are PFI-managed under the hosting agreement — SSH, OS ops, backups are PFI's responsibility. Hardware and data belong to the client; coordinate anything that affects data with them. -- **Worldtree-team dev VM** (`corviduo-dev`) is PFI-hosted (Anaheim subnet) but Worldtree-team-managed at the OS / application layer. PFI handles networking + emergency-ops backstop; OS configuration + deploy workflows + backup decisions live with the architect's team. Treat data-affecting work like SF hosts — coordinate before touching. +- **Worldtree-team dev VM** (`corviduo-dev`) is PFI-hosted (Anaheim subnet). **Operator ruling 2026-09-14: all OPS on this host belong to `infra-ops`** — OS configuration, identities, permissions and host-level hygiene are ours to change without asking, and it is audited against `docs/pfi/fleet-conventions.md` like any other fleet host. (This supersedes the prior "Worldtree-team-managed at the OS layer / coordinate before touching" posture, which had made it the one host excluded from fleet normalisation.) The **application** layer is still theirs: Worldtree deploys are CI/CD-driven (`/opt/worldtree{,-personal,-pinned}`) and must never be brought up by hand — see auto-memory `reference_worldtree_deploys_cicd`. Data-affecting work still gets coordinated. - **Hypervisors** (`pfi-pve`, `nh3-pve`, `esh-pve`, `esh-pve-nas`) are tracked for inventory / capacity planning. Don't deploy Docker stacks directly on them; new workloads land as VMs. `server_inspect.sh` captures host-level detail only — VM/LXC/ZFS enumeration needs Proxmox-native tooling (`qm list`, `pvesh get …`, `zpool list`). ## How to refresh a server's state diff --git a/docs/pfi/fleet-conventions.md b/docs/pfi/fleet-conventions.md new file mode 100644 index 0000000..41a8a33 --- /dev/null +++ b/docs/pfi/fleet-conventions.md @@ -0,0 +1,197 @@ +# Fleet conventions — identities, groups, paths + +Ratified by the operator 2026-09-14. This is the **pin**: new hosts are built +to it, existing hosts converge opportunistically. It is deliberately *not* a +migration mandate — see § Non-goals. + +Audit any host against it with: + +```bash +scripts/elway --playbook playbooks/audit-host-conventions.yaml +``` + +--- + +## 1. Identity + +### 1.1 The human account + +| | | +|---|---| +| **Standard** | `vh`, UID/GID 1000 | +| **Legacy** | `lkraven` on nh3-dev, nh3-docker, nh3-extdev, irv-ml1, ana-docker, esh-docker-vm | + +`vh` matches the operator's mail (`vh@phasefinal.com`), git identity and Gitea +namespace (`vh/*`), so it is the name everything else already uses. + +⚠ **Do not retro-rename the six legacy hosts.** Renaming UID 1000 where the home +directory is populated, systemd user services are lingering, SSH configs point +at it and live agent sessions are running is real blast radius for a cosmetic +gain — and the thing that actually mattered (a personal username owning *shared* +infrastructure) was removed on 2026-09-14 when the docker trees went to +`root:docker`. New hosts get `vh`; old hosts keep `lkraven` until there is an +independent reason to rebuild them. + +### 1.2 The ops account + +| | | +|---|---| +| **Name** | `infra-ops` — on every host, no exceptions | +| **UID/GID** | **850** (pinned; see § 3) | +| **Sudo** | `/etc/sudoers.d/infra-ops` → `infra-ops ALL=(ALL) NOPASSWD:ALL` | +| **Home** | `/home/infra-ops`, owned `infra-ops:infra-ops`, mode **0700** | +| **Groups** | `docker` where Docker is installed | + +`infra-ops` is the fleet's ops identity and is the same party as the `infra-ops` +althing handle. Mode `0700` is not paranoia: these homes hold credentials +(`~/.config/worldtree/personal-admin-token`, `~/.config/cloudflare/…`, +`~/.config/secrets-broker/bootstrap.env`). Verified safe before adoption — +nh3-extdev and corviduo-dev had already been running at `0700` for months, +including systemd units that read from that home. + +⚠ The UID is currently **1001 / 1002 / 1003 / 2001** across the fleet. See § 3 +for why that is tolerable today and what it costs. + +### 1.3 Service accounts + +| | | +|---|---| +| **Name** | `svc-` — e.g. `svc-arbo`, `svc-brokkr`, `svc-llm` | +| **UID/GID** | allocated from **800–849** | +| **Shell** | `/usr/sbin/nologin` unless the account genuinely needs to log in | +| **Home** | `/var/lib/` for daemons; `/home/svc-` only if it needs one | + +```bash +useradd --system --uid 8NN --user-group --shell /usr/sbin/nologin \ + --home-dir /var/lib/ --create-home svc- +``` + +**Legacy, not renamed:** `llmuser`, `sduser`, `brokkr`, `arbotrain`, `nas`, +`deploy`. Six accounts, six naming styles, all sitting in the human UID range +(1000+) where `/etc/passwd` cannot distinguish them from people. That is the +defect the convention fixes going forward. + +⚠ **A service account must not hold interactive root.** As of 2026-09-14 +`llmuser` is in the `sudo` group on ana-docker. Flagged, not yet resolved. + +--- + +## 2. Groups + +| group | GID | membership | +|---|---|---| +| `docker` | **851** (pinned) | the human account + `infra-ops` | +| `sudo` | 27 (Debian default) | the human account only | + +⚠ **`docker` membership is root-equivalent** — anyone in it can bind-mount `/` +into a container. The group makes access *consistent*; it does not create +privilege separation. A future non-root deployer (a contractor, a tenant agent) +needs a dedicated `deploy` group, not an invitation to `docker`. + +Current GIDs are **109 / 110 / 112 / 989** — assigned by `docker-ce` at install +time. Pinning means `groupadd -g 851 docker` *before* installing Docker. + +--- + +## 3. UID/GID allocation map + +Measured across eight hosts on 2026-09-14: dynamically-allocated system accounts +cluster in **989–999** and descend. **800–899 is free on every host**, which is +what makes it safe to pin there. + +``` + 0–99 OS reserved + 100–799 Debian dynamic system allocation — never pin here + 800–849 svc-* service accounts (fleet-pinned) + 850 infra-ops (uid + gid) (fleet-pinned) + 851 docker (gid) (fleet-pinned) + 852–899 reserved for fleet-wide groups + 900–999 Debian allocates system accounts here — leave clear + 1000 the human account (vh) + 1001+ additional humans +``` + +### Why the existing drift is tolerable + +There is **no central identity** anywhere on the fleet — every host is +`passwd: files systemd`. So a UID only has to be consistent where files cross +hosts. They do: `/mnt/smithy` is NFS-mounted on both nh3-dev (`infra-ops`=1001) +and irv-ml1 (`infra-ops`=1003), so UID 1001 on that export means `infra-ops` on +one box and `llmuser` on the other. + +**That collision is not currently biting, and the reason is the finding:** the +export is owned by the Synology's own UIDs (1024, 1031) which resolve on +*neither* host, and every directory on it is `0777`. Cross-host sharing works +today *because* the permissions are wide open. + +So aligning UIDs does not fix something broken — it **earns the right to drop +the 777 on shared storage**. That is a real improvement and a real project. It +is not a prerequisite for anything currently running. + +--- + +## 4. Paths + +| path | ownership | mode | contents | +|---|---|---|---| +| `/opt/docker` | `root:docker` | `2775` | the deploy tree root | +| `/opt/docker/compose//` | `root:docker` | `2775` | `compose.yaml`, `.env` | +| `/opt/docker/conf//` | `root:docker` | `2775` | bind-mounted config | +| `/opt/docker/compose//.env` | `root:docker` | **`0640`** | stack secrets | +| `/home/` | `:` | `0700` | | + +The setgid bit on directories is load-bearing: it makes every file created +afterwards inherit the `docker` group, so the scheme does not decay back into +whoever-touched-it-last ownership. + +⚠ **A stack lives under `/opt/docker/compose//` or it is invisible.** +Every piece of fleet automation walks that path. `talk` sat at `~/talk` until +2026-09-14 and was silently skipped by the docker-ce upgrade's restart loop for +exactly that reason. A stack outside the convention is not merely untidy — it is +excluded from every tool that will ever be written. + +### Permission-sensitive files — never widen these + +Some daemons **refuse to start** when their files are group-accessible, and they +fail at their *next restart* rather than at the moment of the change. That +delayed detonation is worse than a loud failure, so these are excluded by name +*and* by mode from `normalize-docker-tree.yaml`: + +| file | required | breaks | +|---|---|---| +| `acme.json` | `0600` | traefik refuses to start | +| `id_*`, `*.key`, `*.pem`, `*.pfx` | `0600` | ssh refuses a group-readable private key | + +Anything already at `0600`/`0400` is deliberate. Leave it. + +--- + +## 5. Non-goals + +- **Not a migration mandate.** Existing hosts converge when there is an + independent reason to touch them. Nothing here justifies a flag day. +- **Not privilege separation.** Both `docker` membership and `infra-ops` sudo are + root-equivalent today. This document standardises *consistency*, not least + privilege. +- **Not a central identity system.** LDAP/SSSD would make § 3 moot. Deliberately + out of scope — the fleet is small enough that a pinned allocation map is + cheaper than a directory service, and a directory service is a new SPOF. + +--- + +## 6. Open items + +| item | state | +|---|---| +| `llmuser` holds `sudo` on ana-docker | flagged 2026-09-14, unresolved | +| `/mnt/smithy` NFS export is `0777` throughout | blocked on § 3 UID alignment | +| Synapse appservice tokens sit in plaintext on ana-docker | flagged 2026-09-14; belongs in the vault | +| Legacy service accounts in the human UID range | not renamed, by decision | + +## Changelog + +- **2026-09-14** — ratified. Deleted the dormant `linus` account on ana-docker + (passwordless root, last used 2026-04-11, archived to + `/root/account-archive/linus-20260914.tar.gz`). Standardised `infra-ops` home + to `0700` on five hosts. Normalised corviduo-dev's docker tree, the operator + having ruled that all ops on that host belong to `infra-ops`. diff --git a/playbooks/audit-host-conventions.yaml b/playbooks/audit-host-conventions.yaml new file mode 100644 index 0000000..4b91c76 --- /dev/null +++ b/playbooks/audit-host-conventions.yaml @@ -0,0 +1,123 @@ +# audit-host-conventions — report a host's drift from docs/pfi/fleet-conventions.md +# +# READ-ONLY. Changes nothing, ever. Ratified 2026-09-14; the conventions doc is +# the source of truth and this playbook is its instrument. +# +# Deliberately reports rather than enforces. The conventions are a pin for NEW +# hosts, not a migration mandate — existing hosts converge when there is an +# independent reason to touch them (doc § 5 Non-goals). An enforcing playbook +# would quietly turn a standard into a flag day. +# +# scripts/elway --playbook playbooks/audit-host-conventions.yaml + +vars: + pin_infra_ops_uid: "850" + pin_docker_gid: "851" + svc_range_lo: "800" + svc_range_hi: "849" + +steps: + - name: Human account + shell: | + if getent passwd vh >/dev/null; then + echo " OK human account is vh ($(id -u vh))" + elif getent passwd lkraven >/dev/null; then + echo " LEGACY human account is lkraven ($(id -u lkraven)) — standard is vh; do NOT retro-rename (doc 1.1)" + else + echo " DRIFT no vh and no lkraven on this host" + fi + changed_when: "false" + + - name: Ops account — name, uid, home mode, sudo + shell: | + getent passwd infra-ops >/dev/null || { echo " DRIFT no infra-ops account"; exit 0; } + uid=$(id -u infra-ops) + [ "$uid" = "{{ pin_infra_ops_uid }}" ] \ + && echo " OK infra-ops uid=$uid (pinned)" \ + || echo " LEGACY infra-ops uid=$uid — pin is {{ pin_infra_ops_uid }} (doc 1.2 / 3; tolerable, costs the 777 on NFS)" + m=$(stat -c '%a %U:%G' /home/infra-ops 2>/dev/null) + case "$m" in + "700 infra-ops:infra-ops") echo " OK /home/infra-ops $m" ;; + *) echo " DRIFT /home/infra-ops $m — want 700 infra-ops:infra-ops" ;; + esac + sudo -n grep -qE '^infra-ops\s+ALL=\(ALL\)\s+NOPASSWD' /etc/sudoers.d/infra-ops 2>/dev/null \ + && echo " OK sudoers.d/infra-ops NOPASSWD present" \ + || echo " DRIFT sudoers.d/infra-ops missing or not NOPASSWD" + sudo: true + changed_when: "false" + + - name: docker group — gid pin and membership + shell: | + e=$(getent group docker) || { echo " N/A no docker group (host does not run Docker)"; exit 0; } + gid=$(echo "$e" | cut -d: -f3); mem=$(echo "$e" | cut -d: -f4) + [ "$gid" = "{{ pin_docker_gid }}" ] \ + && echo " OK docker gid=$gid (pinned)" \ + || echo " LEGACY docker gid=$gid — pin is {{ pin_docker_gid }} (assigned by docker-ce; pin with groupadd BEFORE install)" + echo "$mem" | tr ',' '\n' | grep -qx infra-ops \ + && echo " OK infra-ops in docker group" \ + || echo " DRIFT infra-ops NOT in docker group — it cannot deploy here" + echo " members: $mem" + changed_when: "false" + + - name: Service accounts — naming and range + # Two failures are reported separately because they have different fixes: + # a non-svc- name is cosmetic, a service account in the human UID range is + # the one that makes /etc/passwd unable to tell a daemon from a person. + shell: | + found=0 + while IFS=: read -r name _ uid gid _ home shell; do + case "$name" in vh|lkraven|infra-ops|nobody) continue ;; esac + found=1 + case "$name" in + svc-*) echo " OK $name uid=$uid" ;; + *) echo " LEGACY $name uid=$uid — convention is svc- in {{ svc_range_lo }}-{{ svc_range_hi }} (doc 1.3)" ;; + esac + [ "$uid" -ge 1000 ] && [ "${name#svc-}" != "$name" ] \ + && echo " DRIFT $name is svc-named but uid=$uid is in the HUMAN range" + done < <(awk -F: '$3>=1000 && $3<65000' /etc/passwd) + [ "$found" = 0 ] && echo " OK no extra accounts" + echo " --- accounts with a path to root ---" + for g in sudo docker; do + getent group "$g" >/dev/null || continue + for u in $(getent group "$g" | cut -d: -f4 | tr ',' ' '); do + case "$u" in vh|lkraven|infra-ops) continue ;; esac + echo " REVIEW $u is in '$g' — root-equivalent; a service account should not be (doc 1.3)" + done + done + changed_when: "false" + + - name: Deploy tree ownership and modes + shell: | + [ -d /opt/docker ] || { echo " N/A no /opt/docker on this host"; exit 0; } + for p in /opt/docker /opt/docker/compose /opt/docker/conf; do + [ -e "$p" ] || continue + s=$(stat -c '%a %U:%G' "$p") + [ "$s" = "2775 root:docker" ] && echo " OK $p $s" || echo " DRIFT $p $s — want 2775 root:docker" + done + bad=$(sudo -n find /opt/docker/compose -maxdepth 2 -name '.env' ! -perm 0640 -printf '%m %p\n' 2>/dev/null | head -5) + [ -z "$bad" ] && echo " OK every stack .env is 0640" || { echo " DRIFT .env not 0640:"; echo "$bad" | sed 's/^/ /'; } + ww=$(sudo -n find /opt/docker -maxdepth 2 -perm -o=w -printf '%m %p\n' 2>/dev/null | head -5) + [ -z "$ww" ] && echo " OK nothing world-writable under /opt/docker" || { echo " DRIFT world-writable:"; echo "$ww" | sed 's/^/ /'; } + sudo: true + changed_when: "false" + + - name: Stacks outside the convention (invisible to fleet automation) + # A stack not under /opt/docker/compose// is skipped by every tool + # that walks that path — silently. This is how `talk` was missed by the + # docker-ce upgrade's restart loop on 2026-09-14 (doc 4). + shell: | + command -v docker >/dev/null || { echo " N/A no docker"; exit 0; } + n=0 + for wd in $(docker ps -q | xargs -r docker inspect \ + --format '{{index .Config.Labels "com.docker.compose.project.working_dir"}}' 2>/dev/null | sort -u); do + [ -z "$wd" ] && continue + case "$wd" in + /opt/docker/compose/*) ;; + *) echo " DRIFT running stack outside the convention: $wd"; n=$((n+1)) ;; + esac + done + [ "$n" = 0 ] && echo " OK every running compose stack lives under /opt/docker/compose/" + loose=$(docker ps --format '{{.Names}}' --filter 'label=com.docker.compose.project' \ + | comm -13 - <(docker ps --format '{{.Names}}' | sort) 2>/dev/null | head -3) + true + changed_when: "false"