esh-pfi-infrastructure/docs/pfi/fleet-conventions.md

Fleet conventions — identities, groups, paths

Ratified by the operator 2026-09-14. This is the pin: new hosts are built to it, existing hosts converge opportunistically. It is deliberately not a migration mandate — see § Non-goals.

Audit any host against it with:

scripts/elway <host> --playbook playbooks/audit-host-conventions.yaml

---

1. Identity

1.1 The human account

Standard	vh, UID/GID 1000
Legacy	lkraven on nh3-dev, nh3-docker, nh3-extdev, irv-ml1, ana-docker, esh-docker-vm

vh matches the operator's mail (vh@phasefinal.com), git identity and Gitea namespace (vh/*), so it is the name everything else already uses.

⚠ Do not retro-rename the six legacy hosts. Renaming UID 1000 where the home directory is populated, systemd user services are lingering, SSH configs point at it and live agent sessions are running is real blast radius for a cosmetic gain — and the thing that actually mattered (a personal username owning shared infrastructure) was removed on 2026-09-14 when the docker trees went to root:docker. New hosts get vh; old hosts keep lkraven until there is an independent reason to rebuild them.

1.2 The ops account

Name	infra-ops — on every host, no exceptions
UID/GID	850 (pinned; see § 3)
Sudo	/etc/sudoers.d/infra-ops → infra-ops ALL=(ALL) NOPASSWD:ALL
Home	/home/infra-ops, owned infra-ops:infra-ops, mode 0700
Groups	docker where Docker is installed

infra-ops is the fleet's ops identity and is the same party as the infra-ops althing handle. Mode 0700 is not paranoia: these homes hold credentials (~/.config/worldtree/personal-admin-token, ~/.config/cloudflare/…, ~/.config/secrets-broker/bootstrap.env). Verified safe before adoption — nh3-extdev and corviduo-dev had already been running at 0700 for months, including systemd units that read from that home.

⚠ The UID is currently 1001 / 1002 / 1003 / 2001 across the fleet. See § 3 for why that is tolerable today and what it costs.

1.3 Service accounts

Name	svc-<service> — e.g. svc-arbo, svc-brokkr, svc-llm
UID/GID	allocated from 800–849
Shell	/usr/sbin/nologin unless the account genuinely needs to log in
Home	/var/lib/<service> for daemons; /home/svc-<service> only if it needs one

useradd --system --uid 8NN --user-group --shell /usr/sbin/nologin \
        --home-dir /var/lib/<service> --create-home svc-<service>

Legacy, not renamed: llmuser, sduser, brokkr, arbotrain, nas, deploy. Six accounts, six naming styles, all sitting in the human UID range (1000+) where /etc/passwd cannot distinguish them from people. That is the defect the convention fixes going forward.

⚠ A service account must not hold interactive root. Enforced 2026-09-14: llmuser was in sudo+docker on ana-docker and sudo on irv-ml1; both stripped. Nothing broke, and the reason is worth knowing — nearly every process attributed to llmuser by pgrep -u was a container process whose in-image UID happens to be 1001. A container's runtime UID has nothing to do with host group membership, so the groups were buying it nothing. The one genuine host workload (lora-training-worker.service on irv-ml1) sets User=/Group= explicitly via systemd, which does not consult the sudo group either.

Generalisation for the next audit: pgrep -u <user> on a Docker host over-attributes badly. Check /proc/<pid>/cgroup for a docker-*.scope before concluding a host account is busy — otherwise a UID collision reads as an active service account and blocks a cleanup that carries no risk.

---

2. Groups

group	GID	membership
docker	851 (pinned)	the human account + infra-ops
sudo	27 (Debian default)	the human account only

⚠ docker membership is root-equivalent — anyone in it can bind-mount / into a container. The group makes access consistent; it does not create privilege separation. A future non-root deployer (a contractor, a tenant agent) needs a dedicated deploy group, not an invitation to docker.

Current GIDs are 109 / 110 / 112 / 989 — assigned by docker-ce at install time. Pinning means groupadd -g 851 docker before installing Docker.

---

3. UID/GID allocation map

Measured across eight hosts on 2026-09-14: dynamically-allocated system accounts cluster in 989–999 and descend. 800–899 is free on every host, which is what makes it safe to pin there.

     0–99    OS reserved
   100–799   Debian dynamic system allocation — never pin here
   800–849   svc-* service accounts        (fleet-pinned)
   850       infra-ops  (uid + gid)        (fleet-pinned)
   851       docker     (gid)              (fleet-pinned)
   852–899   reserved for fleet-wide groups
   900–999   Debian allocates system accounts here — leave clear
  1000       the human account (vh)
  1001+      additional humans

Why the existing drift is tolerable

There is no central identity anywhere on the fleet — every host is passwd: files systemd. So a UID only has to be consistent where files cross hosts. They do: /mnt/smithy is NFS-mounted on both nh3-dev (infra-ops=1001) and irv-ml1 (infra-ops=1003), so UID 1001 on that export means infra-ops on one box and llmuser on the other.

That collision is not currently biting, and the reason is the finding: the export is owned by the Synology's own UIDs (1024, 1031) which resolve on neither host, and every directory on it is 0777. Cross-host sharing works today because the permissions are wide open.

So aligning UIDs does not fix something broken — it earns the right to drop the 777 on shared storage. That is a real improvement and a real project. It is not a prerequisite for anything currently running.

---

4. Paths

path	ownership	mode	contents
/opt/docker	root:docker	2775	the deploy tree root
/opt/docker/compose/<stack>/	root:docker	2775	compose.yaml, .env
/opt/docker/conf/<stack>/	root:docker	2775	bind-mounted config
/opt/docker/compose/<stack>/.env	root:docker	0640	stack secrets
/home/<user>	<user>:<user>	0700

The setgid bit on directories is load-bearing: it makes every file created afterwards inherit the docker group, so the scheme does not decay back into whoever-touched-it-last ownership.

⚠ A stack dir that is a git checkout needs safe.directory for its deploy user. git (>= 2.35.2) refuses to operate in a worktree whose top-level dir is owned by someone else, so moving a checkout to root:docker breaks any deploy that runs git fetch/reset as a normal user, with fatal: detected dubious ownership. That's what stalled yt-voice-clipper's webhook auto-deploy on irv-ml1 after the 2026-09-14 normalization, found 2026-09-24 when v0.3.13 would not land. The fix keeps the convention: as the deploy user, git config --global --add safe.directory /opt/docker/compose/<stack> (one entry per repo; the /* wildcard form needs git >= 2.46, and irv-ml1 has 2.39). Swept 2026-09-24: that was the only checkout under a normalized /opt/docker/compose deployed by a non-root user. esh-docker-vm's /opt/docker/conf/esphome/ is a checkout but is root:root throughout.

⚠ A stack lives under /opt/docker/compose/<stack>/ or it is invisible. Every piece of fleet automation walks that path. talk sat at ~/talk until 2026-09-14 and was silently skipped by the docker-ce upgrade's restart loop for exactly that reason. A stack outside the convention is not merely untidy — it is excluded from every tool that will ever be written.

Permission-sensitive files — never widen these

Some daemons refuse to start when their files are group-accessible, and they fail at their next restart rather than at the moment of the change. That delayed detonation is worse than a loud failure, so these are excluded by name and by mode from normalize-docker-tree.yaml:

file	required	breaks
acme.json	0600	traefik refuses to start
id_*, *.key, *.pem, *.pfx	0600	ssh refuses a group-readable private key

Anything already at 0600/0400 is deliberate. Leave it.

---

5. Non-goals

• Not a migration mandate. Existing hosts converge when there is an independent reason to touch them. Nothing here justifies a flag day.
• Not privilege separation. Both docker membership and infra-ops sudo are root-equivalent today. This document standardises consistency, not least privilege.
• Not a central identity system. LDAP/SSSD would make § 3 moot. Deliberately out of scope — the fleet is small enough that a pinned allocation map is cheaper than a directory service, and a directory service is a new SPOF.

---

6. Open items

item	state
/mnt/smithy NFS export is 0777 throughout	blocked on § 3 UID alignment
Synapse appservice tokens sit in plaintext on ana-docker	flagged 2026-09-14; belongs in the vault
Legacy service accounts in the human UID range	not renamed, by decision

Changelog

• 2026-09-14 — ratified. vh confirmed as the standard for new hosts with no retro-renames. Stripped sudo+docker from llmuser (ana-docker) and sudo (irv-ml1) — see § 1.3; all 63 containers across both hosts stayed healthy and lora-training-worker stayed active. Deleted the dormant linus account on ana-docker (passwordless root, last used 2026-04-11, archived to /root/account-archive/linus-20260914.tar.gz). Standardised infra-ops home to 0700 on five hosts. Normalised corviduo-dev's docker tree, the operator having ruled that all ops on that host belong to infra-ops.