vh
ce7b07f7af
Operator ruling: remove the groups and see what breaks. Nothing did. ana-docker llmuser sudo+docker -> none; irv-ml1 llmuser sudo -> none (it was never in docker there). 45 containers on ana-docker and 18 on irv-ml1 all still running with zero unhealthy, and lora-training-worker stayed active. Extended to irv-ml1 because it is the same account with the same defect and gpasswd -a reverses it in one command; ana-docker was only the host the audit happened to run against first. The durable lesson is why it was safe, and it is a measurement trap rather than a permissions one. reported 19 processes on ana-docker and 3 on irv-ml1, which reads as a busy service account. Nearly all of them were CONTAINER processes whose in-image UID is 1001 and therefore collides with llmuser on the host — /proc/<pid>/cgroup shows docker-*.scope. A container's runtime UID is unrelated to host group membership, so the groups were buying those workloads nothing. The single real host workload sets User=/Group= explicitly through systemd, which does not consult the sudo group either. Recorded in the conventions doc so the next audit checks the cgroup before concluding a host account is busy — otherwise a UID collision blocks a cleanup that carries no risk.
211 lines
8.9 KiB
Markdown
211 lines
8.9 KiB
Markdown
# Fleet conventions — identities, groups, paths
|
||
|
||
Ratified by the operator 2026-09-14. This is the **pin**: new hosts are built
|
||
to it, existing hosts converge opportunistically. It is deliberately *not* a
|
||
migration mandate — see § Non-goals.
|
||
|
||
Audit any host against it with:
|
||
|
||
```bash
|
||
scripts/elway <host> --playbook playbooks/audit-host-conventions.yaml
|
||
```
|
||
|
||
---
|
||
|
||
## 1. Identity
|
||
|
||
### 1.1 The human account
|
||
|
||
| | |
|
||
|---|---|
|
||
| **Standard** | `vh`, UID/GID 1000 |
|
||
| **Legacy** | `lkraven` on nh3-dev, nh3-docker, nh3-extdev, irv-ml1, ana-docker, esh-docker-vm |
|
||
|
||
`vh` matches the operator's mail (`vh@phasefinal.com`), git identity and Gitea
|
||
namespace (`vh/*`), so it is the name everything else already uses.
|
||
|
||
⚠ **Do not retro-rename the six legacy hosts.** Renaming UID 1000 where the home
|
||
directory is populated, systemd user services are lingering, SSH configs point
|
||
at it and live agent sessions are running is real blast radius for a cosmetic
|
||
gain — and the thing that actually mattered (a personal username owning *shared*
|
||
infrastructure) was removed on 2026-09-14 when the docker trees went to
|
||
`root:docker`. New hosts get `vh`; old hosts keep `lkraven` until there is an
|
||
independent reason to rebuild them.
|
||
|
||
### 1.2 The ops account
|
||
|
||
| | |
|
||
|---|---|
|
||
| **Name** | `infra-ops` — on every host, no exceptions |
|
||
| **UID/GID** | **850** (pinned; see § 3) |
|
||
| **Sudo** | `/etc/sudoers.d/infra-ops` → `infra-ops ALL=(ALL) NOPASSWD:ALL` |
|
||
| **Home** | `/home/infra-ops`, owned `infra-ops:infra-ops`, mode **0700** |
|
||
| **Groups** | `docker` where Docker is installed |
|
||
|
||
`infra-ops` is the fleet's ops identity and is the same party as the `infra-ops`
|
||
althing handle. Mode `0700` is not paranoia: these homes hold credentials
|
||
(`~/.config/worldtree/personal-admin-token`, `~/.config/cloudflare/…`,
|
||
`~/.config/secrets-broker/bootstrap.env`). Verified safe before adoption —
|
||
nh3-extdev and corviduo-dev had already been running at `0700` for months,
|
||
including systemd units that read from that home.
|
||
|
||
⚠ The UID is currently **1001 / 1002 / 1003 / 2001** across the fleet. See § 3
|
||
for why that is tolerable today and what it costs.
|
||
|
||
### 1.3 Service accounts
|
||
|
||
| | |
|
||
|---|---|
|
||
| **Name** | `svc-<service>` — e.g. `svc-arbo`, `svc-brokkr`, `svc-llm` |
|
||
| **UID/GID** | allocated from **800–849** |
|
||
| **Shell** | `/usr/sbin/nologin` unless the account genuinely needs to log in |
|
||
| **Home** | `/var/lib/<service>` for daemons; `/home/svc-<service>` only if it needs one |
|
||
|
||
```bash
|
||
useradd --system --uid 8NN --user-group --shell /usr/sbin/nologin \
|
||
--home-dir /var/lib/<service> --create-home svc-<service>
|
||
```
|
||
|
||
**Legacy, not renamed:** `llmuser`, `sduser`, `brokkr`, `arbotrain`, `nas`,
|
||
`deploy`. Six accounts, six naming styles, all sitting in the human UID range
|
||
(1000+) where `/etc/passwd` cannot distinguish them from people. That is the
|
||
defect the convention fixes going forward.
|
||
|
||
⚠ **A service account must not hold interactive root.** Enforced 2026-09-14:
|
||
`llmuser` was in `sudo`+`docker` on ana-docker and `sudo` on irv-ml1; both
|
||
stripped. Nothing broke, and the reason is worth knowing — nearly every process
|
||
attributed to `llmuser` by `pgrep -u` was a **container** process whose in-image
|
||
UID happens to be 1001. A container's runtime UID has nothing to do with host
|
||
group membership, so the groups were buying it nothing. The one genuine host
|
||
workload (`lora-training-worker.service` on irv-ml1) sets `User=`/`Group=`
|
||
explicitly via systemd, which does not consult the `sudo` group either.
|
||
|
||
**Generalisation for the next audit:** `pgrep -u <user>` on a Docker host
|
||
over-attributes badly. Check `/proc/<pid>/cgroup` for a `docker-*.scope` before
|
||
concluding a host account is busy — otherwise a UID collision reads as an
|
||
active service account and blocks a cleanup that carries no risk.
|
||
|
||
---
|
||
|
||
## 2. Groups
|
||
|
||
| group | GID | membership |
|
||
|---|---|---|
|
||
| `docker` | **851** (pinned) | the human account + `infra-ops` |
|
||
| `sudo` | 27 (Debian default) | the human account only |
|
||
|
||
⚠ **`docker` membership is root-equivalent** — anyone in it can bind-mount `/`
|
||
into a container. The group makes access *consistent*; it does not create
|
||
privilege separation. A future non-root deployer (a contractor, a tenant agent)
|
||
needs a dedicated `deploy` group, not an invitation to `docker`.
|
||
|
||
Current GIDs are **109 / 110 / 112 / 989** — assigned by `docker-ce` at install
|
||
time. Pinning means `groupadd -g 851 docker` *before* installing Docker.
|
||
|
||
---
|
||
|
||
## 3. UID/GID allocation map
|
||
|
||
Measured across eight hosts on 2026-09-14: dynamically-allocated system accounts
|
||
cluster in **989–999** and descend. **800–899 is free on every host**, which is
|
||
what makes it safe to pin there.
|
||
|
||
```
|
||
0–99 OS reserved
|
||
100–799 Debian dynamic system allocation — never pin here
|
||
800–849 svc-* service accounts (fleet-pinned)
|
||
850 infra-ops (uid + gid) (fleet-pinned)
|
||
851 docker (gid) (fleet-pinned)
|
||
852–899 reserved for fleet-wide groups
|
||
900–999 Debian allocates system accounts here — leave clear
|
||
1000 the human account (vh)
|
||
1001+ additional humans
|
||
```
|
||
|
||
### Why the existing drift is tolerable
|
||
|
||
There is **no central identity** anywhere on the fleet — every host is
|
||
`passwd: files systemd`. So a UID only has to be consistent where files cross
|
||
hosts. They do: `/mnt/smithy` is NFS-mounted on both nh3-dev (`infra-ops`=1001)
|
||
and irv-ml1 (`infra-ops`=1003), so UID 1001 on that export means `infra-ops` on
|
||
one box and `llmuser` on the other.
|
||
|
||
**That collision is not currently biting, and the reason is the finding:** the
|
||
export is owned by the Synology's own UIDs (1024, 1031) which resolve on
|
||
*neither* host, and every directory on it is `0777`. Cross-host sharing works
|
||
today *because* the permissions are wide open.
|
||
|
||
So aligning UIDs does not fix something broken — it **earns the right to drop
|
||
the 777 on shared storage**. That is a real improvement and a real project. It
|
||
is not a prerequisite for anything currently running.
|
||
|
||
---
|
||
|
||
## 4. Paths
|
||
|
||
| path | ownership | mode | contents |
|
||
|---|---|---|---|
|
||
| `/opt/docker` | `root:docker` | `2775` | the deploy tree root |
|
||
| `/opt/docker/compose/<stack>/` | `root:docker` | `2775` | `compose.yaml`, `.env` |
|
||
| `/opt/docker/conf/<stack>/` | `root:docker` | `2775` | bind-mounted config |
|
||
| `/opt/docker/compose/<stack>/.env` | `root:docker` | **`0640`** | stack secrets |
|
||
| `/home/<user>` | `<user>:<user>` | `0700` | |
|
||
|
||
The setgid bit on directories is load-bearing: it makes every file created
|
||
afterwards inherit the `docker` group, so the scheme does not decay back into
|
||
whoever-touched-it-last ownership.
|
||
|
||
⚠ **A stack lives under `/opt/docker/compose/<stack>/` or it is invisible.**
|
||
Every piece of fleet automation walks that path. `talk` sat at `~/talk` until
|
||
2026-09-14 and was silently skipped by the docker-ce upgrade's restart loop for
|
||
exactly that reason. A stack outside the convention is not merely untidy — it is
|
||
excluded from every tool that will ever be written.
|
||
|
||
### Permission-sensitive files — never widen these
|
||
|
||
Some daemons **refuse to start** when their files are group-accessible, and they
|
||
fail at their *next restart* rather than at the moment of the change. That
|
||
delayed detonation is worse than a loud failure, so these are excluded by name
|
||
*and* by mode from `normalize-docker-tree.yaml`:
|
||
|
||
| file | required | breaks |
|
||
|---|---|---|
|
||
| `acme.json` | `0600` | traefik refuses to start |
|
||
| `id_*`, `*.key`, `*.pem`, `*.pfx` | `0600` | ssh refuses a group-readable private key |
|
||
|
||
Anything already at `0600`/`0400` is deliberate. Leave it.
|
||
|
||
---
|
||
|
||
## 5. Non-goals
|
||
|
||
- **Not a migration mandate.** Existing hosts converge when there is an
|
||
independent reason to touch them. Nothing here justifies a flag day.
|
||
- **Not privilege separation.** Both `docker` membership and `infra-ops` sudo are
|
||
root-equivalent today. This document standardises *consistency*, not least
|
||
privilege.
|
||
- **Not a central identity system.** LDAP/SSSD would make § 3 moot. Deliberately
|
||
out of scope — the fleet is small enough that a pinned allocation map is
|
||
cheaper than a directory service, and a directory service is a new SPOF.
|
||
|
||
---
|
||
|
||
## 6. Open items
|
||
|
||
| item | state |
|
||
|---|---|
|
||
| `/mnt/smithy` NFS export is `0777` throughout | blocked on § 3 UID alignment |
|
||
| Synapse appservice tokens sit in plaintext on ana-docker | flagged 2026-09-14; belongs in the vault |
|
||
| Legacy service accounts in the human UID range | not renamed, by decision |
|
||
|
||
## Changelog
|
||
|
||
- **2026-09-14** — ratified. `vh` confirmed as the standard for new hosts with
|
||
**no retro-renames**. Stripped `sudo`+`docker` from `llmuser` (ana-docker) and
|
||
`sudo` (irv-ml1) — see § 1.3; all 63 containers across both hosts stayed
|
||
healthy and `lora-training-worker` stayed active. Deleted the dormant `linus` account on ana-docker
|
||
(passwordless root, last used 2026-04-11, archived to
|
||
`/root/account-archive/linus-20260914.tar.gz`). Standardised `infra-ops` home
|
||
to `0700` on five hosts. Normalised corviduo-dev's docker tree, the operator
|
||
having ruled that all ops on that host belong to `infra-ops`.
|