feat(ops-log): attribute host changes across two agents sharing one identity

infra-ops and infra-hermes act as the same OS identity and dockerd does not log exec per caller, so host-side changes carry no fingerprint. Git cannot close the gap either: every commit here is attributed to Vuong Hoang by convention, which is correct for authorship and useless for attribution. On 2026-09-18 a second session edited the searxng stack mid-deploy, crash- looping fleet search for ~4 minutes, and the author was unidentifiable. scripts/ops-log records one line per host-changing action and holds a lightweight claim so two agents do not deploy the same stack at once. Four design questions, settled: * Central on nh3-dev, not per-host and not the post office. Both agents run as the same unix user there, so one file is shared with zero provisioning. Per-host needs a writable path on ~25 heterogeneous boxes and stores "we changed host Y" on host Y. journald looked free but shows an unprivileged reader only their own _UID, which would have split the log silently between the infra-ops and lkraven halves of the fleet. * The claim is advisory and enforced in the tooling. deploy-stack.sh refuses a foreign claim across the diff, the prompt and the apply -- the whole review window, which is where the collision happened. Acquire is mkdir, so it is atomic rather than probably-fine. Stale claims auto-break and the break is recorded. * Writers are automatic. deploy-stack.sh and elway record themselves; a log that depends on remembering is the same class of instrument as a health check that passes in both states. * There is a detector. `ops-log audit` asks each host what changed on disk and compares it to the newest log line for that stack, covering the manual ssh-and-edit path the automatic writers structurally cannot. ops-log being absent or broken never blocks a deploy; only a live foreign claim does. `ops-log baseline` marks the 136 stacks that predate the instrument so the detector starts from today rather than reporting the whole fleet forever and training us to ignore it. An unreachable host reports INCOMPLETE and exit 5, never clean.
2026-09-19 05:05:43 -07:00
parent 4d826e17e3
commit ffe7b24935
7 changed files with 960 additions and 22 deletions
@@ -0,0 +1,151 @@
+# The fleet ops log
+
+**What it is:** one appended line per host-changing action, plus a lightweight
+claim so two agents do not deploy the same stack at the same time.
+
+**Tool:** [`scripts/ops-log`](../../scripts/ops-log).
+**Storage:** `<repo>/.ops-log/` on nh3-dev — gitignored, override with `OPS_LOG_DIR`.
+
+---
+
+## Why it exists
+
+`infra-ops` and `infra-hermes` are two different agents that act as the **same
+OS identity** (`ssh infra-ops@<host>`), and dockerd does not log `exec` per
+caller. Host-side changes are therefore fingerprint-less: when a host differs
+from expectation, neither agent can tell whether the other did it, a prior
+session did, or something broke on its own. Git does not close the gap either —
+every commit in this repo is attributed to Vuong Hoang by convention, which is
+correct for authorship and useless for attribution.
+
+With one agent this was theoretical. With two doing routine ops daily it is
+operational.
+
+**The precipitating incident (2026-09-18).** A second session edited the
+searxng stack while another was deploying it. Fleet search crash-looped for
+~4 minutes, and afterwards the author was unidentifiable: the commit said
+Vuong Hoang and the on-host file carried no provenance at all.
+
+---
+
+## Design decisions, and why
+
+Four questions had to be settled. Each one had a defensible alternative; the
+reasoning matters more than the answer, because the alternatives will look
+attractive again the next time someone extends this.
+
+### 1. The log lives centrally on nh3-dev — not per-host, not on the post office
+
+Both agents run as the **same unix user on nh3-dev** (`infra-hermes` is
+`althing-pump-infra-hermes.service`, a *user* unit under `lkraven`). So a single
+file is shared instantly, with zero provisioning, zero permissions story, and no
+network hop.
+
+- **Not per-host.** Provenance next to the thing changed is appealing, but it
+  needs a writable log path created on ~25 heterogeneous hosts (Debian, PVE,
+  DSM) with different sudo situations — and an instrument that is only deployed
+  on some of them lies by omission. It also puts the record of "we changed X on
+  host Y" *on host Y*, which is exactly where it is least available when Y is
+  the thing that broke.
+- **Not syslog/journald.** Tempting (no provisioning at all), but journald only
+  shows an unprivileged reader entries matching their own `_UID`, and the fleet
+  acts as `infra-ops` on some hosts and `lkraven` on others. The log would have
+  silently split in half along an axis nobody would think to check.
+- **Not the post office.** A message bus is not a log: ordering and query are
+  poor, it becomes inbox noise, and a post-office outage would block ops during
+  precisely the incident you are trying to reconstruct.
+
+**The known hole, stated rather than papered over:** an actor operating from a
+box other than nh3-dev is not covered. Today that is only the operator on his
+laptop. If a third agent ever runs elsewhere, this decision is the one to revisit.
+
+### 2. The claim is advisory, and enforced in the tooling
+
+`deploy-stack.sh` **refuses** (exit 3) a stack another agent holds. Nothing
+stops a raw `ssh` — the point is to make the sanctioned path safe, not to build
+a cage that people route around.
+
+Acquisition is `mkdir` of a claim directory, which is atomic, so the check is
+genuinely race-free rather than "probably fine". Claims carry a TTL (default
+30m) and a stale claim is auto-broken — **and the break is logged**, so a claim
+that keeps getting broken is visible rather than silently ineffective.
+
+### 3. The writers are automatic
+
+`deploy-stack.sh` and `elway` record themselves. This was the question that
+mattered most: **a log that depends on remembering is the same class of
+instrument as a health check that passes in both states.**
+
+### 4. There is a detector, not just a rule
+
+Automatic writers cannot cover the manual path — someone ssh'ing in and editing
+a compose file by hand. The answer is not a louder rule; it is
+`ops-log audit`, which asks each host what changed on disk and compares it to
+the newest log line for that stack. Anything that changed with no log line at or
+after it is reported.
+
+Per the same discipline, `audit` refuses to say "clean" for a host it could not
+reach — an unreachable host reports `INCOMPLETE` and exit 5, never 0.
+
+---
+
+## Using it
+
+```bash
+# Before hand-editing a stack on a host, so other agents' tooling refuses it:
+scripts/ops-log claim nh3-docker searxng --why "raising the engine timeout"
+scripts/ops-log release nh3-docker searxng          # when done
+
+# After any change made OUTSIDE deploy-stack.sh / elway:
+scripts/ops-log record --host nh3-docker --action restart --target searxng \
+  --detail "docker restart searxng after the conf edit"
+
+# Reading:
+scripts/ops-log tail --since 24h                    # everything today
+scripts/ops-log tail --host nh3-docker -n 20        # one host
+scripts/ops-log claims                              # who is holding what
+
+# The detector — run it when a host surprises you, or periodically:
+scripts/ops-log audit                               # the six stack hosts
+scripts/ops-log audit nh3-docker --since 7d
+scripts/ops-log audit all                           # every non-tenant host
+```
+
+`deploy-stack.sh` claims and records on its own. Escape hatches:
+`DEPLOY_NO_CLAIM=1` skips the claim, `DEPLOY_CLAIM_TTL=<dur>` lengthens it.
+ops-log being absent or broken never blocks a deploy — only a live foreign
+claim does.
+
+### Exit codes
+
+| code | meaning |
+|---|---|
+| 0 | success / claim acquired / audit clean |
+| 2 | usage error |
+| 3 | claim refused — held by another agent |
+| 4 | audit found unlogged changes |
+| 5 | audit could not reach every host (**incomplete, not clean**) |
+
+---
+
+## What it deliberately does not do
+
+- **It does not cover raw `ssh`.** It cannot. `audit` is the backstop.
+- **It does not claim per-host for elway.** elway records but does not claim:
+  two unrelated playbooks on one host are not a collision, and a host-wide lock
+  would be ignored within a week.
+- **It does not audit `corviduo-dev`.** Worldtree deploys there are CI/CD-driven
+  and rewrite the tree constantly, so it would report unlogged changes forever
+  and train us to ignore the output.
+- **It does not audit the SureFire tenant hosts** (`sf-*`, `sfsrv-ana`), which
+  are client property under the hosting agreement.
+- **It does not cover DNS, Cloudflare, UniFi or FortiGate changes automatically.**
+  Those have no host-side compose tree to diff. Record them by hand with
+  `--host <appliance>` until `dns-sync.py` and friends grow the same hook.
+
+## Open follow-ons
+
+- Hook `scripts/dns-sync.py` (and the UniFi/FortiGate helpers) so
+  control-plane changes record themselves too.
+- Run `audit` on a timer and alert on non-zero, rather than only when someone
+  remembers to look.