vh
cebffae56d
Four attribution failures shipped on the day it was built, and every one of them RECORDED SOMETHING -- just not something findable or attributable. The individual fixes are in the commit log; the PATTERN was not written down anywhere, so a future session would meet each one as an isolated bug rather than as the fifth instance of a known shape. infra-hermes asked for this after the fourth one: 'worth an audit line item so the pattern is visible, not just the fixes.' He was right. The operative warning is at the bottom: absence of an entry is not evidence of absence of the event until you have checked this tool could have recorded it.
184 lines
9.4 KiB
Markdown
184 lines
9.4 KiB
Markdown
# The fleet ops log
|
|
|
|
**What it is:** one appended line per host-changing action, plus a lightweight
|
|
claim so two agents do not deploy the same stack at the same time.
|
|
|
|
**Tool:** [`scripts/ops-log`](../../scripts/ops-log).
|
|
**Storage:** `<repo>/.ops-log/` on nh3-dev — gitignored, override with `OPS_LOG_DIR`.
|
|
|
|
---
|
|
|
|
## Why it exists
|
|
|
|
`infra-ops` and `infra-hermes` are two different agents that act as the **same
|
|
OS identity** (`ssh infra-ops@<host>`), and dockerd does not log `exec` per
|
|
caller. Host-side changes are therefore fingerprint-less: when a host differs
|
|
from expectation, neither agent can tell whether the other did it, a prior
|
|
session did, or something broke on its own. Git does not close the gap either —
|
|
every commit in this repo is attributed to Vuong Hoang by convention, which is
|
|
correct for authorship and useless for attribution.
|
|
|
|
With one agent this was theoretical. With two doing routine ops daily it is
|
|
operational.
|
|
|
|
**The precipitating incident (2026-09-18).** A second session edited the
|
|
searxng stack while another was deploying it. Fleet search crash-looped for
|
|
~4 minutes, and afterwards the author was unidentifiable: the commit said
|
|
Vuong Hoang and the on-host file carried no provenance at all.
|
|
|
|
---
|
|
|
|
## Design decisions, and why
|
|
|
|
Four questions had to be settled. Each one had a defensible alternative; the
|
|
reasoning matters more than the answer, because the alternatives will look
|
|
attractive again the next time someone extends this.
|
|
|
|
### 1. The log lives centrally on nh3-dev — not per-host, not on the post office
|
|
|
|
Both agents run as the **same unix user on nh3-dev** (`infra-hermes` is
|
|
`althing-pump-infra-hermes.service`, a *user* unit under `lkraven`). So a single
|
|
file is shared instantly, with zero provisioning, zero permissions story, and no
|
|
network hop.
|
|
|
|
- **Not per-host.** Provenance next to the thing changed is appealing, but it
|
|
needs a writable log path created on ~25 heterogeneous hosts (Debian, PVE,
|
|
DSM) with different sudo situations — and an instrument that is only deployed
|
|
on some of them lies by omission. It also puts the record of "we changed X on
|
|
host Y" *on host Y*, which is exactly where it is least available when Y is
|
|
the thing that broke.
|
|
- **Not syslog/journald.** Tempting (no provisioning at all), but journald only
|
|
shows an unprivileged reader entries matching their own `_UID`, and the fleet
|
|
acts as `infra-ops` on some hosts and `lkraven` on others. The log would have
|
|
silently split in half along an axis nobody would think to check.
|
|
- **Not the post office.** A message bus is not a log: ordering and query are
|
|
poor, it becomes inbox noise, and a post-office outage would block ops during
|
|
precisely the incident you are trying to reconstruct.
|
|
|
|
**The known hole, stated rather than papered over:** an actor operating from a
|
|
box other than nh3-dev is not covered. Today that is only the operator on his
|
|
laptop. If a third agent ever runs elsewhere, this decision is the one to revisit.
|
|
|
|
### 2. The claim is advisory, and enforced in the tooling
|
|
|
|
`deploy-stack.sh` **refuses** (exit 3) a stack another agent holds. Nothing
|
|
stops a raw `ssh` — the point is to make the sanctioned path safe, not to build
|
|
a cage that people route around.
|
|
|
|
Acquisition is `mkdir` of a claim directory, which is atomic, so the check is
|
|
genuinely race-free rather than "probably fine". Claims carry a TTL (default
|
|
30m) and a stale claim is auto-broken — **and the break is logged**, so a claim
|
|
that keeps getting broken is visible rather than silently ineffective.
|
|
|
|
### 3. The writers are automatic
|
|
|
|
`deploy-stack.sh` and `elway` record themselves. This was the question that
|
|
mattered most: **a log that depends on remembering is the same class of
|
|
instrument as a health check that passes in both states.**
|
|
|
|
### 4. There is a detector, not just a rule
|
|
|
|
Automatic writers cannot cover the manual path — someone ssh'ing in and editing
|
|
a compose file by hand. The answer is not a louder rule; it is
|
|
`ops-log audit`, which asks each host what changed on disk and compares it to
|
|
the newest log line for that stack. Anything that changed with no log line at or
|
|
after it is reported.
|
|
|
|
Per the same discipline, `audit` refuses to say "clean" for a host it could not
|
|
reach — an unreachable host reports `INCOMPLETE` and exit 5, never 0.
|
|
|
|
---
|
|
|
|
## Using it
|
|
|
|
```bash
|
|
# Before hand-editing a stack on a host, so other agents' tooling refuses it:
|
|
scripts/ops-log claim nh3-docker searxng --why "raising the engine timeout"
|
|
scripts/ops-log release nh3-docker searxng # when done
|
|
|
|
# After any change made OUTSIDE deploy-stack.sh / elway:
|
|
scripts/ops-log record --host nh3-docker --action restart --target searxng \
|
|
--detail "docker restart searxng after the conf edit"
|
|
|
|
# Reading:
|
|
scripts/ops-log tail --since 24h # everything today
|
|
scripts/ops-log tail --host nh3-docker -n 20 # one host
|
|
scripts/ops-log claims # who is holding what
|
|
|
|
# The detector — run it when a host surprises you, or periodically:
|
|
scripts/ops-log audit # the six stack hosts
|
|
scripts/ops-log audit nh3-docker --since 7d
|
|
scripts/ops-log audit all # every non-tenant host
|
|
```
|
|
|
|
`deploy-stack.sh` claims and records on its own. Escape hatches:
|
|
`DEPLOY_NO_CLAIM=1` skips the claim, `DEPLOY_CLAIM_TTL=<dur>` lengthens it.
|
|
ops-log being absent or broken never blocks a deploy — only a live foreign
|
|
claim does.
|
|
|
|
### Exit codes
|
|
|
|
| code | meaning |
|
|
|---|---|
|
|
| 0 | success / claim acquired / audit clean |
|
|
| 2 | usage error |
|
|
| 3 | claim refused — held by another agent |
|
|
| 4 | audit found unlogged changes |
|
|
| 5 | audit could not reach every host (**incomplete, not clean**) |
|
|
|
|
---
|
|
|
|
## How this instrument has failed — the pattern, not just the fixes
|
|
|
|
**All four failures below shipped on the day it was built, 2026-09-19, and
|
|
every one of them RECORDED SOMETHING.** None of them recorded something
|
|
*findable* or *attributable*. That is the shape of the failure class here: this
|
|
tool does not fall over loudly, it answers the wrong question quietly. Read
|
|
this table before assuming a missing entry means the event did not happen.
|
|
|
|
| # | Failure | Symptom | Cause | Fixed |
|
|
|---|---|---|---|---|
|
|
| 1 | **Claim released by a sub-tool** | A 45-min operation claim vanished mid-rollout; nothing refused anything | `deploy-stack.sh` claims and releases around its own work, and released a claim it had only refreshed | `3e7d3a3` — `claim` exits 10 when already yours and leaves the holder file untouched |
|
|
| 2 | **Wrong order in the hook chain** | Some commits logged, some not; looked flaky | The commit hook was APPENDED behind graphify's eight `exit 0` paths, so a `graphify-out/`-only or empty commit could never reach it | `9141a41` — prepend after the shebang; attribution runs first, unconditionally |
|
|
| 3 | **No handle in the environment** | A peer's commit logged as `lkraven@nh3-dev` | `ALTHING_HANDLE` was exported only by `althing-infra-hermes-seat-run.sh`, not by `hermes-gateway.service` or the pump unit | Drop-in overrides (infra-hermes) + `4e778ae` — the fallback now says `unattributed(login)` instead of a bare login that reads like an answer |
|
|
| 4 | **Wrong host key on write** | Five records of a real `jobs.cfg` change on esh-pve were invisible to `--host esh-pve` | elway passed its ssh TARGET through as the host, so they landed under `infra-ops@esh-pve` | `f3b68e2` — `normalize_host` strips `user@` on write AND on query, so pre-fix records are findable too |
|
|
|
|
**The rules that fell out of it, in descending order of how much they cost:**
|
|
|
|
1. **A log you cannot query under the obvious name is not a log.** #4 was not
|
|
cosmetic — it produced a confident "this change is unattributed" report
|
|
about a change that had five records.
|
|
2. **Attribution must run FIRST**, never behind another tool's opinion about
|
|
whether an event is interesting.
|
|
3. **An instrument that cannot answer must say so.** A bare unix login where a
|
|
handle belongs is answer-shaped; `unattributed(...)` is not.
|
|
4. **A sub-tool must not drop the claim wrapping its caller.**
|
|
5. **Two samples that look flaky deserve a control-flow audit before a
|
|
concurrency hypothesis.** #2 was diagnosed as an `fcntl` race and was not;
|
|
it was deterministic and five lines up.
|
|
|
|
⚠ **Absence of an entry is not evidence of absence of the event** until you
|
|
have checked that this tool could have recorded it in the first place.
|
|
|
|
## What it deliberately does not do
|
|
|
|
- **It does not cover raw `ssh`.** It cannot. `audit` is the backstop.
|
|
- **It does not claim per-host for elway.** elway records but does not claim:
|
|
two unrelated playbooks on one host are not a collision, and a host-wide lock
|
|
would be ignored within a week.
|
|
- **It does not audit `corviduo-dev`.** Worldtree deploys there are CI/CD-driven
|
|
and rewrite the tree constantly, so it would report unlogged changes forever
|
|
and train us to ignore the output.
|
|
- **It does not audit the SureFire tenant hosts** (`sf-*`, `sfsrv-ana`), which
|
|
are client property under the hosting agreement.
|
|
- **It does not cover DNS, Cloudflare, UniFi or FortiGate changes automatically.**
|
|
Those have no host-side compose tree to diff. Record them by hand with
|
|
`--host <appliance>` until `dns-sync.py` and friends grow the same hook.
|
|
|
|
## Open follow-ons
|
|
|
|
- Hook `scripts/dns-sync.py` (and the UniFi/FortiGate helpers) so
|
|
control-plane changes record themselves too.
|
|
- Run `audit` on a timer and alert on non-zero, rather than only when someone
|
|
remembers to look.
|