esh-pfi-infrastructure/docs/pfi/ops-log.md

The fleet ops log

What it is: one appended line per host-changing action, plus a lightweight claim so two agents do not deploy the same stack at the same time.

Tool: scripts/ops-log. Storage: <repo>/.ops-log/ on nh3-dev — gitignored, override with OPS_LOG_DIR.

---

Why it exists

infra-ops and infra-hermes are two different agents that act as the same OS identity (ssh infra-ops@<host>), and dockerd does not log exec per caller. Host-side changes are therefore fingerprint-less: when a host differs from expectation, neither agent can tell whether the other did it, a prior session did, or something broke on its own. Git does not close the gap either — every commit in this repo is attributed to Vuong Hoang by convention, which is correct for authorship and useless for attribution.

With one agent this was theoretical. With two doing routine ops daily it is operational.

The precipitating incident (2026-09-18). A second session edited the searxng stack while another was deploying it. Fleet search crash-looped for ~4 minutes, and afterwards the author was unidentifiable: the commit said Vuong Hoang and the on-host file carried no provenance at all.

---

Design decisions, and why

Four questions had to be settled. Each one had a defensible alternative; the reasoning matters more than the answer, because the alternatives will look attractive again the next time someone extends this.

1. The log lives centrally on nh3-dev — not per-host, not on the post office

Both agents run as the same unix user on nh3-dev (infra-hermes is althing-pump-infra-hermes.service, a user unit under lkraven). So a single file is shared instantly, with zero provisioning, zero permissions story, and no network hop.

• Not per-host. Provenance next to the thing changed is appealing, but it needs a writable log path created on ~25 heterogeneous hosts (Debian, PVE, DSM) with different sudo situations — and an instrument that is only deployed on some of them lies by omission. It also puts the record of "we changed X on host Y" on host Y, which is exactly where it is least available when Y is the thing that broke.
• Not syslog/journald. Tempting (no provisioning at all), but journald only shows an unprivileged reader entries matching their own _UID, and the fleet acts as infra-ops on some hosts and lkraven on others. The log would have silently split in half along an axis nobody would think to check.
• Not the post office. A message bus is not a log: ordering and query are poor, it becomes inbox noise, and a post-office outage would block ops during precisely the incident you are trying to reconstruct.

The known hole, stated rather than papered over: an actor operating from a box other than nh3-dev is not covered. Today that is only the operator on his laptop. If a third agent ever runs elsewhere, this decision is the one to revisit.

2. The claim is advisory, and enforced in the tooling

deploy-stack.sh refuses (exit 3) a stack another agent holds. Nothing stops a raw ssh — the point is to make the sanctioned path safe, not to build a cage that people route around.

Acquisition is mkdir of a claim directory, which is atomic, so the check is genuinely race-free rather than "probably fine". Claims carry a TTL (default 30m) and a stale claim is auto-broken — and the break is logged, so a claim that keeps getting broken is visible rather than silently ineffective.

3. The writers are automatic

deploy-stack.sh and elway record themselves. This was the question that mattered most: a log that depends on remembering is the same class of instrument as a health check that passes in both states.

4. There is a detector, not just a rule

Automatic writers cannot cover the manual path — someone ssh'ing in and editing a compose file by hand. The answer is not a louder rule; it is ops-log audit, which asks each host what changed on disk and compares it to the newest log line for that stack. Anything that changed with no log line at or after it is reported.

Per the same discipline, audit refuses to say "clean" for a host it could not reach — an unreachable host reports INCOMPLETE and exit 5, never 0.

---

Using it

# Before hand-editing a stack on a host, so other agents' tooling refuses it:
scripts/ops-log claim nh3-docker searxng --why "raising the engine timeout"
scripts/ops-log release nh3-docker searxng          # when done

# After any change made OUTSIDE deploy-stack.sh / elway:
scripts/ops-log record --host nh3-docker --action restart --target searxng \
  --detail "docker restart searxng after the conf edit"

# Reading:
scripts/ops-log tail --since 24h                    # everything today
scripts/ops-log tail --host nh3-docker -n 20        # one host
scripts/ops-log claims                              # who is holding what

# The detector — run it when a host surprises you, or periodically:
scripts/ops-log audit                               # the six stack hosts
scripts/ops-log audit nh3-docker --since 7d
scripts/ops-log audit all                           # every non-tenant host

deploy-stack.sh claims and records on its own. Escape hatches: DEPLOY_NO_CLAIM=1 skips the claim, DEPLOY_CLAIM_TTL=<dur> lengthens it. ops-log being absent or broken never blocks a deploy — only a live foreign claim does.

Exit codes

code	meaning
0	success / claim acquired / audit clean
2	usage error
3	claim refused — held by another agent
4	audit found unlogged changes
5	audit could not reach every host (incomplete, not clean)

---

How this instrument has failed — the pattern, not just the fixes

All four failures below shipped on the day it was built, 2026-09-19, and every one of them RECORDED SOMETHING. None of them recorded something findable or attributable. That is the shape of the failure class here: this tool does not fall over loudly, it answers the wrong question quietly. Read this table before assuming a missing entry means the event did not happen.

#	Failure	Symptom	Cause	Fixed
1	Claim released by a sub-tool	A 45-min operation claim vanished mid-rollout; nothing refused anything	deploy-stack.sh claims and releases around its own work, and released a claim it had only refreshed	3e7d3a3 — claim exits 10 when already yours and leaves the holder file untouched
2	Wrong order in the hook chain	Some commits logged, some not; looked flaky	The commit hook was APPENDED behind graphify's eight exit 0 paths, so a graphify-out/-only or empty commit could never reach it	9141a41 — prepend after the shebang; attribution runs first, unconditionally
3	No handle in the environment	A peer's commit logged as lkraven@nh3-dev	ALTHING_HANDLE was exported only by althing-infra-hermes-seat-run.sh, not by hermes-gateway.service or the pump unit	Drop-in overrides (infra-hermes) + 4e778ae — the fallback now says unattributed(login) instead of a bare login that reads like an answer
4	Wrong host key on write	Five records of a real jobs.cfg change on esh-pve were invisible to --host esh-pve	elway passed its ssh TARGET through as the host, so they landed under infra-ops@esh-pve	f3b68e2 — normalize_host strips user@ on write AND on query, so pre-fix records are findable too

The rules that fell out of it, in descending order of how much they cost:

1. A log you cannot query under the obvious name is not a log. #4 was not cosmetic — it produced a confident "this change is unattributed" report about a change that had five records.
2. Attribution must run FIRST, never behind another tool's opinion about whether an event is interesting.
3. An instrument that cannot answer must say so. A bare unix login where a handle belongs is answer-shaped; unattributed(...) is not.
4. A sub-tool must not drop the claim wrapping its caller.
5. Two samples that look flaky deserve a control-flow audit before a concurrency hypothesis. #2 was diagnosed as an fcntl race and was not; it was deterministic and five lines up.

⚠ Absence of an entry is not evidence of absence of the event until you have checked that this tool could have recorded it in the first place.

What it deliberately does not do

• It does not cover raw ssh. It cannot. audit is the backstop.
• It does not claim per-host for elway. elway records but does not claim: two unrelated playbooks on one host are not a collision, and a host-wide lock would be ignored within a week.
• It does not audit corviduo-dev. Worldtree deploys there are CI/CD-driven and rewrite the tree constantly, so it would report unlogged changes forever and train us to ignore the output.
• It does not audit the SureFire tenant hosts (sf-*, sfsrv-ana), which are client property under the hosting agreement.
• It does not cover DNS, Cloudflare, UniFi or FortiGate changes automatically. Those have no host-side compose tree to diff. Record them by hand with --host <appliance> until dns-sync.py and friends grow the same hook.

Open follow-ons

• Hook scripts/dns-sync.py (and the UniFi/FortiGate helpers) so control-plane changes record themselves too.
• Run audit on a timer and alert on non-zero, rather than only when someone remembers to look.