feat(ops-log): attribute host changes across two agents sharing one identity
infra-ops and infra-hermes act as the same OS identity and dockerd does not
log exec per caller, so host-side changes carry no fingerprint. Git cannot
close the gap either: every commit here is attributed to Vuong Hoang by
convention, which is correct for authorship and useless for attribution.
On 2026-09-18 a second session edited the searxng stack mid-deploy, crash-
looping fleet search for ~4 minutes, and the author was unidentifiable.
scripts/ops-log records one line per host-changing action and holds a
lightweight claim so two agents do not deploy the same stack at once.
Four design questions, settled:
* Central on nh3-dev, not per-host and not the post office. Both agents
run as the same unix user there, so one file is shared with zero
provisioning. Per-host needs a writable path on ~25 heterogeneous boxes
and stores "we changed host Y" on host Y. journald looked free but shows
an unprivileged reader only their own _UID, which would have split the
log silently between the infra-ops and lkraven halves of the fleet.
* The claim is advisory and enforced in the tooling. deploy-stack.sh
refuses a foreign claim across the diff, the prompt and the apply -- the
whole review window, which is where the collision happened. Acquire is
mkdir, so it is atomic rather than probably-fine. Stale claims auto-break
and the break is recorded.
* Writers are automatic. deploy-stack.sh and elway record themselves; a log
that depends on remembering is the same class of instrument as a health
check that passes in both states.
* There is a detector. `ops-log audit` asks each host what changed on disk
and compares it to the newest log line for that stack, covering the
manual ssh-and-edit path the automatic writers structurally cannot.
ops-log being absent or broken never blocks a deploy; only a live foreign
claim does. `ops-log baseline` marks the 136 stacks that predate the
instrument so the detector starts from today rather than reporting the whole
fleet forever and training us to ignore it.
An unreachable host reports INCOMPLETE and exit 5, never clean.
This commit is contained in:
@@ -54,6 +54,38 @@ the operator directly. This is standing policy and not a judgement about him.
|
||||
and dockerd exec is not logged per-caller, so host-side changes are fingerprint-less.
|
||||
That is why the ops log exists (below); use it.
|
||||
|
||||
## The ops log — `scripts/ops-log`
|
||||
|
||||
**Every host-changing action gets a line, and a stack you are hand-editing gets
|
||||
a claim.** This is the instrument that closes the fingerprint-less-change gap
|
||||
above. Full rationale and the design decisions behind it:
|
||||
[`docs/pfi/ops-log.md`](docs/pfi/ops-log.md).
|
||||
|
||||
```bash
|
||||
scripts/ops-log claim nh3-docker searxng --why "raising the engine timeout"
|
||||
scripts/ops-log release nh3-docker searxng
|
||||
scripts/ops-log record --host nh3-docker --action restart --target searxng \
|
||||
--detail "docker restart after the conf edit"
|
||||
scripts/ops-log tail --since 24h # what happened today
|
||||
scripts/ops-log claims # who is holding what right now
|
||||
scripts/ops-log audit # on-host changes with NO log line
|
||||
```
|
||||
|
||||
- **`deploy-stack.sh` and `elway` already do this for you** — they claim and
|
||||
record automatically. You never write a line for work you did through them.
|
||||
- **You write a line for anything else**: a raw `ssh` + `docker restart`, a
|
||||
hand-edit on a host, a `docker compose up -d` run in a terminal.
|
||||
- **`deploy-stack.sh` REFUSES (exit 3) a stack another agent has claimed.** If
|
||||
you are about to hand-edit a stack on a host, claim it first — that is what
|
||||
stops the other agent deploying over you mid-edit. Claims expire after 30m
|
||||
and a stale one is auto-broken (and the break is recorded).
|
||||
- **The log is NOT git.** It lives in `.ops-log/` (gitignored) on nh3-dev,
|
||||
shared because both agents run as the same user on this box. Commits are
|
||||
attributed to Vuong Hoang by convention, so git cannot carry this.
|
||||
- ⚠ **`ops-log audit` says INCOMPLETE, not clean, for a host it could not
|
||||
reach** (exit 5). Read the exit code; a host that was never audited is not
|
||||
an audited host.
|
||||
|
||||
## Persistent memory
|
||||
|
||||
`persistent-memory.md` at the repo root captures durable intent and
|
||||
|
||||
Reference in New Issue
Block a user