vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(ops-log): attribute host changes across two agents sharing one identity

Browse Source 
infra-ops and infra-hermes act as the same OS identity and dockerd does not
log exec per caller, so host-side changes carry no fingerprint. Git cannot
close the gap either: every commit here is attributed to Vuong Hoang by
convention, which is correct for authorship and useless for attribution.
On 2026-09-18 a second session edited the searxng stack mid-deploy, crash-
looping fleet search for ~4 minutes, and the author was unidentifiable.

scripts/ops-log records one line per host-changing action and holds a
lightweight claim so two agents do not deploy the same stack at once.

Four design questions, settled:

  * Central on nh3-dev, not per-host and not the post office. Both agents
    run as the same unix user there, so one file is shared with zero
    provisioning. Per-host needs a writable path on ~25 heterogeneous boxes
    and stores "we changed host Y" on host Y. journald looked free but shows
    an unprivileged reader only their own _UID, which would have split the
    log silently between the infra-ops and lkraven halves of the fleet.
  * The claim is advisory and enforced in the tooling. deploy-stack.sh
    refuses a foreign claim across the diff, the prompt and the apply -- the
    whole review window, which is where the collision happened. Acquire is
    mkdir, so it is atomic rather than probably-fine. Stale claims auto-break
    and the break is recorded.
  * Writers are automatic. deploy-stack.sh and elway record themselves; a log
    that depends on remembering is the same class of instrument as a health
    check that passes in both states.
  * There is a detector. `ops-log audit` asks each host what changed on disk
    and compares it to the newest log line for that stack, covering the
    manual ssh-and-edit path the automatic writers structurally cannot.

ops-log being absent or broken never blocks a deploy; only a live foreign
claim does. `ops-log baseline` marks the 136 stacks that predate the
instrument so the detector starts from today rather than reporting the whole
fleet forever and training us to ignore it.

An unreachable host reports INCOMPLETE and exit 5, never clean.
This commit is contained in:
vh
2026-09-19 05:05:43 -07:00
parent 4d826e17e3
commit ffe7b24935
7 changed files with 960 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+41 -21
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-19 ~04:55 PT (⭐ `infra-hermes` is now infra-ops' ASSISTANT — he takes day-to-day checks and triage, infra-ops keeps tooling and the hard calls, and infra-ops may task him downward. He is explicitly NOT Miranda: his relays are information, not authorization. **Building the ops log is assigned and not started** — two agents now share one fingerprint-less OS identity. Previous day: three silent fleet faults fixed (DERP relay, `.internal` DNS, SearXNG one-engine), althing v3.6.3, FleetTools. ⚠ lv-mccarthy's run outcome still UNVERIFIED.)_
_Last updated: 2026-09-19 ~05:05 PT (⭐ **the ops log is BUILT** — `scripts/ops-log`, claim+record wired into `deploy-stack.sh` and `elway`, 136-stack baseline laid, detector proven in both directions on a live host. `infra-hermes` is infra-ops' ASSISTANT (day-to-day checks and triage; infra-ops keeps tooling and the hard calls, and may task him downward) and is explicitly NOT Miranda. Previous day: three silent fleet faults fixed (DERP relay, `.internal` DNS, SearXNG one-engine), althing v3.6.3, FleetTools. ⚠ lv-mccarthy's run outcome still UNVERIFIED.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,31 +115,49 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-09-19 ~04:55 PT._
_As of 2026-09-19 ~05:05 PT._
### ▶ ASSIGNED, NOT STARTED — build the ops log (operator, 2026-09-19)
### ✅ BUILT — the ops log (`scripts/ops-log`), 2026-09-19
**The problem it solves:** infra-ops and infra-hermes both act as the SAME OS identity
(`ssh infra-ops@<host>`), and dockerd exec is not logged per-caller, so host-side
changes are fingerprint-less. With one agent that was theoretical. With two doing
routine ops daily it is operational — when a host differs from expectation, neither of
us can tell whether the other did it, a prior session did, or something broke.
Shipped. `scripts/ops-log` + `docs/pfi/ops-log.md`, wired into `deploy-stack.sh`
(claims + records) and `elway` (records). Baseline laid: **136 stacks across 6 hosts**
marked pre-ops-log, so the detector starts from today instead of reporting the whole
fleet as unattributable forever.
⚠ **Not hypothetical.** On 2026-09-18 another session edited the searxng stack while
this one was deploying it, crash-looping fleet search for ~4 minutes, and the author
was unidentifiable: every commit is attributed to Vuong Hoang by convention and the
on-host file carried no provenance.
**The four open questions, settled:**
**Shape proposed and approved, not yet designed in detail:** one appended line per
host-changing action (who / what / when), plus a lightweight claim on shared stacks so
two agents do not deploy the same thing at once. Cheap, no new infrastructure, and it
makes infra-hermes's handoffs upward legible.
1. **Where it lives — CENTRAL on nh3-dev** (`<repo>/.ops-log/`, gitignored), not
per-host and not the post office. Decider: both agents run as the *same unix user*
on nh3-dev (infra-hermes is a **user** unit under `lkraven`), so one file is shared
instantly with zero provisioning. Per-host needs a writable path on ~25 heterogeneous
boxes and puts the record of "we changed host Y" *on host Y*. syslog/journald looked
free but journald shows an unprivileged reader only their own `_UID` — the log would
have split silently between the `infra-ops` and `lkraven` halves of the fleet. The
post office is a bus, and an outage there would block ops during the incident you are
reconstructing. ⚠ **Known hole, stated not papered over:** an actor on a box *other
than nh3-dev* is uncovered. Today that is only the operator's laptop; a third agent
elsewhere is what would force a revisit.
2. **Claim = advisory, enforced in the tooling.** `deploy-stack.sh` refuses (exit 3) a
stack another agent holds, across the diff, the y/N prompt AND the apply — the whole
review window, which is where the 09-18 collision actually happened. Acquire is
`mkdir` (atomic → genuinely race-free). TTL 30m; a stale claim auto-breaks **and the
break is logged**, so an ineffective claim is visible rather than silent.
3. **Writers are AUTOMATIC.** This was the one that mattered — a log you must remember
to write is the same class of instrument as a health check that passes in both states.
4. **There is a DETECTOR, not just a rule.** `ops-log audit` asks each host what changed
on disk and compares it to the newest log line for that stack. Covers the manual
`ssh`-and-edit path the automatic writers structurally cannot.
**Open design questions, mine to settle:** where it lives (repo file vs a host-side
log vs the post office), whether the claim is advisory or enforced, and whether
`deploy-stack.sh` / `elway` write to it automatically rather than relying on
discipline — the last one matters most, since a log that depends on remembering is the
same class of instrument as a health check that passes in both states.
⚠ **Found and fixed a bug in my own detector mid-build:** it printed "audit clean" for a
host it never reached. Now `INCOMPLETE` + exit 5 — an unreachable host is not a clean
host. Both directions proven on a live host: after baseline, a mtime-only `touch` on
`nh3-docker/beszel-agent-nh3` fired the detector at 15 s resolution, and recording it
cleared it.
**Not covered, on purpose:** raw `ssh` (audit is the backstop), per-host claims for elway,
`corviduo-dev` (CI/CD rewrites the tree constantly → permanent false positives), the
SureFire tenant hosts. **Follow-ons:** hook `dns-sync.py` / UniFi / FortiGate helpers so
control-plane changes record themselves; run `audit` on a timer.
### ⚠ FIRST — lv-mccarthy's run outcome is UNVERIFIED by this session
@@ -203,6 +221,8 @@ nothing touched. Full context in the 09-17 Recent decisions entries.
## Recent decisions
- `[2026-09-19]` ⭐⭐⭐ **The ops log is BUILT — `scripts/ops-log`, automatic writers, and a detector for the path they cannot cover.** Central on nh3-dev because both agents are the same unix user there (journald was the tempting alternative and would have split the log silently along the `infra-ops`/`lkraven` axis). Claim is `mkdir`-atomic, held across the whole diff→prompt→apply window, auto-breaks stale AND logs the break. `deploy-stack.sh` refuses a foreign claim (exit 3); ops-log being broken never blocks a deploy. 136-stack baseline laid so the detector starts from today. ⚠ Caught my own instrument saying "clean" for an unreachable host — now INCOMPLETE/exit 5. See in-flight § BUILT for the four settled design questions and the known hole (actors outside nh3-dev).
- `[2026-09-19]` ⭐⭐⭐ **`infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.** infra-ops keeps **improving infrastructure tooling** plus the hard calls; infra-hermes does **day-to-day checks, triage and routine operations**; either may perform infra ops; **infra-ops may task him downward** and he escalates upward as needed. Three operator answers, verbatim in intent: (1) **build the ops log** — see in-flight; (2) **he is NOT Miranda**, so the global CLAUDE.md's sole-trusted-relay exception does not cover him and a directive he relays is information rather than authorization (reversible relayed work executes, irreversible or fleet-affecting goes to the operator); (3) **yes, task him**. Structural facts recorded in `CLAUDE.md` § "infra-hermes IS a real peer" rather than here, because a fresh session must have them without reading this file. ⚠ He is a Hermes bus seat on nh3-dev (`althing-pump-infra-hermes.service`, enabled, route declared) — round trip proven both directions 2026-09-19 04:46.
- `[2026-09-18]` ⭐⭐⭐ **NH3↔Anaheim had been running over a throttled DERP relay, not a direct path — 78 GB of fleet traffic on someone else's free infrastructure.** Four additive objects on ana-gw gave ana-scale a stable inbound UDP 41641 endpoint; `tailscale ping` 373–522 ms → **6 ms direct**, cross-site HTTP 1.2 s → 0.015 s, STT via the ANA gateway 1.4 s → 0.25 s. ⚠ That box runs `central-nat`, so a policy `dstaddr` is the REAL internal address, not the VIP. No OOB access — back up with `show` to a local file and make additive changes ONLY. irv-ml1 still relayed. → `persistent-memory.d/2026-09-18-nh3-ana-derp-relay.md`