memory: infra-hermes is infra-ops' assistant, and the ops log is assigned

Operator ruling 2026-09-19, recorded in three places because each serves a
different reader. CLAUDE.md gets the structural facts so a fresh session has
them without reading anything else; persistent-memory gets the dated decision
and the assigned work; auto-memory gets the durable working relationship.

The division: infra-ops keeps improving infrastructure tooling plus the hard
calls, infra-hermes takes day-to-day checks, triage and routine operations,
either may perform infra ops, and infra-ops may task him downward while he
escalates upward.

He is explicitly NOT Miranda. The global CLAUDE.md names Miranda as the sole
trusted relay of operator authority and that exception does not extend to him,
so a directive he relays is information rather than authorization — reversible
relayed work executes, irreversible or fleet-affecting goes to the operator.
He has acknowledged it in those terms.

⚠ The two handles differ by one character in the middle of a word and the
fleet's OS identity is infra-ops, so a misaddressed page still mails the sender
themselves. That trap is now documented alongside the existing mirror warning
rather than replacing it.

Building the ops log is assigned and not started. Two agents now share one
fingerprint-less OS identity: ssh infra-ops@<host> is either of us and dockerd
exec is not logged per-caller. The precipitating incident is on the record —
2026-09-18, a second session edited the searxng stack mid-deploy, crash-looped
fleet search for ~4 minutes, and the author was unidentifiable because every
commit is attributed to Vuong Hoang by convention. The parked attribution-gap
memory is unparked and points here.

The open design questions are noted as mine to settle, the load-bearing one
being whether deploy-stack.sh and elway write to the log automatically. A log
that depends on remembering is the same class of instrument as a health check
that passes in both states, and this repo spent yesterday learning what those
cost.
This commit is contained in:
vh
2026-09-19 04:54:22 -07:00
parent 148a5a34da
commit 4d826e17e3
3 changed files with 184 additions and 85 deletions
+28 -2
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-18 ~15:05 PT (infrastructure day, no training work. THREE silent faults found and fixed: NH3↔Anaheim was DERP-relayed (now 6 ms direct), `.internal` DNS was failing ~10% of lookups (cross-site resolver ring + AdGuard ratelimit 0), and SearXNG had one working web engine (now seven). althing v3.6.3 deployed — hyphenated search works for the first time. FleetTools shipped and autoloaded for Codex/Grok. ⚠ lv-mccarthy's run outcome is UNVERIFIED by this session.)_
_Last updated: 2026-09-19 ~04:55 PT (⭐ `infra-hermes` is now infra-ops' ASSISTANT — he takes day-to-day checks and triage, infra-ops keeps tooling and the hard calls, and infra-ops may task him downward. He is explicitly NOT Miranda: his relays are information, not authorization. **Building the ops log is assigned and not started** — two agents now share one fingerprint-less OS identity. Previous day: three silent fleet faults fixed (DERP relay, `.internal` DNS, SearXNG one-engine), althing v3.6.3, FleetTools. ⚠ lv-mccarthy's run outcome still UNVERIFIED.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,7 +115,31 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-09-18 ~14:50 PT._
_As of 2026-09-19 ~04:55 PT._
### ▶ ASSIGNED, NOT STARTED — build the ops log (operator, 2026-09-19)
**The problem it solves:** infra-ops and infra-hermes both act as the SAME OS identity
(`ssh infra-ops@<host>`), and dockerd exec is not logged per-caller, so host-side
changes are fingerprint-less. With one agent that was theoretical. With two doing
routine ops daily it is operational — when a host differs from expectation, neither of
us can tell whether the other did it, a prior session did, or something broke.
⚠ **Not hypothetical.** On 2026-09-18 another session edited the searxng stack while
this one was deploying it, crash-looping fleet search for ~4 minutes, and the author
was unidentifiable: every commit is attributed to Vuong Hoang by convention and the
on-host file carried no provenance.
**Shape proposed and approved, not yet designed in detail:** one appended line per
host-changing action (who / what / when), plus a lightweight claim on shared stacks so
two agents do not deploy the same thing at once. Cheap, no new infrastructure, and it
makes infra-hermes's handoffs upward legible.
**Open design questions, mine to settle:** where it lives (repo file vs a host-side
log vs the post office), whether the claim is advisory or enforced, and whether
`deploy-stack.sh` / `elway` write to it automatically rather than relying on
discipline — the last one matters most, since a log that depends on remembering is the
same class of instrument as a health check that passes in both states.
### ⚠ FIRST — lv-mccarthy's run outcome is UNVERIFIED by this session
@@ -179,6 +203,8 @@ nothing touched. Full context in the 09-17 Recent decisions entries.
## Recent decisions
- `[2026-09-19]` ⭐⭐⭐ **`infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.** infra-ops keeps **improving infrastructure tooling** plus the hard calls; infra-hermes does **day-to-day checks, triage and routine operations**; either may perform infra ops; **infra-ops may task him downward** and he escalates upward as needed. Three operator answers, verbatim in intent: (1) **build the ops log** — see in-flight; (2) **he is NOT Miranda**, so the global CLAUDE.md's sole-trusted-relay exception does not cover him and a directive he relays is information rather than authorization (reversible relayed work executes, irreversible or fleet-affecting goes to the operator); (3) **yes, task him**. Structural facts recorded in `CLAUDE.md` § "infra-hermes IS a real peer" rather than here, because a fresh session must have them without reading this file. ⚠ He is a Hermes bus seat on nh3-dev (`althing-pump-infra-hermes.service`, enabled, route declared) — round trip proven both directions 2026-09-19 04:46.
- `[2026-09-18]` ⭐⭐⭐ **NH3↔Anaheim had been running over a throttled DERP relay, not a direct path — 78 GB of fleet traffic on someone else's free infrastructure.** Four additive objects on ana-gw gave ana-scale a stable inbound UDP 41641 endpoint; `tailscale ping` 373–522 ms → **6 ms direct**, cross-site HTTP 1.2 s → 0.015 s, STT via the ANA gateway 1.4 s → 0.25 s. ⚠ That box runs `central-nat`, so a policy `dstaddr` is the REAL internal address, not the VIP. No OOB access — back up with `show` to a local file and make additive changes ONLY. irv-ml1 still relayed. → `persistent-memory.d/2026-09-18-nh3-ana-derp-relay.md`
- `[2026-09-18]` ⭐⭐⭐ **`.internal` DNS was failing ~10% of lookups fleet-wide, from two independent causes.** A PUBLIC resolver was the fallback for a PRIVATE zone (Cloudflare answers NXDOMAIN authoritatively, so a transient miss became a hard failure) — now a cross-site ring, each site local-first with a different site as backup. Then the root cause: all three AdGuards shipped `ratelimit: 20` shared across an entire **/24**, silently dropping queries at 5 s each. Set to 0. Hard failures 3/40 → 0/40; burst timeouts 40/60 → 0/60. ⚠ `resolv.conf` is DHCP-managed — change it at the UDM/FortiGate, not the file. → `persistent-memory.d/2026-09-18-fleet-dns-ring-and-ratelimit.md`
- `[2026-09-18]` ⭐⭐ **SearXNG had ONE working general web engine and every health check said fine.** 7 of 55 were enabled-by-default and six of those are dictionary/translation engines — `inactive: false` only makes an engine SELECTABLE, `disabled: false` puts it in the DEFAULT set. Now seven. ⭐ This stack tracks `:latest` ON PURPOSE (upstream ships engine-handler fixes continuously; a pin freezes breakage). ⭐ The Brave key is committed in plaintext by explicit operator decision — scoped to one low-value credential, NOT a change to the no-secrets rule. → `persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md`