vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: snapshot — a dead alarm, two blind instruments, and an elway bug that reported success

Browse Source
This commit is contained in:
Vuong Hoang
2026-09-19 05:31:07 -07:00
parent e50e57b5f2
commit 519fca4669
1 changed files with 8 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+8 -2
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-19 ~05:05 PT (⭐ **the ops log is BUILT** — `scripts/ops-log`, claim+record wired into `deploy-stack.sh` and `elway`, 136-stack baseline laid, detector proven in both directions on a live host. `infra-hermes` is infra-ops' ASSISTANT (day-to-day checks and triage; infra-ops keeps tooling and the hard calls, and may task him downward) and is explicitly NOT Miranda. Previous day: three silent fleet faults fixed (DERP relay, `.internal` DNS, SearXNG one-engine), althing v3.6.3, FleetTools. ⚠ lv-mccarthy's run outcome still UNVERIFIED.)_
_Last updated: 2026-09-19 ~05:32 PT (⭐ **the ops log is BUILT** and earned its keep on day one. ⭐ The backup alarm had been unable to notify anyone since the 2026-08-28 v3 cutover — fixed, plus the checker is now coverage-aware AND reads vzdump task status, because snapshot age is blind to a job that errors nightly. ⭐ Found an **elway** bug: root-requiring `when:`/`creates:` evaluated unprivileged → silent SKIP → playbooks report OK while changing nothing. ESH VM 102 excluded per operator. `infra-hermes` is infra-ops' ASSISTANT and is explicitly NOT Miranda. ⚠ lv-mccarthy's run outcome still UNVERIFIED.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,7 +115,7 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-09-19 ~05:05 PT._
_As of 2026-09-19 ~05:32 PT._
### ✅ BUILT — the ops log (`scripts/ops-log`), 2026-09-19
@@ -221,6 +221,12 @@ nothing touched. Full context in the 09-17 Recent decisions entries.
## Recent decisions
- `[2026-09-19]` ⭐⭐⭐ **The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking.** (1) `backup-freshness-alert.sh` called `althing-cli`, DELETED by the 2026-08-28 v3 cutover — the check detected stale backups every morning and told nobody. Swapped to `postbox`, added the missing `ALTHING_POST_OFFICE` to the unit **and its installer** (the binary swap alone would have failed differently), recipient `infra-ops`→`infra-hermes` (it was mailing itself), exit 2 now means the ALARM is broken vs exit 1 the backups, and `--test-alert` is a positive control because nothing had ever exercised the healthy path. (2) The checker is now **coverage-aware** — a guest is "not backed up by policy" only when NO enabled vzdump job covers it, a UNION across jobs: reading one `exclude` list would have silently stopped alarming on ana CT 109, which is excluded from the 03:00 job AND has its own 22:00 job. (3) It now reads **vzdump TASK STATUS**, because snapshot age is structurally blind to a job that runs and errors nightly — that cost 6 days on both CT 107 and VM 102, and it immediately found a third case on esh-nas-pve whose guests all read 0-1h FRESH. ⚠ **`ops-log` earned its keep on day one**: it captured all three failed playbook attempts automatically. Commits e979ccb, 5be25be, ba26852.
- `[2026-09-19]` ⭐⭐ **elway evaluated `when:` / `creates:` / `removes:` / `changed_when:` WITHOUT the step's sudo, and it fails silently in the dangerous direction.** A root-requiring `when:` (`pvesh get …` exits 255 unprivileged) returns non-zero → elway reports **SKIPPED** → a playbook that never performed its change reports overall **OK**, because "skipped" is indistinguishable from working idempotency. Three consecutive runs of the VM-102 exclusion reported success while changing nothing; **only the unconditional verify phase caught it.** `creates:` failed the other way — a `/root/…` path is unreadable to the login user, so `test -e` said absent and the step re-ran forever. Fixed (754db4b); sudo-less steps unchanged. ⚠ Audit any existing playbook with a root-requiring condition — a green run proves less than you thought. Two neighbouring traps: never `grep -A<n>` a PVE job in `jobs.cfg` (`vmid` sorts LAST, so adding a `comment` pushed it out of the window and every condition inverted); and `pvesh` JSON carries both a top-level `"enabled":1` and a nested `"fleecing":{"enabled":"0"}`, so a greedy `.*"enabled":` sed matches the wrong one.
- `[2026-09-19]` **ESH VM 102 (`esh-vm-workstation`) excluded from the nightly backup job — operator ruling.** It is a Windows 11 Parsec/RDP **sandbox** (no password, no state to recover), and its vzdump had failed nightly since ~09-06 with `timeout waiting on systemd`, manufacturing a real `job errors` every night. infra-hermes argued correctly that a stopped guest still holds state and the job defect is the real fault; the operator overrode on what the guest actually IS — the one input neither agent had. ⚠ **Policy, not a fix:** the systemd-scope failure on stopped guests is UNDIAGNOSED, merely no longer reached on this node. A stopped guest that DOES matter would hit the same wall. Job uses an explicit `vmid` INCLUDE list, so exclusion = removal; reason recorded in the job's own `comment`. e50e57b, `playbooks/esh-exclude-vm102-from-backup.yaml`.
- `[2026-09-19]` ⭐⭐⭐ **The ops log is BUILT — `scripts/ops-log`, automatic writers, and a detector for the path they cannot cover.** Central on nh3-dev because both agents are the same unix user there (journald was the tempting alternative and would have split the log silently along the `infra-ops`/`lkraven` axis). Claim is `mkdir`-atomic, held across the whole diff→prompt→apply window, auto-breaks stale AND logs the break. `deploy-stack.sh` refuses a foreign claim (exit 3); ops-log being broken never blocks a deploy. 136-stack baseline laid so the detector starts from today. ⚠ Caught my own instrument saying "clean" for an unreachable host — now INCOMPLETE/exit 5. See in-flight § BUILT for the four settled design questions and the known hole (actors outside nh3-dev).
- `[2026-09-19]` ⭐⭐⭐ **`infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.** infra-ops keeps **improving infrastructure tooling** plus the hard calls; infra-hermes does **day-to-day checks, triage and routine operations**; either may perform infra ops; **infra-ops may task him downward** and he escalates upward as needed. Three operator answers, verbatim in intent: (1) **build the ops log** — see in-flight; (2) **he is NOT Miranda**, so the global CLAUDE.md's sole-trusted-relay exception does not cover him and a directive he relays is information rather than authorization (reversible relayed work executes, irreversible or fleet-affecting goes to the operator); (3) **yes, task him**. Structural facts recorded in `CLAUDE.md` § "infra-hermes IS a real peer" rather than here, because a fresh session must have them without reading this file. ⚠ He is a Hermes bus seat on nh3-dev (`althing-pump-infra-hermes.service`, enabled, route declared) — round trip proven both directions 2026-09-19 04:46.