memory: snapshot — the 3.7.0 rollout, and three tries to attribute a commit

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-09-19 ~05:32 PT (⭐ **the ops log is BUILT** and earned its keep on day one. ⭐ The backup alarm had been unable to notify anyone since the 2026-08-28 v3 cutover — fixed, plus the checker is now coverage-aware AND reads vzdump task status, because snapshot age is blind to a job that errors nightly. ⭐ Found an **elway** bug: root-requiring `when:`/`creates:` evaluated unprivileged → silent SKIP → playbooks report OK while changing nothing. ESH VM 102 excluded per operator. `infra-hermes` is infra-ops' ASSISTANT and is explicitly NOT Miranda. ⚠ lv-mccarthy's run outcome still UNVERIFIED.)_
+_Last updated: 2026-09-19 ~07:15 PT (⭐ **the ops log is BUILT** and caught its own tooling's failure during a live bus upgrade. ⭐ althing **3.7.0** deployed to the post office + nh3-extdev, forseti-verified. ⭐ The backup alarm had been unable to notify ANYONE since the 2026-08-28 v3 cutover — fixed, plus coverage-awareness and vzdump task-status checking. ⭐ **elway** bug: root-requiring `when:`/`creates:` evaluated unprivileged → silent SKIP → playbooks report OK while changing nothing. ⭐ Commit attribution across three agents now recorded by handle. ⚠ lv-mccarthy's run outcome STILL UNVERIFIED.)_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,7 +115,7 @@ no longer deployed sidecars here. See Recent decisions.)
 
 ## Current state / in-flight
 
-_As of 2026-09-19 ~05:32 PT._
+_As of 2026-09-19 ~07:15 PT._
 
 ### ✅ BUILT — the ops log (`scripts/ops-log`), 2026-09-19
 
@@ -221,6 +221,10 @@ nothing touched. Full context in the 09-17 Recent decisions entries.
 
 ## Recent decisions
 
+- `[2026-09-19]` ⭐⭐ **althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning.** Post office on nh3-docker (built from `althing@6db955f`, manifest `sha256:df0709b3`, **10.9 s** recreate, volume preserved) and the nh3-extdev system wheel + herald. forseti independently verified both. ⚠ Everything CONTENT-verified, never tag-verified: `postbox --version` inside the image before the push and inside the running container after; data continuity proven by reading forseti's own message back out of the running 3.7.0 store. ⚠ **`deploy-stack.sh` rsyncs with `--delete`, so the host-side `.bak-<version>` compose convention is GONE** — it converges rather than accretes; rollback is git history + the retained 3.6.3 registry digest. ⚠ **The claim bug:** a 45-minute operation claim was refreshed *and then released* by deploy-stack.sh's exit trap, silently dropping the protection mid-rollout. Fixed 3e7d3a3 — `ops-log claim` exits **10** when the claim is already the caller's and leaves the holder file UNTOUCHED (a refresh would overwrite the reason and TTL the original claimant chose).
+
+- `[2026-09-19]` ⭐⭐⭐ **Three agents commit as one git author, and closing that gap took three instruments to get right.** An unattributable commit (`e43e262`) appeared in the push set between two of mine — unidentifiable from git (all commits are Vuong Hoang by convention), from the ops log (host changes only) or from the bus. Turned out to be infra-hermes; forseti disclaimed it and the files' own `~/.hermes/hermes-agent` imports gave it away. Three fixes, each exposing the next: **(1)** `0dc8e90` a post-commit hook recording the althing handle; **(2)** `4e778ae` its fallback said `lkraven@nh3-dev`, which *reads like an answer while being the absence of one* — now `unattributed(login)`, and handles never contain `@` or parens; **(3)** `9141a41` the hook was **appended** behind graphify's **eight `exit 0` paths**, so a `graphify-out/`-only or empty commit could NEVER be recorded. ⚠ **Root cause was control flow, not the lock race proposed** — see `feedback_control_flow_before_concurrency`. Attribution must run FIRST, never behind another hook's interestingness filter. ⚠ Also: `ALTHING_HANDLE` lived only in `althing-infra-hermes-seat-run.sh`, not in `hermes-gateway.service` / the pump unit — hermes fixed it with drop-ins. ⚠ **`systemctl show -p Environment` reports the CONFIGURED env, not the running process's** — it read correct while `/proc/<pid>/environ` did not have the variable at all. Check the process, never the config.
+
 - `[2026-09-19]` ⭐⭐⭐ **The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking.** (1) `backup-freshness-alert.sh` called `althing-cli`, DELETED by the 2026-08-28 v3 cutover — the check detected stale backups every morning and told nobody. Swapped to `postbox`, added the missing `ALTHING_POST_OFFICE` to the unit **and its installer** (the binary swap alone would have failed differently), recipient `infra-ops`→`infra-hermes` (it was mailing itself), exit 2 now means the ALARM is broken vs exit 1 the backups, and `--test-alert` is a positive control because nothing had ever exercised the healthy path. (2) The checker is now **coverage-aware** — a guest is "not backed up by policy" only when NO enabled vzdump job covers it, a UNION across jobs: reading one `exclude` list would have silently stopped alarming on ana CT 109, which is excluded from the 03:00 job AND has its own 22:00 job. (3) It now reads **vzdump TASK STATUS**, because snapshot age is structurally blind to a job that runs and errors nightly — that cost 6 days on both CT 107 and VM 102, and it immediately found a third case on esh-nas-pve whose guests all read 0-1h FRESH. ⚠ **`ops-log` earned its keep on day one**: it captured all three failed playbook attempts automatically. Commits e979ccb, 5be25be, ba26852.
 
 - `[2026-09-19]` ⭐⭐ **elway evaluated `when:` / `creates:` / `removes:` / `changed_when:` WITHOUT the step's sudo, and it fails silently in the dangerous direction.** A root-requiring `when:` (`pvesh get …` exits 255 unprivileged) returns non-zero → elway reports **SKIPPED** → a playbook that never performed its change reports overall **OK**, because "skipped" is indistinguishable from working idempotency. Three consecutive runs of the VM-102 exclusion reported success while changing nothing; **only the unconditional verify phase caught it.** `creates:` failed the other way — a `/root/…` path is unreadable to the login user, so `test -e` said absent and the step re-ran forever. Fixed (754db4b); sudo-less steps unchanged. ⚠ Audit any existing playbook with a root-requiring condition — a green run proves less than you thought. Two neighbouring traps: never `grep -A<n>` a PVE job in `jobs.cfg` (`vmid` sorts LAST, so adding a `comment` pushed it out of the window and every condition inverted); and `pvesh` JSON carries both a top-level `"enabled":1` and a nested `"fleecing":{"enabled":"0"}`, so a greedy `.*"enabled":` sed matches the wrong one.