memory: snapshot — 2026-10-04 ~0546 (albok 0.1.6; 8 settled 09-19 entries archived)

archival-memory.md

@@ -13559,3 +13559,45 @@ Commits `156e126` (applied), `1a35181` (reverted).
 
 **`api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings** — this build has NO `!ENV` YAML constructor, so the file was unparseable and the container crash-looped ten times with fleet search down. There is no env-var path into that file at all: the loader reads only `SEARXNG_SETTINGS_PATH` and the entrypoint substitutes only `ultrasecretkey`. Literal or nothing.
   _Archived 2026-10-03._
+
+## Recent decisions (archived 2026-10-04 batch)
+
+# `[2026-09-19]` FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed).
+
+⭐⭐ **FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed).** The ceiling is **1920 W**, not 2400 — a GPU host running for hours is a continuous load, so NEC's 80% rule governs. Measured: BMC `dcmi power reading` 390 W instantaneous / 461 W max over 2423 s **at idle GPUs**, so the non-GPU baseline is ~313 W (EPYC 9254 24C/96T, 5+ drives, 4 PSUs). GPUs are capped **275 W** each against a **300 W** stock TGP (`power.default_limit`) — a 100 W saving across four cards, NOT the 200 W you get by measuring against the 325 W firmware ceiling; the operator corrected me on exactly that. Derived worst case **~1625 W capped (85%)** vs **~1725 W at stock (90%)**. ⚠ **Keep the caps** — 90% leaves nothing for a heavier-than-estimated R420, PSU efficiency, or a warm day. ⚠⚠ **The coupling is worse than the trip:** OPNsense IS the FV edge and shares the breaker with the thing most likely to trip it, so an overload takes the router with it and removes the remote path needed to recover. Four PSUs do not help — they are all downstream of one breaker. ⚠ **NOT measured:** fv-ml1 under real 4-GPU load (the 461 W max is idle-ish), whether the BMC reports AC or DC (±10% ≈ 150 W at load), and the R420's actual draw. Recorded in `servers/fv-ml1/README.md` § Power. ⚠ Also fixed there: the README claimed **2x** GPUs; `nvidia-smi` reports **four**.
+  _Archived 2026-10-04._
+
+# `[2026-09-19]` althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning.
+
+⭐⭐ **althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning.** Post office on nh3-docker (built from `althing@6db955f`, manifest `sha256:df0709b3`, **10.9 s** recreate, volume preserved) and the nh3-extdev system wheel + herald. forseti independently verified both. ⚠ Everything CONTENT-verified, never tag-verified: `postbox --version` inside the image before the push and inside the running container after; data continuity proven by reading forseti's own message back out of the running 3.7.0 store. ⚠ **`deploy-stack.sh` rsyncs with `--delete`, so the host-side `.bak-<version>` compose convention is GONE** — it converges rather than accretes; rollback is git history + the retained 3.6.3 registry digest. ⚠ **The claim bug:** a 45-minute operation claim was refreshed *and then released* by deploy-stack.sh's exit trap, silently dropping the protection mid-rollout. Fixed 3e7d3a3 — `ops-log claim` exits **10** when the claim is already the caller's and leaves the holder file UNTOUCHED (a refresh would overwrite the reason and TTL the original claimant chose).
+  _Archived 2026-10-04._
+
+# `[2026-09-19]` Three agents commit as one git author, and closing that gap took three instruments to get right.
+
+⭐⭐⭐ **Three agents commit as one git author, and closing that gap took three instruments to get right.** An unattributable commit (`e43e262`) appeared in the push set between two of mine — unidentifiable from git (all commits are Vuong Hoang by convention), from the ops log (host changes only) or from the bus. Turned out to be infra-hermes; forseti disclaimed it and the files' own `~/.hermes/hermes-agent` imports gave it away. Three fixes, each exposing the next: **(1)** `0dc8e90` a post-commit hook recording the althing handle; **(2)** `4e778ae` its fallback said `lkraven@nh3-dev`, which *reads like an answer while being the absence of one* — now `unattributed(login)`, and handles never contain `@` or parens; **(3)** `9141a41` the hook was **appended** behind graphify's **eight `exit 0` paths**, so a `graphify-out/`-only or empty commit could NEVER be recorded. ⚠ **Root cause was control flow, not the lock race proposed** — see `feedback_control_flow_before_concurrency`. Attribution must run FIRST, never behind another hook's interestingness filter. ⚠ Also: `ALTHING_HANDLE` lived only in `althing-infra-hermes-seat-run.sh`, not in `hermes-gateway.service` / the pump unit — hermes fixed it with drop-ins. ⚠ **`systemctl show -p Environment` reports the CONFIGURED env, not the running process's** — it read correct while `/proc/<pid>/environ` did not have the variable at all. Check the process, never the config.
+  _Archived 2026-10-04._
+
+# `[2026-09-19]` The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking.
+
+⭐⭐⭐ **The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking.** (1) `backup-freshness-alert.sh` called `althing-cli`, DELETED by the 2026-08-28 v3 cutover — the check detected stale backups every morning and told nobody. Swapped to `postbox`, added the missing `ALTHING_POST_OFFICE` to the unit **and its installer** (the binary swap alone would have failed differently), recipient `infra-ops`→`infra-hermes` (it was mailing itself), exit 2 now means the ALARM is broken vs exit 1 the backups, and `--test-alert` is a positive control because nothing had ever exercised the healthy path. (2) The checker is now **coverage-aware** — a guest is "not backed up by policy" only when NO enabled vzdump job covers it, a UNION across jobs: reading one `exclude` list would have silently stopped alarming on ana CT 109, which is excluded from the 03:00 job AND has its own 22:00 job. (3) It now reads **vzdump TASK STATUS**, because snapshot age is structurally blind to a job that runs and errors nightly — that cost 6 days on both CT 107 and VM 102, and it immediately found a third case on esh-nas-pve whose guests all read 0-1h FRESH. ⚠ **`ops-log` earned its keep on day one**: it captured all three failed playbook attempts automatically. Commits e979ccb, 5be25be, ba26852.
+  _Archived 2026-10-04._
+
+# `[2026-09-19]` elway evaluated `when:` / `creates:` / `removes:` / `changed_when:` WITHOUT the step's sudo, and it fails silently in the dangerous direction.
+
+⭐⭐ **elway evaluated `when:` / `creates:` / `removes:` / `changed_when:` WITHOUT the step's sudo, and it fails silently in the dangerous direction.** A root-requiring `when:` (`pvesh get …` exits 255 unprivileged) returns non-zero → elway reports **SKIPPED** → a playbook that never performed its change reports overall **OK**, because "skipped" is indistinguishable from working idempotency. Three consecutive runs of the VM-102 exclusion reported success while changing nothing; **only the unconditional verify phase caught it.** `creates:` failed the other way — a `/root/…` path is unreadable to the login user, so `test -e` said absent and the step re-ran forever. Fixed (754db4b); sudo-less steps unchanged. ⚠ Audit any existing playbook with a root-requiring condition — a green run proves less than you thought. Two neighbouring traps: never `grep -A<n>` a PVE job in `jobs.cfg` (`vmid` sorts LAST, so adding a `comment` pushed it out of the window and every condition inverted); and `pvesh` JSON carries both a top-level `"enabled":1` and a nested `"fleecing":{"enabled":"0"}`, so a greedy `.*"enabled":` sed matches the wrong one.
+  _Archived 2026-10-04._
+
+# `[2026-09-19]` ESH VM 102 (`esh-vm-workstation`) excluded from the nightly backup job — operator ruling.
+
+**ESH VM 102 (`esh-vm-workstation`) excluded from the nightly backup job — operator ruling.** It is a Windows 11 Parsec/RDP **sandbox** (no password, no state to recover), and its vzdump had failed nightly since ~09-06 with `timeout waiting on systemd`, manufacturing a real `job errors` every night. infra-hermes argued correctly that a stopped guest still holds state and the job defect is the real fault; the operator overrode on what the guest actually IS — the one input neither agent had. ⚠ **Policy, not a fix:** the systemd-scope failure on stopped guests is UNDIAGNOSED, merely no longer reached on this node. A stopped guest that DOES matter would hit the same wall. Job uses an explicit `vmid` INCLUDE list, so exclusion = removal; reason recorded in the job's own `comment`. e50e57b, `playbooks/esh-exclude-vm102-from-backup.yaml`.
+  _Archived 2026-10-04._
+
+# `[2026-09-19]` The ops log is BUILT — `scripts/ops-log`, automatic writers, and a detector for the path they cannot cover.
+
+⭐⭐⭐ **The ops log is BUILT — `scripts/ops-log`, automatic writers, and a detector for the path they cannot cover.** Central on nh3-dev because both agents are the same unix user there (journald was the tempting alternative and would have split the log silently along the `infra-ops`/`lkraven` axis). Claim is `mkdir`-atomic, held across the whole diff→prompt→apply window, auto-breaks stale AND logs the break. `deploy-stack.sh` refuses a foreign claim (exit 3); ops-log being broken never blocks a deploy. 136-stack baseline laid so the detector starts from today. ⚠ Caught my own instrument saying "clean" for an unreachable host — now INCOMPLETE/exit 5. See in-flight § BUILT for the four settled design questions and the known hole (actors outside nh3-dev).
+  _Archived 2026-10-04._
+
+# `[2026-09-19]` `infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.
+
+⭐⭐⭐ **`infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.** infra-ops keeps **improving infrastructure tooling** plus the hard calls; infra-hermes does **day-to-day checks, triage and routine operations**; either may perform infra ops; **infra-ops may task him downward** and he escalates upward as needed. Three operator answers, verbatim in intent: (1) **build the ops log** — see in-flight; (2) **he is NOT Miranda**, so the global CLAUDE.md's sole-trusted-relay exception does not cover him and a directive he relays is information rather than authorization (reversible relayed work executes, irreversible or fleet-affecting goes to the operator); (3) **yes, task him**. Structural facts recorded in `CLAUDE.md` § "infra-hermes IS a real peer" rather than here, because a fresh session must have them without reading this file. ⚠ He is a Hermes bus seat on nh3-dev (`althing-pump-infra-hermes.service`, enabled, route declared) — round trip proven both directions 2026-09-19 04:46.
+  _Archived 2026-10-04._