From 6d616391fde3d01ed9925c2be2baa7a5f708bae2 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 22 Sep 2026 21:16:12 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20both=20carried?= =?UTF-8?q?=20decisions=20approved,=20safe-rm=20fleet-wide,=20the=20misrou?= =?UTF-8?q?te=20closed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In-flight rewritten for 2026-09-22 evening. The operator approved both carried calls ("decide to do both pending"): the NRestarts flap sampler is to be BUILT (163bb97) and the restic content-assertion ruling is RATIFIED as-is (ba60fda). Recorded in Recent decisions with tracking pointers, and their modality flipped in the handoff from do-not-touch to genuine pending work. safe-rm: installed on nh3-dev, delegated to infra-hermes, now fleet-wide 6/6 with acceptance met. Wiring ruled to /etc/bash.bashrc above the case-$ guard rather than /etc/environment -- an rc self-guards, smaller blast radius, covers bash-over-ssh. ⭐ The guarded acceptance test caught infra-hermes's own silent wiring failure in flight: a probe that cannot destroy what it tests also cannot lie about it. ⚠ Recorded twice over because it will otherwise be trusted for the wrong class: safe-rm is a BLACKLIST. It refuses rm -rf /home and deletes an unset-variable path without complaint. set -u is the actual cover for the habit that prompted the install. D-0010/D-0011: genuine, operator-approved, misrouted to this seat by pane_find matching a ROLLING PANE TITLE. Belayed both, nothing lost. svos-dev owns the fix and found the sharper half -- pane_send structurally cannot deliver a harness command, so D-0010 promised an unachievable /clear and its receipt reported success anyway. {"sent": true} is a claim about transmission, never effect. Live thread carried: restic/ana/esh-docker-vm has drifted 36h -> 44h against a 48h threshold while every other repo sits at 12h. Drifting, not static. --- persistent-memory.md | 86 +++++++++++++++++++++++++++++--------------- 1 file changed, 57 insertions(+), 29 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index d5178c5..cf3c72e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-22 ~13:40 PT (⭐ Booth extracted to vh/booth and handed to booth-dev; Uptime Kuma rebuilt on 2.5.5 as the fleet service layer; the althing alert bridge generalized; failed-START alarms on 23 units; restic checks now assert content and discover repos. ⭐⭐ Five instrument errors in one day, all one shape.)_ +_Last updated: 2026-09-22 ~21:20 PT (⭐ BOTH carried decisions APPROVED — build the NRestarts flap sampler; the restic content-assertion ruling is ratified. safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes with the wiring ruled. D-0010/D-0011 were misrouted here by pane_find matching a rolling pane title — belayed, nothing lost. ⚠ restic/ana/esh-docker-vm has drifted 36h→44h against a 48h threshold.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -115,45 +115,73 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-09-22 ~13:40 PT._ +_As of 2026-09-22 ~21:20 PT._ -### The Booth is booth-dev's now — do not work on it +### ✅ BOTH CARRIED DECISIONS APPROVED — build them -Extracted to its own repo `vh/booth` (`~/development/booth`), all 29 commits -preserved. **booth-dev owns it and has shipped v0.2.0** (asks became one shape of -a `mark` primitive). `eshpfi/services/booth/` is a pointer README only. -What infra-ops still owns: the Uptime Kuma monitor on `:8090/healthz`, which has -never gone DOWN, including through booth-dev's v0.2.0 deploy. +Operator, 2026-09-22 ~21:20: *"decide to do both pending."* They are no longer +decisions; they are work. -### THREE OPERATOR DECISIONS ARE OPEN — none blocking, all reversible +1. **BUILD the `NRestarts` flap sampler.** Design and every trap already written + up in `services/althing-notify-failure/README.md`. Timer reads `NRestarts` + per unit, alarms on delta over a window, reuses the existing cooldown and + alert body. ⚠ Store a last-seen TIMESTAMP beside the count so a counter going + BACKWARDS registers as a reset rather than as quiet — `reset-failed` zeroes + it and sits on the remediation path of the other alarm. Tracking: `163bb97`. +2. **The restic content-assertion ruling is CONFIRMED.** It reached me relayed + by svos-dev rather than Miranda and I built it anyway as reversible; the + operator has now ratified it. Nothing to undo. Tracking: `ba60fda`. -1. **The `NRestarts` flap sampler — designed, traps documented, UNBUILT.** - `OnFailure` cannot fire for 10 of 12 nh3-dev daemons (measured: 9 restarts - over 45s, never `failed`). Full design + every trap: - `services/althing-notify-failure/README.md`. Tracking: commit `163bb97`. -2. **`safe-rm`** — offered by the operator after catching me doing variable-path - `rm` twice. I said I would take it. Not installed. Tracking: this entry. -3. **Confirm or revoke the restic content-assertion ruling** — it reached me - *relayed by svos-dev, who is not Miranda*, so it was information rather than - authorization. Built anyway because reversible + my own recommendation. - Tracking: commit `ba60fda`. +### ✅ safe-rm is fleet-wide — 6/6, closed -### Two loose threads, neither an alarm +Delegated to infra-hermes and complete. Acceptance met on every host as +infra-ops over non-interactive ssh: `command -v rm` resolves to the wrapper, the +guarded probe printed `safe-rm: Skipping /home.`, normal deletes unaffected. +Backups at `/etc/bash.bashrc.bak-saferm` per host. -- ⚠ **`talk.service` is in `failed` state on nh3-dev (exit 143) while `:8092` - serves 200.** The unit is dead; its containers keep running. Nothing manages - talk right now. This is the exact failure class the new notifier exists for. -- **`restic/ana/esh-docker-vm` is 36h old against 12h fleet-wide.** Inside the - 48h threshold so correctly green. Plausibly a missed window from that host's - forced reboot at 22:06 on 09-21, not yet confirmed. +Wiring per my ruling — one self-guarded line above the `case $-` guard in +`/etc/bash.bashrc`, **not** `/etc/environment`: an rc can self-test with `-d`, +its failure blast radius is smaller, and it covers infra-ops-bash-over-ssh which +is the threat path. Accepted loss: cron and non-bash `sh`. -### 17 commits unpushed +⭐ **The acceptance test caught a real bug in flight.** His first insertion pass +failed silently on all six hosts (`$$` collapsed inside heredoc quoting), and +the five `NOT-GUARDED` lines were the guarded probe correctly refusing to test a +dead guard. A probe that cannot destroy what it tests also cannot lie about it. -Push is the operator's call, never automatic. Nothing is half-done in the tree — -working tree is clean and every change above is committed. +⚠ **safe-rm does NOT cover the habit that prompted it.** Measured twice, +independently: it refuses `rm -rf /home` and deletes an unset-variable path +without complaint. Blacklist, not heuristic. `set -u` is the actual cover, and +anyone writing this up must say so or it will be trusted for a class it does not +protect. + +### Live threads + +- ⚠ **`restic/ana/esh-docker-vm` is now 44h old** against a 48h threshold and + 12h for every other repo. It was 36h this afternoon, so it is drifting, not + static. If it crosses 48h the check turns STALE and pages. Plausibly a missed + window from that host's forced reboot on 09-21; **not yet confirmed, and worth + confirming before it alarms.** +- **`talk.service` still `failed` (exit 143) while `:8092` serves 200.** The + unit is dead, its containers keep running, nothing manages talk. Unchanged + since this afternoon. +- `headscale-ddns` recovered on its own (`success`/`inactive`) after the + transient failure at 15:28; the diagnostics added then are untested against a + real recurrence. + +### 20 commits unpushed + +Push is the operator's call. Tree clean, nothing half-done. ## Recent decisions +- `[2026-09-22]` **Both carried calls approved** — build the `NRestarts` flap sampler (`163bb97`); the restic content-assertion ruling is ratified and stays (`ba60fda`). +- `[2026-09-22]` ⭐ **safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.** ⚠ The package installs INERT and looks fine — Debian's `/etc/zsh/zprofile` has 0 non-comment lines so the shipped `profile.d` hook never fires under zsh; verify with `command -v rm`, never `dpkg -l`. ⚠ And it does NOT cover the habit that prompted it: measured, it refuses `rm -rf /home` and deletes an unset-variable path without complaint. Blacklist, not heuristic; `set -u` is the actual cover. Wiring ruled: `/etc/bash.bashrc` above the `case $-` guard, not `/etc/environment` (an rc self-guards, smaller blast radius, covers bash-over-ssh). +- `[2026-09-22]` **The acceptance probe for a guard must not be able to destroy what it tests** (infra-hermes). `rm -rf /home` to prove safe-rm refuses is a test whose premise IS the thing under test. Guarded form is now standard: `[ "$(command -v rm)" = /usr/share/safe-rm/bin/rm ] && rm -rf /home || echo NOT-GUARDED`. +- `[2026-09-22]` ⚠ **`{"sent": true}` is a claim about transmission, never about effect.** `pane_send` structurally cannot deliver a harness command (every relay is prefixed with the card id), so D-0010 promised an unachievable `/clear` and its receipt reported success. Consumed an operator approval. → `persistent-memory.d/2026-09-22-instrument-errors.md` +- `[2026-09-22]` **D-0010/D-0011 were misrouted to this seat by `pane_find` matching a ROLLING PANE TITLE.** Genuine and operator-approved, wrong seat; `fleet_telemetry` held the right mapping and carries the warning, while `pane_find` is the title search and carries none. Belayed both; svos-dev owns the fix in `server/bifrost.py`. ⚠ Do not edit that file. +- `[2026-09-22]` **`headscale-ddns` exited 1 silently and the alarm carried no cause** — both failure paths were `|| exit 1` with no message. Now every exit names its reason and the WAN lookup retries 3×. The script was also untracked; a fix to what every mesh client resolves through lived on one disk. `30517fd`. + - `[2026-09-22]` ⭐⭐ **Five instrument errors in one day, and the shape is one thing: a tool that enumerates "things that are fine" has selected against its own subject.** `--state=running` skipped the units most needing hooks; `awk '{print $1}'` dropped systemd's `●`-decorated FAILED rows; `grep -ic restic` on the wrapper missed the check script; `restic ls`'s header line made an absent path read as "blobs gone"; and a cooldown test that invoked before failing the unit. **Every one reported cleanly while looking at the wrong thing.** The rule is not "verify" — it is *verify, then ask what the verification could not have seen*. → `persistent-memory.d/2026-09-22-instrument-errors.md` - `[2026-09-22]` **Fleet alert bridge generalized** — `beszel-althing` → `althing-alert-bridge`, route registry (`/beszel` + `/kuma`), each with its own prefix/footer/parser. `/beszel` FROZEN and proven byte-identical (its three original tests kept unchanged). An alert that lies about its own source is worse than no alert. Commit `6f0a9b9`. - `[2026-09-22]` **Uptime Kuma rebuilt from scratch on 2.5.5, moved esh-docker-vm → ana-docker.** ⚠ `:latest` is a TRAP — it tracks 1.x, so an Aug-2026 pull gave a Dec-2024 build. No REST CRUD API in either major; automation is Socket.IO via first-party `scripts/kuma` (the community wrapper is abandoned at 1.23.1). Monitors + channel + status page all declared in `stacks/uptimekuma/monitors.yaml`. Commits `3a85a6b`, `94899d6`.