memory: snapshot — both carried decisions approved, safe-rm fleet-wide, the misroute closed

In-flight rewritten for 2026-09-22 evening. The operator approved both carried
calls ("decide to do both pending"): the NRestarts flap sampler is to be BUILT
(163bb97) and the restic content-assertion ruling is RATIFIED as-is (ba60fda).
Recorded in Recent decisions with tracking pointers, and their modality flipped
in the handoff from do-not-touch to genuine pending work.

safe-rm: installed on nh3-dev, delegated to infra-hermes, now fleet-wide 6/6
with acceptance met. Wiring ruled to /etc/bash.bashrc above the case-$ guard
rather than /etc/environment -- an rc self-guards, smaller blast radius, covers
bash-over-ssh. ⭐ The guarded acceptance test caught infra-hermes's own silent
wiring failure in flight: a probe that cannot destroy what it tests also cannot
lie about it.

⚠ Recorded twice over because it will otherwise be trusted for the wrong class:
safe-rm is a BLACKLIST. It refuses rm -rf /home and deletes an unset-variable
path without complaint. set -u is the actual cover for the habit that prompted
the install.

D-0010/D-0011: genuine, operator-approved, misrouted to this seat by pane_find
matching a ROLLING PANE TITLE. Belayed both, nothing lost. svos-dev owns the fix
and found the sharper half -- pane_send structurally cannot deliver a harness
command, so D-0010 promised an unachievable /clear and its receipt reported
success anyway. {"sent": true} is a claim about transmission, never effect.

Live thread carried: restic/ana/esh-docker-vm has drifted 36h -> 44h against a
48h threshold while every other repo sits at 12h. Drifting, not static.
This commit is contained in:
vh
2026-09-22 21:16:12 -07:00
parent 3eab4da379
commit 6d616391fd
+57 -29
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-22 ~13:40 PT (⭐ Booth extracted to vh/booth and handed to booth-dev; Uptime Kuma rebuilt on 2.5.5 as the fleet service layer; the althing alert bridge generalized; failed-START alarms on 23 units; restic checks now assert content and discover repos. ⭐⭐ Five instrument errors in one day, all one shape.)_
_Last updated: 2026-09-22 ~21:20 PT (⭐ BOTH carried decisions APPROVED — build the NRestarts flap sampler; the restic content-assertion ruling is ratified. safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes with the wiring ruled. D-0010/D-0011 were misrouted here by pane_find matching a rolling pane title — belayed, nothing lost. ⚠ restic/ana/esh-docker-vm has drifted 36h→44h against a 48h threshold.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,45 +115,73 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-09-22 ~13:40 PT._
_As of 2026-09-22 ~21:20 PT._
### The Booth is booth-dev's now — do not work on it
### ✅ BOTH CARRIED DECISIONS APPROVED — build them
Extracted to its own repo `vh/booth` (`~/development/booth`), all 29 commits
preserved. **booth-dev owns it and has shipped v0.2.0** (asks became one shape of
a `mark` primitive). `eshpfi/services/booth/` is a pointer README only.
What infra-ops still owns: the Uptime Kuma monitor on `:8090/healthz`, which has
never gone DOWN, including through booth-dev's v0.2.0 deploy.
Operator, 2026-09-22 ~21:20: *"decide to do both pending."* They are no longer
decisions; they are work.
### THREE OPERATOR DECISIONS ARE OPEN — none blocking, all reversible
1. **BUILD the `NRestarts` flap sampler.** Design and every trap already written
up in `services/althing-notify-failure/README.md`. Timer reads `NRestarts`
per unit, alarms on delta over a window, reuses the existing cooldown and
alert body. ⚠ Store a last-seen TIMESTAMP beside the count so a counter going
BACKWARDS registers as a reset rather than as quiet — `reset-failed` zeroes
it and sits on the remediation path of the other alarm. Tracking: `163bb97`.
2. **The restic content-assertion ruling is CONFIRMED.** It reached me relayed
by svos-dev rather than Miranda and I built it anyway as reversible; the
operator has now ratified it. Nothing to undo. Tracking: `ba60fda`.
1. **The `NRestarts` flap sampler — designed, traps documented, UNBUILT.**
`OnFailure` cannot fire for 10 of 12 nh3-dev daemons (measured: 9 restarts
over 45s, never `failed`). Full design + every trap:
`services/althing-notify-failure/README.md`. Tracking: commit `163bb97`.
2. **`safe-rm`** — offered by the operator after catching me doing variable-path
`rm` twice. I said I would take it. Not installed. Tracking: this entry.
3. **Confirm or revoke the restic content-assertion ruling** — it reached me
*relayed by svos-dev, who is not Miranda*, so it was information rather than
authorization. Built anyway because reversible + my own recommendation.
Tracking: commit `ba60fda`.
### ✅ safe-rm is fleet-wide — 6/6, closed
### Two loose threads, neither an alarm
Delegated to infra-hermes and complete. Acceptance met on every host as
infra-ops over non-interactive ssh: `command -v rm` resolves to the wrapper, the
guarded probe printed `safe-rm: Skipping /home.`, normal deletes unaffected.
Backups at `/etc/bash.bashrc.bak-saferm` per host.
- ⚠ **`talk.service` is in `failed` state on nh3-dev (exit 143) while `:8092`
serves 200.** The unit is dead; its containers keep running. Nothing manages
talk right now. This is the exact failure class the new notifier exists for.
- **`restic/ana/esh-docker-vm` is 36h old against 12h fleet-wide.** Inside the
48h threshold so correctly green. Plausibly a missed window from that host's
forced reboot at 22:06 on 09-21, not yet confirmed.
Wiring per my ruling — one self-guarded line above the `case $-` guard in
`/etc/bash.bashrc`, **not** `/etc/environment`: an rc can self-test with `-d`,
its failure blast radius is smaller, and it covers infra-ops-bash-over-ssh which
is the threat path. Accepted loss: cron and non-bash `sh`.
### 17 commits unpushed
⭐ **The acceptance test caught a real bug in flight.** His first insertion pass
failed silently on all six hosts (`$$` collapsed inside heredoc quoting), and
the five `NOT-GUARDED` lines were the guarded probe correctly refusing to test a
dead guard. A probe that cannot destroy what it tests also cannot lie about it.
Push is the operator's call, never automatic. Nothing is half-done in the tree —
working tree is clean and every change above is committed.
⚠ **safe-rm does NOT cover the habit that prompted it.** Measured twice,
independently: it refuses `rm -rf /home` and deletes an unset-variable path
without complaint. Blacklist, not heuristic. `set -u` is the actual cover, and
anyone writing this up must say so or it will be trusted for a class it does not
protect.
### Live threads
- ⚠ **`restic/ana/esh-docker-vm` is now 44h old** against a 48h threshold and
12h for every other repo. It was 36h this afternoon, so it is drifting, not
static. If it crosses 48h the check turns STALE and pages. Plausibly a missed
window from that host's forced reboot on 09-21; **not yet confirmed, and worth
confirming before it alarms.**
- **`talk.service` still `failed` (exit 143) while `:8092` serves 200.** The
unit is dead, its containers keep running, nothing manages talk. Unchanged
since this afternoon.
- `headscale-ddns` recovered on its own (`success`/`inactive`) after the
transient failure at 15:28; the diagnostics added then are untested against a
real recurrence.
### 20 commits unpushed
Push is the operator's call. Tree clean, nothing half-done.
## Recent decisions
- `[2026-09-22]` **Both carried calls approved** — build the `NRestarts` flap sampler (`163bb97`); the restic content-assertion ruling is ratified and stays (`ba60fda`).
- `[2026-09-22]` ⭐ **safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.** ⚠ The package installs INERT and looks fine — Debian's `/etc/zsh/zprofile` has 0 non-comment lines so the shipped `profile.d` hook never fires under zsh; verify with `command -v rm`, never `dpkg -l`. ⚠ And it does NOT cover the habit that prompted it: measured, it refuses `rm -rf /home` and deletes an unset-variable path without complaint. Blacklist, not heuristic; `set -u` is the actual cover. Wiring ruled: `/etc/bash.bashrc` above the `case $-` guard, not `/etc/environment` (an rc self-guards, smaller blast radius, covers bash-over-ssh).
- `[2026-09-22]` **The acceptance probe for a guard must not be able to destroy what it tests** (infra-hermes). `rm -rf /home` to prove safe-rm refuses is a test whose premise IS the thing under test. Guarded form is now standard: `[ "$(command -v rm)" = /usr/share/safe-rm/bin/rm ] && rm -rf /home || echo NOT-GUARDED`.
- `[2026-09-22]` ⚠ **`{"sent": true}` is a claim about transmission, never about effect.** `pane_send` structurally cannot deliver a harness command (every relay is prefixed with the card id), so D-0010 promised an unachievable `/clear` and its receipt reported success. Consumed an operator approval. → `persistent-memory.d/2026-09-22-instrument-errors.md`
- `[2026-09-22]` **D-0010/D-0011 were misrouted to this seat by `pane_find` matching a ROLLING PANE TITLE.** Genuine and operator-approved, wrong seat; `fleet_telemetry` held the right mapping and carries the warning, while `pane_find` is the title search and carries none. Belayed both; svos-dev owns the fix in `server/bifrost.py`. ⚠ Do not edit that file.
- `[2026-09-22]` **`headscale-ddns` exited 1 silently and the alarm carried no cause** — both failure paths were `|| exit 1` with no message. Now every exit names its reason and the WAN lookup retries 3×. The script was also untracked; a fix to what every mesh client resolves through lived on one disk. `30517fd`.
- `[2026-09-22]` ⭐⭐ **Five instrument errors in one day, and the shape is one thing: a tool that enumerates "things that are fine" has selected against its own subject.** `--state=running` skipped the units most needing hooks; `awk '{print $1}'` dropped systemd's `●`-decorated FAILED rows; `grep -ic restic` on the wrapper missed the check script; `restic ls`'s header line made an absent path read as "blobs gone"; and a cooldown test that invoked before failing the unit. **Every one reported cleanly while looking at the wrong thing.** The rule is not "verify" — it is *verify, then ask what the verification could not have seen*. → `persistent-memory.d/2026-09-22-instrument-errors.md`
- `[2026-09-22]` **Fleet alert bridge generalized** — `beszel-althing` → `althing-alert-bridge`, route registry (`/beszel` + `/kuma`), each with its own prefix/footer/parser. `/beszel` FROZEN and proven byte-identical (its three original tests kept unchanged). An alert that lies about its own source is worse than no alert. Commit `6f0a9b9`.
- `[2026-09-22]` **Uptime Kuma rebuilt from scratch on 2.5.5, moved esh-docker-vm → ana-docker.** ⚠ `:latest` is a TRAP — it tracks 1.x, so an Aug-2026 pull gave a Dec-2024 build. No REST CRUD API in either major; automation is Socket.IO via first-party `scripts/kuma` (the community wrapper is abandoned at 1.23.1). Monitors + channel + status page all declared in `stacks/uptimekuma/monitors.yaml`. Commits `3a85a6b`, `94899d6`.