memory: snapshot — Booth handed to booth-dev, Kuma rebuilt, alerts generalized, five instrument errors

In-flight rewritten for 2026-09-22. The Booth is booth-dev's now (extracted to
vh/booth, v0.2.0 shipped by them); infra-ops retains only its Uptime Kuma
monitor.

Eight decisions recorded, and three OPEN OPERATOR DECISIONS filed under Recent
decisions with commit tracking pointers rather than under in-flight, per the
deferred-work rule: the NRestarts flap sampler (163bb97), safe-rm, and
confirming the restic content-assertion ruling that reached me relayed by
svos-dev rather than Miranda (ba60fda).

Detail file for the day's throughline: five instrument errors, one shape -- a
tool that enumerates 'things that are fine' has selected against its own
subject. Four were mine, one a peer's, and mine included making theirs, on the
same day we were both cataloguing the habit. Written down because a rule both
parties broke while discussing the rule needs to live where a fresh session
reads it.

Two loose threads carried: talk.service failed while still serving on :8092,
and restic/ana/esh-docker-vm 36h behind the fleet.

persistent-memory.d/2026-09-22-instrument-errors.md

@@ -0,0 +1,71 @@
+# Five instrument errors in one day — and they are one shape
+
+`[2026-09-22]` · infra-ops, in a running exchange with svos-dev
+
+## The shape
+
+> **A tool that enumerates "things that are fine" and acts on them has selected
+> against its own subject.** (svos-dev's framing, taken close to verbatim.)
+
+None of these failed randomly. Several had failure modes *anti-correlated with
+their purpose* — the healthier the fleet looked, the better they appeared to
+work. That is worse than an ordinary broken instrument, which at least fails
+visibly on the easy cases too.
+
+## The five
+
+1. **`--state=running` in `install.sh`** (`ef70b2f`). Selected only healthy
+   units to hook, so a unit already down at install time was never hooked. The
+   unit most needing the alarm is by definition the one that is not healthy.
+   Found because `talk.service` was sitting in `failed` and uncovered.
+
+2. **`awk '{print $1}'` over `systemctl list-units`** (`ef70b2f`). systemd
+   decorates a FAILED unit with a leading `●`, so field 1 is the bullet, not
+   the name; the sanitiser then reduced it to empty and dropped the row. The
+   parser silently lost *exactly the rows that matter*. Fix: `--plain`.
+
+3. **`grep -ic restic backup-freshness-alert.sh` → 0** (`ba60fda`). That is the
+   WRAPPER. The checking lives in `check-backup-freshness.sh`, which had always
+   checked seven hosts. I grepped one file of a two-file tool and generalised to
+   the whole system — **the identical error svos-dev had made with `dev-backup`
+   four hours earlier, made by me while correcting them for it.**
+
+4. **`restic ls` always prints a header line** (`ba60fda`). Absent path returns
+   1 line, real path returned 6; counting with `grep -c .` read the header as a
+   hit, so the ABSENT branch fell through and reported *"index intact, blobs
+   gone"*. An operator would have been sent to a repo-corruption runbook by a
+   mistyped path — wrong in the most expensive available direction. Found ONLY
+   by running the negative control. Fix: `grep -c '^/'`.
+
+5. **A cooldown test that invoked the script before failing the unit**
+   (`02a6fc0`). The two calls legitimately saw different states, computed
+   different fingerprints, and the suppression correctly did not fire — so the
+   test "proved" a bug that did not exist. The code was right; the test was
+   wrong.
+
+Adjacent, same family: **"nothing to count is not the same as not counting."**
+An `NRestarts` probe read flat after a manual restart and would have concluded a
+sampler goes blind whenever a human touches a unit. Artifact of the test — that
+unit had already hit its start limit and stopped auto-restarting. The instrument
+was reading correctly from a subject that had stopped producing.
+
+## The rule this cost
+
+**Not "verify". *Verify, then ask what the verification could not have seen.***
+
+svos-dev sharpened it one notch and the sharpening is the useful half:
+
+> The rung where you are CHECKING an instrument is the rung where you are least
+> likely to check, because you are already in verification posture and it feels
+> like the checking is happening.
+
+## Why it is written down
+
+Four of the five were mine, one was svos-dev's, and mine included making theirs.
+Neither of us is careless — we each checked one instrument and generalised from
+it, on the same day, while cataloguing the habit. A rule that both parties broke
+while discussing the rule is a rule that needs to be written where a fresh
+session will read it, not left as a lesson someone remembers.
+
+Related: [[2026-09-21-booth-two-dead-controls]] — same family, found by the
+operator rather than by me, because I read templates instead of rendering them.

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-09-21 ~21:05 PT (⭐ **the ops log is BUILT** and then failed four ways in its first hours — every one recording something unfindable; the day's subject was instruments that report without looking. ⭐ Draupnir engine COMPLETE and acceptance-tested on irv-ml1. ⭐ Booth gained blur + a closed keep round trip after shipping two dead controls. ⭐ claude-bot is an org Owner; cicada+draupnir moved to `pfi`; vh-token use standing-authorized from the vault. ⭐ nh3-dev 84%→72%, and a LoRA adapter rescued from a 3-day-swept /tmp. ⭐⭐⭐ lv-mccarthy SHIPPED (ckpt300, fourth voice on the seat) — memorisation identical to the author, voice best-of-five, at a cost of 35% out-of-band generations; the two-epoch recipe is 0 for 3 with the loss curve CONFIDENTLY wrong; I held the ship three times and only the first hold was right; and every voice on the seat has been emitting empty <think> tags since 2026-09-16.)_
+_Last updated: 2026-09-22 ~13:40 PT (⭐ Booth extracted to vh/booth and handed to booth-dev; Uptime Kuma rebuilt on 2.5.5 as the fleet service layer; the althing alert bridge generalized; failed-START alarms on 23 units; restic checks now assert content and discover repos. ⭐⭐ Five instrument errors in one day, all one shape.)_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,84 +115,54 @@ no longer deployed sidecars here. See Recent decisions.)
 
 ## Current state / in-flight
 
-_As of 2026-09-21 ~21:05 PT._
+_As of 2026-09-22 ~13:40 PT._
 
-### NEXT UP — the Booth (`services/booth`)
+### The Booth is booth-dev's now — do not work on it
 
-The operator is moving to Booth work this session. Nothing is specified yet; ask
+Extracted to its own repo `vh/booth` (`~/development/booth`), all 29 commits
-what they want rather than assuming from the backlog below. Context worth having
+preserved. **booth-dev owns it and has shipped v0.2.0** (asks became one shape of
-before the first question:
+a `mark` primitive). `eshpfi/services/booth/` is a pointer README only.
+What infra-ops still owns: the Uptime Kuma monitor on `:8090/healthz`, which has
+never gone DOWN, including through booth-dev's v0.2.0 deploy.
 
-- **What it is:** ephemeral media board on nh3-dev, `http://10.100.10.50:8090`,
+### THREE OPERATOR DECISIONS ARE OPEN — none blocking, all reversible
-  drop a folder in `~/booth-data`, 24 h TTL, A/B renders. `booth link <url> "<desc>"`
-  is the fleet link-board every session is told to post operator-facing links to.
-- ⚠ **Its last two sessions each shipped a control that did nothing** — a reveal
-  handler Jinja discarded for sitting after `{% endblock %}`, and a `×` a sibling
-  form covered by 30×22 px. **Both were found by the operator, not by me, because
-  I read templates instead of rendering them.** `scripts/layout-probe.py` exists
-  now and took four iterations to become trustworthy — use it.
-  → `persistent-memory.d/2026-09-21-booth-two-dead-controls.md`
-- The ASKS primitive (`booth ask|asks|answer --wait`) and inline asks-in-a-custom-
-  report are both live; see the 2026-09-09 decisions.
 
-### ✅ lv-mccarthy SHIPPED — ckpt300 is the fourth voice on `voices-seat`
+1. **The `NRestarts` flap sampler — designed, traps documented, UNBUILT.**
+   `OnFailure` cannot fire for 10 of 12 nh3-dev daemons (measured: 9 restarts
+   over 45s, never `failed`). Full design + every trap:
+   `services/althing-notify-failure/README.md`. Tracking: commit `163bb97`.
+2. **`safe-rm`** — offered by the operator after catching me doing variable-path
+   `rm` twice. I said I would take it. Not installed. Tracking: this entry.
+3. **Confirm or revoke the restic content-assertion ruling** — it reached me
+   *relayed by svos-dev, who is not Miranda*, so it was information rather than
+   authorization. Built anyway because reversible + my own recommendation.
+   Tracking: commit `ba60fda`.
 
-Live on fv-ml1 GPU0 `:8027` as `lv-mccarthy`, verified by read-back: seat healthy,
+### Two loose threads, neither an alarm
-five models, GPU0 96,012 MiB (a LoRA costs nothing beside the existing seat), live
-smoke 96 words / 0 quote marks / `wasnt`, other three voices unregressed. Full
-record and every number: `persistent-memory.d/2026-09-21-lv-mccarthy-gate.md`.
-⚠ `pfi-gx10` does not resolve from nh3-dev by name — it is **10.100.50.60**.
 
-**Two things carried forward, neither blocking:**
+- ⚠ **`talk.service` is in `failed` state on nh3-dev (exit 143) while `:8092`
+  serves 200.** The unit is dead; its containers keep running. Nothing manages
+  talk right now. This is the exact failure class the new notifier exists for.
+- **`restic/ana/esh-docker-vm` is 36h old against 12h fleet-wide.** Inside the
+  48h threshold so correctly green. Plausibly a missed window from that host's
+  forced reboot at 22:06 on 09-21, not yet confirmed.
 
-1. **The length cost is real and shipped documented, not fixed.** In-band 0.65
+### 17 commits unpushed
-   against base's 0.89; **35% of generations miss the requested 90–140 band against
-   base's 11%.** No checkpoint choice fixes it — every adapted arm is 33–48% out.
-   **The open follow-up is a retrain targeting length (pair construction or the
-   length target), and the operator's steer was to fold it into Faulkner's pair
-   build rather than run it standalone**, since Faulkner is the measured next voice
-   and would otherwise inherit the same defect.
-2. ⚠ **Check Skaldsong for the think-tag defect.** Every voice on the seat prefixes
-   an empty `<think></think>` unless the caller sends
-   `chat_template_kwargs: {"enable_thinking": false}`. It is the Qwen3 chat
-   template, so it has affected lv-yarros/bronte/hemingway since 2026-09-16. No
-   gate number is affected; a caller that omits it gets 17 junk characters at the
-   head of every passage, and any word-count over that string counts tags as prose.
 
-### Draupnir engine — COMPLETE on irv-ml1, acceptance passing
+Push is the operator's call, never automatic. Nothing is half-done in the tree —
-
+working tree is clean and every change above is committed.
-build123d 0.12.0 + OCP, numpy 2.4.6, trimesh 5.1.0 on py3.11.2; FreeCAD 1.0.0
-AppImage headless; OrcaSlicer 2.4.2 containerised at `~/bin/orca-slice`;
-artifact root `/mnt/smithy/draupnir` 2775, 500 GB budget. The contrastive
-control pair PASSES. **The operator has moved to a code session inside the
-Draupnir repo**, so the next questions come from there rather than from
-brokkr-smithy-dev, and everything is in the repo (b8db50b) rather than only in
-the althing thread.
-
-### The fleet is quiet and nothing is blocked on me
-
-- **Backups:** `RESULT: all backups fresh`. The two `⏸` policy exclusions
-  (ana-scale CT 114, esh-vm-workstation VM 102) are correct and visible.
-  infra-hermes holds the watch and names CT 107 explicitly rather than trusting
-  the absence of red — ⚠ that lock was RELEASED BY HAND, not self-healed, and
-  its mechanism is still unexplained.
-- **Disk:** nh3-dev at 72%, 66 GB free after the 2026-09-21 triage.
-- **ops-log:** clean audit across 6 hosts; commit attribution working.
-- **main is PUSHED and clean** — operator confirmed 2026-09-22 ~04:00Z; `git rev-list
-  --count origin/main..HEAD` = 0, verified rather than assumed.
-  only the hook-generated `graphify-out/GRAPH_REPORT.md` is dirty.
-
-### Open, low-urgency
-
-- ⚠ VM 102's efidisk carries **UEFI 2011 certs expired June 2026**. Needs the
-  sandbox down and BitLocker protectors suspended first. Operator's machine,
-  operator's call; surfaced, not acted on.
-- Nine unnecessary packages on irv-ml1 (`libwebkit2gtk-4.1-0` + deps) from a
-  serial dependency chase. Left deliberately — `autoremove` on a box running
-  twelve production services is a second risk, not a cleanup.
 
 ## Recent decisions
 
+- `[2026-09-22]` ⭐⭐ **Five instrument errors in one day, and the shape is one thing: a tool that enumerates "things that are fine" has selected against its own subject.** `--state=running` skipped the units most needing hooks; `awk '{print $1}'` dropped systemd's `●`-decorated FAILED rows; `grep -ic restic` on the wrapper missed the check script; `restic ls`'s header line made an absent path read as "blobs gone"; and a cooldown test that invoked before failing the unit. **Every one reported cleanly while looking at the wrong thing.** The rule is not "verify" — it is *verify, then ask what the verification could not have seen*. → `persistent-memory.d/2026-09-22-instrument-errors.md`
+- `[2026-09-22]` **Fleet alert bridge generalized** — `beszel-althing` → `althing-alert-bridge`, route registry (`/beszel` + `/kuma`), each with its own prefix/footer/parser. `/beszel` FROZEN and proven byte-identical (its three original tests kept unchanged). An alert that lies about its own source is worse than no alert. Commit `6f0a9b9`.
+- `[2026-09-22]` **Uptime Kuma rebuilt from scratch on 2.5.5, moved esh-docker-vm → ana-docker.** ⚠ `:latest` is a TRAP — it tracks 1.x, so an Aug-2026 pull gave a Dec-2024 build. No REST CRUD API in either major; automation is Socket.IO via first-party `scripts/kuma` (the community wrapper is abandoned at 1.23.1). Monitors + channel + status page all declared in `stacks/uptimekuma/monitors.yaml`. Commits `3a85a6b`, `94899d6`.
+- `[2026-09-22]` **Beszel and Uptime Kuma are DISJOINT, not redundant** — Beszel's alerts bind to a *system* with a threshold; there is no URL column, so it is structurally incapable of "this endpoint should return 200". Homepage is DISPLAY ONLY (38 polls, alerts nobody). The empty seam between them is where homepage sat dead for three days. Settled; do not re-litigate.
+- `[2026-09-22]` **Failed-START alarms on 23 nh3-dev units** (`services/althing-notify-failure/`). ⚠ **Coverage is far narrower than it looks** — only the 7 timer oneshots and `svos.service` can actually reach `failed`; ten daemons flap forever instead. `svos.service`'s divergent 5-min window is LOAD-BEARING (operator ruling 2026-09-11) — **do not harmonise it**. Commits `1be39d1`, `163bb97`.
+- `[2026-09-22]` **Backup coverage is a property of the SYSTEM, never one job's scope** — establish it by querying the repo for the path in a real snapshot, never by reading a job's `SRC=`. A peer read `dev-backup` alone and told the operator, unhedged, that Miranda's conversation had "never been backed up"; all five paths were in that night's restic snapshot. `docs/runbooks/backups.md`.
+- `[2026-09-22]` **restic checks now assert CONTENT and are DISCOVERED not enumerated** — conjunctive (recent AND present AND restores non-zero bytes); repos discovered from each NAS, hand list demoted to an EXPECTED set that distinguishes "vanished" from "stale". nh3-dev had never been on the watch list at all. Commits `ba60fda`, `4979869`.
+- `[2026-09-22]` **irv-ml1: Irvine is a TENANCY behind a Fortinet PFI does not control** — its TLS inspection breaks Tailscale relay/control intermittently (41 cert warnings/week). **No fix on our side; do not go looking.** ⚠ Diagnose reachability with `tailscale ping`, NOT status output — headscale said `online` and status said `active, 19.7GB` while nothing answered. Commit `38bb20c`.
+
 - `[2026-09-21]` ⭐⭐⭐ **lv-mccarthy GATED and NOT SHIPPED — voice passes, memorisation is the cleanest in the line, and the length discipline is gone.** 720 generations, 3 arms. Voice +0.152 at **2.9×** floor for ckpt450 (ckpt900 +0.172 at only 1.2×, its spread one outlier seed), and **~3/4 of the gain survives stripping every punctuation mark**, so it is not the cheap win. ⭐ Memorisation: **ckpt450 at 0.12 against the author's own held-out 0.12 — identical, longest match 11 words against the author's coincidental 12**, all 96 matches READ and every one stock grammar (`he looked at the wolf and he looked at him`); the name-shaped hits are the RENAMED inventions. Axis C is the blocker: **20% / 28% of generations overshoot the 90–140 band against base's 1%**, worst case a degenerate loop at 279 words. ⚠ My own prereg's axis C transcribed `score_beats.py`'s **v1** criteria including "in-band up on base", which the operator RETIRED 2026-09-15 because base maxes it — under the v2 (ran-on only) ckpt450 passes by **0.01 against a 0.200 floor**, but that reading was found AFTER the numbers and was not used. Fix the prereg prospectively. → `persistent-memory.d/2026-09-21-lv-mccarthy-gate.md`
 - `[2026-09-21]` ⭐⭐⭐ **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one cost nothing. Here the curve RESOLVED epoch 2 as better — ckpt900 +4.9× the 0.00393 median neighbour jitter above ckpt450, nowhere near tied — and epoch 2 lost every axis that resolves: **4.0× wider seed spread** (0.148 vs 0.037), **1.8× the author's memorisation rate vs 1.0×**, more ran-on (0.28 vs 0.20), worse on-beat. Its only win is a 0.019 voice point estimate, inside the floor, and its spread is ONE outlier seed (0.605 vs 0.457/0.531/0.554) — the third occurrence of that shape in the later checkpoint after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. **Durable: on this schedule the eval-loss minimum is not the ship candidate, and the curve's CONFIDENCE about it carries no information.** Default this for Faulkner/Morrison/Chandler rather than re-deriving it.
 - `[2026-09-21]` ⭐⭐ **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** `memorization_check.py` used **base-unadapted** as its negative control; base writes summary while the adapted arms write pastiche, so it cannot collide with a register it does not imitate and its zero is unearned. The lv-hemingway gate established this, computed the correct control (the author's own held-out text) **by hand**, and left no instrument — so the finding was not reproducible. Now `--train-only --heldout-reference`, refusing the unsafe combination, validated by reproducing Hemingway's hand-computed row **to the digit**. ⭐ On McCarthy it is the difference between reporting ckpt450 as memorising (0.12 vs base 0.00 = 12×) and clean (0.12 vs the author's 0.12 = 1.0×). ⭐ And the "the register makes collisions inevitable" story that was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12) — measured, not assumed; neither number transfers between authors. Also new: `show_memorisation_matches.py`, because rate and exposure are different questions and the reading was hand-done too. Commits `b4ba731` `7eadbd6`.