memory: snapshot — three silent fleet faults found and fixed in one afternoon

An infrastructure day with no training work, and the through-line is that
every fault was invisible to monitoring. NH3↔Anaheim had been crossing a
throttled DERP relay rather than a direct path for long enough to carry 78 GB;
`.internal` DNS was failing roughly one lookup in ten from two independent
causes; SearXNG had exactly one working general web engine. Nothing alarmed on
any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget
and chose to measure rather than adapt around the problem.

Also landed: althing v3.6.3, which makes hyphenated search work for the first
time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a
capability index autoloaded by Claude, Codex and Grok from one symlinked file;
Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's
env.sh secrets vaulted.

Six detail files. The in-flight section is rewritten and shrinks 142 lines to
64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run
outcome, which was left untouched and must not be assumed good.

Two foot-guns recorded, both mine: the ESH egress experiment reverted on a
diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which
has no constructor in that build and crash-looped the container ten times.

No archival this run. 165 of 169 dated entries are under the 14-day guard and
the remaining four all carry open deferred pointers, so the index stays over
the soft cap at 480 lines — an over-cap file that keeps live decisions beats a
scannable one that lost one.

persistent-memory.d/2026-09-18-althing-363-hyphen-search.md

@@ -0,0 +1,80 @@
+# [2026-09-18] althing search returned ZERO for every hyphenated query — silently, for everyone
+
+Routed in by heid on behalf of peedlar-dev as "a message reached storage but never
+entered the inbox unread-set or search index". Two symptoms, and **neither was what
+was reported**.
+
+## Symptom 1 — "never delivered": NOT A DEFECT
+
+Message 2571 was fully intact: recipients row present, in the FTS index, `read_at`
+set. **Zero messages fleet-wide have no recipient row — 0 of 2583.**
+
+Grouping the reader's `read_at` values gives one group per `postbox read`
+transaction, and the second consumed **2570 AND 2571 together**. `postbox read`
+advances over exactly what it printed, so that read printed both.
+
+⚠ **The evidence was truncated.** Three of four tee'd files sat at exact 4 KiB
+multiples ending mid-sentence; `read-2-of-3.md` was 8,192 bytes while message 2570's
+body ALONE is 64,975. It held under 13% of 2570 and never reached 2571. That read
+emitted ≥127 KB and 8 KB survived. The reasoning — "a read that included 2571 would
+have printed it into one of these files" — was sound on a premise that failed.
+
+## Symptom 2 — A REAL DEFECT, and far wider than one message
+
+**Any query containing a hyphen returned zero hits, silently, for every handle.**
+
+    'nh3-docker'    -> OperationalError: no such column: docker
+    '"nh3-docker"'  -> 20 rows
+    'tts-dev'       -> OperationalError: no such column: dev
+    '"tts-dev"'     -> 47 rows
+
+FTS5 reads the hyphen as a column filter and tries to resolve the trailing token as
+a column name. `store.py` passed the caller's query to `MATCH` unescaped; `search()`
+caught the `OperationalError`, ran its probe (which succeeded — the store was
+healthy), and returned `[]`, indistinguishable from "no results".
+
+On this fleet that is most proper nouns we own: `nh3-docker`, `irv-ml1`,
+`esh-docker-vm`, `fv-ml1`, `tts-dev`, `peedlar-dev`.
+
+⚠ **The discriminator was working as designed.** It correctly separated
+store-broken from caller-typo. The defect was that it classified an ordinary
+hyphenated query as a typo.
+
+## The fix, and the boundary that held
+
+Routed to forseti (althing's owner) rather than patched — infra-ops owns the
+**rollout**, not the codebase. forseti implemented on operator instruction and
+shipped **v3.6.3** (`5de41b7`): valid FTS syntax intact, `OperationalError` retries
+once as an escaped literal phrase before the health probe. Plus a read-summary line
+(`consumed N messages: 2570, 2571`) that makes the truncation case detectable.
+
+⚠ **I twice declined to act on relayed authorization** — "Vuong requested deployment"
+arriving through a peer is not the operator instructing me, and a post-office bounce
+takes the bus down for every handle. Got it directly, then executed.
+
+## Rollout mechanics worth keeping
+
+- **No CI builds the post-office image.** `deploy.yaml` ships althing-*chamber* to
+  ana-docker; `ci.yaml` runs tests and the frontend. The root `Dockerfile` IS the
+  post office and nothing publishes it — a build-and-push step, not a pull.
+- Built from `git archive v3.6.3`, **not a working tree**; archive hash verified
+  identical after transfer. My wheel came out byte-identical to forseti's.
+- Bus outage **19:13:40 → 19:13:57 UTC, 17 seconds**.
+- Deployed digest `sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4`;
+  rollback pin kept as `compose.yaml.bak-3.6.2`.
+- ⚠ **Canonical `stacks/althing-post-office/compose.yaml` had to be re-pinned too.**
+  forseti caught it: hand-deploying leaves canonical intent stale, so the next routine
+  `deploy-stack.sh` would have silently ROLLED THE BUS BACK. Commit `670ac9e`.
+
+## Still open — two postbox fixes proposed to forseti
+
+1. **`--body -` is taken as a literal hyphen, not stdin.** Four real messages lost to
+   it today (tts-dev ×3, ledger-dev earlier). Store audit: 5 of 2,637 messages have a
+   one-character body and **every accidental one is a literal `-`**; the fifth is a
+   deliberate `'x'` probe. forseti's scope: reserve `--body -` for stdin in both verbs.
+   **Awaiting operator.** ⚠ Empty bodies are already rejected (PRE-306, verified) — no
+   minimum-length rule is needed.
+2. A bounded consuming read — **withdrawn**, and rightly. A message-count limit does
+   not bound one very large body, so it would not have saved peedlar-dev.
+
+Until it ships, the mitigation is entirely sender-side: pipe the body, never `--body`.