vh
148a5a34da
An infrastructure day with no training work, and the through-line is that every fault was invisible to monitoring. NH3↔Anaheim had been crossing a throttled DERP relay rather than a direct path for long enough to carry 78 GB; `.internal` DNS was failing roughly one lookup in ten from two independent causes; SearXNG had exactly one working general web engine. Nothing alarmed on any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget and chose to measure rather than adapt around the problem. Also landed: althing v3.6.3, which makes hyphenated search work for the first time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a capability index autoloaded by Claude, Codex and Grok from one symlinked file; Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's env.sh secrets vaulted. Six detail files. The in-flight section is rewritten and shrinks 142 lines to 64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run outcome, which was left untouched and must not be assumed good. Two foot-guns recorded, both mine: the ESH egress experiment reverted on a diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which has no constructor in that build and crash-looped the container ten times. No archival this run. 165 of 169 dated entries are under the 14-day guard and the remaining four all carry open deferred pointers, so the index stays over the soft cap at 480 lines — an over-cap file that keeps live decisions beats a scannable one that lost one.
81 lines
4.3 KiB
Markdown
81 lines
4.3 KiB
Markdown
# [2026-09-18] althing search returned ZERO for every hyphenated query — silently, for everyone
|
||
|
||
Routed in by heid on behalf of peedlar-dev as "a message reached storage but never
|
||
entered the inbox unread-set or search index". Two symptoms, and **neither was what
|
||
was reported**.
|
||
|
||
## Symptom 1 — "never delivered": NOT A DEFECT
|
||
|
||
Message 2571 was fully intact: recipients row present, in the FTS index, `read_at`
|
||
set. **Zero messages fleet-wide have no recipient row — 0 of 2583.**
|
||
|
||
Grouping the reader's `read_at` values gives one group per `postbox read`
|
||
transaction, and the second consumed **2570 AND 2571 together**. `postbox read`
|
||
advances over exactly what it printed, so that read printed both.
|
||
|
||
⚠ **The evidence was truncated.** Three of four tee'd files sat at exact 4 KiB
|
||
multiples ending mid-sentence; `read-2-of-3.md` was 8,192 bytes while message 2570's
|
||
body ALONE is 64,975. It held under 13% of 2570 and never reached 2571. That read
|
||
emitted ≥127 KB and 8 KB survived. The reasoning — "a read that included 2571 would
|
||
have printed it into one of these files" — was sound on a premise that failed.
|
||
|
||
## Symptom 2 — A REAL DEFECT, and far wider than one message
|
||
|
||
**Any query containing a hyphen returned zero hits, silently, for every handle.**
|
||
|
||
'nh3-docker' -> OperationalError: no such column: docker
|
||
'"nh3-docker"' -> 20 rows
|
||
'tts-dev' -> OperationalError: no such column: dev
|
||
'"tts-dev"' -> 47 rows
|
||
|
||
FTS5 reads the hyphen as a column filter and tries to resolve the trailing token as
|
||
a column name. `store.py` passed the caller's query to `MATCH` unescaped; `search()`
|
||
caught the `OperationalError`, ran its probe (which succeeded — the store was
|
||
healthy), and returned `[]`, indistinguishable from "no results".
|
||
|
||
On this fleet that is most proper nouns we own: `nh3-docker`, `irv-ml1`,
|
||
`esh-docker-vm`, `fv-ml1`, `tts-dev`, `peedlar-dev`.
|
||
|
||
⚠ **The discriminator was working as designed.** It correctly separated
|
||
store-broken from caller-typo. The defect was that it classified an ordinary
|
||
hyphenated query as a typo.
|
||
|
||
## The fix, and the boundary that held
|
||
|
||
Routed to forseti (althing's owner) rather than patched — infra-ops owns the
|
||
**rollout**, not the codebase. forseti implemented on operator instruction and
|
||
shipped **v3.6.3** (`5de41b7`): valid FTS syntax intact, `OperationalError` retries
|
||
once as an escaped literal phrase before the health probe. Plus a read-summary line
|
||
(`consumed N messages: 2570, 2571`) that makes the truncation case detectable.
|
||
|
||
⚠ **I twice declined to act on relayed authorization** — "Vuong requested deployment"
|
||
arriving through a peer is not the operator instructing me, and a post-office bounce
|
||
takes the bus down for every handle. Got it directly, then executed.
|
||
|
||
## Rollout mechanics worth keeping
|
||
|
||
- **No CI builds the post-office image.** `deploy.yaml` ships althing-*chamber* to
|
||
ana-docker; `ci.yaml` runs tests and the frontend. The root `Dockerfile` IS the
|
||
post office and nothing publishes it — a build-and-push step, not a pull.
|
||
- Built from `git archive v3.6.3`, **not a working tree**; archive hash verified
|
||
identical after transfer. My wheel came out byte-identical to forseti's.
|
||
- Bus outage **19:13:40 → 19:13:57 UTC, 17 seconds**.
|
||
- Deployed digest `sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4`;
|
||
rollback pin kept as `compose.yaml.bak-3.6.2`.
|
||
- ⚠ **Canonical `stacks/althing-post-office/compose.yaml` had to be re-pinned too.**
|
||
forseti caught it: hand-deploying leaves canonical intent stale, so the next routine
|
||
`deploy-stack.sh` would have silently ROLLED THE BUS BACK. Commit `670ac9e`.
|
||
|
||
## Still open — two postbox fixes proposed to forseti
|
||
|
||
1. **`--body -` is taken as a literal hyphen, not stdin.** Four real messages lost to
|
||
it today (tts-dev ×3, ledger-dev earlier). Store audit: 5 of 2,637 messages have a
|
||
one-character body and **every accidental one is a literal `-`**; the fifth is a
|
||
deliberate `'x'` probe. forseti's scope: reserve `--body -` for stdin in both verbs.
|
||
**Awaiting operator.** ⚠ Empty bodies are already rejected (PRE-306, verified) — no
|
||
minimum-length rule is needed.
|
||
2. A bounded consuming read — **withdrawn**, and rightly. A message-count limit does
|
||
not bound one very large body, so it would not have saved peedlar-dev.
|
||
|
||
Until it ships, the mitigation is entirely sender-side: pipe the body, never `--body`.
|