Files

T

vh 148a5a34da memory: snapshot — three silent fleet faults found and fixed in one afternoon

An infrastructure day with no training work, and the through-line is that
every fault was invisible to monitoring. NH3↔Anaheim had been crossing a
throttled DERP relay rather than a direct path for long enough to carry 78 GB;
`.internal` DNS was failing roughly one lookup in ten from two independent
causes; SearXNG had exactly one working general web engine. Nothing alarmed on
any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget
and chose to measure rather than adapt around the problem.

Also landed: althing v3.6.3, which makes hyphenated search work for the first
time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a
capability index autoloaded by Claude, Codex and Grok from one symlinked file;
Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's
env.sh secrets vaulted.

Six detail files. The in-flight section is rewritten and shrinks 142 lines to
64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run
outcome, which was left untouched and must not be assumed good.

Two foot-guns recorded, both mine: the ESH egress experiment reverted on a
diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which
has no constructor in that build and crash-looped the container ten times.

No archival this run. 165 of 169 dated entries are under the 14-day guard and
the remaining four all carry open deferred pointers, so the index stays over
the soft cap at 480 lines — an over-cap file that keeps live decisions beats a
scannable one that lost one.

2026-09-18 20:09:01 -07:00

4.3 KiB

Raw Blame History

[2026-09-18] althing search returned ZERO for every hyphenated query — silently, for everyone

Routed in by heid on behalf of peedlar-dev as "a message reached storage but never entered the inbox unread-set or search index". Two symptoms, and neither was what was reported.

Symptom 1 — "never delivered": NOT A DEFECT

Message 2571 was fully intact: recipients row present, in the FTS index, read_at set. Zero messages fleet-wide have no recipient row — 0 of 2583.

Grouping the reader's read_at values gives one group per postbox read transaction, and the second consumed 2570 AND 2571 together. postbox read advances over exactly what it printed, so that read printed both.

⚠ The evidence was truncated. Three of four tee'd files sat at exact 4 KiB multiples ending mid-sentence; read-2-of-3.md was 8,192 bytes while message 2570's body ALONE is 64,975. It held under 13% of 2570 and never reached 2571. That read emitted ≥127 KB and 8 KB survived. The reasoning — "a read that included 2571 would have printed it into one of these files" — was sound on a premise that failed.

Symptom 2 — A REAL DEFECT, and far wider than one message

Any query containing a hyphen returned zero hits, silently, for every handle.

'nh3-docker'    -> OperationalError: no such column: docker
'"nh3-docker"'  -> 20 rows
'tts-dev'       -> OperationalError: no such column: dev
'"tts-dev"'     -> 47 rows

FTS5 reads the hyphen as a column filter and tries to resolve the trailing token as a column name. store.py passed the caller's query to MATCH unescaped; search() caught the OperationalError, ran its probe (which succeeded — the store was healthy), and returned [], indistinguishable from "no results".

On this fleet that is most proper nouns we own: nh3-docker, irv-ml1, esh-docker-vm, fv-ml1, tts-dev, peedlar-dev.

⚠ The discriminator was working as designed. It correctly separated store-broken from caller-typo. The defect was that it classified an ordinary hyphenated query as a typo.

The fix, and the boundary that held

Routed to forseti (althing's owner) rather than patched — infra-ops owns the rollout, not the codebase. forseti implemented on operator instruction and shipped v3.6.3 (5de41b7): valid FTS syntax intact, OperationalError retries once as an escaped literal phrase before the health probe. Plus a read-summary line (consumed N messages: 2570, 2571) that makes the truncation case detectable.

⚠ I twice declined to act on relayed authorization — "Vuong requested deployment" arriving through a peer is not the operator instructing me, and a post-office bounce takes the bus down for every handle. Got it directly, then executed.

Rollout mechanics worth keeping

No CI builds the post-office image. deploy.yaml ships althing-chamber to ana-docker; ci.yaml runs tests and the frontend. The root Dockerfile IS the post office and nothing publishes it — a build-and-push step, not a pull.
Built from git archive v3.6.3, not a working tree; archive hash verified identical after transfer. My wheel came out byte-identical to forseti's.
Bus outage 19:13:40 → 19:13:57 UTC, 17 seconds.
Deployed digest sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4; rollback pin kept as compose.yaml.bak-3.6.2.
⚠ Canonical stacks/althing-post-office/compose.yaml had to be re-pinned too. forseti caught it: hand-deploying leaves canonical intent stale, so the next routine deploy-stack.sh would have silently ROLLED THE BUS BACK. Commit 670ac9e.

Still open — two postbox fixes proposed to forseti

--body - is taken as a literal hyphen, not stdin. Four real messages lost to it today (tts-dev ×3, ledger-dev earlier). Store audit: 5 of 2,637 messages have a one-character body and every accidental one is a literal -; the fifth is a deliberate 'x' probe. forseti's scope: reserve --body - for stdin in both verbs. Awaiting operator. ⚠ Empty bodies are already rejected (PRE-306, verified) — no minimum-length rule is needed.
A bounded consuming read — withdrawn, and rightly. A message-count limit does not bound one very large body, so it would not have saved peedlar-dev.

Until it ships, the mitigation is entirely sender-side: pipe the body, never --body.

4.3 KiB Raw Blame History Unescape Escape