memory: snapshot — three silent fleet faults found and fixed in one afternoon

An infrastructure day with no training work, and the through-line is that
every fault was invisible to monitoring. NH3↔Anaheim had been crossing a
throttled DERP relay rather than a direct path for long enough to carry 78 GB;
`.internal` DNS was failing roughly one lookup in ten from two independent
causes; SearXNG had exactly one working general web engine. Nothing alarmed on
any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget
and chose to measure rather than adapt around the problem.

Also landed: althing v3.6.3, which makes hyphenated search work for the first
time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a
capability index autoloaded by Claude, Codex and Grok from one symlinked file;
Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's
env.sh secrets vaulted.

Six detail files. The in-flight section is rewritten and shrinks 142 lines to
64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run
outcome, which was left untouched and must not be assumed good.

Two foot-guns recorded, both mine: the ESH egress experiment reverted on a
diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which
has no constructor in that build and crash-looped the container ten times.

No archival this run. 165 of 169 dated entries are under the 14-day guard and
the remaining four all carry open deferred pointers, so the index stays over
the soft cap at 480 lines — an over-cap file that keeps live decisions beats a
scannable one that lost one.

persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md

@@ -0,0 +1,81 @@
+# [2026-09-18] SearXNG had ONE working general web engine — now seven, and the health script never noticed
+
+The fleet `web_search` MCP tool every nh3-dev session uses was answering from
+`google cse` alone, and had been for at least a day, with every instrument green.
+
+## The root cause, which is subtler than "engines were disabled"
+
+Of **55 general-category engines, 7 were enabled by default** — and six of those
+seven are dictionary, translation, currency and encyclopedia engines that return
+nothing for an ordinary web query. `google cse` was the only general WEB engine on
+the instance. When its quota exhausted, a query returned **HTTP 200 with an empty
+`results` array**.
+
+⚠ **The distinction that hid it: `inactive: false` only makes an engine
+SELECTABLE; `disabled: false` is what puts it in the DEFAULT set.** The other 48
+were selectable-but-off, which an API client has no way to change.
+
+## Three wrong causes published before the right one
+
+Recorded because the pattern repeated all day and each was reasoned, not careless.
+
+1. **"The ESH egress move broke it."** Search egress had been moved on 09-17 to a
+   SOCKS5 proxy on esh-scale, and a 2026-09-03 comment claimed NH3's residential
+   address avoided CAPTCHAs. Reverting to direct NH3 egress produced a
+   **byte-identical** failure. Falsified.
+2. **"Stale suspension timers."** A live `!ddg` bang probe on a freshly restarted
+   container also returned CAPTCHA. Falsified.
+3. **"The version bump fixed it."** Updating `2026.9.3` → `2026.9.18` appeared to
+   restore all four engines — but **another session had removed the captcha'd
+   scrapers from the engine set four minutes before that measurement**, and I did
+   not check what else had moved. Two changes, one measurement, attributed to mine.
+
+Every one asserted causation from correlation with **no baseline measurement of the
+working state**.
+
+## The fix
+
+Five keyless engines added to the default set, each bang-probed on this egress
+first rather than trusted from a list:
+
+    duckduckgo web  10      mojeek     Suspended     excluded
+    bing            10      qwant      CAPTCHA       excluded
+    yep             20      startpage  Suspended     excluded
+    yahoo            7      brave      Suspended     excluded
+    wiby            12      seznam     timeout       excluded
+
+Plus `braveapi` with a real key. **General web engines: 1 → 7.**
+
+Verified under the live failure condition rather than a simulated one — `google cse`
+was STILL quota-suspended and three consecutive queries returned 38–41 results from
+4–5 engines each. Later with braveapi: 55–63 results from six.
+
+## Two durable rules this settles
+
+⭐ **This stack tracks `:latest` ON PURPOSE — do not "improve" it by pinning.**
+Upstream ships engine-handler fixes continuously because providers change bot
+gating continuously; being current IS the mitigation, and a digest pin would have
+frozen the breakage in place. (The althing post office is pinned for the opposite
+reason — it is the message bus and must not move underfoot.)
+
+⭐ **The Brave API key is committed in plaintext by explicit operator decision**
+(2026-09-18): *"if the service is useless, so is the key"* — free tier, rate-limited
+service, not worth the machinery of keeping it out of git. It is also vaulted at
+`nh3-docker/searxng-brave-api-key`. **This is scoped to one low-value credential and
+is not a change to the no-secrets-in-git rule.** It cannot be un-committed; rotation
+means a new key at Brave, never a history rewrite, because the repo is shared.
+
+⚠ There is **no env-var path into `searxng-settings.yml`** in this build. The loader
+reads only `SEARXNG_SETTINGS_PATH`; the entrypoint substitutes only `ultrasecretkey`.
+`SEARXNG_BRAVE_API_KEY` reaches the container and is never read — it is kept in the
+compose, unused, with a comment saying so.
+
+## ⚠ Still open: the health script cannot see any of this
+
+`scripts/searxng-health.sh` printed `✓ searching` with six engines answering AND
+with one. It gates on "did any results come back" and prints failed engines as
+informational. Search ran on a single engine for a day with every instrument green.
+**Fix it to fail on blocked engines before trusting it again.**
+
+Commits `9a428fd` `274d3e2` `9219942` `5b20b02`. See
+[[2026-09-18-searxng-esh-egress-reverted]].