vh
148a5a34da
An infrastructure day with no training work, and the through-line is that every fault was invisible to monitoring. NH3↔Anaheim had been crossing a throttled DERP relay rather than a direct path for long enough to carry 78 GB; `.internal` DNS was failing roughly one lookup in ten from two independent causes; SearXNG had exactly one working general web engine. Nothing alarmed on any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget and chose to measure rather than adapt around the problem. Also landed: althing v3.6.3, which makes hyphenated search work for the first time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a capability index autoloaded by Claude, Codex and Grok from one symlinked file; Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's env.sh secrets vaulted. Six detail files. The in-flight section is rewritten and shrinks 142 lines to 64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run outcome, which was left untouched and must not be assumed good. Two foot-guns recorded, both mine: the ESH egress experiment reverted on a diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which has no constructor in that build and crash-looped the container ten times. No archival this run. 165 of 169 dated entries are under the 14-day guard and the remaining four all carry open deferred pointers, so the index stays over the soft cap at 480 lines — an over-cap file that keeps live decisions beats a scannable one that lost one.
82 lines
4.2 KiB
Markdown
82 lines
4.2 KiB
Markdown
# [2026-09-18] SearXNG had ONE working general web engine — now seven, and the health script never noticed
|
||
|
||
The fleet `web_search` MCP tool every nh3-dev session uses was answering from
|
||
`google cse` alone, and had been for at least a day, with every instrument green.
|
||
|
||
## The root cause, which is subtler than "engines were disabled"
|
||
|
||
Of **55 general-category engines, 7 were enabled by default** — and six of those
|
||
seven are dictionary, translation, currency and encyclopedia engines that return
|
||
nothing for an ordinary web query. `google cse` was the only general WEB engine on
|
||
the instance. When its quota exhausted, a query returned **HTTP 200 with an empty
|
||
`results` array**.
|
||
|
||
⚠ **The distinction that hid it: `inactive: false` only makes an engine
|
||
SELECTABLE; `disabled: false` is what puts it in the DEFAULT set.** The other 48
|
||
were selectable-but-off, which an API client has no way to change.
|
||
|
||
## Three wrong causes published before the right one
|
||
|
||
Recorded because the pattern repeated all day and each was reasoned, not careless.
|
||
|
||
1. **"The ESH egress move broke it."** Search egress had been moved on 09-17 to a
|
||
SOCKS5 proxy on esh-scale, and a 2026-09-03 comment claimed NH3's residential
|
||
address avoided CAPTCHAs. Reverting to direct NH3 egress produced a
|
||
**byte-identical** failure. Falsified.
|
||
2. **"Stale suspension timers."** A live `!ddg` bang probe on a freshly restarted
|
||
container also returned CAPTCHA. Falsified.
|
||
3. **"The version bump fixed it."** Updating `2026.9.3` → `2026.9.18` appeared to
|
||
restore all four engines — but **another session had removed the captcha'd
|
||
scrapers from the engine set four minutes before that measurement**, and I did
|
||
not check what else had moved. Two changes, one measurement, attributed to mine.
|
||
|
||
Every one asserted causation from correlation with **no baseline measurement of the
|
||
working state**.
|
||
|
||
## The fix
|
||
|
||
Five keyless engines added to the default set, each bang-probed on this egress
|
||
first rather than trusted from a list:
|
||
|
||
duckduckgo web 10 mojeek Suspended excluded
|
||
bing 10 qwant CAPTCHA excluded
|
||
yep 20 startpage Suspended excluded
|
||
yahoo 7 brave Suspended excluded
|
||
wiby 12 seznam timeout excluded
|
||
|
||
Plus `braveapi` with a real key. **General web engines: 1 → 7.**
|
||
|
||
Verified under the live failure condition rather than a simulated one — `google cse`
|
||
was STILL quota-suspended and three consecutive queries returned 38–41 results from
|
||
4–5 engines each. Later with braveapi: 55–63 results from six.
|
||
|
||
## Two durable rules this settles
|
||
|
||
⭐ **This stack tracks `:latest` ON PURPOSE — do not "improve" it by pinning.**
|
||
Upstream ships engine-handler fixes continuously because providers change bot
|
||
gating continuously; being current IS the mitigation, and a digest pin would have
|
||
frozen the breakage in place. (The althing post office is pinned for the opposite
|
||
reason — it is the message bus and must not move underfoot.)
|
||
|
||
⭐ **The Brave API key is committed in plaintext by explicit operator decision**
|
||
(2026-09-18): *"if the service is useless, so is the key"* — free tier, rate-limited
|
||
service, not worth the machinery of keeping it out of git. It is also vaulted at
|
||
`nh3-docker/searxng-brave-api-key`. **This is scoped to one low-value credential and
|
||
is not a change to the no-secrets-in-git rule.** It cannot be un-committed; rotation
|
||
means a new key at Brave, never a history rewrite, because the repo is shared.
|
||
|
||
⚠ There is **no env-var path into `searxng-settings.yml`** in this build. The loader
|
||
reads only `SEARXNG_SETTINGS_PATH`; the entrypoint substitutes only `ultrasecretkey`.
|
||
`SEARXNG_BRAVE_API_KEY` reaches the container and is never read — it is kept in the
|
||
compose, unused, with a comment saying so.
|
||
|
||
## ⚠ Still open: the health script cannot see any of this
|
||
|
||
`scripts/searxng-health.sh` printed `✓ searching` with six engines answering AND
|
||
with one. It gates on "did any results come back" and prints failed engines as
|
||
informational. Search ran on a single engine for a day with every instrument green.
|
||
**Fix it to fail on blocked engines before trusting it again.**
|
||
|
||
Commits `9a428fd` `274d3e2` `9219942` `5b20b02`. See
|
||
[[2026-09-18-searxng-esh-egress-reverted]].
|