vh
148a5a34da
An infrastructure day with no training work, and the through-line is that every fault was invisible to monitoring. NH3↔Anaheim had been crossing a throttled DERP relay rather than a direct path for long enough to carry 78 GB; `.internal` DNS was failing roughly one lookup in ten from two independent causes; SearXNG had exactly one working general web engine. Nothing alarmed on any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget and chose to measure rather than adapt around the problem. Also landed: althing v3.6.3, which makes hyphenated search work for the first time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a capability index autoloaded by Claude, Codex and Grok from one symlinked file; Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's env.sh secrets vaulted. Six detail files. The in-flight section is rewritten and shrinks 142 lines to 64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run outcome, which was left untouched and must not be assumed good. Two foot-guns recorded, both mine: the ESH egress experiment reverted on a diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which has no constructor in that build and crash-looped the container ten times. No archival this run. 165 of 169 dated entries are under the 14-day guard and the remaining four all carry open deferred pointers, so the index stays over the soft cap at 480 lines — an over-cap file that keeps live decisions beats a scannable one that lost one.
4.2 KiB
[2026-09-18] SearXNG had ONE working general web engine — now seven, and the health script never noticed
The fleet web_search MCP tool every nh3-dev session uses was answering from
google cse alone, and had been for at least a day, with every instrument green.
The root cause, which is subtler than "engines were disabled"
Of 55 general-category engines, 7 were enabled by default — and six of those
seven are dictionary, translation, currency and encyclopedia engines that return
nothing for an ordinary web query. google cse was the only general WEB engine on
the instance. When its quota exhausted, a query returned HTTP 200 with an empty
results array.
⚠ The distinction that hid it: inactive: false only makes an engine
SELECTABLE; disabled: false is what puts it in the DEFAULT set. The other 48
were selectable-but-off, which an API client has no way to change.
Three wrong causes published before the right one
Recorded because the pattern repeated all day and each was reasoned, not careless.
- "The ESH egress move broke it." Search egress had been moved on 09-17 to a SOCKS5 proxy on esh-scale, and a 2026-09-03 comment claimed NH3's residential address avoided CAPTCHAs. Reverting to direct NH3 egress produced a byte-identical failure. Falsified.
- "Stale suspension timers." A live
!ddgbang probe on a freshly restarted container also returned CAPTCHA. Falsified. - "The version bump fixed it." Updating
2026.9.3→2026.9.18appeared to restore all four engines — but another session had removed the captcha'd scrapers from the engine set four minutes before that measurement, and I did not check what else had moved. Two changes, one measurement, attributed to mine.
Every one asserted causation from correlation with no baseline measurement of the working state.
The fix
Five keyless engines added to the default set, each bang-probed on this egress first rather than trusted from a list:
duckduckgo web 10 mojeek Suspended excluded
bing 10 qwant CAPTCHA excluded
yep 20 startpage Suspended excluded
yahoo 7 brave Suspended excluded
wiby 12 seznam timeout excluded
Plus braveapi with a real key. General web engines: 1 → 7.
Verified under the live failure condition rather than a simulated one — google cse
was STILL quota-suspended and three consecutive queries returned 38–41 results from
4–5 engines each. Later with braveapi: 55–63 results from six.
Two durable rules this settles
⭐ This stack tracks :latest ON PURPOSE — do not "improve" it by pinning.
Upstream ships engine-handler fixes continuously because providers change bot
gating continuously; being current IS the mitigation, and a digest pin would have
frozen the breakage in place. (The althing post office is pinned for the opposite
reason — it is the message bus and must not move underfoot.)
⭐ The Brave API key is committed in plaintext by explicit operator decision
(2026-09-18): "if the service is useless, so is the key" — free tier, rate-limited
service, not worth the machinery of keeping it out of git. It is also vaulted at
nh3-docker/searxng-brave-api-key. This is scoped to one low-value credential and
is not a change to the no-secrets-in-git rule. It cannot be un-committed; rotation
means a new key at Brave, never a history rewrite, because the repo is shared.
⚠ There is no env-var path into searxng-settings.yml in this build. The loader
reads only SEARXNG_SETTINGS_PATH; the entrypoint substitutes only ultrasecretkey.
SEARXNG_BRAVE_API_KEY reaches the container and is never read — it is kept in the
compose, unused, with a comment saying so.
⚠ Still open: the health script cannot see any of this
scripts/searxng-health.sh printed ✓ searching with six engines answering AND
with one. It gates on "did any results come back" and prints failed engines as
informational. Search ran on a single engine for a day with every instrument green.
Fix it to fail on blocked engines before trusting it again.
Commits 9a428fd 274d3e2 9219942 5b20b02. See
2026-09-18-searxng-esh-egress-reverted.