# [2026-09-18] SearXNG had ONE working general web engine — now seven, and the health script never noticed

The fleet `web_search` MCP tool every nh3-dev session uses was answering from
`google cse` alone, and had been for at least a day, with every instrument green.

## The root cause, which is subtler than "engines were disabled"

Of **55 general-category engines, 7 were enabled by default** — and six of those
seven are dictionary, translation, currency and encyclopedia engines that return
nothing for an ordinary web query. `google cse` was the only general WEB engine on
the instance. When its quota exhausted, a query returned **HTTP 200 with an empty
`results` array**.

⚠ **The distinction that hid it: `inactive: false` only makes an engine
SELECTABLE; `disabled: false` is what puts it in the DEFAULT set.** The other 48
were selectable-but-off, which an API client has no way to change.

## Three wrong causes published before the right one

Recorded because the pattern repeated all day and each was reasoned, not careless.

1. **"The ESH egress move broke it."** Search egress had been moved on 09-17 to a
   SOCKS5 proxy on esh-scale, and a 2026-09-03 comment claimed NH3's residential
   address avoided CAPTCHAs. Reverting to direct NH3 egress produced a
   **byte-identical** failure. Falsified.
2. **"Stale suspension timers."** A live `!ddg` bang probe on a freshly restarted
   container also returned CAPTCHA. Falsified.
3. **"The version bump fixed it."** Updating `2026.9.3` → `2026.9.18` appeared to
   restore all four engines — but **another session had removed the captcha'd
   scrapers from the engine set four minutes before that measurement**, and I did
   not check what else had moved. Two changes, one measurement, attributed to mine.

Every one asserted causation from correlation with **no baseline measurement of the
working state**.

## The fix

Five keyless engines added to the default set, each bang-probed on this egress
first rather than trusted from a list:

    duckduckgo web  10      mojeek     Suspended     excluded
    bing            10      qwant      CAPTCHA       excluded
    yep             20      startpage  Suspended     excluded
    yahoo            7      brave      Suspended     excluded
    wiby            12      seznam     timeout       excluded

Plus `braveapi` with a real key. **General web engines: 1 → 7.**

Verified under the live failure condition rather than a simulated one — `google cse`
was STILL quota-suspended and three consecutive queries returned 38–41 results from
4–5 engines each. Later with braveapi: 55–63 results from six.

## Two durable rules this settles

⭐ **This stack tracks `:latest` ON PURPOSE — do not "improve" it by pinning.**
Upstream ships engine-handler fixes continuously because providers change bot
gating continuously; being current IS the mitigation, and a digest pin would have
frozen the breakage in place. (The althing post office is pinned for the opposite
reason — it is the message bus and must not move underfoot.)

⭐ **The Brave API key is committed in plaintext by explicit operator decision**
(2026-09-18): *"if the service is useless, so is the key"* — free tier, rate-limited
service, not worth the machinery of keeping it out of git. It is also vaulted at
`nh3-docker/searxng-brave-api-key`. **This is scoped to one low-value credential and
is not a change to the no-secrets-in-git rule.** It cannot be un-committed; rotation
means a new key at Brave, never a history rewrite, because the repo is shared.

⚠ There is **no env-var path into `searxng-settings.yml`** in this build. The loader
reads only `SEARXNG_SETTINGS_PATH`; the entrypoint substitutes only `ultrasecretkey`.
`SEARXNG_BRAVE_API_KEY` reaches the container and is never read — it is kept in the
compose, unused, with a comment saying so.

## ⚠ Still open: the health script cannot see any of this

`scripts/searxng-health.sh` printed `✓ searching` with six engines answering AND
with one. It gates on "did any results come back" and prints failed engines as
informational. Search ran on a single engine for a day with every instrument green.
**Fix it to fail on blocked engines before trusting it again.**

Commits `9a428fd` `274d3e2` `9219942` `5b20b02`. See
[[2026-09-18-searxng-esh-egress-reverted]].