vh
274d3e2443
Root cause of the silent-empty-results failure peedlar-dev reported. Of 55 general-category engines, only seven were enabled-by-default, and six of those are dictionary, translation, currency or encyclopedia engines that return nothing for an ordinary web query. `google cse` was the instance's ONLY general web engine, so a single quota exhaustion produced HTTP 200 with an empty results array and no error, for every consumer on the fleet. The distinction that matters: `inactive: false` only makes an engine selectable, `disabled: false` puts it in the default set. The other 48 were selectable-but-off, which an API client has no way to change. Enables five keyless engines, each bang-probed first and returning real results with no API key: duckduckgo web 10, bing 10, yep 20, yahoo 7, wiby 12. General web engines go 1 -> 6. Deliberately excluded: mojeek, qwant, startpage and the brave scraper, all of which CAPTCHA or rate-limit this egress, and seznam, which times out. Verified under the live failure condition rather than a simulated one. google cse is still quota-suspended right now, and three consecutive queries returned 38-41 results from 4-5 engines each. The single point of failure is gone while the failing engine is still failing. Also adopts the concurrent v4 settings work from the other session — marginalia on its public key, and the captcha'd-scraper removals — plus the fix for the crash-loop that work introduced: this build has no !ENV YAML constructor, so `api_key: !ENV SEARXNG_BRAVE_API_KEY` made the file unparseable and the container restarted ten times with search down fleet-wide. That block stays commented; the vaulted Brave key is valid but has no supported path into the settings file, which is a separate decision.
searxng
Privacy-respecting metasearch. Runs on nh3-docker (10.100.50.40:9996,
searxng.nh3.internal), and is exposed to every Claude Code session on nh3-dev
as the web_search MCP tool.
Current egress: direct from NH3
Hosting and egress are both nh3-docker. Outbound search requests leave from
70.230.226.88, NH3's residential address, with no proxy in outgoing:.
This is the configuration that works; the section below records why, and the
one after it records an attempt to change it that failed.
The 2026-09-18 engine outage — cause was the build, not egress
For at least a day, searxng answered from one engine: brave and startpage
suspended, duckduckgo CAPTCHA, google cse alone returning results. Fixed by
updating the image — 2026.9.3+a1144dda3 → 2026.9.18+c0042add3. All four
engines came back immediately and held across 11 consecutive queries.
Upstream ships engine-handler fixes constantly, because search providers change
their bot gating constantly. That is why this stack tracks :latest rather
than pinning a digest: for searxng specifically, being current is the
mitigation, and a pin would have frozen the breakage in place.
⚠ Two wrong diagnoses were published before the right one. Read this before theorising about egress again.
- The failures were first blamed on the ESH egress move (below), on the strength of a 2026-09-03 comment claiming NH3's residential address avoided CAPTCHAs. Reverting to direct NH3 egress produced a byte-identical result — same three engines down — which falsified it.
- A live
!ddgbang probe on a freshly restarted container also returned CAPTCHA, ruling out a stale circuit-breaker timer as the explanation.
Both claims asserted causation from correlation without measuring a baseline. The version bump was the only change that moved the number.
⚠ Neither health check can see this class of failure. /healthz cannot,
and scripts/searxng-health.sh prints ✓ searching whether four engines
answer or one, because it gates on "did any results come back" and treats
failed engines as informational — it prints them, then passes anyway. Search
ran on one engine for a day with every instrument green. Make the script fail
on blocked engines before trusting it again.
The ESH egress experiment (2026-09-17 → reverted 2026-09-18)
Search egress was moved to socks5h://10.0.50.65:1080 on esh-scale
(CT 108 on esh-pve) — an application-level proxy, no host route or exit-node
changes. Reverted after one day.
It did not cause the engine outage above, and it did not fix anything either: engine behaviour was identical with it and without it. It was reverted on its own merits — it bought no measurable improvement while making all fleet search depend on ESH WAN and mesh availability, and the simpler configuration is the better one.
ESH egress measured 128.177.138.182 in use and 154.50.58.126 at cutover;
ESH's WAN address moves and nothing pins it.
The proxy side was built correctly and none of this was its fault: microsocks
ran as nobody under searxng-egress.service, bound 10.0.50.65:1080 only,
and allowed source 10.100.50.40 alone. The canonical unit is kept at
configs/esh-scale/searxng-egress.service
in case a future egress experiment wants it; the service is stopped and
disabled on esh-scale, and tailscaled there was not touched.
Historical placement: NH3 rather than the colo
Measured 2026-09-03:
ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA
nh3-docker egress 70.230.226.88 residential -> no gate on that account
Search engines gate datacenter ranges. Running the aggregator from the
residential-egress site was the original approach. The operator subsequently
requested ESH egress; outgoing.proxies now explicitly selects it as above.
⚠ It is not a complete fix: brave, duckduckgo and startpage still
CAPTCHA from here. google cse carries general search at ~20 results/query, and
yandex, wiby, github, stackoverflow and marginalia all work. General
search is therefore effectively single-engine — if google cse breaks, the
instance goes quiet, which is exactly the failure below.
⚠ /healthz cannot tell you whether search works
On 2026-09-03 the old ana-docker instance was found returning zero results for every query, for an unknown number of weeks, while:
- the container reported
healthyfor 7 straight days, 0 restarts; - the Homepage card was green;
/healthzreturned 200 every 30 seconds.
It was running 2026.4.17 while current was 2026.9.3 — 4.5 months of engine
scrapers rotting against sites that had changed their markup. :latest means
"latest at pull time", and nothing re-pulls on its own.
Proven by experiment before touching anything: the new image, same settings file, same host, same query returned 20 results where the running one returned 0.
scripts/searxng-health.sh asserts the property, not the mechanism — three
unrelated queries must each return results > 0. Run it after any change, and
periodically; it is the only thing that catches rot.
Update
ssh infra-ops@nh3-docker 'cd /opt/docker/compose/searxng && \
sudo docker compose pull && sudo docker compose up -d'
scripts/searxng-health.sh
The MCP tool
services/searxng-mcp/ — installed with uv tool install and registered at
user scope (claude mcp add --scope user searxng searxng-mcp), so every
Claude Code session gets web_search with no per-project setup.
⚠ Zero results raise an error rather than returning an empty list. An empty list is indistinguishable from a broken aggregator, and that ambiguity is what let the old instance fail silently for weeks. Same reasoning as althing's "an unreachable post office is an OUTAGE, never an empty inbox".
⚠ Written against mcp 2.x, where FastMCP became MCPServer and the v1
@app.list_tools() decorator API is gone. v1 examples fail at import with
'Server' object has no attribute 'list_tools'.
⚠ uv tool install --force alone served a cached build and silently
reinstalled the old code — the installed file still had the v1 API after the
source no longer did. --reinstall --no-cache was required, and the check that
caught it was md5sum of source vs installed.
Secret
SEARXNG_SECRET lives in /opt/docker/compose/searxng/.env (0600, root) on the
host and is vaulted at nh3-docker/searxng-secret. Never in git.