# searxng Privacy-respecting metasearch. **Runs on nh3-docker** (`10.100.50.40:9996`, `searxng.nh3.internal`), and is exposed to every Claude Code session on nh3-dev as the `web_search` MCP tool. ## Current egress: direct from NH3 Hosting and egress are both nh3-docker. Outbound search requests leave from **70.230.226.88**, NH3's residential address, with no proxy in `outgoing:`. This is the configuration that works; the section below records why, and the one after it records an attempt to change it that failed. ## The 2026-09-18 engine outage — cause was the build, not egress For at least a day, searxng answered from **one engine**: brave and startpage suspended, duckduckgo CAPTCHA, google cse alone returning results. **Fixed by updating the image** — `2026.9.3+a1144dda3` → `2026.9.18+c0042add3`. All four engines came back immediately and held across 11 consecutive queries. Upstream ships engine-handler fixes constantly, because search providers change their bot gating constantly. **That is why this stack tracks `:latest` rather than pinning a digest**: for searxng specifically, being current *is* the mitigation, and a pin would have frozen the breakage in place. ⚠ **Two wrong diagnoses were published before the right one. Read this before theorising about egress again.** 1. The failures were first blamed on the ESH egress move (below), on the strength of a 2026-09-03 comment claiming NH3's residential address avoided CAPTCHAs. Reverting to direct NH3 egress produced a **byte-identical** result — same three engines down — which falsified it. 2. A live `!ddg` bang probe on a freshly restarted container also returned CAPTCHA, ruling out a stale circuit-breaker timer as the explanation. Both claims asserted causation from correlation without measuring a baseline. The version bump was the only change that moved the number. ⚠ **Neither health check can see this class of failure.** `/healthz` cannot, and `scripts/searxng-health.sh` prints `✓ searching` whether four engines answer or one, because it gates on "did any results come back" and treats failed engines as informational — it prints them, then passes anyway. Search ran on one engine for a day with every instrument green. **Make the script fail on blocked engines before trusting it again.** ## The ESH egress experiment (2026-09-17 → reverted 2026-09-18) Search egress was moved to `socks5h://10.0.50.65:1080` on **esh-scale** (CT 108 on esh-pve) — an application-level proxy, no host route or exit-node changes. Reverted after one day. It did **not** cause the engine outage above, and it did not fix anything either: engine behaviour was identical with it and without it. It was reverted on its own merits — it bought no measurable improvement while making all fleet search depend on ESH WAN and mesh availability, and the simpler configuration is the better one. ESH egress measured `128.177.138.182` in use and `154.50.58.126` at cutover; ESH's WAN address moves and nothing pins it. The proxy side was built correctly and none of this was its fault: microsocks ran as `nobody` under `searxng-egress.service`, bound `10.0.50.65:1080` only, and allowed source `10.100.50.40` alone. The canonical unit is kept at [`configs/esh-scale/searxng-egress.service`](../../configs/esh-scale/searxng-egress.service) in case a future egress experiment wants it; the service is stopped and disabled on esh-scale, and tailscaled there was not touched. ## Historical placement: NH3 rather than the colo Measured 2026-09-03: ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA nh3-docker egress 70.230.226.88 residential -> no gate on that account Search engines gate datacenter ranges. Running the aggregator from the residential-egress site was the original approach. The operator subsequently requested ESH egress; `outgoing.proxies` now explicitly selects it as above. ⚠ It is **not** a complete fix: `brave`, `duckduckgo` and `startpage` still CAPTCHA from here. `google cse` carries general search at ~20 results/query, and `yandex`, `wiby`, `github`, `stackoverflow` and `marginalia` all work. General search is therefore effectively single-engine — if `google cse` breaks, the instance goes quiet, which is exactly the failure below. ## ⚠ /healthz cannot tell you whether search works On 2026-09-03 the old ana-docker instance was found returning **zero results for every query**, for an unknown number of weeks, while: - the container reported `healthy` for 7 straight days, 0 restarts; - the Homepage card was green; - `/healthz` returned 200 every 30 seconds. It was running **2026.4.17 while current was 2026.9.3** — 4.5 months of engine scrapers rotting against sites that had changed their markup. `:latest` means "latest at pull time", and nothing re-pulls on its own. Proven by experiment before touching anything: the new image, same settings file, same host, same query returned **20 results where the running one returned 0**. **`scripts/searxng-health.sh` asserts the property, not the mechanism** — three unrelated queries must each return results > 0. Run it after any change, and periodically; it is the only thing that catches rot. ## Update ssh infra-ops@nh3-docker 'cd /opt/docker/compose/searxng && \ sudo docker compose pull && sudo docker compose up -d' scripts/searxng-health.sh ## The MCP tool `services/searxng-mcp/` — installed with `uv tool install` and registered at **user scope** (`claude mcp add --scope user searxng searxng-mcp`), so every Claude Code session gets `web_search` with no per-project setup. ⚠ **Zero results raise an error rather than returning an empty list.** An empty list is indistinguishable from a broken aggregator, and that ambiguity is what let the old instance fail silently for weeks. Same reasoning as althing's "an unreachable post office is an OUTAGE, never an empty inbox". ⚠ Written against **mcp 2.x**, where `FastMCP` became `MCPServer` and the v1 `@app.list_tools()` decorator API is gone. v1 examples fail at import with `'Server' object has no attribute 'list_tools'`. ⚠ `uv tool install --force` alone served a **cached build** and silently reinstalled the old code — the installed file still had the v1 API after the source no longer did. `--reinstall --no-cache` was required, and the check that caught it was `md5sum` of source vs installed. ## Secret `SEARXNG_SECRET` lives in `/opt/docker/compose/searxng/.env` (0600, root) on the host and is vaulted at `nh3-docker/searxng-secret`. Never in git.