# searxng

Privacy-respecting metasearch. **Runs on nh3-docker** (`10.100.50.40:9996`,
`searxng.nh3.internal`), and is exposed to every Claude Code session on nh3-dev
as the `web_search` MCP tool.

## Current egress: ESH (2026-09-17)

Hosting remains on nh3-docker; outbound search requests and their DNS resolution
use `socks5h://10.0.50.65:1080` on **esh-scale**, the ESH exit-node router
(CT 108 on esh-pve, `root@10.0.250.35`). This is an application proxy, not a
host-wide Tailscale exit-node selection. No host/subnet routes were changed.
The application network layer measured ESH egress **154.50.58.126** at cutover;
nh3-docker's direct egress remains **70.230.226.88**. ESH's public IP can change
with WAN failover; no public IP is pinned in this configuration.

The Debian `microsocks` package runs as `nobody` under
`searxng-egress.service`, enabled at boot. The canonical unit is
[`configs/esh-scale/searxng-egress.service`](../../configs/esh-scale/searxng-egress.service).
It binds only `10.0.50.65:1080`. Only source `10.100.50.40` bypasses SOCKS
authentication; other sources require a random, undistributed password regenerated
at each start. Allowed-host egress and denied-host rejection were both tested.
No gateway port-forward or firewall/routing changes were made.

**Dependency:** ESH WAN/mesh or proxy outages now break outbound search. There is
no automatic direct-NH3 fallback. `/healthz` alone cannot detect this; run the
three-query health script below. Engine CAPTCHAs may persist despite changing IP.

**Rollback:** on nh3-docker restore
`/opt/docker/conf/searxng/searxng-settings.yml.pre-esh-20260917` over
`/opt/docker/conf/searxng/searxng-settings.yml`, then `sudo docker restart searxng`
and run `scripts/searxng-health.sh`. Remove `outgoing.proxies` in the canonical
settings too. Once no longer used, disable the dedicated proxy via
`ssh root@10.0.250.35 'pct exec 108 -- systemctl disable --now searxng-egress.service'`.
This does not stop Tailscale or alter ESH subnet routing.

## Historical placement: NH3 rather than the colo

Measured 2026-09-03:

    ana-docker egress   38.120.12.42    datacenter   -> DuckDuckGo/Startpage CAPTCHA
    nh3-docker egress   70.230.226.88   residential  -> no gate on that account

Search engines gate datacenter ranges. Running the aggregator from the
residential-egress site was the original approach. The operator subsequently
requested ESH egress; `outgoing.proxies` now explicitly selects it as above.

⚠ It is **not** a complete fix: `brave`, `duckduckgo` and `startpage` still
CAPTCHA from here. `google cse` carries general search at ~20 results/query, and
`yandex`, `wiby`, `github`, `stackoverflow` and `marginalia` all work. General
search is therefore effectively single-engine — if `google cse` breaks, the
instance goes quiet, which is exactly the failure below.

## ⚠ /healthz cannot tell you whether search works

On 2026-09-03 the old ana-docker instance was found returning **zero results for
every query**, for an unknown number of weeks, while:

- the container reported `healthy` for 7 straight days, 0 restarts;
- the Homepage card was green;
- `/healthz` returned 200 every 30 seconds.

It was running **2026.4.17 while current was 2026.9.3** — 4.5 months of engine
scrapers rotting against sites that had changed their markup. `:latest` means
"latest at pull time", and nothing re-pulls on its own.

Proven by experiment before touching anything: the new image, same settings
file, same host, same query returned **20 results where the running one returned
0**.

**`scripts/searxng-health.sh` asserts the property, not the mechanism** — three
unrelated queries must each return results > 0. Run it after any change, and
periodically; it is the only thing that catches rot.

## Update

    ssh infra-ops@nh3-docker 'cd /opt/docker/compose/searxng && \
      sudo docker compose pull && sudo docker compose up -d'
    scripts/searxng-health.sh

## The MCP tool

`services/searxng-mcp/` — installed with `uv tool install` and registered at
**user scope** (`claude mcp add --scope user searxng searxng-mcp`), so every
Claude Code session gets `web_search` with no per-project setup.

⚠ **Zero results raise an error rather than returning an empty list.** An empty
list is indistinguishable from a broken aggregator, and that ambiguity is what
let the old instance fail silently for weeks. Same reasoning as althing's
"an unreachable post office is an OUTAGE, never an empty inbox".

⚠ Written against **mcp 2.x**, where `FastMCP` became `MCPServer` and the v1
`@app.list_tools()` decorator API is gone. v1 examples fail at import with
`'Server' object has no attribute 'list_tools'`.

⚠ `uv tool install --force` alone served a **cached build** and silently
reinstalled the old code — the installed file still had the v1 API after the
source no longer did. `--reinstall --no-cache` was required, and the check that
caught it was `md5sum` of source vs installed.

## Secret

`SEARXNG_SECRET` lives in `/opt/docker/compose/searxng/.env` (0600, root) on the
host and is vaulted at `nh3-docker/searxng-secret`. Never in git.