Files

T

vh 156e12619d feat(searxng): route search egress through the esh-scale SOCKS5 proxy

Committing work deployed on 2026-09-17 that had been left uncommitted, so
canonical intent stops disagreeing with the running host. The deployed
/opt/docker/conf/searxng/searxng-settings.yml is byte-identical to the
canonical file here, verified before this commit.

Search requests and their DNS now exit via socks5h://10.0.50.65:1080 on
esh-scale (CT 108), an application-level proxy rather than a host-wide exit
node; no route or firewall changes. microsocks runs as nobody under
searxng-egress.service, binds only 10.0.50.65:1080, and bypasses SOCKS auth
for source 10.100.50.40 alone — every other source must supply a password
regenerated at each start and never distributed. Verified active and enabled.

There is deliberately no direct-NH3 fallback: an ESH outage must fail the
search rather than silently revert egress.

⚠ THE CHANGE HAS NOT ACHIEVED ITS PURPOSE AS DEPLOYED. Two independent live
queries, 2026-09-18, both report brave "Suspended: too many requests",
duckduckgo "CAPTCHA" and startpage "Suspended: CAPTCHA", leaving google cse
as the only answering engine. Moving egress off NH3's residential address is
what this change did, and CAPTCHA avoidance was the stated reason searxng sits
at NH3 at all. The README anticipated the risk in its Dependency note; it has
materialised. Rollback procedure is in the README and the pre-change config is
kept on the host as searxng-settings.yml.pre-esh-20260917.

Measured egress also drifted from the value recorded at cutover: the README
notes 154.50.58.126, the proxy now exits 128.177.138.182. Expected — the README
pins no public IP and calls out WAN failover — but recorded here so the number
in the doc is not mistaken for current.

Also retargets seat-inventory.py's default host from the mesh address
100.64.0.7 to fv-ml1's LAN address 10.251.50.54, routed by the site gateway.

2026-09-18 12:30:42 -07:00

5.0 KiB

Raw Blame History

searxng

Privacy-respecting metasearch. Runs on nh3-docker (10.100.50.40:9996, searxng.nh3.internal), and is exposed to every Claude Code session on nh3-dev as the web_search MCP tool.

Current egress: ESH (2026-09-17)

Hosting remains on nh3-docker; outbound search requests and their DNS resolution use socks5h://10.0.50.65:1080 on esh-scale, the ESH exit-node router (CT 108 on esh-pve, root@10.0.250.35). This is an application proxy, not a host-wide Tailscale exit-node selection. No host/subnet routes were changed. The application network layer measured ESH egress 154.50.58.126 at cutover; nh3-docker's direct egress remains 70.230.226.88. ESH's public IP can change with WAN failover; no public IP is pinned in this configuration.

The Debian microsocks package runs as nobody under searxng-egress.service, enabled at boot. The canonical unit is configs/esh-scale/searxng-egress.service. It binds only 10.0.50.65:1080. Only source 10.100.50.40 bypasses SOCKS authentication; other sources require a random, undistributed password regenerated at each start. Allowed-host egress and denied-host rejection were both tested. No gateway port-forward or firewall/routing changes were made.

Dependency: ESH WAN/mesh or proxy outages now break outbound search. There is no automatic direct-NH3 fallback. /healthz alone cannot detect this; run the three-query health script below. Engine CAPTCHAs may persist despite changing IP.

Rollback: on nh3-docker restore /opt/docker/conf/searxng/searxng-settings.yml.pre-esh-20260917 over /opt/docker/conf/searxng/searxng-settings.yml, then sudo docker restart searxng and run scripts/searxng-health.sh. Remove outgoing.proxies in the canonical settings too. Once no longer used, disable the dedicated proxy via ssh root@10.0.250.35 'pct exec 108 -- systemctl disable --now searxng-egress.service'. This does not stop Tailscale or alter ESH subnet routing.

Historical placement: NH3 rather than the colo

Measured 2026-09-03:

ana-docker egress   38.120.12.42    datacenter   -> DuckDuckGo/Startpage CAPTCHA
nh3-docker egress   70.230.226.88   residential  -> no gate on that account

Search engines gate datacenter ranges. Running the aggregator from the residential-egress site was the original approach. The operator subsequently requested ESH egress; outgoing.proxies now explicitly selects it as above.

⚠ It is not a complete fix: brave, duckduckgo and startpage still CAPTCHA from here. google cse carries general search at ~20 results/query, and yandex, wiby, github, stackoverflow and marginalia all work. General search is therefore effectively single-engine — if google cse breaks, the instance goes quiet, which is exactly the failure below.

⚠ /healthz cannot tell you whether search works

On 2026-09-03 the old ana-docker instance was found returning zero results for every query, for an unknown number of weeks, while:

the container reported healthy for 7 straight days, 0 restarts;
the Homepage card was green;
/healthz returned 200 every 30 seconds.

It was running 2026.4.17 while current was 2026.9.3 — 4.5 months of engine scrapers rotting against sites that had changed their markup. :latest means "latest at pull time", and nothing re-pulls on its own.

Proven by experiment before touching anything: the new image, same settings file, same host, same query returned 20 results where the running one returned 0.

scripts/searxng-health.sh asserts the property, not the mechanism — three unrelated queries must each return results > 0. Run it after any change, and periodically; it is the only thing that catches rot.

Update

ssh infra-ops@nh3-docker 'cd /opt/docker/compose/searxng && \
  sudo docker compose pull && sudo docker compose up -d'
scripts/searxng-health.sh

The MCP tool

services/searxng-mcp/ — installed with uv tool install and registered at user scope (claude mcp add --scope user searxng searxng-mcp), so every Claude Code session gets web_search with no per-project setup.

⚠ Zero results raise an error rather than returning an empty list. An empty list is indistinguishable from a broken aggregator, and that ambiguity is what let the old instance fail silently for weeks. Same reasoning as althing's "an unreachable post office is an OUTAGE, never an empty inbox".

⚠ Written against mcp 2.x, where FastMCP became MCPServer and the v1 @app.list_tools() decorator API is gone. v1 examples fail at import with 'Server' object has no attribute 'list_tools'.

⚠ uv tool install --force alone served a cached build and silently reinstalled the old code — the installed file still had the v1 API after the source no longer did. --reinstall --no-cache was required, and the check that caught it was md5sum of source vs installed.

Secret

SEARXNG_SECRET lives in /opt/docker/compose/searxng/.env (0600, root) on the host and is vaulted at nh3-docker/searxng-secret. Never in git.

5.0 KiB Raw Blame History