vh
156e12619d
Committing work deployed on 2026-09-17 that had been left uncommitted, so canonical intent stops disagreeing with the running host. The deployed /opt/docker/conf/searxng/searxng-settings.yml is byte-identical to the canonical file here, verified before this commit. Search requests and their DNS now exit via socks5h://10.0.50.65:1080 on esh-scale (CT 108), an application-level proxy rather than a host-wide exit node; no route or firewall changes. microsocks runs as nobody under searxng-egress.service, binds only 10.0.50.65:1080, and bypasses SOCKS auth for source 10.100.50.40 alone — every other source must supply a password regenerated at each start and never distributed. Verified active and enabled. There is deliberately no direct-NH3 fallback: an ESH outage must fail the search rather than silently revert egress. ⚠ THE CHANGE HAS NOT ACHIEVED ITS PURPOSE AS DEPLOYED. Two independent live queries, 2026-09-18, both report brave "Suspended: too many requests", duckduckgo "CAPTCHA" and startpage "Suspended: CAPTCHA", leaving google cse as the only answering engine. Moving egress off NH3's residential address is what this change did, and CAPTCHA avoidance was the stated reason searxng sits at NH3 at all. The README anticipated the risk in its Dependency note; it has materialised. Rollback procedure is in the README and the pre-change config is kept on the host as searxng-settings.yml.pre-esh-20260917. Measured egress also drifted from the value recorded at cutover: the README notes 154.50.58.126, the proxy now exits 128.177.138.182. Expected — the README pins no public IP and calls out WAN failover — but recorded here so the number in the doc is not mistaken for current. Also retargets seat-inventory.py's default host from the mesh address 100.64.0.7 to fv-ml1's LAN address 10.251.50.54, routed by the site gateway.
searxng
Privacy-respecting metasearch. Runs on nh3-docker (10.100.50.40:9996,
searxng.nh3.internal), and is exposed to every Claude Code session on nh3-dev
as the web_search MCP tool.
Current egress: ESH (2026-09-17)
Hosting remains on nh3-docker; outbound search requests and their DNS resolution
use socks5h://10.0.50.65:1080 on esh-scale, the ESH exit-node router
(CT 108 on esh-pve, root@10.0.250.35). This is an application proxy, not a
host-wide Tailscale exit-node selection. No host/subnet routes were changed.
The application network layer measured ESH egress 154.50.58.126 at cutover;
nh3-docker's direct egress remains 70.230.226.88. ESH's public IP can change
with WAN failover; no public IP is pinned in this configuration.
The Debian microsocks package runs as nobody under
searxng-egress.service, enabled at boot. The canonical unit is
configs/esh-scale/searxng-egress.service.
It binds only 10.0.50.65:1080. Only source 10.100.50.40 bypasses SOCKS
authentication; other sources require a random, undistributed password regenerated
at each start. Allowed-host egress and denied-host rejection were both tested.
No gateway port-forward or firewall/routing changes were made.
Dependency: ESH WAN/mesh or proxy outages now break outbound search. There is
no automatic direct-NH3 fallback. /healthz alone cannot detect this; run the
three-query health script below. Engine CAPTCHAs may persist despite changing IP.
Rollback: on nh3-docker restore
/opt/docker/conf/searxng/searxng-settings.yml.pre-esh-20260917 over
/opt/docker/conf/searxng/searxng-settings.yml, then sudo docker restart searxng
and run scripts/searxng-health.sh. Remove outgoing.proxies in the canonical
settings too. Once no longer used, disable the dedicated proxy via
ssh root@10.0.250.35 'pct exec 108 -- systemctl disable --now searxng-egress.service'.
This does not stop Tailscale or alter ESH subnet routing.
Historical placement: NH3 rather than the colo
Measured 2026-09-03:
ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA
nh3-docker egress 70.230.226.88 residential -> no gate on that account
Search engines gate datacenter ranges. Running the aggregator from the
residential-egress site was the original approach. The operator subsequently
requested ESH egress; outgoing.proxies now explicitly selects it as above.
⚠ It is not a complete fix: brave, duckduckgo and startpage still
CAPTCHA from here. google cse carries general search at ~20 results/query, and
yandex, wiby, github, stackoverflow and marginalia all work. General
search is therefore effectively single-engine — if google cse breaks, the
instance goes quiet, which is exactly the failure below.
⚠ /healthz cannot tell you whether search works
On 2026-09-03 the old ana-docker instance was found returning zero results for every query, for an unknown number of weeks, while:
- the container reported
healthyfor 7 straight days, 0 restarts; - the Homepage card was green;
/healthzreturned 200 every 30 seconds.
It was running 2026.4.17 while current was 2026.9.3 — 4.5 months of engine
scrapers rotting against sites that had changed their markup. :latest means
"latest at pull time", and nothing re-pulls on its own.
Proven by experiment before touching anything: the new image, same settings file, same host, same query returned 20 results where the running one returned 0.
scripts/searxng-health.sh asserts the property, not the mechanism — three
unrelated queries must each return results > 0. Run it after any change, and
periodically; it is the only thing that catches rot.
Update
ssh infra-ops@nh3-docker 'cd /opt/docker/compose/searxng && \
sudo docker compose pull && sudo docker compose up -d'
scripts/searxng-health.sh
The MCP tool
services/searxng-mcp/ — installed with uv tool install and registered at
user scope (claude mcp add --scope user searxng searxng-mcp), so every
Claude Code session gets web_search with no per-project setup.
⚠ Zero results raise an error rather than returning an empty list. An empty list is indistinguishable from a broken aggregator, and that ambiguity is what let the old instance fail silently for weeks. Same reasoning as althing's "an unreachable post office is an OUTAGE, never an empty inbox".
⚠ Written against mcp 2.x, where FastMCP became MCPServer and the v1
@app.list_tools() decorator API is gone. v1 examples fail at import with
'Server' object has no attribute 'list_tools'.
⚠ uv tool install --force alone served a cached build and silently
reinstalled the old code — the installed file still had the v1 API after the
source no longer did. --reinstall --no-cache was required, and the check that
caught it was md5sum of source vs installed.
Secret
SEARXNG_SECRET lives in /opt/docker/compose/searxng/.env (0600, root) on the
host and is vaulted at nh3-docker/searxng-secret. Never in git.