diff --git a/stacks/searxng/README.md b/stacks/searxng/README.md index 97d305d..8c7f756 100644 --- a/stacks/searxng/README.md +++ b/stacks/searxng/README.md @@ -4,35 +4,48 @@ Privacy-respecting metasearch. **Runs on nh3-docker** (`10.100.50.40:9996`, `searxng.nh3.internal`), and is exposed to every Claude Code session on nh3-dev as the `web_search` MCP tool. -## Current egress: ESH (2026-09-17) +## Current egress: direct from NH3 -Hosting remains on nh3-docker; outbound search requests and their DNS resolution -use `socks5h://10.0.50.65:1080` on **esh-scale**, the ESH exit-node router -(CT 108 on esh-pve, `root@10.0.250.35`). This is an application proxy, not a -host-wide Tailscale exit-node selection. No host/subnet routes were changed. -The application network layer measured ESH egress **154.50.58.126** at cutover; -nh3-docker's direct egress remains **70.230.226.88**. ESH's public IP can change -with WAN failover; no public IP is pinned in this configuration. +Hosting and egress are both nh3-docker. Outbound search requests leave from +**70.230.226.88**, NH3's residential address, with no proxy in `outgoing:`. +This is the configuration that works; the section below records why, and the +one after it records an attempt to change it that failed. -The Debian `microsocks` package runs as `nobody` under -`searxng-egress.service`, enabled at boot. The canonical unit is -[`configs/esh-scale/searxng-egress.service`](../../configs/esh-scale/searxng-egress.service). -It binds only `10.0.50.65:1080`. Only source `10.100.50.40` bypasses SOCKS -authentication; other sources require a random, undistributed password regenerated -at each start. Allowed-host egress and denied-host rejection were both tested. -No gateway port-forward or firewall/routing changes were made. +## The ESH egress experiment (2026-09-17 → reverted 2026-09-18) -**Dependency:** ESH WAN/mesh or proxy outages now break outbound search. There is -no automatic direct-NH3 fallback. `/healthz` alone cannot detect this; run the -three-query health script below. Engine CAPTCHAs may persist despite changing IP. +Search egress was moved to `socks5h://10.0.50.65:1080` on **esh-scale** +(CT 108 on esh-pve) — an application-level proxy, no host route or exit-node +changes. **Reverted after one day: it cost three of the four engines.** -**Rollback:** on nh3-docker restore -`/opt/docker/conf/searxng/searxng-settings.yml.pre-esh-20260917` over -`/opt/docker/conf/searxng/searxng-settings.yml`, then `sudo docker restart searxng` -and run `scripts/searxng-health.sh`. Remove `outgoing.proxies` in the canonical -settings too. Once no longer used, disable the dedicated proxy via -`ssh root@10.0.250.35 'pct exec 108 -- systemctl disable --now searxng-egress.service'`. -This does not stop Tailscale or alter ESH subnet routing. +Measured 2026-09-18 across five queries (two ad-hoc plus all three in +`scripts/searxng-health.sh`), identical every time: + + brave Suspended: too many requests + duckduckgo CAPTCHA + startpage Suspended: CAPTCHA + google cse 20 results <- the only engine answering + +The cause is the one this stack already knew about. SearXNG runs at NH3 +*because* its residential egress avoids the CAPTCHA gating that gets applied +to datacenter ranges; routing out through ESH (`128.177.138.182` when +measured, `154.50.58.126` at cutover — ESH's WAN address moves) gave that +property up. The Dependency note written at cutover called the risk and it +materialised. + +⚠ **The failure was invisible to both health checks.** `/healthz` cannot see +it, and `scripts/searxng-health.sh` — the script this README told you to run +instead — prints `✓ searching` in this exact state, because it gates on "did +any results come back" and treats failed engines as informational. Search ran +on one engine for a day with every instrument reporting healthy. If egress is +ever changed again, fix the script to fail on blocked engines *first*, or the +next regression is equally silent. + +The proxy side was built correctly and none of this was its fault: microsocks +ran as `nobody` under `searxng-egress.service`, bound `10.0.50.65:1080` only, +and allowed source `10.100.50.40` alone. The canonical unit is kept at +[`configs/esh-scale/searxng-egress.service`](../../configs/esh-scale/searxng-egress.service) +in case a future egress experiment wants it; the service is disabled on +esh-scale. ## Historical placement: NH3 rather than the colo diff --git a/stacks/searxng/conf/searxng-settings.yml b/stacks/searxng/conf/searxng-settings.yml index b065f77..03f3208 100644 --- a/stacks/searxng/conf/searxng-settings.yml +++ b/stacks/searxng/conf/searxng-settings.yml @@ -3,9 +3,10 @@ # ⚠ WHY NH3 AND NOT THE COLO. Measured 2026-09-03: # ana-docker egress 38.120.12.42 (datacenter) -> DuckDuckGo + Startpage CAPTCHA # nh3-docker egress 70.230.226.88 (residential) -> no CAPTCHA -# Since 2026-09-17, search requests exit via a restricted SOCKS5 listener on -# esh-scale (10.0.50.65:1080), per operator request. Hosting remains at NH3. -# No host default-route or mesh routing changes. See stacks/searxng/README.md. +# Search engines gate datacenter ranges. Same reason the fleet keeps a +# residential SOCKS5 egress proxy on nh3-dev for yt-dlp. Running the search +# aggregator from a residential-egress site removes the problem at the source +# rather than proxying around it. use_default_settings: engines: @@ -71,8 +72,8 @@ outgoing: pool_connections: 100 pool_maxsize: 20 enable_http2: true - # ESH-only search egress; resolve engine hostnames at the proxy. - # No direct fallback: an ESH outage must not silently switch back to NH3. - proxies: - all://: - - socks5h://10.0.50.65:1080 + # No proxy needed: this host already egresses residentially (see header). + # If that ever changes, the fleet's NH3 SOCKS5 proxy is the fallback: + # proxies: + # all://: + # - socks5h://10.100.10.50:1080