From 156e12619db8dd304825ee4bf443471af68b452d Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 18 Sep 2026 12:30:42 -0700 Subject: [PATCH] feat(searxng): route search egress through the esh-scale SOCKS5 proxy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Committing work deployed on 2026-09-17 that had been left uncommitted, so canonical intent stops disagreeing with the running host. The deployed /opt/docker/conf/searxng/searxng-settings.yml is byte-identical to the canonical file here, verified before this commit. Search requests and their DNS now exit via socks5h://10.0.50.65:1080 on esh-scale (CT 108), an application-level proxy rather than a host-wide exit node; no route or firewall changes. microsocks runs as nobody under searxng-egress.service, binds only 10.0.50.65:1080, and bypasses SOCKS auth for source 10.100.50.40 alone — every other source must supply a password regenerated at each start and never distributed. Verified active and enabled. There is deliberately no direct-NH3 fallback: an ESH outage must fail the search rather than silently revert egress. ⚠ THE CHANGE HAS NOT ACHIEVED ITS PURPOSE AS DEPLOYED. Two independent live queries, 2026-09-18, both report brave "Suspended: too many requests", duckduckgo "CAPTCHA" and startpage "Suspended: CAPTCHA", leaving google cse as the only answering engine. Moving egress off NH3's residential address is what this change did, and CAPTCHA avoidance was the stated reason searxng sits at NH3 at all. The README anticipated the risk in its Dependency note; it has materialised. Rollback procedure is in the README and the pre-change config is kept on the host as searxng-settings.yml.pre-esh-20260917. Measured egress also drifted from the value recorded at cutover: the README notes 154.50.58.126, the proxy now exits 128.177.138.182. Expected — the README pins no public IP and calls out WAN failover — but recorded here so the number in the doc is not mistaken for current. Also retargets seat-inventory.py's default host from the mesh address 100.64.0.7 to fv-ml1's LAN address 10.251.50.54, routed by the site gateway. --- configs/esh-scale/searxng-egress.service | 22 ++++++++++++++ scripts/seat-inventory.py | 2 +- stacks/searxng/README.md | 38 ++++++++++++++++++++---- stacks/searxng/conf/searxng-settings.yml | 17 +++++------ 4 files changed, 64 insertions(+), 15 deletions(-) create mode 100644 configs/esh-scale/searxng-egress.service diff --git a/configs/esh-scale/searxng-egress.service b/configs/esh-scale/searxng-egress.service new file mode 100644 index 0000000..f31c70f --- /dev/null +++ b/configs/esh-scale/searxng-egress.service @@ -0,0 +1,22 @@ +[Unit] +Description=SearXNG-only SOCKS5 egress via ESH +After=network-online.target tailscaled.service +Wants=network-online.target + +[Service] +Type=simple +User=nobody +Group=nogroup +# -w bypasses authentication ONLY for nh3-docker. Everyone else must provide +# an unknown, freshly randomized password (never stored or distributed). +ExecStart=/bin/sh -ec 'exec /usr/bin/microsocks -i 10.0.50.65 -p 1080 -b 10.0.50.65 -w 10.100.50.40 -u denied -P "$$(cat /proc/sys/kernel/random/uuid)"' +Restart=on-failure +RestartSec=5 +NoNewPrivileges=true +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +RestrictAddressFamilies=AF_INET AF_UNIX + +[Install] +WantedBy=multi-user.target diff --git a/scripts/seat-inventory.py b/scripts/seat-inventory.py index 04e6652..45b8d04 100755 --- a/scripts/seat-inventory.py +++ b/scripts/seat-inventory.py @@ -16,7 +16,7 @@ containers cannot. """ import argparse, json, re, subprocess, sys, datetime, pathlib -DEFAULT_HOST = "100.64.0.7" # fv-ml1 over the mesh +DEFAULT_HOST = "10.251.50.54" # fv-ml1 (LAN addr; routed by vb-gateway over the mesh) DEFAULT_OUT = "docs/pfi/fv-ml1-gpu-seat-inventory.md" GATEWAY = "10.250.50.70" # LiteLLM, for alias resolution diff --git a/stacks/searxng/README.md b/stacks/searxng/README.md index 91c679f..97d305d 100644 --- a/stacks/searxng/README.md +++ b/stacks/searxng/README.md @@ -4,7 +4,37 @@ Privacy-respecting metasearch. **Runs on nh3-docker** (`10.100.50.40:9996`, `searxng.nh3.internal`), and is exposed to every Claude Code session on nh3-dev as the `web_search` MCP tool. -## ⚠ Why NH3 and not the colo +## Current egress: ESH (2026-09-17) + +Hosting remains on nh3-docker; outbound search requests and their DNS resolution +use `socks5h://10.0.50.65:1080` on **esh-scale**, the ESH exit-node router +(CT 108 on esh-pve, `root@10.0.250.35`). This is an application proxy, not a +host-wide Tailscale exit-node selection. No host/subnet routes were changed. +The application network layer measured ESH egress **154.50.58.126** at cutover; +nh3-docker's direct egress remains **70.230.226.88**. ESH's public IP can change +with WAN failover; no public IP is pinned in this configuration. + +The Debian `microsocks` package runs as `nobody` under +`searxng-egress.service`, enabled at boot. The canonical unit is +[`configs/esh-scale/searxng-egress.service`](../../configs/esh-scale/searxng-egress.service). +It binds only `10.0.50.65:1080`. Only source `10.100.50.40` bypasses SOCKS +authentication; other sources require a random, undistributed password regenerated +at each start. Allowed-host egress and denied-host rejection were both tested. +No gateway port-forward or firewall/routing changes were made. + +**Dependency:** ESH WAN/mesh or proxy outages now break outbound search. There is +no automatic direct-NH3 fallback. `/healthz` alone cannot detect this; run the +three-query health script below. Engine CAPTCHAs may persist despite changing IP. + +**Rollback:** on nh3-docker restore +`/opt/docker/conf/searxng/searxng-settings.yml.pre-esh-20260917` over +`/opt/docker/conf/searxng/searxng-settings.yml`, then `sudo docker restart searxng` +and run `scripts/searxng-health.sh`. Remove `outgoing.proxies` in the canonical +settings too. Once no longer used, disable the dedicated proxy via +`ssh root@10.0.250.35 'pct exec 108 -- systemctl disable --now searxng-egress.service'`. +This does not stop Tailscale or alter ESH subnet routing. + +## Historical placement: NH3 rather than the colo Measured 2026-09-03: @@ -12,10 +42,8 @@ Measured 2026-09-03: nh3-docker egress 70.230.226.88 residential -> no gate on that account Search engines gate datacenter ranges. Running the aggregator from the -residential-egress site removes the problem at the source rather than proxying -around it — the same reason the fleet keeps a residential SOCKS5 proxy on -nh3-dev for yt-dlp. If NH3's egress ever changes, `outgoing.proxies` in -`conf/searxng-settings.yml` has the fallback commented in place. +residential-egress site was the original approach. The operator subsequently +requested ESH egress; `outgoing.proxies` now explicitly selects it as above. ⚠ It is **not** a complete fix: `brave`, `duckduckgo` and `startpage` still CAPTCHA from here. `google cse` carries general search at ~20 results/query, and diff --git a/stacks/searxng/conf/searxng-settings.yml b/stacks/searxng/conf/searxng-settings.yml index 03f3208..b065f77 100644 --- a/stacks/searxng/conf/searxng-settings.yml +++ b/stacks/searxng/conf/searxng-settings.yml @@ -3,10 +3,9 @@ # ⚠ WHY NH3 AND NOT THE COLO. Measured 2026-09-03: # ana-docker egress 38.120.12.42 (datacenter) -> DuckDuckGo + Startpage CAPTCHA # nh3-docker egress 70.230.226.88 (residential) -> no CAPTCHA -# Search engines gate datacenter ranges. Same reason the fleet keeps a -# residential SOCKS5 egress proxy on nh3-dev for yt-dlp. Running the search -# aggregator from a residential-egress site removes the problem at the source -# rather than proxying around it. +# Since 2026-09-17, search requests exit via a restricted SOCKS5 listener on +# esh-scale (10.0.50.65:1080), per operator request. Hosting remains at NH3. +# No host default-route or mesh routing changes. See stacks/searxng/README.md. use_default_settings: engines: @@ -72,8 +71,8 @@ outgoing: pool_connections: 100 pool_maxsize: 20 enable_http2: true - # No proxy needed: this host already egresses residentially (see header). - # If that ever changes, the fleet's NH3 SOCKS5 proxy is the fallback: - # proxies: - # all://: - # - socks5h://10.100.10.50:1080 + # ESH-only search egress; resolve engine hostnames at the proxy. + # No direct fallback: an ESH outage must not silently switch back to NH3. + proxies: + all://: + - socks5h://10.0.50.65:1080