revert(searxng): return search egress to direct NH3

Reverts the outgoing.proxies block added in 156e126. Canonical restored from
that commit's parent and verified byte-identical to the host's
searxng-settings.yml.pre-esh-20260917 backup, then deployed via
scripts/deploy-stack.sh so canonical and host converge rather than drift. The
esh-scale searxng-egress.service is stopped and disabled; tailscaled on that
container was not touched.

⚠ THE ROLLBACK DID NOT RESTORE THE ENGINES, WHICH FALSIFIES THE REASON GIVEN
FOR IT. 156e126 recorded that moving egress to ESH had cost three of four
engines. Measured after this revert, with egress confirmed back on
70.230.226.88 and the same instrument used for the before-measurement, the
result is identical: brave and startpage suspended, duckduckgo CAPTCHA, google
cse the only engine answering. Per-engine bang probes confirm duckduckgo is
CAPTCHA-ing the residential address live, so this is not a stale suspension
timer.

The engine failures therefore have some other cause and predate or are
independent of the ESH move. The claim in 156e126 asserted causation from a
correlation without measuring the pre-change state; the only evidence for
"residential egress avoids CAPTCHAs" was a comment dated 2026-09-03, which is
no longer true of this address.

The revert still stands on its own merits: ESH egress bought no measurable
improvement while adding a hard dependency on ESH WAN and mesh availability
for all fleet search, so the simpler configuration is the better one. It is
simply not the fix for the engines.

README rewritten to match: direct NH3 is documented as current, the ESH
attempt is kept as history with its measured outcome, and the health script's
blind spot is called out — scripts/searxng-health.sh prints a passing result
while three engines are blocked, because it gates on "any results returned"
and treats failed engines as informational. That script needs to fail on
blocked engines before any future egress change, or the next regression is
equally invisible.
This commit is contained in:
vh
2026-09-18 12:34:52 -07:00
parent 6ddb453b20
commit 1a35181b67
2 changed files with 47 additions and 33 deletions
+38 -25
View File
@@ -4,35 +4,48 @@ Privacy-respecting metasearch. **Runs on nh3-docker** (`10.100.50.40:9996`,
`searxng.nh3.internal`), and is exposed to every Claude Code session on nh3-dev
as the `web_search` MCP tool.
## Current egress: ESH (2026-09-17)
## Current egress: direct from NH3
Hosting remains on nh3-docker; outbound search requests and their DNS resolution
use `socks5h://10.0.50.65:1080` on **esh-scale**, the ESH exit-node router
(CT 108 on esh-pve, `root@10.0.250.35`). This is an application proxy, not a
host-wide Tailscale exit-node selection. No host/subnet routes were changed.
The application network layer measured ESH egress **154.50.58.126** at cutover;
nh3-docker's direct egress remains **70.230.226.88**. ESH's public IP can change
with WAN failover; no public IP is pinned in this configuration.
Hosting and egress are both nh3-docker. Outbound search requests leave from
**70.230.226.88**, NH3's residential address, with no proxy in `outgoing:`.
This is the configuration that works; the section below records why, and the
one after it records an attempt to change it that failed.
The Debian `microsocks` package runs as `nobody` under
`searxng-egress.service`, enabled at boot. The canonical unit is
[`configs/esh-scale/searxng-egress.service`](../../configs/esh-scale/searxng-egress.service).
It binds only `10.0.50.65:1080`. Only source `10.100.50.40` bypasses SOCKS
authentication; other sources require a random, undistributed password regenerated
at each start. Allowed-host egress and denied-host rejection were both tested.
No gateway port-forward or firewall/routing changes were made.
## The ESH egress experiment (2026-09-17 → reverted 2026-09-18)
**Dependency:** ESH WAN/mesh or proxy outages now break outbound search. There is
no automatic direct-NH3 fallback. `/healthz` alone cannot detect this; run the
three-query health script below. Engine CAPTCHAs may persist despite changing IP.
Search egress was moved to `socks5h://10.0.50.65:1080` on **esh-scale**
(CT 108 on esh-pve) — an application-level proxy, no host route or exit-node
changes. **Reverted after one day: it cost three of the four engines.**
**Rollback:** on nh3-docker restore
`/opt/docker/conf/searxng/searxng-settings.yml.pre-esh-20260917` over
`/opt/docker/conf/searxng/searxng-settings.yml`, then `sudo docker restart searxng`
and run `scripts/searxng-health.sh`. Remove `outgoing.proxies` in the canonical
settings too. Once no longer used, disable the dedicated proxy via
`ssh root@10.0.250.35 'pct exec 108 -- systemctl disable --now searxng-egress.service'`.
This does not stop Tailscale or alter ESH subnet routing.
Measured 2026-09-18 across five queries (two ad-hoc plus all three in
`scripts/searxng-health.sh`), identical every time:
brave Suspended: too many requests
duckduckgo CAPTCHA
startpage Suspended: CAPTCHA
google cse 20 results <- the only engine answering
The cause is the one this stack already knew about. SearXNG runs at NH3
*because* its residential egress avoids the CAPTCHA gating that gets applied
to datacenter ranges; routing out through ESH (`128.177.138.182` when
measured, `154.50.58.126` at cutover — ESH's WAN address moves) gave that
property up. The Dependency note written at cutover called the risk and it
materialised.
⚠ **The failure was invisible to both health checks.** `/healthz` cannot see
it, and `scripts/searxng-health.sh` — the script this README told you to run
instead — prints `✓ searching` in this exact state, because it gates on "did
any results come back" and treats failed engines as informational. Search ran
on one engine for a day with every instrument reporting healthy. If egress is
ever changed again, fix the script to fail on blocked engines *first*, or the
next regression is equally silent.
The proxy side was built correctly and none of this was its fault: microsocks
ran as `nobody` under `searxng-egress.service`, bound `10.0.50.65:1080` only,
and allowed source `10.100.50.40` alone. The canonical unit is kept at
[`configs/esh-scale/searxng-egress.service`](../../configs/esh-scale/searxng-egress.service)
in case a future egress experiment wants it; the service is disabled on
esh-scale.
## Historical placement: NH3 rather than the colo
+9 -8
View File
@@ -3,9 +3,10 @@
# ⚠ WHY NH3 AND NOT THE COLO. Measured 2026-09-03:
# ana-docker egress 38.120.12.42 (datacenter) -> DuckDuckGo + Startpage CAPTCHA
# nh3-docker egress 70.230.226.88 (residential) -> no CAPTCHA
# Since 2026-09-17, search requests exit via a restricted SOCKS5 listener on
# esh-scale (10.0.50.65:1080), per operator request. Hosting remains at NH3.
# No host default-route or mesh routing changes. See stacks/searxng/README.md.
# Search engines gate datacenter ranges. Same reason the fleet keeps a
# residential SOCKS5 egress proxy on nh3-dev for yt-dlp. Running the search
# aggregator from a residential-egress site removes the problem at the source
# rather than proxying around it.
use_default_settings:
engines:
@@ -71,8 +72,8 @@ outgoing:
pool_connections: 100
pool_maxsize: 20
enable_http2: true
# ESH-only search egress; resolve engine hostnames at the proxy.
# No direct fallback: an ESH outage must not silently switch back to NH3.
proxies:
all://:
- socks5h://10.0.50.65:1080
# No proxy needed: this host already egresses residentially (see header).
# If that ever changes, the fleet's NH3 SOCKS5 proxy is the fallback:
# proxies:
# all://:
# - socks5h://10.100.10.50:1080