diff --git a/stacks/searxng/README.md b/stacks/searxng/README.md index 8c7f756..359d78f 100644 --- a/stacks/searxng/README.md +++ b/stacks/searxng/README.md @@ -11,41 +11,59 @@ Hosting and egress are both nh3-docker. Outbound search requests leave from This is the configuration that works; the section below records why, and the one after it records an attempt to change it that failed. +## The 2026-09-18 engine outage — cause was the build, not egress + +For at least a day, searxng answered from **one engine**: brave and startpage +suspended, duckduckgo CAPTCHA, google cse alone returning results. **Fixed by +updating the image** — `2026.9.3+a1144dda3` → `2026.9.18+c0042add3`. All four +engines came back immediately and held across 11 consecutive queries. + +Upstream ships engine-handler fixes constantly, because search providers change +their bot gating constantly. **That is why this stack tracks `:latest` rather +than pinning a digest**: for searxng specifically, being current *is* the +mitigation, and a pin would have frozen the breakage in place. + +⚠ **Two wrong diagnoses were published before the right one. Read this before +theorising about egress again.** + +1. The failures were first blamed on the ESH egress move (below), on the + strength of a 2026-09-03 comment claiming NH3's residential address avoided + CAPTCHAs. Reverting to direct NH3 egress produced a **byte-identical** + result — same three engines down — which falsified it. +2. A live `!ddg` bang probe on a freshly restarted container also returned + CAPTCHA, ruling out a stale circuit-breaker timer as the explanation. + +Both claims asserted causation from correlation without measuring a baseline. +The version bump was the only change that moved the number. + +⚠ **Neither health check can see this class of failure.** `/healthz` cannot, +and `scripts/searxng-health.sh` prints `✓ searching` whether four engines +answer or one, because it gates on "did any results come back" and treats +failed engines as informational — it prints them, then passes anyway. Search +ran on one engine for a day with every instrument green. **Make the script fail +on blocked engines before trusting it again.** + ## The ESH egress experiment (2026-09-17 → reverted 2026-09-18) Search egress was moved to `socks5h://10.0.50.65:1080` on **esh-scale** (CT 108 on esh-pve) — an application-level proxy, no host route or exit-node -changes. **Reverted after one day: it cost three of the four engines.** +changes. Reverted after one day. -Measured 2026-09-18 across five queries (two ad-hoc plus all three in -`scripts/searxng-health.sh`), identical every time: +It did **not** cause the engine outage above, and it did not fix anything +either: engine behaviour was identical with it and without it. It was reverted +on its own merits — it bought no measurable improvement while making all fleet +search depend on ESH WAN and mesh availability, and the simpler configuration +is the better one. - brave Suspended: too many requests - duckduckgo CAPTCHA - startpage Suspended: CAPTCHA - google cse 20 results <- the only engine answering - -The cause is the one this stack already knew about. SearXNG runs at NH3 -*because* its residential egress avoids the CAPTCHA gating that gets applied -to datacenter ranges; routing out through ESH (`128.177.138.182` when -measured, `154.50.58.126` at cutover — ESH's WAN address moves) gave that -property up. The Dependency note written at cutover called the risk and it -materialised. - -⚠ **The failure was invisible to both health checks.** `/healthz` cannot see -it, and `scripts/searxng-health.sh` — the script this README told you to run -instead — prints `✓ searching` in this exact state, because it gates on "did -any results come back" and treats failed engines as informational. Search ran -on one engine for a day with every instrument reporting healthy. If egress is -ever changed again, fix the script to fail on blocked engines *first*, or the -next regression is equally silent. +ESH egress measured `128.177.138.182` in use and `154.50.58.126` at cutover; +ESH's WAN address moves and nothing pins it. The proxy side was built correctly and none of this was its fault: microsocks ran as `nobody` under `searxng-egress.service`, bound `10.0.50.65:1080` only, and allowed source `10.100.50.40` alone. The canonical unit is kept at [`configs/esh-scale/searxng-egress.service`](../../configs/esh-scale/searxng-egress.service) -in case a future egress experiment wants it; the service is disabled on -esh-scale. +in case a future egress experiment wants it; the service is stopped and +disabled on esh-scale, and tailscaled there was not touched. ## Historical placement: NH3 rather than the colo