fix(searxng): update to 2026.9.18 — all four engines restored

searxng had been answering from google cse alone for at least a day, with
brave and startpage suspended and duckduckgo returning CAPTCHA. Updating the
image from 2026.9.3+a1144dda3 to 2026.9.18+c0042add3 restored all four
engines immediately, and they held across 11 consecutive queries run after the
change specifically to rule out a freshly-reset circuit breaker flattering the
first measurement.

  before  searxng/searxng@sha256:3602e6ddbeba037f5d800d1ed9d296a8b93c9f5b3cf9d05fa179d0e766dd59a1
  after   searxng/searxng@sha256:e0027a772aeeea55bf642256aae6fb3344ffa5f25ca665898c2ea821101334c4

The image stays on :latest rather than being digest-pinned. For this stack that
is deliberate and now demonstrated: upstream ships engine-handler fixes as
providers change their bot gating, so being current is the mitigation, and a
pin would have frozen the breakage in place. The post office is pinned for the
opposite reason — it is the fleet message bus and must not move under us.

README corrected. It had carried two successive wrong diagnoses, both blaming
egress, and now records the real cause plus the two measurements that
falsified them: reverting to direct NH3 egress reproduced the failure exactly,
and a live !ddg probe on a freshly restarted container also CAPTCHA'd, ruling
out a stale suspension timer. Both wrong claims asserted causation from
correlation without a baseline.

The health-script blind spot is unchanged and still called out:
scripts/searxng-health.sh reports the same passing result whether four engines
answer or one.
This commit is contained in:
2026-09-18 12:46:21 -07:00
parent ca5f0a91c0
commit 9a428fded9
+42 -24
View File
@@ -11,41 +11,59 @@ Hosting and egress are both nh3-docker. Outbound search requests leave from
This is the configuration that works; the section below records why, and the
one after it records an attempt to change it that failed.
## The 2026-09-18 engine outage — cause was the build, not egress
For at least a day, searxng answered from **one engine**: brave and startpage
suspended, duckduckgo CAPTCHA, google cse alone returning results. **Fixed by
updating the image** — `2026.9.3+a1144dda3``2026.9.18+c0042add3`. All four
engines came back immediately and held across 11 consecutive queries.
Upstream ships engine-handler fixes constantly, because search providers change
their bot gating constantly. **That is why this stack tracks `:latest` rather
than pinning a digest**: for searxng specifically, being current *is* the
mitigation, and a pin would have frozen the breakage in place.
⚠ **Two wrong diagnoses were published before the right one. Read this before
theorising about egress again.**
1. The failures were first blamed on the ESH egress move (below), on the
strength of a 2026-09-03 comment claiming NH3's residential address avoided
CAPTCHAs. Reverting to direct NH3 egress produced a **byte-identical**
result — same three engines down — which falsified it.
2. A live `!ddg` bang probe on a freshly restarted container also returned
CAPTCHA, ruling out a stale circuit-breaker timer as the explanation.
Both claims asserted causation from correlation without measuring a baseline.
The version bump was the only change that moved the number.
**Neither health check can see this class of failure.** `/healthz` cannot,
and `scripts/searxng-health.sh` prints `✓ searching` whether four engines
answer or one, because it gates on "did any results come back" and treats
failed engines as informational — it prints them, then passes anyway. Search
ran on one engine for a day with every instrument green. **Make the script fail
on blocked engines before trusting it again.**
## The ESH egress experiment (2026-09-17 → reverted 2026-09-18)
Search egress was moved to `socks5h://10.0.50.65:1080` on **esh-scale**
(CT 108 on esh-pve) — an application-level proxy, no host route or exit-node
changes. **Reverted after one day: it cost three of the four engines.**
changes. Reverted after one day.
Measured 2026-09-18 across five queries (two ad-hoc plus all three in
`scripts/searxng-health.sh`), identical every time:
It did **not** cause the engine outage above, and it did not fix anything
either: engine behaviour was identical with it and without it. It was reverted
on its own merits — it bought no measurable improvement while making all fleet
search depend on ESH WAN and mesh availability, and the simpler configuration
is the better one.
brave Suspended: too many requests
duckduckgo CAPTCHA
startpage Suspended: CAPTCHA
google cse 20 results <- the only engine answering
The cause is the one this stack already knew about. SearXNG runs at NH3
*because* its residential egress avoids the CAPTCHA gating that gets applied
to datacenter ranges; routing out through ESH (`128.177.138.182` when
measured, `154.50.58.126` at cutover — ESH's WAN address moves) gave that
property up. The Dependency note written at cutover called the risk and it
materialised.
**The failure was invisible to both health checks.** `/healthz` cannot see
it, and `scripts/searxng-health.sh` — the script this README told you to run
instead — prints `✓ searching` in this exact state, because it gates on "did
any results come back" and treats failed engines as informational. Search ran
on one engine for a day with every instrument reporting healthy. If egress is
ever changed again, fix the script to fail on blocked engines *first*, or the
next regression is equally silent.
ESH egress measured `128.177.138.182` in use and `154.50.58.126` at cutover;
ESH's WAN address moves and nothing pins it.
The proxy side was built correctly and none of this was its fault: microsocks
ran as `nobody` under `searxng-egress.service`, bound `10.0.50.65:1080` only,
and allowed source `10.100.50.40` alone. The canonical unit is kept at
[`configs/esh-scale/searxng-egress.service`](../../configs/esh-scale/searxng-egress.service)
in case a future egress experiment wants it; the service is disabled on
esh-scale.
in case a future egress experiment wants it; the service is stopped and
disabled on esh-scale, and tailscaled there was not touched.
## Historical placement: NH3 rather than the colo