searxng had been answering from google cse alone for at least a day, with
brave and startpage suspended and duckduckgo returning CAPTCHA. Updating the
image from 2026.9.3+a1144dda3 to 2026.9.18+c0042add3 restored all four
engines immediately, and they held across 11 consecutive queries run after the
change specifically to rule out a freshly-reset circuit breaker flattering the
first measurement.
before searxng/searxng@sha256:3602e6ddbeba037f5d800d1ed9d296a8b93c9f5b3cf9d05fa179d0e766dd59a1
after searxng/searxng@sha256:e0027a772aeeea55bf642256aae6fb3344ffa5f25ca665898c2ea821101334c4
The image stays on :latest rather than being digest-pinned. For this stack that
is deliberate and now demonstrated: upstream ships engine-handler fixes as
providers change their bot gating, so being current is the mitigation, and a
pin would have frozen the breakage in place. The post office is pinned for the
opposite reason — it is the fleet message bus and must not move under us.
README corrected. It had carried two successive wrong diagnoses, both blaming
egress, and now records the real cause plus the two measurements that
falsified them: reverting to direct NH3 egress reproduced the failure exactly,
and a live !ddg probe on a freshly restarted container also CAPTCHA'd, ruling
out a stale suspension timer. Both wrong claims asserted causation from
correlation without a baseline.
The health-script blind spot is unchanged and still called out:
scripts/searxng-health.sh reports the same passing result whether four engines
answer or one.
Reverts the outgoing.proxies block added in 156e126. Canonical restored from
that commit's parent and verified byte-identical to the host's
searxng-settings.yml.pre-esh-20260917 backup, then deployed via
scripts/deploy-stack.sh so canonical and host converge rather than drift. The
esh-scale searxng-egress.service is stopped and disabled; tailscaled on that
container was not touched.
⚠ THE ROLLBACK DID NOT RESTORE THE ENGINES, WHICH FALSIFIES THE REASON GIVEN
FOR IT. 156e126 recorded that moving egress to ESH had cost three of four
engines. Measured after this revert, with egress confirmed back on
70.230.226.88 and the same instrument used for the before-measurement, the
result is identical: brave and startpage suspended, duckduckgo CAPTCHA, google
cse the only engine answering. Per-engine bang probes confirm duckduckgo is
CAPTCHA-ing the residential address live, so this is not a stale suspension
timer.
The engine failures therefore have some other cause and predate or are
independent of the ESH move. The claim in 156e126 asserted causation from a
correlation without measuring the pre-change state; the only evidence for
"residential egress avoids CAPTCHAs" was a comment dated 2026-09-03, which is
no longer true of this address.
The revert still stands on its own merits: ESH egress bought no measurable
improvement while adding a hard dependency on ESH WAN and mesh availability
for all fleet search, so the simpler configuration is the better one. It is
simply not the fix for the engines.
README rewritten to match: direct NH3 is documented as current, the ESH
attempt is kept as history with its measured outcome, and the health script's
blind spot is called out — scripts/searxng-health.sh prints a passing result
while three engines are blocked, because it gates on "any results returned"
and treats failed engines as informational. That script needs to fail on
blocked engines before any future egress change, or the next regression is
equally invisible.
Committing work deployed on 2026-09-17 that had been left uncommitted, so
canonical intent stops disagreeing with the running host. The deployed
/opt/docker/conf/searxng/searxng-settings.yml is byte-identical to the
canonical file here, verified before this commit.
Search requests and their DNS now exit via socks5h://10.0.50.65:1080 on
esh-scale (CT 108), an application-level proxy rather than a host-wide exit
node; no route or firewall changes. microsocks runs as nobody under
searxng-egress.service, binds only 10.0.50.65:1080, and bypasses SOCKS auth
for source 10.100.50.40 alone — every other source must supply a password
regenerated at each start and never distributed. Verified active and enabled.
There is deliberately no direct-NH3 fallback: an ESH outage must fail the
search rather than silently revert egress.
⚠ THE CHANGE HAS NOT ACHIEVED ITS PURPOSE AS DEPLOYED. Two independent live
queries, 2026-09-18, both report brave "Suspended: too many requests",
duckduckgo "CAPTCHA" and startpage "Suspended: CAPTCHA", leaving google cse
as the only answering engine. Moving egress off NH3's residential address is
what this change did, and CAPTCHA avoidance was the stated reason searxng sits
at NH3 at all. The README anticipated the risk in its Dependency note; it has
materialised. Rollback procedure is in the README and the pre-change config is
kept on the host as searxng-settings.yml.pre-esh-20260917.
Measured egress also drifted from the value recorded at cutover: the README
notes 154.50.58.126, the proxy now exits 128.177.138.182. Expected — the README
pins no public IP and calls out WAN failover — but recorded here so the number
in the doc is not mistaken for current.
Also retargets seat-inventory.py's default host from the mesh address
100.64.0.7 to fv-ml1's LAN address 10.251.50.54, routed by the site gateway.
The ana-docker instance was returning zero results for every query while
reporting healthy — 4.5 months stale (2026.4.17 against a current 2026.9.3),
its engine scrapers rotted against sites that had changed. /healthz proves
the web app answers and says nothing about whether search works, so seven
days of green sat on top of a search box that found nothing.
Moved to nh3-docker rather than updated in place, because the colo egress is
the other half of the problem: 38.120.12.42 is a datacenter address that
DuckDuckGo and Startpage CAPTCHA, while nh3-docker egresses residentially at
70.230.226.88. Same reasoning as the fleet's residential proxy for yt-dlp,
applied at the source instead of around it.
Config corrected along the way: base_url said searxng.pfi.local, a name
retired on 2026-08-19, while the environment said something else — the env
won so nothing broke and the file quietly lied. The karmasearch.videos
removal key never matched, because the engine's real name has a space.
scripts/searxng-health.sh asserts results > 0 across three unrelated
queries. That is the check that would have caught this, and the only kind
that can: the mechanism was healthy throughout.
services/searxng-mcp exposes it as `web_search` at user scope, so every
Claude Code session has it. Zero results raise rather than returning an
empty list — an empty list is indistinguishable from a broken aggregator,
which is precisely how this hid.
Old instance stopped and removed; DNS alias repointed to searxng.nh3.internal.
vh