Reverts the outgoing.proxies block added in 156e126. Canonical restored from
that commit's parent and verified byte-identical to the host's
searxng-settings.yml.pre-esh-20260917 backup, then deployed via
scripts/deploy-stack.sh so canonical and host converge rather than drift. The
esh-scale searxng-egress.service is stopped and disabled; tailscaled on that
container was not touched.
⚠ THE ROLLBACK DID NOT RESTORE THE ENGINES, WHICH FALSIFIES THE REASON GIVEN
FOR IT. 156e126 recorded that moving egress to ESH had cost three of four
engines. Measured after this revert, with egress confirmed back on
70.230.226.88 and the same instrument used for the before-measurement, the
result is identical: brave and startpage suspended, duckduckgo CAPTCHA, google
cse the only engine answering. Per-engine bang probes confirm duckduckgo is
CAPTCHA-ing the residential address live, so this is not a stale suspension
timer.
The engine failures therefore have some other cause and predate or are
independent of the ESH move. The claim in 156e126 asserted causation from a
correlation without measuring the pre-change state; the only evidence for
"residential egress avoids CAPTCHAs" was a comment dated 2026-09-03, which is
no longer true of this address.
The revert still stands on its own merits: ESH egress bought no measurable
improvement while adding a hard dependency on ESH WAN and mesh availability
for all fleet search, so the simpler configuration is the better one. It is
simply not the fix for the engines.
README rewritten to match: direct NH3 is documented as current, the ESH
attempt is kept as history with its measured outcome, and the health script's
blind spot is called out — scripts/searxng-health.sh prints a passing result
while three engines are blocked, because it gates on "any results returned"
and treats failed engines as informational. That script needs to fail on
blocked engines before any future egress change, or the next regression is
equally invisible.
Committing work deployed on 2026-09-17 that had been left uncommitted, so
canonical intent stops disagreeing with the running host. The deployed
/opt/docker/conf/searxng/searxng-settings.yml is byte-identical to the
canonical file here, verified before this commit.
Search requests and their DNS now exit via socks5h://10.0.50.65:1080 on
esh-scale (CT 108), an application-level proxy rather than a host-wide exit
node; no route or firewall changes. microsocks runs as nobody under
searxng-egress.service, binds only 10.0.50.65:1080, and bypasses SOCKS auth
for source 10.100.50.40 alone — every other source must supply a password
regenerated at each start and never distributed. Verified active and enabled.
There is deliberately no direct-NH3 fallback: an ESH outage must fail the
search rather than silently revert egress.
⚠ THE CHANGE HAS NOT ACHIEVED ITS PURPOSE AS DEPLOYED. Two independent live
queries, 2026-09-18, both report brave "Suspended: too many requests",
duckduckgo "CAPTCHA" and startpage "Suspended: CAPTCHA", leaving google cse
as the only answering engine. Moving egress off NH3's residential address is
what this change did, and CAPTCHA avoidance was the stated reason searxng sits
at NH3 at all. The README anticipated the risk in its Dependency note; it has
materialised. Rollback procedure is in the README and the pre-change config is
kept on the host as searxng-settings.yml.pre-esh-20260917.
Measured egress also drifted from the value recorded at cutover: the README
notes 154.50.58.126, the proxy now exits 128.177.138.182. Expected — the README
pins no public IP and calls out WAN failover — but recorded here so the number
in the doc is not mistaken for current.
Also retargets seat-inventory.py's default host from the mesh address
100.64.0.7 to fv-ml1's LAN address 10.251.50.54, routed by the site gateway.
The ana-docker instance was returning zero results for every query while
reporting healthy — 4.5 months stale (2026.4.17 against a current 2026.9.3),
its engine scrapers rotted against sites that had changed. /healthz proves
the web app answers and says nothing about whether search works, so seven
days of green sat on top of a search box that found nothing.
Moved to nh3-docker rather than updated in place, because the colo egress is
the other half of the problem: 38.120.12.42 is a datacenter address that
DuckDuckGo and Startpage CAPTCHA, while nh3-docker egresses residentially at
70.230.226.88. Same reasoning as the fleet's residential proxy for yt-dlp,
applied at the source instead of around it.
Config corrected along the way: base_url said searxng.pfi.local, a name
retired on 2026-08-19, while the environment said something else — the env
won so nothing broke and the file quietly lied. The karmasearch.videos
removal key never matched, because the engine's real name has a space.
scripts/searxng-health.sh asserts results > 0 across three unrelated
queries. That is the check that would have caught this, and the only kind
that can: the mechanism was healthy throughout.
services/searxng-mcp exposes it as `web_search` at user scope, so every
Claude Code session has it. Zero results raise rather than returning an
empty list — an empty list is indistinguishable from a broken aggregator,
which is precisely how this hid.
Old instance stopped and removed; DNS alias repointed to searxng.nh3.internal.
searxng: the healthcheck passed '--tries' and '--spider' as separate argv
entries, so wget consumed '--spider' as the value of '--tries'. Spider mode
never engaged and every 30s probe downloaded the response to disk; the
container's working directory had accumulated 295,287 healthz.N files since
April, and the directory scan to pick the next free filename is what
intermittently blew the 10s timeout and flapped the dashboard card to
UNHEALTHY. Restored '--tries=1'. The junk was in the writable layer, so the
recreate cleared it. Now healthy, fails=0, 200 in 0.16s.
seafile: none of the three services declared a restart policy, so Docker
defaulted them to 'no'. The daemon stopped all three within 200ms on
2026-05-06 and nothing brought them back — a three-month outage whose only
trace was an EXITED card. Exit 255 is what a container ignoring SIGTERM
reports when the daemon stops it, not a crash. Added restart: unless-stopped.
Stack is back up; mysql gates on its healthcheck as designed and seahub
started without the race. 302 -> login page.
Both stacks were running unmanaged on ana-docker and are now tracked here.
homepage: AI tab reordered by clickability per operator — chat frontends,
ComfyUI and the control plane on top; vLLM /docs seats and TTS endpoints
below. Corrects the previous commit's UNRESOLVED tab-bar section: it was
warm-up time after a recreate, not a defect.
vh