vh
1a35181b67
Reverts the outgoing.proxies block added in156e126. Canonical restored from that commit's parent and verified byte-identical to the host's searxng-settings.yml.pre-esh-20260917 backup, then deployed via scripts/deploy-stack.sh so canonical and host converge rather than drift. The esh-scale searxng-egress.service is stopped and disabled; tailscaled on that container was not touched. ⚠ THE ROLLBACK DID NOT RESTORE THE ENGINES, WHICH FALSIFIES THE REASON GIVEN FOR IT.156e126recorded that moving egress to ESH had cost three of four engines. Measured after this revert, with egress confirmed back on 70.230.226.88 and the same instrument used for the before-measurement, the result is identical: brave and startpage suspended, duckduckgo CAPTCHA, google cse the only engine answering. Per-engine bang probes confirm duckduckgo is CAPTCHA-ing the residential address live, so this is not a stale suspension timer. The engine failures therefore have some other cause and predate or are independent of the ESH move. The claim in156e126asserted causation from a correlation without measuring the pre-change state; the only evidence for "residential egress avoids CAPTCHAs" was a comment dated 2026-09-03, which is no longer true of this address. The revert still stands on its own merits: ESH egress bought no measurable improvement while adding a hard dependency on ESH WAN and mesh availability for all fleet search, so the simpler configuration is the better one. It is simply not the fix for the engines. README rewritten to match: direct NH3 is documented as current, the ESH attempt is kept as history with its measured outcome, and the health script's blind spot is called out — scripts/searxng-health.sh prints a passing result while three engines are blocked, because it gates on "any results returned" and treats failed engines as informational. That script needs to fail on blocked engines before any future egress change, or the next regression is equally invisible.
118 lines
5.4 KiB
Markdown
118 lines
5.4 KiB
Markdown
# searxng
|
|
|
|
Privacy-respecting metasearch. **Runs on nh3-docker** (`10.100.50.40:9996`,
|
|
`searxng.nh3.internal`), and is exposed to every Claude Code session on nh3-dev
|
|
as the `web_search` MCP tool.
|
|
|
|
## Current egress: direct from NH3
|
|
|
|
Hosting and egress are both nh3-docker. Outbound search requests leave from
|
|
**70.230.226.88**, NH3's residential address, with no proxy in `outgoing:`.
|
|
This is the configuration that works; the section below records why, and the
|
|
one after it records an attempt to change it that failed.
|
|
|
|
## The ESH egress experiment (2026-09-17 → reverted 2026-09-18)
|
|
|
|
Search egress was moved to `socks5h://10.0.50.65:1080` on **esh-scale**
|
|
(CT 108 on esh-pve) — an application-level proxy, no host route or exit-node
|
|
changes. **Reverted after one day: it cost three of the four engines.**
|
|
|
|
Measured 2026-09-18 across five queries (two ad-hoc plus all three in
|
|
`scripts/searxng-health.sh`), identical every time:
|
|
|
|
brave Suspended: too many requests
|
|
duckduckgo CAPTCHA
|
|
startpage Suspended: CAPTCHA
|
|
google cse 20 results <- the only engine answering
|
|
|
|
The cause is the one this stack already knew about. SearXNG runs at NH3
|
|
*because* its residential egress avoids the CAPTCHA gating that gets applied
|
|
to datacenter ranges; routing out through ESH (`128.177.138.182` when
|
|
measured, `154.50.58.126` at cutover — ESH's WAN address moves) gave that
|
|
property up. The Dependency note written at cutover called the risk and it
|
|
materialised.
|
|
|
|
⚠ **The failure was invisible to both health checks.** `/healthz` cannot see
|
|
it, and `scripts/searxng-health.sh` — the script this README told you to run
|
|
instead — prints `✓ searching` in this exact state, because it gates on "did
|
|
any results come back" and treats failed engines as informational. Search ran
|
|
on one engine for a day with every instrument reporting healthy. If egress is
|
|
ever changed again, fix the script to fail on blocked engines *first*, or the
|
|
next regression is equally silent.
|
|
|
|
The proxy side was built correctly and none of this was its fault: microsocks
|
|
ran as `nobody` under `searxng-egress.service`, bound `10.0.50.65:1080` only,
|
|
and allowed source `10.100.50.40` alone. The canonical unit is kept at
|
|
[`configs/esh-scale/searxng-egress.service`](../../configs/esh-scale/searxng-egress.service)
|
|
in case a future egress experiment wants it; the service is disabled on
|
|
esh-scale.
|
|
|
|
## Historical placement: NH3 rather than the colo
|
|
|
|
Measured 2026-09-03:
|
|
|
|
ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA
|
|
nh3-docker egress 70.230.226.88 residential -> no gate on that account
|
|
|
|
Search engines gate datacenter ranges. Running the aggregator from the
|
|
residential-egress site was the original approach. The operator subsequently
|
|
requested ESH egress; `outgoing.proxies` now explicitly selects it as above.
|
|
|
|
⚠ It is **not** a complete fix: `brave`, `duckduckgo` and `startpage` still
|
|
CAPTCHA from here. `google cse` carries general search at ~20 results/query, and
|
|
`yandex`, `wiby`, `github`, `stackoverflow` and `marginalia` all work. General
|
|
search is therefore effectively single-engine — if `google cse` breaks, the
|
|
instance goes quiet, which is exactly the failure below.
|
|
|
|
## ⚠ /healthz cannot tell you whether search works
|
|
|
|
On 2026-09-03 the old ana-docker instance was found returning **zero results for
|
|
every query**, for an unknown number of weeks, while:
|
|
|
|
- the container reported `healthy` for 7 straight days, 0 restarts;
|
|
- the Homepage card was green;
|
|
- `/healthz` returned 200 every 30 seconds.
|
|
|
|
It was running **2026.4.17 while current was 2026.9.3** — 4.5 months of engine
|
|
scrapers rotting against sites that had changed their markup. `:latest` means
|
|
"latest at pull time", and nothing re-pulls on its own.
|
|
|
|
Proven by experiment before touching anything: the new image, same settings
|
|
file, same host, same query returned **20 results where the running one returned
|
|
0**.
|
|
|
|
**`scripts/searxng-health.sh` asserts the property, not the mechanism** — three
|
|
unrelated queries must each return results > 0. Run it after any change, and
|
|
periodically; it is the only thing that catches rot.
|
|
|
|
## Update
|
|
|
|
ssh infra-ops@nh3-docker 'cd /opt/docker/compose/searxng && \
|
|
sudo docker compose pull && sudo docker compose up -d'
|
|
scripts/searxng-health.sh
|
|
|
|
## The MCP tool
|
|
|
|
`services/searxng-mcp/` — installed with `uv tool install` and registered at
|
|
**user scope** (`claude mcp add --scope user searxng searxng-mcp`), so every
|
|
Claude Code session gets `web_search` with no per-project setup.
|
|
|
|
⚠ **Zero results raise an error rather than returning an empty list.** An empty
|
|
list is indistinguishable from a broken aggregator, and that ambiguity is what
|
|
let the old instance fail silently for weeks. Same reasoning as althing's
|
|
"an unreachable post office is an OUTAGE, never an empty inbox".
|
|
|
|
⚠ Written against **mcp 2.x**, where `FastMCP` became `MCPServer` and the v1
|
|
`@app.list_tools()` decorator API is gone. v1 examples fail at import with
|
|
`'Server' object has no attribute 'list_tools'`.
|
|
|
|
⚠ `uv tool install --force` alone served a **cached build** and silently
|
|
reinstalled the old code — the installed file still had the v1 API after the
|
|
source no longer did. `--reinstall --no-cache` was required, and the check that
|
|
caught it was `md5sum` of source vs installed.
|
|
|
|
## Secret
|
|
|
|
`SEARXNG_SECRET` lives in `/opt/docker/compose/searxng/.env` (0600, root) on the
|
|
host and is vaulted at `nh3-docker/searxng-secret`. Never in git.
|