vh
9a428fded9
searxng had been answering from google cse alone for at least a day, with brave and startpage suspended and duckduckgo returning CAPTCHA. Updating the image from 2026.9.3+a1144dda3 to 2026.9.18+c0042add3 restored all four engines immediately, and they held across 11 consecutive queries run after the change specifically to rule out a freshly-reset circuit breaker flattering the first measurement. before searxng/searxng@sha256:3602e6ddbeba037f5d800d1ed9d296a8b93c9f5b3cf9d05fa179d0e766dd59a1 after searxng/searxng@sha256:e0027a772aeeea55bf642256aae6fb3344ffa5f25ca665898c2ea821101334c4 The image stays on :latest rather than being digest-pinned. For this stack that is deliberate and now demonstrated: upstream ships engine-handler fixes as providers change their bot gating, so being current is the mitigation, and a pin would have frozen the breakage in place. The post office is pinned for the opposite reason — it is the fleet message bus and must not move under us. README corrected. It had carried two successive wrong diagnoses, both blaming egress, and now records the real cause plus the two measurements that falsified them: reverting to direct NH3 egress reproduced the failure exactly, and a live !ddg probe on a freshly restarted container also CAPTCHA'd, ruling out a stale suspension timer. Both wrong claims asserted causation from correlation without a baseline. The health-script blind spot is unchanged and still called out: scripts/searxng-health.sh reports the same passing result whether four engines answer or one.
136 lines
6.4 KiB
Markdown
136 lines
6.4 KiB
Markdown
# searxng
|
|
|
|
Privacy-respecting metasearch. **Runs on nh3-docker** (`10.100.50.40:9996`,
|
|
`searxng.nh3.internal`), and is exposed to every Claude Code session on nh3-dev
|
|
as the `web_search` MCP tool.
|
|
|
|
## Current egress: direct from NH3
|
|
|
|
Hosting and egress are both nh3-docker. Outbound search requests leave from
|
|
**70.230.226.88**, NH3's residential address, with no proxy in `outgoing:`.
|
|
This is the configuration that works; the section below records why, and the
|
|
one after it records an attempt to change it that failed.
|
|
|
|
## The 2026-09-18 engine outage — cause was the build, not egress
|
|
|
|
For at least a day, searxng answered from **one engine**: brave and startpage
|
|
suspended, duckduckgo CAPTCHA, google cse alone returning results. **Fixed by
|
|
updating the image** — `2026.9.3+a1144dda3` → `2026.9.18+c0042add3`. All four
|
|
engines came back immediately and held across 11 consecutive queries.
|
|
|
|
Upstream ships engine-handler fixes constantly, because search providers change
|
|
their bot gating constantly. **That is why this stack tracks `:latest` rather
|
|
than pinning a digest**: for searxng specifically, being current *is* the
|
|
mitigation, and a pin would have frozen the breakage in place.
|
|
|
|
⚠ **Two wrong diagnoses were published before the right one. Read this before
|
|
theorising about egress again.**
|
|
|
|
1. The failures were first blamed on the ESH egress move (below), on the
|
|
strength of a 2026-09-03 comment claiming NH3's residential address avoided
|
|
CAPTCHAs. Reverting to direct NH3 egress produced a **byte-identical**
|
|
result — same three engines down — which falsified it.
|
|
2. A live `!ddg` bang probe on a freshly restarted container also returned
|
|
CAPTCHA, ruling out a stale circuit-breaker timer as the explanation.
|
|
|
|
Both claims asserted causation from correlation without measuring a baseline.
|
|
The version bump was the only change that moved the number.
|
|
|
|
⚠ **Neither health check can see this class of failure.** `/healthz` cannot,
|
|
and `scripts/searxng-health.sh` prints `✓ searching` whether four engines
|
|
answer or one, because it gates on "did any results come back" and treats
|
|
failed engines as informational — it prints them, then passes anyway. Search
|
|
ran on one engine for a day with every instrument green. **Make the script fail
|
|
on blocked engines before trusting it again.**
|
|
|
|
## The ESH egress experiment (2026-09-17 → reverted 2026-09-18)
|
|
|
|
Search egress was moved to `socks5h://10.0.50.65:1080` on **esh-scale**
|
|
(CT 108 on esh-pve) — an application-level proxy, no host route or exit-node
|
|
changes. Reverted after one day.
|
|
|
|
It did **not** cause the engine outage above, and it did not fix anything
|
|
either: engine behaviour was identical with it and without it. It was reverted
|
|
on its own merits — it bought no measurable improvement while making all fleet
|
|
search depend on ESH WAN and mesh availability, and the simpler configuration
|
|
is the better one.
|
|
|
|
ESH egress measured `128.177.138.182` in use and `154.50.58.126` at cutover;
|
|
ESH's WAN address moves and nothing pins it.
|
|
|
|
The proxy side was built correctly and none of this was its fault: microsocks
|
|
ran as `nobody` under `searxng-egress.service`, bound `10.0.50.65:1080` only,
|
|
and allowed source `10.100.50.40` alone. The canonical unit is kept at
|
|
[`configs/esh-scale/searxng-egress.service`](../../configs/esh-scale/searxng-egress.service)
|
|
in case a future egress experiment wants it; the service is stopped and
|
|
disabled on esh-scale, and tailscaled there was not touched.
|
|
|
|
## Historical placement: NH3 rather than the colo
|
|
|
|
Measured 2026-09-03:
|
|
|
|
ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA
|
|
nh3-docker egress 70.230.226.88 residential -> no gate on that account
|
|
|
|
Search engines gate datacenter ranges. Running the aggregator from the
|
|
residential-egress site was the original approach. The operator subsequently
|
|
requested ESH egress; `outgoing.proxies` now explicitly selects it as above.
|
|
|
|
⚠ It is **not** a complete fix: `brave`, `duckduckgo` and `startpage` still
|
|
CAPTCHA from here. `google cse` carries general search at ~20 results/query, and
|
|
`yandex`, `wiby`, `github`, `stackoverflow` and `marginalia` all work. General
|
|
search is therefore effectively single-engine — if `google cse` breaks, the
|
|
instance goes quiet, which is exactly the failure below.
|
|
|
|
## ⚠ /healthz cannot tell you whether search works
|
|
|
|
On 2026-09-03 the old ana-docker instance was found returning **zero results for
|
|
every query**, for an unknown number of weeks, while:
|
|
|
|
- the container reported `healthy` for 7 straight days, 0 restarts;
|
|
- the Homepage card was green;
|
|
- `/healthz` returned 200 every 30 seconds.
|
|
|
|
It was running **2026.4.17 while current was 2026.9.3** — 4.5 months of engine
|
|
scrapers rotting against sites that had changed their markup. `:latest` means
|
|
"latest at pull time", and nothing re-pulls on its own.
|
|
|
|
Proven by experiment before touching anything: the new image, same settings
|
|
file, same host, same query returned **20 results where the running one returned
|
|
0**.
|
|
|
|
**`scripts/searxng-health.sh` asserts the property, not the mechanism** — three
|
|
unrelated queries must each return results > 0. Run it after any change, and
|
|
periodically; it is the only thing that catches rot.
|
|
|
|
## Update
|
|
|
|
ssh infra-ops@nh3-docker 'cd /opt/docker/compose/searxng && \
|
|
sudo docker compose pull && sudo docker compose up -d'
|
|
scripts/searxng-health.sh
|
|
|
|
## The MCP tool
|
|
|
|
`services/searxng-mcp/` — installed with `uv tool install` and registered at
|
|
**user scope** (`claude mcp add --scope user searxng searxng-mcp`), so every
|
|
Claude Code session gets `web_search` with no per-project setup.
|
|
|
|
⚠ **Zero results raise an error rather than returning an empty list.** An empty
|
|
list is indistinguishable from a broken aggregator, and that ambiguity is what
|
|
let the old instance fail silently for weeks. Same reasoning as althing's
|
|
"an unreachable post office is an OUTAGE, never an empty inbox".
|
|
|
|
⚠ Written against **mcp 2.x**, where `FastMCP` became `MCPServer` and the v1
|
|
`@app.list_tools()` decorator API is gone. v1 examples fail at import with
|
|
`'Server' object has no attribute 'list_tools'`.
|
|
|
|
⚠ `uv tool install --force` alone served a **cached build** and silently
|
|
reinstalled the old code — the installed file still had the v1 API after the
|
|
source no longer did. `--reinstall --no-cache` was required, and the check that
|
|
caught it was `md5sum` of source vs installed.
|
|
|
|
## Secret
|
|
|
|
`SEARXNG_SECRET` lives in `/opt/docker/compose/searxng/.env` (0600, root) on the
|
|
host and is vaulted at `nh3-docker/searxng-secret`. Never in git.
|