vh
156e12619d
Committing work deployed on 2026-09-17 that had been left uncommitted, so canonical intent stops disagreeing with the running host. The deployed /opt/docker/conf/searxng/searxng-settings.yml is byte-identical to the canonical file here, verified before this commit. Search requests and their DNS now exit via socks5h://10.0.50.65:1080 on esh-scale (CT 108), an application-level proxy rather than a host-wide exit node; no route or firewall changes. microsocks runs as nobody under searxng-egress.service, binds only 10.0.50.65:1080, and bypasses SOCKS auth for source 10.100.50.40 alone — every other source must supply a password regenerated at each start and never distributed. Verified active and enabled. There is deliberately no direct-NH3 fallback: an ESH outage must fail the search rather than silently revert egress. ⚠ THE CHANGE HAS NOT ACHIEVED ITS PURPOSE AS DEPLOYED. Two independent live queries, 2026-09-18, both report brave "Suspended: too many requests", duckduckgo "CAPTCHA" and startpage "Suspended: CAPTCHA", leaving google cse as the only answering engine. Moving egress off NH3's residential address is what this change did, and CAPTCHA avoidance was the stated reason searxng sits at NH3 at all. The README anticipated the risk in its Dependency note; it has materialised. Rollback procedure is in the README and the pre-change config is kept on the host as searxng-settings.yml.pre-esh-20260917. Measured egress also drifted from the value recorded at cutover: the README notes 154.50.58.126, the proxy now exits 128.177.138.182. Expected — the README pins no public IP and calls out WAN failover — but recorded here so the number in the doc is not mistaken for current. Also retargets seat-inventory.py's default host from the mesh address 100.64.0.7 to fv-ml1's LAN address 10.251.50.54, routed by the site gateway.
105 lines
5.0 KiB
Markdown
105 lines
5.0 KiB
Markdown
# searxng
|
|
|
|
Privacy-respecting metasearch. **Runs on nh3-docker** (`10.100.50.40:9996`,
|
|
`searxng.nh3.internal`), and is exposed to every Claude Code session on nh3-dev
|
|
as the `web_search` MCP tool.
|
|
|
|
## Current egress: ESH (2026-09-17)
|
|
|
|
Hosting remains on nh3-docker; outbound search requests and their DNS resolution
|
|
use `socks5h://10.0.50.65:1080` on **esh-scale**, the ESH exit-node router
|
|
(CT 108 on esh-pve, `root@10.0.250.35`). This is an application proxy, not a
|
|
host-wide Tailscale exit-node selection. No host/subnet routes were changed.
|
|
The application network layer measured ESH egress **154.50.58.126** at cutover;
|
|
nh3-docker's direct egress remains **70.230.226.88**. ESH's public IP can change
|
|
with WAN failover; no public IP is pinned in this configuration.
|
|
|
|
The Debian `microsocks` package runs as `nobody` under
|
|
`searxng-egress.service`, enabled at boot. The canonical unit is
|
|
[`configs/esh-scale/searxng-egress.service`](../../configs/esh-scale/searxng-egress.service).
|
|
It binds only `10.0.50.65:1080`. Only source `10.100.50.40` bypasses SOCKS
|
|
authentication; other sources require a random, undistributed password regenerated
|
|
at each start. Allowed-host egress and denied-host rejection were both tested.
|
|
No gateway port-forward or firewall/routing changes were made.
|
|
|
|
**Dependency:** ESH WAN/mesh or proxy outages now break outbound search. There is
|
|
no automatic direct-NH3 fallback. `/healthz` alone cannot detect this; run the
|
|
three-query health script below. Engine CAPTCHAs may persist despite changing IP.
|
|
|
|
**Rollback:** on nh3-docker restore
|
|
`/opt/docker/conf/searxng/searxng-settings.yml.pre-esh-20260917` over
|
|
`/opt/docker/conf/searxng/searxng-settings.yml`, then `sudo docker restart searxng`
|
|
and run `scripts/searxng-health.sh`. Remove `outgoing.proxies` in the canonical
|
|
settings too. Once no longer used, disable the dedicated proxy via
|
|
`ssh root@10.0.250.35 'pct exec 108 -- systemctl disable --now searxng-egress.service'`.
|
|
This does not stop Tailscale or alter ESH subnet routing.
|
|
|
|
## Historical placement: NH3 rather than the colo
|
|
|
|
Measured 2026-09-03:
|
|
|
|
ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA
|
|
nh3-docker egress 70.230.226.88 residential -> no gate on that account
|
|
|
|
Search engines gate datacenter ranges. Running the aggregator from the
|
|
residential-egress site was the original approach. The operator subsequently
|
|
requested ESH egress; `outgoing.proxies` now explicitly selects it as above.
|
|
|
|
⚠ It is **not** a complete fix: `brave`, `duckduckgo` and `startpage` still
|
|
CAPTCHA from here. `google cse` carries general search at ~20 results/query, and
|
|
`yandex`, `wiby`, `github`, `stackoverflow` and `marginalia` all work. General
|
|
search is therefore effectively single-engine — if `google cse` breaks, the
|
|
instance goes quiet, which is exactly the failure below.
|
|
|
|
## ⚠ /healthz cannot tell you whether search works
|
|
|
|
On 2026-09-03 the old ana-docker instance was found returning **zero results for
|
|
every query**, for an unknown number of weeks, while:
|
|
|
|
- the container reported `healthy` for 7 straight days, 0 restarts;
|
|
- the Homepage card was green;
|
|
- `/healthz` returned 200 every 30 seconds.
|
|
|
|
It was running **2026.4.17 while current was 2026.9.3** — 4.5 months of engine
|
|
scrapers rotting against sites that had changed their markup. `:latest` means
|
|
"latest at pull time", and nothing re-pulls on its own.
|
|
|
|
Proven by experiment before touching anything: the new image, same settings
|
|
file, same host, same query returned **20 results where the running one returned
|
|
0**.
|
|
|
|
**`scripts/searxng-health.sh` asserts the property, not the mechanism** — three
|
|
unrelated queries must each return results > 0. Run it after any change, and
|
|
periodically; it is the only thing that catches rot.
|
|
|
|
## Update
|
|
|
|
ssh infra-ops@nh3-docker 'cd /opt/docker/compose/searxng && \
|
|
sudo docker compose pull && sudo docker compose up -d'
|
|
scripts/searxng-health.sh
|
|
|
|
## The MCP tool
|
|
|
|
`services/searxng-mcp/` — installed with `uv tool install` and registered at
|
|
**user scope** (`claude mcp add --scope user searxng searxng-mcp`), so every
|
|
Claude Code session gets `web_search` with no per-project setup.
|
|
|
|
⚠ **Zero results raise an error rather than returning an empty list.** An empty
|
|
list is indistinguishable from a broken aggregator, and that ambiguity is what
|
|
let the old instance fail silently for weeks. Same reasoning as althing's
|
|
"an unreachable post office is an OUTAGE, never an empty inbox".
|
|
|
|
⚠ Written against **mcp 2.x**, where `FastMCP` became `MCPServer` and the v1
|
|
`@app.list_tools()` decorator API is gone. v1 examples fail at import with
|
|
`'Server' object has no attribute 'list_tools'`.
|
|
|
|
⚠ `uv tool install --force` alone served a **cached build** and silently
|
|
reinstalled the old code — the installed file still had the v1 API after the
|
|
source no longer did. `--reinstall --no-cache` was required, and the check that
|
|
caught it was `md5sum` of source vs installed.
|
|
|
|
## Secret
|
|
|
|
`SEARXNG_SECRET` lives in `/opt/docker/compose/searxng/.env` (0600, root) on the
|
|
host and is vaulted at `nh3-docker/searxng-secret`. Never in git.
|