memory: snapshot — searxng move, gx10 rack networking, mac fleet provisioning

2026-09-03 22:22:11 -07:00
parent a95717e810
commit 43135a2768
7 changed files with 361 additions and 197 deletions
@@ -0,0 +1,67 @@
+- `[2026-09-03]` **SearXNG was returning ZERO results for every query while reporting `healthy` — moved to nh3-docker, updated, and exposed to every CC session as an MCP tool.**
+
+**The failure.** The ana-docker instance answered `/healthz` every 30s, showed
+`Up 7 days (healthy)` with 0 restarts, and had a green Homepage card — while
+returning **0 results for every query tested**. It was running **2026.4.17
+against a current 2026.9.3**: 4.5 months of engine scrapers rotting against
+sites that had changed their markup. SearXNG ships near-daily releases for
+exactly this reason.
+
+⚠ **`:latest` means "latest AT PULL TIME".** Nothing re-pulls on its own. A
+container created in April on `:latest` is pinned to April forever.
+
+⚠ **`/healthz` proves the web app answers and says NOTHING about whether search
+works.** That is the whole lesson. Same shape as the nh3-dev "failing disk" that
+was a stalled backup, the runner audit that trusted liveness for identity, and
+the statusline bell that measured a mechanism.
+
+**Proven before acting**: the new image, same settings file, same host, same
+query, in a throwaway container → **20 results where the running one returned
+0**. Network was ruled out first — from inside the container DNS resolved and
+mojeek/wikipedia were reachable, so engines were reachable and the parsers were
+the broken part.
+
+**Why NH3 and not an in-place update** (operator's call, and the measurement
+backs it):
+
+    ana-docker egress   38.120.12.42   datacenter    -> DuckDuckGo/Startpage CAPTCHA
+    nh3-docker egress   70.230.226.88  residential   -> not gated the same way
+
+Search engines gate datacenter ranges. Same reason the fleet keeps a residential
+SOCKS5 proxy on nh3-dev for yt-dlp — applied at the source instead of proxied
+around. `outgoing.proxies` has the fallback commented in place if NH3's egress
+ever changes.
+
+⚠ **Not a complete fix.** `brave`, `duckduckgo`, `startpage` still CAPTCHA from
+NH3. `google cse` carries general search at ~20 results/query; `yandex`, `wiby`,
+`github`, `stackoverflow`, `marginalia` work. **General search is effectively
+single-engine** — if google cse breaks, it goes quiet again.
+
+**Two config defects, both silent:** `base_url` still named
+`searxng.pfi.local`, retired 2026-08-19, while the env said otherwise (env wins,
+so nothing broke and the file lied to every reader); and the
+`karmasearch.videos` removal key never matched because the engine's real name
+has a space in it.
+
+**`scripts/searxng-health.sh` asserts results > 0** across three unrelated
+queries. That is the only check that could have caught this — the mechanism was
+healthy throughout.
+
+**The MCP tool** — `services/searxng-mcp`, `uv tool install`, registered
+`claude mcp add --scope user searxng searxng-mcp`, so every CC session gets
+`web_search`. ⚠ Zero results **raise** rather than returning an empty list: an
+empty list is indistinguishable from a broken aggregator, which is precisely how
+this hid. Same principle as althing's "unreachable post office is an OUTAGE,
+never an empty inbox".
+
+⚠ Written against **mcp 2.x** (`FastMCP` → `MCPServer`; the v1
+`@app.list_tools()` decorator is gone and fails at import). ⚠ **`uv tool install
+--force` served a CACHED build** and silently reinstalled the old code — the
+installed file still had the v1 API after the source no longer did.
+`--reinstall --no-cache` fixed it; `md5sum` of source vs installed is what
+caught it.
+
+Old instance stopped and removed; DNS alias repointed to
+`searxng.nh3.internal` → 10.100.50.40. Secret vaulted at
+`nh3-docker/searxng-secret`. Commit `0f748ea`. See [[2026-09-03-gx10-rack-network]]
+for the other UniFi-side change the same day.