vh
0f748ea54e
The ana-docker instance was returning zero results for every query while reporting healthy — 4.5 months stale (2026.4.17 against a current 2026.9.3), its engine scrapers rotted against sites that had changed. /healthz proves the web app answers and says nothing about whether search works, so seven days of green sat on top of a search box that found nothing. Moved to nh3-docker rather than updated in place, because the colo egress is the other half of the problem: 38.120.12.42 is a datacenter address that DuckDuckGo and Startpage CAPTCHA, while nh3-docker egresses residentially at 70.230.226.88. Same reasoning as the fleet's residential proxy for yt-dlp, applied at the source instead of around it. Config corrected along the way: base_url said searxng.pfi.local, a name retired on 2026-08-19, while the environment said something else — the env won so nothing broke and the file quietly lied. The karmasearch.videos removal key never matched, because the engine's real name has a space. scripts/searxng-health.sh asserts results > 0 across three unrelated queries. That is the check that would have caught this, and the only kind that can: the mechanism was healthy throughout. services/searxng-mcp exposes it as `web_search` at user scope, so every Claude Code session has it. Zero results raise rather than returning an empty list — an empty list is indistinguishable from a broken aggregator, which is precisely how this hid. Old instance stopped and removed; DNS alias repointed to searxng.nh3.internal.
64 lines
2.9 KiB
Bash
Executable File
64 lines
2.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Is SearXNG actually searching?
|
|
#
|
|
# scripts/searxng-health.sh check the fleet instance
|
|
# scripts/searxng-health.sh --url http://host:9996
|
|
#
|
|
# ⚠ THIS EXISTS BECAUSE /healthz CANNOT ANSWER THE QUESTION. On 2026-09-03 the
|
|
# old ana-docker instance was found returning ZERO results for every query, for
|
|
# an unknown number of weeks, while its container reported `healthy` for 7 days
|
|
# straight and its dashboard card was green. /healthz proves the web app
|
|
# answers; it says nothing about whether a single engine works.
|
|
#
|
|
# SearXNG rots quietly: engine scrapers break as upstream sites change markup,
|
|
# and the project ships near-daily releases to keep up. An instance pinned to
|
|
# `:latest` that nobody re-pulls is frozen at whatever `latest` meant on the day
|
|
# it was created — that one was 4.5 months behind.
|
|
#
|
|
# So this asserts the property, not the mechanism: RESULTS > 0.
|
|
set -euo pipefail
|
|
|
|
URL="http://10.100.50.40:9996"
|
|
[[ "${1:-}" == "--url" ]] && { URL="${2:?--url needs a value}"; shift 2; }
|
|
|
|
fail=0
|
|
say() { printf '%s\n' "$*"; }
|
|
|
|
say "── ${URL}"
|
|
|
|
# 1. reachable at all? An unreachable instance is an OUTAGE, not "no results".
|
|
if ! ver=$(curl -s -m 15 "$URL/config" | python3 -c 'import sys,json;print(json.load(sys.stdin)["version"])' 2>/dev/null); then
|
|
say " ✗ unreachable — this is an outage, not an empty index"
|
|
exit 1
|
|
fi
|
|
say " version: $ver"
|
|
|
|
# 2. is that version current? `:latest` is only latest at pull time.
|
|
if latest=$(curl -s -m 20 "https://hub.docker.com/v2/repositories/searxng/searxng/tags/?page_size=1&ordering=last_updated" \
|
|
| python3 -c 'import sys,json;print(json.load(sys.stdin)["results"][0]["last_updated"][:10])' 2>/dev/null); then
|
|
say " registry :latest last pushed: $latest (running build predates any later push)"
|
|
fi
|
|
|
|
# 3. THE CHECK THAT MATTERS. Three unrelated queries, because one query
|
|
# returning nothing can legitimately mean nothing matched; three cannot.
|
|
for q in "proxmox backup" "python asyncio" "linux kernel"; do
|
|
out=$(curl -s -m 45 --get --data-urlencode "q=$q" --data "format=json" "$URL/search" 2>/dev/null) || out=""
|
|
n=$(printf '%s' "$out" | python3 -c 'import sys,json;print(len(json.load(sys.stdin).get("results") or []))' 2>/dev/null || echo 0)
|
|
errs=$(printf '%s' "$out" | python3 -c 'import sys,json;print(",".join(e[0] for e in (json.load(sys.stdin).get("unresponsive_engines") or [])) or "-")' 2>/dev/null || echo "?")
|
|
if [[ "$n" -gt 0 ]]; then
|
|
say " ✓ '$q' -> $n results (failed engines: $errs)"
|
|
else
|
|
say " ✗ '$q' -> ZERO results (failed engines: $errs)"
|
|
fail=1
|
|
fi
|
|
done
|
|
|
|
if (( fail )); then
|
|
say ""
|
|
say " ✗ SearXNG answers HTTP but finds nothing. Almost always staleness:"
|
|
say " ssh infra-ops@nh3-docker 'cd /opt/docker/compose/searxng && \\"
|
|
say " sudo docker compose pull && sudo docker compose up -d'"
|
|
exit 1
|
|
fi
|
|
say " ✓ searching"
|