memory: snapshot — three silent fleet faults found and fixed in one afternoon
An infrastructure day with no training work, and the through-line is that every fault was invisible to monitoring. NH3↔Anaheim had been crossing a throttled DERP relay rather than a direct path for long enough to carry 78 GB; `.internal` DNS was failing roughly one lookup in ten from two independent causes; SearXNG had exactly one working general web engine. Nothing alarmed on any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget and chose to measure rather than adapt around the problem. Also landed: althing v3.6.3, which makes hyphenated search work for the first time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a capability index autoloaded by Claude, Codex and Grok from one symlinked file; Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's env.sh secrets vaulted. Six detail files. The in-flight section is rewritten and shrinks 142 lines to 64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run outcome, which was left untouched and must not be assumed good. Two foot-guns recorded, both mine: the ESH egress experiment reverted on a diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which has no constructor in that build and crash-looped the container ten times. No archival this run. 165 of 169 dated entries are under the 14-day guard and the remaining four all carry open deferred pointers, so the index stays over the soft cap at 480 lines — an over-cap file that keeps live decisions beats a scannable one that lost one.
This commit is contained in:
@@ -0,0 +1,80 @@
|
||||
# [2026-09-18] althing search returned ZERO for every hyphenated query — silently, for everyone
|
||||
|
||||
Routed in by heid on behalf of peedlar-dev as "a message reached storage but never
|
||||
entered the inbox unread-set or search index". Two symptoms, and **neither was what
|
||||
was reported**.
|
||||
|
||||
## Symptom 1 — "never delivered": NOT A DEFECT
|
||||
|
||||
Message 2571 was fully intact: recipients row present, in the FTS index, `read_at`
|
||||
set. **Zero messages fleet-wide have no recipient row — 0 of 2583.**
|
||||
|
||||
Grouping the reader's `read_at` values gives one group per `postbox read`
|
||||
transaction, and the second consumed **2570 AND 2571 together**. `postbox read`
|
||||
advances over exactly what it printed, so that read printed both.
|
||||
|
||||
⚠ **The evidence was truncated.** Three of four tee'd files sat at exact 4 KiB
|
||||
multiples ending mid-sentence; `read-2-of-3.md` was 8,192 bytes while message 2570's
|
||||
body ALONE is 64,975. It held under 13% of 2570 and never reached 2571. That read
|
||||
emitted ≥127 KB and 8 KB survived. The reasoning — "a read that included 2571 would
|
||||
have printed it into one of these files" — was sound on a premise that failed.
|
||||
|
||||
## Symptom 2 — A REAL DEFECT, and far wider than one message
|
||||
|
||||
**Any query containing a hyphen returned zero hits, silently, for every handle.**
|
||||
|
||||
'nh3-docker' -> OperationalError: no such column: docker
|
||||
'"nh3-docker"' -> 20 rows
|
||||
'tts-dev' -> OperationalError: no such column: dev
|
||||
'"tts-dev"' -> 47 rows
|
||||
|
||||
FTS5 reads the hyphen as a column filter and tries to resolve the trailing token as
|
||||
a column name. `store.py` passed the caller's query to `MATCH` unescaped; `search()`
|
||||
caught the `OperationalError`, ran its probe (which succeeded — the store was
|
||||
healthy), and returned `[]`, indistinguishable from "no results".
|
||||
|
||||
On this fleet that is most proper nouns we own: `nh3-docker`, `irv-ml1`,
|
||||
`esh-docker-vm`, `fv-ml1`, `tts-dev`, `peedlar-dev`.
|
||||
|
||||
⚠ **The discriminator was working as designed.** It correctly separated
|
||||
store-broken from caller-typo. The defect was that it classified an ordinary
|
||||
hyphenated query as a typo.
|
||||
|
||||
## The fix, and the boundary that held
|
||||
|
||||
Routed to forseti (althing's owner) rather than patched — infra-ops owns the
|
||||
**rollout**, not the codebase. forseti implemented on operator instruction and
|
||||
shipped **v3.6.3** (`5de41b7`): valid FTS syntax intact, `OperationalError` retries
|
||||
once as an escaped literal phrase before the health probe. Plus a read-summary line
|
||||
(`consumed N messages: 2570, 2571`) that makes the truncation case detectable.
|
||||
|
||||
⚠ **I twice declined to act on relayed authorization** — "Vuong requested deployment"
|
||||
arriving through a peer is not the operator instructing me, and a post-office bounce
|
||||
takes the bus down for every handle. Got it directly, then executed.
|
||||
|
||||
## Rollout mechanics worth keeping
|
||||
|
||||
- **No CI builds the post-office image.** `deploy.yaml` ships althing-*chamber* to
|
||||
ana-docker; `ci.yaml` runs tests and the frontend. The root `Dockerfile` IS the
|
||||
post office and nothing publishes it — a build-and-push step, not a pull.
|
||||
- Built from `git archive v3.6.3`, **not a working tree**; archive hash verified
|
||||
identical after transfer. My wheel came out byte-identical to forseti's.
|
||||
- Bus outage **19:13:40 → 19:13:57 UTC, 17 seconds**.
|
||||
- Deployed digest `sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4`;
|
||||
rollback pin kept as `compose.yaml.bak-3.6.2`.
|
||||
- ⚠ **Canonical `stacks/althing-post-office/compose.yaml` had to be re-pinned too.**
|
||||
forseti caught it: hand-deploying leaves canonical intent stale, so the next routine
|
||||
`deploy-stack.sh` would have silently ROLLED THE BUS BACK. Commit `670ac9e`.
|
||||
|
||||
## Still open — two postbox fixes proposed to forseti
|
||||
|
||||
1. **`--body -` is taken as a literal hyphen, not stdin.** Four real messages lost to
|
||||
it today (tts-dev ×3, ledger-dev earlier). Store audit: 5 of 2,637 messages have a
|
||||
one-character body and **every accidental one is a literal `-`**; the fifth is a
|
||||
deliberate `'x'` probe. forseti's scope: reserve `--body -` for stdin in both verbs.
|
||||
**Awaiting operator.** ⚠ Empty bodies are already rejected (PRE-306, verified) — no
|
||||
minimum-length rule is needed.
|
||||
2. A bounded consuming read — **withdrawn**, and rightly. A message-count limit does
|
||||
not bound one very large body, so it would not have saved peedlar-dev.
|
||||
|
||||
Until it ships, the mitigation is entirely sender-side: pipe the body, never `--body`.
|
||||
@@ -0,0 +1,94 @@
|
||||
# [2026-09-18] `.internal` DNS was failing ~10% of lookups — two causes, both fleet-wide
|
||||
|
||||
Two independent faults, fixed in order. Together they were costing roughly one
|
||||
`.internal` lookup in ten either a hard failure or a five-second stall, on every
|
||||
DHCP client at every site.
|
||||
|
||||
## Fault 1 — a PUBLIC resolver was the fallback for a PRIVATE zone
|
||||
|
||||
DHCP handed out `10.100.50.40, 1.1.1.1` on both NH3 VLANs. When the internal
|
||||
resolver missed a packet, the resolver waited out its timeout, fell through to
|
||||
Cloudflare, and got an **authoritative NXDOMAIN** — so a transient miss became a
|
||||
definitive "no such host" rather than a retry. Binary failure: instant, or five
|
||||
seconds then `gaierror`.
|
||||
|
||||
**Anaheim was worse.** Its FortiGate handed clients *itself* as resolver and
|
||||
forwarded to `1.1.1.1`/`1.0.0.1`, so **no ANA host could resolve `.internal` at
|
||||
all** — `ana-docker`, which HOSTS the Anaheim AdGuard, had `1.1.1.1` in its own
|
||||
`resolv.conf`.
|
||||
|
||||
⚠ **`/etc/resolv.conf` on these hosts is DHCP-managed** (dhclient on ens18). Hand
|
||||
editing it survives exactly until the next lease renewal — tts-dev's original
|
||||
framing was "fix nh3-dev's resolv.conf", which would have worked, been verified,
|
||||
and then silently reverted. Change it at the source:
|
||||
|
||||
- **UniFi**: `dhcpd_dns_1` / `dhcpd_dns_2` per network object.
|
||||
- **FortiGate**: `config system dhcp server`, `set dns-service specify` +
|
||||
`dns-server1` / `dns-server2`.
|
||||
- Then `sudo dhclient -1 -v ens18` to pick it up **without releasing** the lease.
|
||||
|
||||
**Operator's ruling was broader than the proposal**: each site's backup resolver
|
||||
should be *another site's* resolver. ESH already worked this way; the other two
|
||||
were set, and the ring closes.
|
||||
|
||||
ESH primary 10.0.50.45 backup 10.100.50.40 (NH3) [pre-existing]
|
||||
NH3 primary 10.100.50.40 backup 10.250.50.70 (ANA) [changed]
|
||||
ANA primary 10.250.50.70 backup 10.0.50.45 (ESH) [changed]
|
||||
|
||||
All six members verified to answer the private zone. ⚠ The SureFire tenant DHCP
|
||||
server (`sfsrv`) and the fortilink switch-management server were deliberately left
|
||||
alone — tenant property.
|
||||
|
||||
**Result: hard failures 3/40 → 0/40. The 5 s stalls remained.**
|
||||
|
||||
## Fault 2 — THE ROOT CAUSE: AdGuard rate-limiting, identical on all three
|
||||
|
||||
ratelimit: 20
|
||||
ratelimit_subnet_len_ipv4: 24
|
||||
ratelimit_whitelist: []
|
||||
|
||||
**20 queries per second shared across an entire /24 — per subnet, not per client.**
|
||||
Every host on a VLAN draws from one bucket, so one busy container starves the rest,
|
||||
and anything over the line is **silently dropped**, costing the client its full 5 s
|
||||
resolver timeout.
|
||||
|
||||
Demonstrated rather than inferred — 60 concurrent queries at one resolver:
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| median | 5044 ms | 72 ms |
|
||||
| timed out (>4 s) | **40 of 60** | **0 of 60** |
|
||||
|
||||
Set to `ratelimit: 0` on all three. These are LAN-only resolvers behind three
|
||||
firewalls; that setting exists to blunt internet-side DNS amplification, which is
|
||||
not this.
|
||||
|
||||
⚠ **Restarting AdGuard is required for a config change and briefly drops that site's
|
||||
primary resolver — do the three ONE AT A TIME** so the cross-site ring always has a
|
||||
live member. That the ring made this rollout safe is the point of having built it
|
||||
first, an hour earlier.
|
||||
|
||||
## End-to-end, the original symptom
|
||||
|
||||
40 TCP connects to a service by name vs by IP; the by-IP arm is the control.
|
||||
|
||||
| | original | after ring only | after ratelimit 0 |
|
||||
|---|---|---|---|
|
||||
| by NAME p90 | 120 ms | 5017 ms | **13.5 ms** |
|
||||
| by NAME >1 s | 3/40 | 5/40 | **0/40** |
|
||||
| by NAME failed | 2/40 | 0/40 | **0/40** |
|
||||
| by IP p90 (control) | 25 ms | — | 11.7 ms |
|
||||
|
||||
tts-dev confirmed from talk's own request path: name and IP **1.5 ms apart at the
|
||||
median**. Resolution has stopped being a cost rather than become a smaller one.
|
||||
|
||||
## What this says about monitoring
|
||||
|
||||
Nothing on this fleet watches DNS success rate. The fleet's actual DNS monitor
|
||||
today was a voice loop with a 1545 ms budget whose owner chose to measure instead
|
||||
of adapting around the problem. Both faults were invisible to Beszel and Uptime
|
||||
Kuma — every host was up, every service answered.
|
||||
|
||||
Related: [[2026-09-18-nh3-ana-derp-relay]] — the ANA AdGuard is only a sensible
|
||||
cross-site backup because that fix took NH3→ANA from ~1.2 s to ~15 ms the same
|
||||
afternoon. The two changes compose.
|
||||
@@ -0,0 +1,57 @@
|
||||
# [2026-09-18] FleetTools — one capability index, autoloaded by three agent families
|
||||
|
||||
Every agent on nh3-dev needs the same answers — what runs here, how do I call it,
|
||||
what will bite me — but that knowledge lived in `~/.claude/CLAUDE.md`, which only
|
||||
Claude sessions load, interleaved with operator preferences other families have no
|
||||
use for.
|
||||
|
||||
## Shape
|
||||
|
||||
**Two-tier, matching the persistent-memory split.** `FLEETTOOLS.md` is a ~135-line
|
||||
index an agent reads whole; each entry links to a detail file it opens only when it
|
||||
needs that tool. Reading the index costs about a fifth of reading the tree.
|
||||
|
||||
- Canonical: `docs/fleettools/` in this repo (git-tracked)
|
||||
- Discovery: `~/FLEETTOOLS.md` → symlink to the index
|
||||
- **Detail links are ABSOLUTE paths**, not relative markdown — a non-Claude agent
|
||||
`cat`s the path rather than following a link, and may be anywhere on the filesystem.
|
||||
|
||||
Covers: althing, the Booth, `secret`, LiteLLM, inference seats + Asset Engine,
|
||||
speech, Arbo, elway, fleet SSH, observability, graphify, Playwright, the Henge, and
|
||||
a segregated Claude-only file (ratecheck, remote-ssh MCP, task-board, skills).
|
||||
|
||||
## Autoload — verified per family, not assumed
|
||||
|
||||
- **Codex** reads `$CODEX_HOME/AGENTS.md` — the binary carries the string
|
||||
`"Failed to read global AGENTS.md instructions from"`.
|
||||
- **Grok** always scans `$GROK_HOME/rules/` and loads **every `*.md` in it regardless
|
||||
of name** — from its own embedded docs.
|
||||
|
||||
Both locations were empty. `AGENT-BOOTSTRAP.md` is **symlinked** into each rather than
|
||||
copied — one file, three families, no drift surface.
|
||||
|
||||
⚠ Grok also scans `~/.claude/rules/` and recognises `CLAUDE.md` via harness-compat
|
||||
settings. That directory is empty today, but Grok can see Claude's config.
|
||||
|
||||
## Rule zero, which is the load-bearing part
|
||||
|
||||
**Query live inventories, never a written list.** The index points at
|
||||
Homepage `/api/services`, asset-engine `/api/v1/services`, LiteLLM `/v1/models`, and
|
||||
every FastAPI seat's `/openapi.json`. A copied service table would be stale within a
|
||||
month — the LiteLLM roster was already at 40 models where the old CLAUDE.md note said
|
||||
~30.
|
||||
|
||||
## The CLAUDE.md swap
|
||||
|
||||
`~/.claude/CLAUDE.md`'s "Global tools available" section went from **231 lines to 39**
|
||||
(file 1218 → 1026, a 16% cut to what every Claude session loads). Three things stayed
|
||||
inline deliberately because they govern BEHAVIOUR rather than lookup: run `ratecheck`
|
||||
before expensive work, post operator-facing links to the Booth board, and the vault is
|
||||
the credential source of truth. **A rule behind a file read is a rule that stops
|
||||
firing.** Backup at `~/.claude/CLAUDE.md.bak-20260918-074048`.
|
||||
|
||||
The shared all-agents LiteLLM key stays inline there too — every session needs it and a
|
||||
vault round-trip measured over two minutes — but it is now ALSO vaulted at
|
||||
`litellm/all-agents-shared-key`, since it had been single-copy.
|
||||
|
||||
Commits `53c3e80` `21d24c5`.
|
||||
@@ -0,0 +1,91 @@
|
||||
# [2026-09-18] NH3↔Anaheim was DERP-relayed, not direct — a UDP 41641 port-forward on ana-gw fixed it
|
||||
|
||||
Every NH3→Anaheim flow had been crossing Tailscale's LA DERP relay rather than a
|
||||
direct path, long enough to have carried **78 GB tx on the NH3 side alone**. DERP
|
||||
is a deliberately throttled fallback — a reachability mechanism, not a data plane —
|
||||
so this imposed both a fixed round-trip penalty and a bandwidth ceiling on LiteLLM,
|
||||
Beszel, task-board, vor and the Henge alike.
|
||||
|
||||
## How it surfaced, and how it did NOT
|
||||
|
||||
tts-dev reported ext-stt transcriptions 4–18× slower through the LiteLLM gateway
|
||||
than straight at the Parakeet host, with a ~1.9 s fixed floor plus a per-byte cost.
|
||||
They had already cleared GPU contention, the ASR itself and output quality with
|
||||
real controls.
|
||||
|
||||
**No monitor caught it.** Beszel had every host green. Nothing on this fleet checks
|
||||
whether a mesh path is direct.
|
||||
|
||||
## The diagnosis, and two wrong causes published on the way
|
||||
|
||||
Their hypothesis was a dead first target in a LiteLLM fallback list. Checked: `ext-stt`
|
||||
has exactly ONE deployment, no fallback, nothing to retry against. Langfuse looked like
|
||||
a great candidate for a fixed floor — env vars still set, container gone — but it was
|
||||
retired from the callback list 2026-06-20 and nothing calls it per-request.
|
||||
|
||||
The decisive arm was one neither of us had run: **measure from ana-docker itself**, so
|
||||
both arms share the identical onward path to FV.
|
||||
|
||||
ana-docker -> fv-ml1 DIRECT 0.289 / 0.218 / 0.273 / 0.220 s
|
||||
ana-docker -> gateway -> fv-ml1 0.463 / 0.211 / 0.222 / 0.222 s
|
||||
|
||||
The gateway adds ~0. tts-dev measured 2618 ms for that clip; from the gateway's own
|
||||
host it is 220 ms. Their two arms differed in TWO things — whether LiteLLM was in the
|
||||
path AND which destination the packets crossed the network to reach — and the entire
|
||||
delta was attributed to the variable of interest.
|
||||
|
||||
nh3-dev -> fv-ml1 direct 0.269 / 0.219 / 0.216 s
|
||||
nh3-dev -> ANA gateway 1.449 / 1.399 / 1.404 s
|
||||
|
||||
nh3-scale$ tailscale ping 100.64.0.3
|
||||
pong via DERP(lax) in 507ms, 518, 521, 522, 373, 423, 464, 493
|
||||
direct connection not established
|
||||
|
||||
## Why hole-punching failed
|
||||
|
||||
ana-scale advertised `38.120.12.42:41641`, but its netcheck mapped to `:60798` — the
|
||||
Anaheim NAT was not preserving the port — and `PortMapping` was empty, so no UPnP or
|
||||
NAT-PMP was establishing one. `MappingVariesByDestIP: false`, so not symmetric NAT;
|
||||
simply no reachable inbound endpoint. ESH↔Anaheim and NH3↔ESH were already direct;
|
||||
only this pair failed.
|
||||
|
||||
## The fix — four ADDITIVE objects on ana-gw (10.250.0.1)
|
||||
|
||||
⚠ **This box runs `central-nat enable`, so a policy's `dstaddr` is the REAL internal
|
||||
address and not the VIP.** The existing `wg-to-ana-wg` VIP+policy pair is the house
|
||||
template; writing what a FortiGate normally wants would have failed.
|
||||
|
||||
| object | value |
|
||||
|---|---|
|
||||
| `firewall address` | `ana-scale-ip` → 10.250.50.45/32, iface `servers` |
|
||||
| `firewall service custom` | `Tailscale-41641` → `udp-portrange 41641` |
|
||||
| `firewall vip` | `tailscale-to-ana-scale` → 38.120.12.42:41641 udp → 10.250.50.45:41641, extintf wan1 |
|
||||
| `firewall policy` id 75 | wan1→servers, all→`ana-scale-ip`, `Tailscale-41641`, accept |
|
||||
|
||||
## Measured before → after
|
||||
|
||||
| path | before | after |
|
||||
|---|---|---|
|
||||
| `tailscale ping` nh3-scale→ana-scale | 373–522 ms via DERP(lax) | **6 ms direct** |
|
||||
| STT via the ANA gateway (96 kB clip) | 1.399–1.449 s | **0.237–0.270 s** |
|
||||
| Beszel HTTP nh3-dev→ana-docker | 0.94–1.29 s | **0.014–0.016 s** |
|
||||
|
||||
tts-dev confirmed independently from talk's own vantage: 2618 → 249 ms and 4119 →
|
||||
548 ms, converged with their direct arm.
|
||||
|
||||
## Operational notes for next time
|
||||
|
||||
⚠ **This edge has NO out-of-band access.** Back up first — `ssh infra-ops@10.250.0.1
|
||||
'show' > file` captures the full non-default config (11,320 lines) without needing a
|
||||
tftp server; kept at `~/backups/ana-gw-config-BEFORE-<stamp>.txt`. Credentials vaulted
|
||||
at `fortigate/ana-gw-infra-ops-password`, reached with
|
||||
`sshpass -e ssh -o PubkeyAuthentication=no`. **Additive objects only** — never edit an
|
||||
existing policy or interface over a 327 ms link you cannot recover.
|
||||
|
||||
⚠ **irv-ml1 (100.64.0.6) remains `relay "lax"`.** Same class, different site, not fixed
|
||||
by this. Measured 24-25 ms on HTTP from nh3-dev, so it is not costing what Anaheim was.
|
||||
I warned tts-dev their TTS stack was paying for it; they measured and I was wrong.
|
||||
|
||||
Documented in `docs/pfi/headscale-mesh-plan.md`; commit `5a9fad8`. See
|
||||
[[2026-09-18-fleet-dns-ring-and-ratelimit]] — the ANA AdGuard only became a viable
|
||||
cross-site DNS backup because of this fix.
|
||||
@@ -0,0 +1,47 @@
|
||||
# [2026-09-18] SearXNG's ESH SOCKS5 egress — one day live, reverted, and it fixed nothing
|
||||
|
||||
On 2026-09-17, at operator request, SearXNG's search egress was moved from
|
||||
nh3-docker's direct residential path to `socks5h://10.0.50.65:1080` on **esh-scale**
|
||||
(CT 108 on esh-pve) — an application-level proxy, no host route or exit-node change.
|
||||
Reverted 2026-09-18.
|
||||
|
||||
## Why it was reverted, and why the STATED reason was wrong
|
||||
|
||||
It was reverted on a diagnosis that turned out to be false: that the move had cost
|
||||
three of four search engines to CAPTCHAs. Reverting to direct NH3 egress produced a
|
||||
**byte-identical** result — same three engines down — which falsified it. A live
|
||||
`!ddg` bang probe on a freshly restarted container also CAPTCHA'd, ruling out stale
|
||||
suspension timers too.
|
||||
|
||||
The real cause was elsewhere (see [[2026-09-18-searxng-one-engine-to-seven]]).
|
||||
|
||||
**The revert still stands on its own merits**: ESH egress bought no measurable
|
||||
improvement while making all fleet search depend on ESH WAN and mesh availability.
|
||||
The simpler configuration is the better one. It is simply not the fix for the engines.
|
||||
|
||||
## What was built, because it was built correctly
|
||||
|
||||
The proxy side was not at fault and is worth keeping as a pattern. `microsocks` ran
|
||||
as `nobody` under `searxng-egress.service`, bound `10.0.50.65:1080` only, and allowed
|
||||
source `10.100.50.40` alone — everything else had to supply a password regenerated at
|
||||
each start and never distributed. Allowed-host egress and denied-host rejection were
|
||||
both tested. The unit is kept at `configs/esh-scale/searxng-egress.service`; the
|
||||
service is **stopped and disabled** on esh-scale, and `tailscaled` there was not
|
||||
touched (CT 108 is ESH's whole-site mesh SPOF).
|
||||
|
||||
Measured egress was `128.177.138.182` in use and `154.50.58.126` at cutover — ESH's
|
||||
WAN address moves and nothing pins it, so the number in the stack README is historical.
|
||||
|
||||
## The transferable lesson
|
||||
|
||||
⚠ **I asserted causation from correlation with no baseline.** The only evidence that
|
||||
residential egress avoided CAPTCHAs was a config comment dated 2026-09-03, which was
|
||||
no longer true of that address. Five samples of the post-change state and zero of the
|
||||
working state is not a comparison. The rollback WAS the counterfactual, and it
|
||||
falsified the claim I had already published in a commit message.
|
||||
|
||||
This was the first of three wrong causal attributions in a single afternoon. The
|
||||
common shape: measure after a change, attribute the delta to *my* change, never check
|
||||
what else moved.
|
||||
|
||||
Commits `156e126` (applied), `1a35181` (reverted).
|
||||
@@ -0,0 +1,81 @@
|
||||
# [2026-09-18] SearXNG had ONE working general web engine — now seven, and the health script never noticed
|
||||
|
||||
The fleet `web_search` MCP tool every nh3-dev session uses was answering from
|
||||
`google cse` alone, and had been for at least a day, with every instrument green.
|
||||
|
||||
## The root cause, which is subtler than "engines were disabled"
|
||||
|
||||
Of **55 general-category engines, 7 were enabled by default** — and six of those
|
||||
seven are dictionary, translation, currency and encyclopedia engines that return
|
||||
nothing for an ordinary web query. `google cse` was the only general WEB engine on
|
||||
the instance. When its quota exhausted, a query returned **HTTP 200 with an empty
|
||||
`results` array**.
|
||||
|
||||
⚠ **The distinction that hid it: `inactive: false` only makes an engine
|
||||
SELECTABLE; `disabled: false` is what puts it in the DEFAULT set.** The other 48
|
||||
were selectable-but-off, which an API client has no way to change.
|
||||
|
||||
## Three wrong causes published before the right one
|
||||
|
||||
Recorded because the pattern repeated all day and each was reasoned, not careless.
|
||||
|
||||
1. **"The ESH egress move broke it."** Search egress had been moved on 09-17 to a
|
||||
SOCKS5 proxy on esh-scale, and a 2026-09-03 comment claimed NH3's residential
|
||||
address avoided CAPTCHAs. Reverting to direct NH3 egress produced a
|
||||
**byte-identical** failure. Falsified.
|
||||
2. **"Stale suspension timers."** A live `!ddg` bang probe on a freshly restarted
|
||||
container also returned CAPTCHA. Falsified.
|
||||
3. **"The version bump fixed it."** Updating `2026.9.3` → `2026.9.18` appeared to
|
||||
restore all four engines — but **another session had removed the captcha'd
|
||||
scrapers from the engine set four minutes before that measurement**, and I did
|
||||
not check what else had moved. Two changes, one measurement, attributed to mine.
|
||||
|
||||
Every one asserted causation from correlation with **no baseline measurement of the
|
||||
working state**.
|
||||
|
||||
## The fix
|
||||
|
||||
Five keyless engines added to the default set, each bang-probed on this egress
|
||||
first rather than trusted from a list:
|
||||
|
||||
duckduckgo web 10 mojeek Suspended excluded
|
||||
bing 10 qwant CAPTCHA excluded
|
||||
yep 20 startpage Suspended excluded
|
||||
yahoo 7 brave Suspended excluded
|
||||
wiby 12 seznam timeout excluded
|
||||
|
||||
Plus `braveapi` with a real key. **General web engines: 1 → 7.**
|
||||
|
||||
Verified under the live failure condition rather than a simulated one — `google cse`
|
||||
was STILL quota-suspended and three consecutive queries returned 38–41 results from
|
||||
4–5 engines each. Later with braveapi: 55–63 results from six.
|
||||
|
||||
## Two durable rules this settles
|
||||
|
||||
⭐ **This stack tracks `:latest` ON PURPOSE — do not "improve" it by pinning.**
|
||||
Upstream ships engine-handler fixes continuously because providers change bot
|
||||
gating continuously; being current IS the mitigation, and a digest pin would have
|
||||
frozen the breakage in place. (The althing post office is pinned for the opposite
|
||||
reason — it is the message bus and must not move underfoot.)
|
||||
|
||||
⭐ **The Brave API key is committed in plaintext by explicit operator decision**
|
||||
(2026-09-18): *"if the service is useless, so is the key"* — free tier, rate-limited
|
||||
service, not worth the machinery of keeping it out of git. It is also vaulted at
|
||||
`nh3-docker/searxng-brave-api-key`. **This is scoped to one low-value credential and
|
||||
is not a change to the no-secrets-in-git rule.** It cannot be un-committed; rotation
|
||||
means a new key at Brave, never a history rewrite, because the repo is shared.
|
||||
|
||||
⚠ There is **no env-var path into `searxng-settings.yml`** in this build. The loader
|
||||
reads only `SEARXNG_SETTINGS_PATH`; the entrypoint substitutes only `ultrasecretkey`.
|
||||
`SEARXNG_BRAVE_API_KEY` reaches the container and is never read — it is kept in the
|
||||
compose, unused, with a comment saying so.
|
||||
|
||||
## ⚠ Still open: the health script cannot see any of this
|
||||
|
||||
`scripts/searxng-health.sh` printed `✓ searching` with six engines answering AND
|
||||
with one. It gates on "did any results come back" and prints failed engines as
|
||||
informational. Search ran on a single engine for a day with every instrument green.
|
||||
**Fix it to fail on blocked engines before trusting it again.**
|
||||
|
||||
Commits `9a428fd` `274d3e2` `9219942` `5b20b02`. See
|
||||
[[2026-09-18-searxng-esh-egress-reverted]].
|
||||
Reference in New Issue
Block a user