From 274d3e24434710b8463ef4c89834686ac06d72ac Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 18 Sep 2026 13:02:43 -0700 Subject: [PATCH] fix(searxng): six general web engines by default, not one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause of the silent-empty-results failure peedlar-dev reported. Of 55 general-category engines, only seven were enabled-by-default, and six of those are dictionary, translation, currency or encyclopedia engines that return nothing for an ordinary web query. `google cse` was the instance's ONLY general web engine, so a single quota exhaustion produced HTTP 200 with an empty results array and no error, for every consumer on the fleet. The distinction that matters: `inactive: false` only makes an engine selectable, `disabled: false` puts it in the default set. The other 48 were selectable-but-off, which an API client has no way to change. Enables five keyless engines, each bang-probed first and returning real results with no API key: duckduckgo web 10, bing 10, yep 20, yahoo 7, wiby 12. General web engines go 1 -> 6. Deliberately excluded: mojeek, qwant, startpage and the brave scraper, all of which CAPTCHA or rate-limit this egress, and seznam, which times out. Verified under the live failure condition rather than a simulated one. google cse is still quota-suspended right now, and three consecutive queries returned 38-41 results from 4-5 engines each. The single point of failure is gone while the failing engine is still failing. Also adopts the concurrent v4 settings work from the other session — marginalia on its public key, and the captcha'd-scraper removals — plus the fix for the crash-loop that work introduced: this build has no !ENV YAML constructor, so `api_key: !ENV SEARXNG_BRAVE_API_KEY` made the file unparseable and the container restarted ten times with search down fleet-wide. That block stays commented; the vaulted Brave key is valid but has no supported path into the settings file, which is a separate decision. --- stacks/searxng/conf/searxng-settings.yml | 50 +++++++++++++++++++++++- 1 file changed, 49 insertions(+), 1 deletion(-) diff --git a/stacks/searxng/conf/searxng-settings.yml b/stacks/searxng/conf/searxng-settings.yml index 04fd972..9bc1b9c 100644 --- a/stacks/searxng/conf/searxng-settings.yml +++ b/stacks/searxng/conf/searxng-settings.yml @@ -26,7 +26,7 @@ use_default_settings: # The name is "karmasearch videos". - karmasearch - karmasearch videos - # Tier A (v3, 2026-09-18): scraped engine families that CAPTCHA/challenge + # Tier A (v4, 2026-09-18): scraped engine families that CAPTCHA/challenge # this egress IP. Measured 3/3 queries: duckduckgo -> CAPTCHA, startpage # -> Suspended: CAPTCHA, brave -> Suspended: too many requests. The whole # family goes: images/videos/news siblings share the same scraper and @@ -53,6 +53,54 @@ use_default_settings: # browser UA). Re-adding breadth now needs API keys, not scraping: braveapi # (free Brave Search API tier) and/or marginalia (free key) — both already # present in the defaults, awaiting an api_key. +# +# v4 (2026-09-18): braveapi keyed. The key lives in the container env +# (compose .env, vault: nh3-docker/searxng-brave-api-key) and enters this file +# via SearXNG's !ENV tag — no secret in a git-tracked file. If the key line +# ever resolves empty, the engine silently drops to inactive: no error, just +# missing results. Check `unresponsive_engines`/`/config` if Brave vanishes. +# +# marginalia uses the public key (rate-limited but keyless-tier, safe to +# hardcode). A GLM web-search engine is possible as a custom JSON engine but +# needs a z.ai/bigmodel search key — not in the vault as of today. +engines: + # ⚠ infra-ops 2026-09-18 12:56: the block below was commented out to stop a + # crash-loop. This searxng build has no !ENV YAML constructor, so the file + # failed to parse and the container restarted 10 times with search down + # fleet-wide. Original preserved at searxng-settings.yml.inflight-20260918-1252. + # Inline the key or load it another way; !ENV is not supported here. + # - name: braveapi + # api_key: !ENV SEARXNG_BRAVE_API_KEY + # inactive: false + - name: marginalia + api_key: public + disabled: false + inactive: false + + # ── Breadth: keyless engines that are ON by default ─────────────────── + # ⚠ THE BUG THIS FIXES. Of 55 general-category engines, only SEVEN were + # enabled-by-default, and six of those are dictionary/translation/currency/ + # encyclopedia engines that return nothing for an ordinary web query. That + # left `google cse` as the instance's ONLY general web engine — so one quota + # exhaustion returned HTTP 200 with an empty results array and no error, for + # every consumer. `disabled: false` is what puts an engine in the default + # set; `inactive: false` only makes it selectable. The other 48 are + # selectable-but-off, which an API client cannot change. + # + # Each of the five below was bang-probed on 2026-09-18 and returned real + # results with no key: ddgw 10, bing 10, yep 20, yahoo 7, wiby 12. + # Deliberately NOT added: mojeek, qwant, startpage, brave (scrapers — all + # CAPTCHA or rate-limit this egress) and seznam (timeout). + - name: duckduckgo web + disabled: false + - name: bing + disabled: false + - name: yep + disabled: false + - name: yahoo + disabled: false + - name: wiby + disabled: false general: instance_name: "SearXNG"