From 148a5a34da6c4c0c8c9f4db3284b2c512a842584 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 18 Sep 2026 20:09:01 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20three=20silent?= =?UTF-8?q?=20fleet=20faults=20found=20and=20fixed=20in=20one=20afternoon?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An infrastructure day with no training work, and the through-line is that every fault was invisible to monitoring. NH3↔Anaheim had been crossing a throttled DERP relay rather than a direct path for long enough to carry 78 GB; `.internal` DNS was failing roughly one lookup in ten from two independent causes; SearXNG had exactly one working general web engine. Nothing alarmed on any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget and chose to measure rather than adapt around the problem. Also landed: althing v3.6.3, which makes hyphenated search work for the first time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a capability index autoloaded by Claude, Codex and Grok from one symlinked file; Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's env.sh secrets vaulted. Six detail files. The in-flight section is rewritten and shrinks 142 lines to 64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run outcome, which was left untouched and must not be assumed good. Two foot-guns recorded, both mine: the ESH egress experiment reverted on a diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which has no constructor in that build and crash-looped the container ten times. No archival this run. 165 of 169 dated entries are under the 14-day guard and the remaining four all carry open deferred pointers, so the index stays over the soft cap at 480 lines — an over-cap file that keeps live decisions beats a scannable one that lost one. --- graphify-out/GRAPH_REPORT.md | 185 ++++++++--------- .../2026-09-18-althing-363-hyphen-search.md | 80 ++++++++ ...2026-09-18-fleet-dns-ring-and-ratelimit.md | 94 +++++++++ .../2026-09-18-fleettools-agent-index.md | 57 ++++++ .../2026-09-18-nh3-ana-derp-relay.md | 91 +++++++++ .../2026-09-18-searxng-esh-egress-reverted.md | 47 +++++ .../2026-09-18-searxng-one-engine-to-seven.md | 81 ++++++++ persistent-memory.md | 187 ++++++------------ 8 files changed, 605 insertions(+), 217 deletions(-) create mode 100644 persistent-memory.d/2026-09-18-althing-363-hyphen-search.md create mode 100644 persistent-memory.d/2026-09-18-fleet-dns-ring-and-ratelimit.md create mode 100644 persistent-memory.d/2026-09-18-fleettools-agent-index.md create mode 100644 persistent-memory.d/2026-09-18-nh3-ana-derp-relay.md create mode 100644 persistent-memory.d/2026-09-18-searxng-esh-egress-reverted.md create mode 100644 persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md diff --git a/graphify-out/GRAPH_REPORT.md b/graphify-out/GRAPH_REPORT.md index ae97e73..e4761fc 100644 --- a/graphify-out/GRAPH_REPORT.md +++ b/graphify-out/GRAPH_REPORT.md @@ -1,16 +1,16 @@ # Graph Report - eshpfi-management (2026-09-18) ## Corpus Check -- 623 files · ~839,853 words +- 623 files · ~840,412 words - Verdict: corpus is large enough that graph structure adds value. ## Summary -- 5471 nodes · 5974 edges · 631 communities (477 shown, 154 thin omitted) +- 5472 nodes · 5975 edges · 632 communities (478 shown, 154 thin omitted) - Extraction: 98% EXTRACTED · 2% INFERRED · 0% AMBIGUOUS · INFERRED: 97 edges (avg confidence: 0.81) - Token cost: 0 input · 0 output ## Graph Freshness -- Built from commit: `d812bfe9` +- Built from commit: `ee0fa1b7` - Run `git rev-parse HEAD` and compare to check if the graph is stale. - Run `graphify update .` after code changes (no API cost). @@ -152,13 +152,13 @@ - Recent decisions (archived 2026-08-03 batch) - [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit - `[2026-09-15]` FV cross-site routing fixed — one NAT rule scoped to Anaheim only -- category +- stream_chunks - build_gallery - lv-mccarthy — corpus → gate → pairs - 7. Critical Warnings by Model - think-leak — reproducers for the h300 unterminated-`` defect - services.schema.json -- method +- CatalogReproducibility - test_invocation.py - scriberr — self-hosted transcription + diarization (fv-ml1, GPU1) - elway @@ -431,7 +431,7 @@ - `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression - searxng-health.sh - searxng-mcp -- content_type +- adapter/server.py - pfi-gx10 — ERP-seat SFT run 3c, staged and ready to launch - 2026-09-05-floor-claim-n2-retraction.md - FastAPI @@ -444,11 +444,11 @@ - `[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against the wrong number, twice in one hour - launch-run-03c.sh - `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early -- fields +- .submit - `[2026-09-04]` esh-nas SMB account for the AudioGridder box — and the NAS is effectively open to the whole LAN - remote-ssh - verify_corpus.py -- catalog_version +- max - booth - tok_repro.py - wrap_verbatim_html @@ -458,7 +458,7 @@ - `[2026-09-10]` althing 3.6.2 rolled — post office + both heralds, and it was TWO nodes not seven - FV to Anaheim: scoped Tailscale NAT repair - setup-nh3-scale-socks-egress.sh -- generated_at +- max_length - 2026-09-10-beszel-fleet-wiring.md - bench_lora_step.py - probe_carrier.py @@ -552,7 +552,7 @@ - irv-ml1 dead-address sweep — EXECUTED 2026-09-15 (operator-directed) - fv-mesh-watchdog.sh - grok-token-broker -- accepted_types +- mime - kb - 2026-09-10-an-author-voice-adapter-transfers-subject.md - 2026-09-10-babybronte-h02-adapter-the-voice-transferred.md @@ -578,7 +578,7 @@ - breeze-tts sizing and the fv-ml1 GPU 0 placement question (2026-09-15) - 2026-09-15-client-abandon-cancellation-boundary.md - gender_by_proximity.py -- streamable +- mime_from_field - verify_quant.py - [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live - booth/app.py @@ -587,28 +587,29 @@ - description - score_beats.py - Direct inference seats + the Asset Engine control plane -- license_warning +- optional - LiteLLM gateway — the fleet's OpenAI-compatible front door - Fleet SSH — the infra-ops identity and host map - build_beat_fixture.py - The Booth — ephemeral media board + operator decisions -- lifecycle +- options - min -- properties +- $ref - pin_known_gender.py - Speech — TTS and ASR endpoints +- output_field - repair_corpus_bronte.py - 2026-09-16-grok-broker-shelved.md - 2026-09-17-lv-bronte-gate.md +- required +- type - Path - memorization_selfsim.py - eval-hemingway.sh - memorization_dump_matches.py - voice-prep.py - althing — inter-agent message bus -- estimated_latency -- host -- version +- sanitize_tts_text - apply_gender_overrides.py - rename.py - elway — SSH playbook runner (CHANGING things) @@ -647,7 +648,7 @@ ## Import Cycles - None detected. -## Communities (631 total, 154 thin omitted) +## Communities (632 total, 154 thin omitted) ### Community 0 - "quant_modelopt.py" Cohesion: 0.53 @@ -670,8 +671,8 @@ Cohesion: 0.07 Nodes (29): ana-docker (VM 10.250.50.70), ana-filebot (CT 112 on pfi-pve) — file automation, ana-nas (CT 109 on pfi-pve, 10.250.50.50), ana-wg (CT 113 on pfi-pve) — WireGuard VPN, Cross-cutting: what to check FIRST for ambiguous outages, esh-docker-vm + vm-esh-nas, esh-pve + esh-pve-nas (10.0.250.35 + 10.0.50.55), Fleet disaster-recovery runbook (+21 more) ### Community 5 - "properties" -Cohesion: 0.07 -Nodes (29): anyOf, default, title, anyOf, default, title, properties, anyOf (+21 more) +Cohesion: 0.15 +Nodes (13): anyOf, default, title, anyOf, default, title, properties, audio_field (+5 more) ### Community 6 - "1. Locked architectural decisions" Cohesion: 0.07 @@ -706,8 +707,8 @@ Cohesion: 0.11 Nodes (17): 1. Create rest-server-ana htpasswd entry, 2. Install secrets on esh-vm-db, 3. Initialize the repo, 4. Install prerequisites (restic, resticprofile, mongosh client), 5. Deploy profile + hook, 6. Schedule + verify, Deploy (one-time), Full host config (+9 more) ### Community 14 - "properties" -Cohesion: 0.08 -Nodes (24): properties, default, title, anyOf, default, anyOf, default, title (+16 more) +Cohesion: 0.18 +Nodes (11): anyOf, default, title, properties, default, title, title, type (+3 more) ### Community 15 - "Deploy (one-time setup)" Cohesion: 0.12 @@ -842,8 +843,8 @@ Cohesion: 0.22 Nodes (9): properties, anyOf, default, title, cold_start_s, warm_per_unit, anyOf, default (+1 more) ### Community 48 - "CatalogModel" -Cohesion: 0.14 -Nodes (14): additionalProperties, properties, required, title, type, CatalogModel, anyOf, default (+6 more) +Cohesion: 0.12 +Nodes (17): additionalProperties, properties, required, title, type, CatalogModel, title, type (+9 more) ### Community 49 - "FieldOption" Cohesion: 0.13 @@ -918,8 +919,8 @@ Cohesion: 0.25 Nodes (7): backrest, Backup scope, when wired up, Deploy, First-time configuration (in the UI), Role in the fleet, Scaling knobs, Why not `offen/docker-volume-backup`? ### Community 67 - "omnivoice/app.py" -Cohesion: 0.06 -Nodes (46): Any, ChunkConfig, ChunkResult, ClockFn, GenerateFn, _base_gen_kwargs(), _chunk_config(), GenParams (+38 more) +Cohesion: 0.09 +Nodes (26): Any, ChunkConfig, ChunkResult, _base_gen_kwargs(), _chunk_config(), GenParams, _log_chunk(), _pcm16() (+18 more) ### Community 68 - "calibre-web-automated" Cohesion: 0.25 @@ -998,8 +999,8 @@ Cohesion: 0.25 Nodes (7): API, Deploy, Gotchas, Switching to the 7B variant, VibeVoice, Voices, Why this stack exists ### Community 87 - "properties" -Cohesion: 0.15 -Nodes (13): properties, title, type, $ref, title, type, endpoint, model (+5 more) +Cohesion: 0.05 +Nodes (42): properties, enum, title, type, default, title, type, title (+34 more) ### Community 88 - "Dia / Dia2" Cohesion: 0.29 @@ -1090,8 +1091,8 @@ Cohesion: 0.40 Nodes (4): Deeper context, Fleet bootstrap — read this first, Non-negotiables, Three endpoints that answer "what exists right now" ### Community 110 - "CatalogSectionGroup" -Cohesion: 0.14 -Nodes (14): additionalProperties, description, properties, required, title, type, CatalogSectionGroup, anyOf (+6 more) +Cohesion: 0.18 +Nodes (11): additionalProperties, description, properties, required, title, type, CatalogSectionGroup, anyOf (+3 more) ### Community 111 - "3. Per-model breakdowns" Cohesion: 0.06 @@ -1118,8 +1119,8 @@ Cohesion: 0.29 Nodes (7): 5.1. Create Debian VM on nh3-pve, 5.2. Install PBS, 5.3. Datastore backing — Synology NFS (chosen 2026-04-22), 5.4. Create datastore, 5.5. Create a verify job on the mirror, Phase 5 done-state, Phase 5 — Provision PBS-NH3 (DR mirror) ### Community 117 - "Headscale overlay mesh — plan (2026-09-06)" -Cohesion: 0.10 -Nodes (20): 06:05Z — return path built and verified (operator approved 05:50Z), 06:20Z — first roaming client verified, 06:35Z — ESH egress whitelisted in CrowdSec (operator-directed, temporary), 2026-09-06 — exit nodes advertised (multi-location egress proxy), 2026-09-06 — irv-ml1 CUTOVER COMPLETE + dante retired (done remotely, 5-day no-access), 2026-09-06 — irv-ml1 folded in ADDITIVELY (operator inaccessible 5 days — lifeline preserved), 2026-09-06 — selective masquerade applied (supersedes the SNAT-reversal note above), 2026-09-06 — throughput baseline (mesh vs the old IPsec) (+12 more) +Cohesion: 0.09 +Nodes (21): 06:05Z — return path built and verified (operator approved 05:50Z), 06:20Z — first roaming client verified, 06:35Z — ESH egress whitelisted in CrowdSec (operator-directed, temporary), 2026-09-06 — exit nodes advertised (multi-location egress proxy), 2026-09-06 — irv-ml1 CUTOVER COMPLETE + dante retired (done remotely, 5-day no-access), 2026-09-06 — irv-ml1 folded in ADDITIVELY (operator inaccessible 5 days — lifeline preserved), 2026-09-06 — selective masquerade applied (supersedes the SNAT-reversal note above), 2026-09-06 — throughput baseline (mesh vs the old IPsec) (+13 more) ### Community 118 - "Phase 0 — Prerequisites (one-time, before Phase 1)" Cohesion: 0.33 @@ -1193,9 +1194,9 @@ Nodes (49): [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gb Cohesion: 0.08 Nodes (20): `[2026-09-15]` FV cross-site routing fixed — one NAT rule scoped to Anaheim only, Fix, Root cause, ⭐ The diagnostic signature, so the next person skips the evening, Wrong turns worth not repeating, `[2026-09-15]` Break-glass mesh path on fv-ml1 (inverted from a restore-watchdog), Design choices that matter, ⭐ INVERTED the same night, on the operator's suggestion (+12 more) -### Community 137 - "category" -Cohesion: 0.50 -Nodes (4): enum, title, type, category +### Community 137 - "stream_chunks" +Cohesion: 0.19 +Nodes (17): ClockFn, GenerateFn, ChunkConfig, _ema(), _est_gen_time(), plan_chunk(), protect_first_audio(), Split into sentence units, preserving punctuation. Whitespace-collapsed. (+9 more) ### Community 138 - "build_gallery" Cohesion: 0.09 @@ -1214,12 +1215,12 @@ Cohesion: 0.22 Nodes (8): Did our abliteration cause it? No — the base did (~83% / ~17%), Resolution — Cold-Fusion abandoned, seat rolled back to `heresy` (2026-08-21), Scripts, The fix, The trigger is TEMPERATURE, not presence_penalty, think-leak — reproducers for the h300 unterminated-`` defect, Using these on any future seat, What actually happens ### Community 142 - "services.schema.json" -Cohesion: 0.25 -Nodes (7): additionalProperties, description, $id, required, $schema, title, type +Cohesion: 0.10 +Nodes (19): additionalProperties, default, title, type, description, anyOf, default, title (+11 more) -### Community 143 - "method" -Cohesion: 0.50 -Nodes (4): default, title, type, method +### Community 143 - "CatalogReproducibility" +Cohesion: 0.12 +Nodes (17): additionalProperties, properties, required, title, type, CatalogReproducibility, title, type (+9 more) ### Community 144 - "test_invocation.py" Cohesion: 0.25 @@ -1338,8 +1339,8 @@ Cohesion: 0.15 Nodes (12): 1. The MTP head is abliterated *in-band*, 2. The vision tower is preserved byte-identical, Abliteration recipe — Qwen3.8-27B (MTP-aware, vision-preserving), How this maps onto our pipeline, Measured behaviour (their numbers, for reference), Provenance, Refusal-direction selection, The attention-sink dimension — the one that bricks the model (+4 more) ### Community 190 - "JobManager" -Cohesion: 0.17 -Nodes (8): Popen, Path, Job, JobManager, _pid_alive(), Singleton owner of the one active job + retained history. Thread-safe., Start a train. Raises InvalidTrainRequest (→422) or Busy (→409)., Phase 2 publish step: COPY the succeeded LoRA into ComfyUI's loras search path a +Cohesion: 0.20 +Nodes (7): Popen, Path, Job, JobManager, _pid_alive(), Singleton owner of the one active job + retained history. Thread-safe., Phase 2 publish step: COPY the succeeded LoRA into ComfyUI's loras search path a ### Community 193 - "LoRA training worker" Cohesion: 0.22 @@ -1755,7 +1756,7 @@ Nodes (4): ask(), classify(), load_prompts(), main() ### Community 327 - "worker/app.py" Cohesion: 0.15 -Nodes (15): RuntimeError, get_gpu_status(), post_train(), FastAPI surface for the LoRA training worker (§4.1 API — arbo is the client). E, TrainRequest, _bus_id_by_index(), _cmdline(), gpu_status() (+7 more) +Nodes (13): get_gpu_status(), FastAPI surface for the LoRA training worker (§4.1 API — arbo is the client). E, Static configuration for the LoRA training worker. Everything load-bearing is a, _bus_id_by_index(), _cmdline(), gpu_status(), _query_compute_apps(), _query_devices() (+5 more) ### Community 329 - "Beszel fleet monitoring" Cohesion: 0.33 @@ -1786,8 +1787,8 @@ Cohesion: 0.25 Nodes (7): built_at, corpus, normalisation, source, total_chapters, total_words, works ### Community 337 - "properties" -Cohesion: 0.06 -Nodes (35): additionalProperties, properties, required, title, type, additionalProperties, properties, required (+27 more) +Cohesion: 0.11 +Nodes (18): additionalProperties, properties, required, title, type, CatalogAuditEntry, title, type (+10 more) ### Community 338 - "archival-memory.md" Cohesion: 0.10 @@ -2097,9 +2098,9 @@ Nodes (6): `[2026-08-19]` Four unmanaged stacks found on live hosts — and two Cohesion: 0.40 Nodes (4): `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression, The lora_B signal worth carrying forward, The verdict (brokkr-smithy-dev, run03-gate/RESULT-run03c-gate.md), What the port proved about the box -### Community 440 - "content_type" -Cohesion: 0.50 -Nodes (4): default, title, type, content_type +### Community 440 - "adapter/server.py" +Cohesion: 0.29 +Nodes (7): _get_model(), OpenAI-ish /v1/audio/speech adapter in front of the Zonos Python SDK. Why this, _speaker_embedding(), speech(), SpeechRequest, JSONResponse, Zonos ### Community 441 - "pfi-gx10 — ERP-seat SFT run 3c, staged and ready to launch" Cohesion: 0.05 @@ -2110,8 +2111,8 @@ Cohesion: 0.40 Nodes (3): `[2026-09-05]` A peer's "2.7x stack effect" was a coin flip — the operator caught it and the arithmetic is worth keeping, `[2026-09-04/05]` vLLM RUNS on sm_121 — and the blocker was `ninja` not being on PATH, not the silicon, Run 4 — the corpus arm ### Community 443 - "FastAPI" -Cohesion: 0.12 -Nodes (18): ACEStepInput, ACEStepOutput, generate_audio(), initialize_pipeline(), Generate music; respond with the audio bytes inline. Pre-2026-05-11 this re, ACEStepPipeline, _get_model(), OpenAI-ish /v1/audio/speech adapter in front of the Zonos Python SDK. Why this (+10 more) +Cohesion: 0.16 +Nodes (13): ACEStepInput, ACEStepOutput, generate_audio(), initialize_pipeline(), Generate music; respond with the audio bytes inline. Pre-2026-05-11 this re, ACEStepPipeline, BaseModel, FastAPI (+5 more) ### Community 445 - "`[2026-09-15]` ESPHome modernised for ha-dev; `kb` search tool for the personal Worldtree KB" Cohesion: 0.50 @@ -2137,17 +2138,17 @@ Nodes (3): `[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against Cohesion: 0.50 Nodes (3): `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early, ⚠ IT DEGRADED, AND MY "PLATEAU" READ WAS WRONG, Method notes worth keeping -### Community 454 - "fields" -Cohesion: 0.50 -Nodes (4): items, title, type, fields +### Community 454 - ".submit" +Cohesion: 0.33 +Nodes (4): RuntimeError, Busy, Start a train. Raises InvalidTrainRequest (→422) or Busy (→409)., A second train was dispatched while one is active → HTTP 409. ### Community 456 - "remote-ssh" Cohesion: 0.50 Nodes (3): SSH_MCP_STRICT_ALLOWLIST, node, remote-ssh -### Community 458 - "catalog_version" +### Community 458 - "max" Cohesion: 0.50 -Nodes (4): default, title, type, catalog_version +Nodes (4): anyOf, default, title, max ### Community 462 - "wrap_verbatim_html" Cohesion: 0.22 @@ -2173,13 +2174,13 @@ Nodes (6): `[2026-09-10]` althing 3.6.2 rolled — post office + both heralds, a Cohesion: 0.25 Nodes (7): Backup and rollback, Broadened to the whole FV LAN, same session, Evidence and verification, ⭐ EXTENDED 2026-09-15 — NH3, ESH and Irvine now have the same rule, FV to Anaheim: scoped Tailscale NAT repair, Persistent configuration, What this looks like when it breaks, so the next person skips the evening -### Community 469 - "generated_at" +### Community 469 - "max_length" Cohesion: 0.50 -Nodes (4): anyOf, default, title, generated_at +Nodes (4): anyOf, default, title, max_length ### Community 474 - "invocation.py" -Cohesion: 0.14 -Nodes (17): Path, test_published_relative_path(), test_published_relative_path_explicit_train_id_wins(), ValueError, Static configuration for the LoRA training worker. Everything load-bearing is a, LoRA training worker — a host service (runs as llmuser on irv-ml1) that runs sd-, InvalidTrainRequest, published_relative_path() (+9 more) +Cohesion: 0.19 +Nodes (14): Path, test_published_relative_path(), test_published_relative_path_explicit_train_id_wins(), ValueError, InvalidTrainRequest, published_relative_path(), Fixed-invocation command builder — the enforcement point for INV-T7. arbo hands, The ComfyUI-relative loras path for a succeeded LoRA (Phase 2 publish step). (+6 more) ### Community 483 - "recover_direction" Cohesion: 0.52 @@ -2329,9 +2330,9 @@ Nodes (6): ⚠⚠ FOUR MORE LIVE BREAKAGES on OTHER hosts — the sweep's real f Cohesion: 0.29 Nodes (6): Current state — ⛔ SHELVED, deliberately, 2026-09-16, For a consumer, grok-token-broker, Invariants, The problem, ⛔ Why the loop is disarmed until measured -### Community 564 - "accepted_types" +### Community 564 - "mime" Cohesion: 0.50 -Nodes (4): anyOf, default, title, accepted_types +Nodes (4): anyOf, default, title, mime ### Community 582 - "build_corpus_hemingway.py" Cohesion: 0.20 @@ -2353,9 +2354,9 @@ Nodes (8): breeze-tts sizing and the fv-ml1 GPU 0 placement question (2026-09-15 Cohesion: 0.31 Nodes (9): decide(), load_text(), main(), Path, D2b for BabyHemingway: resolve entity gender by MAJORITY VOTE over nearby…, One pass over the corpus for EVERY surface at once. The obvious shape -- rescan…, Score a name's local pronoun mix AGAINST THE CORPUS BASE RATE, not against…, score() (+1 more) -### Community 590 - "streamable" +### Community 590 - "mime_from_field" Cohesion: 0.50 -Nodes (4): streamable, default, title, type +Nodes (4): anyOf, default, title, mime_from_field ### Community 591 - "verify_quant.py" Cohesion: 0.40 @@ -2381,9 +2382,9 @@ Nodes (7): load(), main(), per_seed(), Score beat arms against a decision rule f Cohesion: 0.33 Nodes (5): Asset Engine — `http://10.250.50.70:8200`, Direct inference seats + the Asset Engine control plane, GPU hosts, ⚠ Query the live inventory, never a written list, Traps -### Community 599 - "license_warning" +### Community 599 - "optional" Cohesion: 0.50 -Nodes (4): anyOf, default, title, license_warning +Nodes (4): default, title, type, optional ### Community 600 - "LiteLLM gateway — the fleet's OpenAI-compatible front door" Cohesion: 0.33 @@ -2397,41 +2398,45 @@ Nodes (5): Fleet SSH — the infra-ops identity and host map, Identity, Mesh, Si Cohesion: 0.40 Nodes (4): Ask the operator to decide (no chat round-trip), Show the operator something, Standing convention — the link board, The Booth — ephemeral media board + operator decisions -### Community 604 - "lifecycle" -Cohesion: 0.67 -Nodes (3): anyOf, default, lifecycle +### Community 604 - "options" +Cohesion: 0.50 +Nodes (4): anyOf, default, title, options ### Community 605 - "min" Cohesion: 0.50 Nodes (4): anyOf, default, title, min -### Community 606 - "properties" -Cohesion: 0.15 -Nodes (14): $ref, properties, reproducibility_audit, section_groups, services, items, title, type (+6 more) +### Community 606 - "$ref" +Cohesion: 0.14 +Nodes (14): items, title, type, $ref, fields, section_groups, services, items (+6 more) ### Community 608 - "Speech — TTS and ASR endpoints" Cohesion: 0.40 Nodes (4): ASR, Speech — TTS and ASR endpoints, Traps, TTS +### Community 609 - "output_field" +Cohesion: 0.50 +Nodes (4): anyOf, default, title, output_field + ### Community 612 - "2026-09-17-lv-bronte-gate.md" Cohesion: 0.05 Nodes (41): `[2026-09-16]` lv-hemingway corpus: half the work was EXCLUSION, and the gate found what a hand count would not, `[2026-09-16]` The lv-* voice line: Option C proved, lv-yarros shipped, lv-hemingway training, `[2026-09-16]` voices-seat: LoRA over merge, measured — and GPU 0 is now full, `[2026-09-17]` The beat-writing model restores the author's real names — a leak the corpus gate structurally cannot see, `[2026-09-17]` lv-bronte: corpus gated for real, adapter trained, SHIPPED with a FAILED voice axis on the record, ⚠ A DEFECT IN THE v2 RULE ITSELF, worth fixing before the next author, ⚠⚠ AMENDED 2026-09-17 — THE VOICE AXIS PASSES UNDER THE CORRECTED FLOOR RULE, Artefacts (+33 more) +### Community 613 - "required" +Cohesion: 0.50 +Nodes (4): required, default, title, type + +### Community 614 - "type" +Cohesion: 0.50 +Nodes (4): type, enum, title, type + ### Community 620 - "althing — inter-agent message bus" Cohesion: 0.50 Nodes (3): althing — inter-agent message bus, Commands, Traps -### Community 621 - "estimated_latency" -Cohesion: 0.67 -Nodes (3): anyOf, default, estimated_latency - -### Community 622 - "host" -Cohesion: 0.67 -Nodes (3): title, type, host - -### Community 623 - "version" -Cohesion: 0.67 -Nodes (3): version, title, type +### Community 621 - "sanitize_tts_text" +Cohesion: 0.50 +Nodes (3): Language-safe text sanitizer for the OmniVoice TTS wrapper. Strips artifacts th, Return ``text`` cleaned of artifacts that shouldn't be spoken, language-safe., sanitize_tts_text() ### Community 625 - "rename.py" Cohesion: 0.60 @@ -2450,7 +2455,7 @@ Cohesion: 0.25 Nodes (8): _cli(), Regression: the branch ended with `[ "$n" -eq 0 ] && echo ...`, so it returned…, Regression: ids are 8 hex chars and roughly one in forty is all digits. Those…, test_cli_links_exits_zero_on_a_NON_empty_board(), test_cli_unlink_by_id_even_when_the_id_is_all_digits(), test_cli_unlink_by_index(), test_cli_unlink_of_a_stale_id_leaves_the_board_alone(), test_cli_unlink_rejects_a_non_id_non_index() ## Knowledge Gaps -- **3021 isolated node(s):** `⚠ YOU ARE `infra-ops` — the fleet operations agent`, `Persistent memory`, `Model quantization`, `Training runs`, `Purpose` (+3016 more) +- **3022 isolated node(s):** `Why Headscale (decided 2026-09-06)`, `Site facts that drive placement`, `Topology`, `Access & credentials — all in hand (verified 2026-09-06)`, `Phases` (+3017 more) These have ≤1 connection - possible missing edges or undocumented components. - **154 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes. @@ -2458,16 +2463,16 @@ Nodes (8): _cli(), Regression: the branch ended with `[ "$n" -eq 0 ] && echo ... _Questions this graph is uniquely positioned to answer:_ - **Why does `create_app()` connect `booth/app.py` to `FastAPI`, `test_booth.py`, `Path`?** - _High betweenness centrality (0.003) - this node is a cross-community bridge._ -- **Why does `main()` connect `abliterate.py` to `convert_hf_to_native.py`, `Path`?** - _High betweenness centrality (0.002) - this node is a cross-community bridge._ + _High betweenness centrality (0.004) - this node is a cross-community bridge._ - **Why does ``[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c` connect ``[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c` to `archival-memory.md`?** _High betweenness centrality (0.002) - this node is a cross-community bridge._ -- **What connects `⚠ YOU ARE `infra-ops` — the fleet operations agent`, `Persistent memory`, `Model quantization` to the rest of the system?** - _3021 weakly-connected nodes found - possible documentation gaps or missing edges._ +- **Why does `CatalogService` connect `$defs` to `properties`?** + _High betweenness centrality (0.001) - this node is a cross-community bridge._ +- **What connects `Why Headscale (decided 2026-09-06)`, `Site facts that drive placement`, `Topology` to the rest of the system?** + _3022 weakly-connected nodes found - possible documentation gaps or missing edges._ - **Should `Status + Open Issues` be split into smaller, more focused modules?** _Cohesion score 0.05555555555555555 - nodes in this community are weakly interconnected._ - **Should `Fleet disaster-recovery runbook` be split into smaller, more focused modules?** _Cohesion score 0.06666666666666667 - nodes in this community are weakly interconnected._ -- **Should `properties` be split into smaller, more focused modules?** - _Cohesion score 0.06896551724137931 - nodes in this community are weakly interconnected._ \ No newline at end of file +- **Should `1. Locked architectural decisions` be split into smaller, more focused modules?** + _Cohesion score 0.07407407407407407 - nodes in this community are weakly interconnected._ \ No newline at end of file diff --git a/persistent-memory.d/2026-09-18-althing-363-hyphen-search.md b/persistent-memory.d/2026-09-18-althing-363-hyphen-search.md new file mode 100644 index 0000000..aa74e59 --- /dev/null +++ b/persistent-memory.d/2026-09-18-althing-363-hyphen-search.md @@ -0,0 +1,80 @@ +# [2026-09-18] althing search returned ZERO for every hyphenated query — silently, for everyone + +Routed in by heid on behalf of peedlar-dev as "a message reached storage but never +entered the inbox unread-set or search index". Two symptoms, and **neither was what +was reported**. + +## Symptom 1 — "never delivered": NOT A DEFECT + +Message 2571 was fully intact: recipients row present, in the FTS index, `read_at` +set. **Zero messages fleet-wide have no recipient row — 0 of 2583.** + +Grouping the reader's `read_at` values gives one group per `postbox read` +transaction, and the second consumed **2570 AND 2571 together**. `postbox read` +advances over exactly what it printed, so that read printed both. + +⚠ **The evidence was truncated.** Three of four tee'd files sat at exact 4 KiB +multiples ending mid-sentence; `read-2-of-3.md` was 8,192 bytes while message 2570's +body ALONE is 64,975. It held under 13% of 2570 and never reached 2571. That read +emitted ≥127 KB and 8 KB survived. The reasoning — "a read that included 2571 would +have printed it into one of these files" — was sound on a premise that failed. + +## Symptom 2 — A REAL DEFECT, and far wider than one message + +**Any query containing a hyphen returned zero hits, silently, for every handle.** + + 'nh3-docker' -> OperationalError: no such column: docker + '"nh3-docker"' -> 20 rows + 'tts-dev' -> OperationalError: no such column: dev + '"tts-dev"' -> 47 rows + +FTS5 reads the hyphen as a column filter and tries to resolve the trailing token as +a column name. `store.py` passed the caller's query to `MATCH` unescaped; `search()` +caught the `OperationalError`, ran its probe (which succeeded — the store was +healthy), and returned `[]`, indistinguishable from "no results". + +On this fleet that is most proper nouns we own: `nh3-docker`, `irv-ml1`, +`esh-docker-vm`, `fv-ml1`, `tts-dev`, `peedlar-dev`. + +⚠ **The discriminator was working as designed.** It correctly separated +store-broken from caller-typo. The defect was that it classified an ordinary +hyphenated query as a typo. + +## The fix, and the boundary that held + +Routed to forseti (althing's owner) rather than patched — infra-ops owns the +**rollout**, not the codebase. forseti implemented on operator instruction and +shipped **v3.6.3** (`5de41b7`): valid FTS syntax intact, `OperationalError` retries +once as an escaped literal phrase before the health probe. Plus a read-summary line +(`consumed N messages: 2570, 2571`) that makes the truncation case detectable. + +⚠ **I twice declined to act on relayed authorization** — "Vuong requested deployment" +arriving through a peer is not the operator instructing me, and a post-office bounce +takes the bus down for every handle. Got it directly, then executed. + +## Rollout mechanics worth keeping + +- **No CI builds the post-office image.** `deploy.yaml` ships althing-*chamber* to + ana-docker; `ci.yaml` runs tests and the frontend. The root `Dockerfile` IS the + post office and nothing publishes it — a build-and-push step, not a pull. +- Built from `git archive v3.6.3`, **not a working tree**; archive hash verified + identical after transfer. My wheel came out byte-identical to forseti's. +- Bus outage **19:13:40 → 19:13:57 UTC, 17 seconds**. +- Deployed digest `sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4`; + rollback pin kept as `compose.yaml.bak-3.6.2`. +- ⚠ **Canonical `stacks/althing-post-office/compose.yaml` had to be re-pinned too.** + forseti caught it: hand-deploying leaves canonical intent stale, so the next routine + `deploy-stack.sh` would have silently ROLLED THE BUS BACK. Commit `670ac9e`. + +## Still open — two postbox fixes proposed to forseti + +1. **`--body -` is taken as a literal hyphen, not stdin.** Four real messages lost to + it today (tts-dev ×3, ledger-dev earlier). Store audit: 5 of 2,637 messages have a + one-character body and **every accidental one is a literal `-`**; the fifth is a + deliberate `'x'` probe. forseti's scope: reserve `--body -` for stdin in both verbs. + **Awaiting operator.** ⚠ Empty bodies are already rejected (PRE-306, verified) — no + minimum-length rule is needed. +2. A bounded consuming read — **withdrawn**, and rightly. A message-count limit does + not bound one very large body, so it would not have saved peedlar-dev. + +Until it ships, the mitigation is entirely sender-side: pipe the body, never `--body`. diff --git a/persistent-memory.d/2026-09-18-fleet-dns-ring-and-ratelimit.md b/persistent-memory.d/2026-09-18-fleet-dns-ring-and-ratelimit.md new file mode 100644 index 0000000..64904b6 --- /dev/null +++ b/persistent-memory.d/2026-09-18-fleet-dns-ring-and-ratelimit.md @@ -0,0 +1,94 @@ +# [2026-09-18] `.internal` DNS was failing ~10% of lookups — two causes, both fleet-wide + +Two independent faults, fixed in order. Together they were costing roughly one +`.internal` lookup in ten either a hard failure or a five-second stall, on every +DHCP client at every site. + +## Fault 1 — a PUBLIC resolver was the fallback for a PRIVATE zone + +DHCP handed out `10.100.50.40, 1.1.1.1` on both NH3 VLANs. When the internal +resolver missed a packet, the resolver waited out its timeout, fell through to +Cloudflare, and got an **authoritative NXDOMAIN** — so a transient miss became a +definitive "no such host" rather than a retry. Binary failure: instant, or five +seconds then `gaierror`. + +**Anaheim was worse.** Its FortiGate handed clients *itself* as resolver and +forwarded to `1.1.1.1`/`1.0.0.1`, so **no ANA host could resolve `.internal` at +all** — `ana-docker`, which HOSTS the Anaheim AdGuard, had `1.1.1.1` in its own +`resolv.conf`. + +⚠ **`/etc/resolv.conf` on these hosts is DHCP-managed** (dhclient on ens18). Hand +editing it survives exactly until the next lease renewal — tts-dev's original +framing was "fix nh3-dev's resolv.conf", which would have worked, been verified, +and then silently reverted. Change it at the source: + +- **UniFi**: `dhcpd_dns_1` / `dhcpd_dns_2` per network object. +- **FortiGate**: `config system dhcp server`, `set dns-service specify` + + `dns-server1` / `dns-server2`. +- Then `sudo dhclient -1 -v ens18` to pick it up **without releasing** the lease. + +**Operator's ruling was broader than the proposal**: each site's backup resolver +should be *another site's* resolver. ESH already worked this way; the other two +were set, and the ring closes. + + ESH primary 10.0.50.45 backup 10.100.50.40 (NH3) [pre-existing] + NH3 primary 10.100.50.40 backup 10.250.50.70 (ANA) [changed] + ANA primary 10.250.50.70 backup 10.0.50.45 (ESH) [changed] + +All six members verified to answer the private zone. ⚠ The SureFire tenant DHCP +server (`sfsrv`) and the fortilink switch-management server were deliberately left +alone — tenant property. + +**Result: hard failures 3/40 → 0/40. The 5 s stalls remained.** + +## Fault 2 — THE ROOT CAUSE: AdGuard rate-limiting, identical on all three + + ratelimit: 20 + ratelimit_subnet_len_ipv4: 24 + ratelimit_whitelist: [] + +**20 queries per second shared across an entire /24 — per subnet, not per client.** +Every host on a VLAN draws from one bucket, so one busy container starves the rest, +and anything over the line is **silently dropped**, costing the client its full 5 s +resolver timeout. + +Demonstrated rather than inferred — 60 concurrent queries at one resolver: + +| | before | after | +|---|---|---| +| median | 5044 ms | 72 ms | +| timed out (>4 s) | **40 of 60** | **0 of 60** | + +Set to `ratelimit: 0` on all three. These are LAN-only resolvers behind three +firewalls; that setting exists to blunt internet-side DNS amplification, which is +not this. + +⚠ **Restarting AdGuard is required for a config change and briefly drops that site's +primary resolver — do the three ONE AT A TIME** so the cross-site ring always has a +live member. That the ring made this rollout safe is the point of having built it +first, an hour earlier. + +## End-to-end, the original symptom + +40 TCP connects to a service by name vs by IP; the by-IP arm is the control. + +| | original | after ring only | after ratelimit 0 | +|---|---|---|---| +| by NAME p90 | 120 ms | 5017 ms | **13.5 ms** | +| by NAME >1 s | 3/40 | 5/40 | **0/40** | +| by NAME failed | 2/40 | 0/40 | **0/40** | +| by IP p90 (control) | 25 ms | — | 11.7 ms | + +tts-dev confirmed from talk's own request path: name and IP **1.5 ms apart at the +median**. Resolution has stopped being a cost rather than become a smaller one. + +## What this says about monitoring + +Nothing on this fleet watches DNS success rate. The fleet's actual DNS monitor +today was a voice loop with a 1545 ms budget whose owner chose to measure instead +of adapting around the problem. Both faults were invisible to Beszel and Uptime +Kuma — every host was up, every service answered. + +Related: [[2026-09-18-nh3-ana-derp-relay]] — the ANA AdGuard is only a sensible +cross-site backup because that fix took NH3→ANA from ~1.2 s to ~15 ms the same +afternoon. The two changes compose. diff --git a/persistent-memory.d/2026-09-18-fleettools-agent-index.md b/persistent-memory.d/2026-09-18-fleettools-agent-index.md new file mode 100644 index 0000000..b1d345e --- /dev/null +++ b/persistent-memory.d/2026-09-18-fleettools-agent-index.md @@ -0,0 +1,57 @@ +# [2026-09-18] FleetTools — one capability index, autoloaded by three agent families + +Every agent on nh3-dev needs the same answers — what runs here, how do I call it, +what will bite me — but that knowledge lived in `~/.claude/CLAUDE.md`, which only +Claude sessions load, interleaved with operator preferences other families have no +use for. + +## Shape + +**Two-tier, matching the persistent-memory split.** `FLEETTOOLS.md` is a ~135-line +index an agent reads whole; each entry links to a detail file it opens only when it +needs that tool. Reading the index costs about a fifth of reading the tree. + +- Canonical: `docs/fleettools/` in this repo (git-tracked) +- Discovery: `~/FLEETTOOLS.md` → symlink to the index +- **Detail links are ABSOLUTE paths**, not relative markdown — a non-Claude agent + `cat`s the path rather than following a link, and may be anywhere on the filesystem. + +Covers: althing, the Booth, `secret`, LiteLLM, inference seats + Asset Engine, +speech, Arbo, elway, fleet SSH, observability, graphify, Playwright, the Henge, and +a segregated Claude-only file (ratecheck, remote-ssh MCP, task-board, skills). + +## Autoload — verified per family, not assumed + +- **Codex** reads `$CODEX_HOME/AGENTS.md` — the binary carries the string + `"Failed to read global AGENTS.md instructions from"`. +- **Grok** always scans `$GROK_HOME/rules/` and loads **every `*.md` in it regardless + of name** — from its own embedded docs. + +Both locations were empty. `AGENT-BOOTSTRAP.md` is **symlinked** into each rather than +copied — one file, three families, no drift surface. + +⚠ Grok also scans `~/.claude/rules/` and recognises `CLAUDE.md` via harness-compat +settings. That directory is empty today, but Grok can see Claude's config. + +## Rule zero, which is the load-bearing part + +**Query live inventories, never a written list.** The index points at +Homepage `/api/services`, asset-engine `/api/v1/services`, LiteLLM `/v1/models`, and +every FastAPI seat's `/openapi.json`. A copied service table would be stale within a +month — the LiteLLM roster was already at 40 models where the old CLAUDE.md note said +~30. + +## The CLAUDE.md swap + +`~/.claude/CLAUDE.md`'s "Global tools available" section went from **231 lines to 39** +(file 1218 → 1026, a 16% cut to what every Claude session loads). Three things stayed +inline deliberately because they govern BEHAVIOUR rather than lookup: run `ratecheck` +before expensive work, post operator-facing links to the Booth board, and the vault is +the credential source of truth. **A rule behind a file read is a rule that stops +firing.** Backup at `~/.claude/CLAUDE.md.bak-20260918-074048`. + +The shared all-agents LiteLLM key stays inline there too — every session needs it and a +vault round-trip measured over two minutes — but it is now ALSO vaulted at +`litellm/all-agents-shared-key`, since it had been single-copy. + +Commits `53c3e80` `21d24c5`. diff --git a/persistent-memory.d/2026-09-18-nh3-ana-derp-relay.md b/persistent-memory.d/2026-09-18-nh3-ana-derp-relay.md new file mode 100644 index 0000000..c2788ee --- /dev/null +++ b/persistent-memory.d/2026-09-18-nh3-ana-derp-relay.md @@ -0,0 +1,91 @@ +# [2026-09-18] NH3↔Anaheim was DERP-relayed, not direct — a UDP 41641 port-forward on ana-gw fixed it + +Every NH3→Anaheim flow had been crossing Tailscale's LA DERP relay rather than a +direct path, long enough to have carried **78 GB tx on the NH3 side alone**. DERP +is a deliberately throttled fallback — a reachability mechanism, not a data plane — +so this imposed both a fixed round-trip penalty and a bandwidth ceiling on LiteLLM, +Beszel, task-board, vor and the Henge alike. + +## How it surfaced, and how it did NOT + +tts-dev reported ext-stt transcriptions 4–18× slower through the LiteLLM gateway +than straight at the Parakeet host, with a ~1.9 s fixed floor plus a per-byte cost. +They had already cleared GPU contention, the ASR itself and output quality with +real controls. + +**No monitor caught it.** Beszel had every host green. Nothing on this fleet checks +whether a mesh path is direct. + +## The diagnosis, and two wrong causes published on the way + +Their hypothesis was a dead first target in a LiteLLM fallback list. Checked: `ext-stt` +has exactly ONE deployment, no fallback, nothing to retry against. Langfuse looked like +a great candidate for a fixed floor — env vars still set, container gone — but it was +retired from the callback list 2026-06-20 and nothing calls it per-request. + +The decisive arm was one neither of us had run: **measure from ana-docker itself**, so +both arms share the identical onward path to FV. + + ana-docker -> fv-ml1 DIRECT 0.289 / 0.218 / 0.273 / 0.220 s + ana-docker -> gateway -> fv-ml1 0.463 / 0.211 / 0.222 / 0.222 s + +The gateway adds ~0. tts-dev measured 2618 ms for that clip; from the gateway's own +host it is 220 ms. Their two arms differed in TWO things — whether LiteLLM was in the +path AND which destination the packets crossed the network to reach — and the entire +delta was attributed to the variable of interest. + + nh3-dev -> fv-ml1 direct 0.269 / 0.219 / 0.216 s + nh3-dev -> ANA gateway 1.449 / 1.399 / 1.404 s + + nh3-scale$ tailscale ping 100.64.0.3 + pong via DERP(lax) in 507ms, 518, 521, 522, 373, 423, 464, 493 + direct connection not established + +## Why hole-punching failed + +ana-scale advertised `38.120.12.42:41641`, but its netcheck mapped to `:60798` — the +Anaheim NAT was not preserving the port — and `PortMapping` was empty, so no UPnP or +NAT-PMP was establishing one. `MappingVariesByDestIP: false`, so not symmetric NAT; +simply no reachable inbound endpoint. ESH↔Anaheim and NH3↔ESH were already direct; +only this pair failed. + +## The fix — four ADDITIVE objects on ana-gw (10.250.0.1) + +⚠ **This box runs `central-nat enable`, so a policy's `dstaddr` is the REAL internal +address and not the VIP.** The existing `wg-to-ana-wg` VIP+policy pair is the house +template; writing what a FortiGate normally wants would have failed. + +| object | value | +|---|---| +| `firewall address` | `ana-scale-ip` → 10.250.50.45/32, iface `servers` | +| `firewall service custom` | `Tailscale-41641` → `udp-portrange 41641` | +| `firewall vip` | `tailscale-to-ana-scale` → 38.120.12.42:41641 udp → 10.250.50.45:41641, extintf wan1 | +| `firewall policy` id 75 | wan1→servers, all→`ana-scale-ip`, `Tailscale-41641`, accept | + +## Measured before → after + +| path | before | after | +|---|---|---| +| `tailscale ping` nh3-scale→ana-scale | 373–522 ms via DERP(lax) | **6 ms direct** | +| STT via the ANA gateway (96 kB clip) | 1.399–1.449 s | **0.237–0.270 s** | +| Beszel HTTP nh3-dev→ana-docker | 0.94–1.29 s | **0.014–0.016 s** | + +tts-dev confirmed independently from talk's own vantage: 2618 → 249 ms and 4119 → +548 ms, converged with their direct arm. + +## Operational notes for next time + +⚠ **This edge has NO out-of-band access.** Back up first — `ssh infra-ops@10.250.0.1 +'show' > file` captures the full non-default config (11,320 lines) without needing a +tftp server; kept at `~/backups/ana-gw-config-BEFORE-.txt`. Credentials vaulted +at `fortigate/ana-gw-infra-ops-password`, reached with +`sshpass -e ssh -o PubkeyAuthentication=no`. **Additive objects only** — never edit an +existing policy or interface over a 327 ms link you cannot recover. + +⚠ **irv-ml1 (100.64.0.6) remains `relay "lax"`.** Same class, different site, not fixed +by this. Measured 24-25 ms on HTTP from nh3-dev, so it is not costing what Anaheim was. +I warned tts-dev their TTS stack was paying for it; they measured and I was wrong. + +Documented in `docs/pfi/headscale-mesh-plan.md`; commit `5a9fad8`. See +[[2026-09-18-fleet-dns-ring-and-ratelimit]] — the ANA AdGuard only became a viable +cross-site DNS backup because of this fix. diff --git a/persistent-memory.d/2026-09-18-searxng-esh-egress-reverted.md b/persistent-memory.d/2026-09-18-searxng-esh-egress-reverted.md new file mode 100644 index 0000000..caed581 --- /dev/null +++ b/persistent-memory.d/2026-09-18-searxng-esh-egress-reverted.md @@ -0,0 +1,47 @@ +# [2026-09-18] SearXNG's ESH SOCKS5 egress — one day live, reverted, and it fixed nothing + +On 2026-09-17, at operator request, SearXNG's search egress was moved from +nh3-docker's direct residential path to `socks5h://10.0.50.65:1080` on **esh-scale** +(CT 108 on esh-pve) — an application-level proxy, no host route or exit-node change. +Reverted 2026-09-18. + +## Why it was reverted, and why the STATED reason was wrong + +It was reverted on a diagnosis that turned out to be false: that the move had cost +three of four search engines to CAPTCHAs. Reverting to direct NH3 egress produced a +**byte-identical** result — same three engines down — which falsified it. A live +`!ddg` bang probe on a freshly restarted container also CAPTCHA'd, ruling out stale +suspension timers too. + +The real cause was elsewhere (see [[2026-09-18-searxng-one-engine-to-seven]]). + +**The revert still stands on its own merits**: ESH egress bought no measurable +improvement while making all fleet search depend on ESH WAN and mesh availability. +The simpler configuration is the better one. It is simply not the fix for the engines. + +## What was built, because it was built correctly + +The proxy side was not at fault and is worth keeping as a pattern. `microsocks` ran +as `nobody` under `searxng-egress.service`, bound `10.0.50.65:1080` only, and allowed +source `10.100.50.40` alone — everything else had to supply a password regenerated at +each start and never distributed. Allowed-host egress and denied-host rejection were +both tested. The unit is kept at `configs/esh-scale/searxng-egress.service`; the +service is **stopped and disabled** on esh-scale, and `tailscaled` there was not +touched (CT 108 is ESH's whole-site mesh SPOF). + +Measured egress was `128.177.138.182` in use and `154.50.58.126` at cutover — ESH's +WAN address moves and nothing pins it, so the number in the stack README is historical. + +## The transferable lesson + +⚠ **I asserted causation from correlation with no baseline.** The only evidence that +residential egress avoided CAPTCHAs was a config comment dated 2026-09-03, which was +no longer true of that address. Five samples of the post-change state and zero of the +working state is not a comparison. The rollback WAS the counterfactual, and it +falsified the claim I had already published in a commit message. + +This was the first of three wrong causal attributions in a single afternoon. The +common shape: measure after a change, attribute the delta to *my* change, never check +what else moved. + +Commits `156e126` (applied), `1a35181` (reverted). diff --git a/persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md b/persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md new file mode 100644 index 0000000..efc81d0 --- /dev/null +++ b/persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md @@ -0,0 +1,81 @@ +# [2026-09-18] SearXNG had ONE working general web engine — now seven, and the health script never noticed + +The fleet `web_search` MCP tool every nh3-dev session uses was answering from +`google cse` alone, and had been for at least a day, with every instrument green. + +## The root cause, which is subtler than "engines were disabled" + +Of **55 general-category engines, 7 were enabled by default** — and six of those +seven are dictionary, translation, currency and encyclopedia engines that return +nothing for an ordinary web query. `google cse` was the only general WEB engine on +the instance. When its quota exhausted, a query returned **HTTP 200 with an empty +`results` array**. + +⚠ **The distinction that hid it: `inactive: false` only makes an engine +SELECTABLE; `disabled: false` is what puts it in the DEFAULT set.** The other 48 +were selectable-but-off, which an API client has no way to change. + +## Three wrong causes published before the right one + +Recorded because the pattern repeated all day and each was reasoned, not careless. + +1. **"The ESH egress move broke it."** Search egress had been moved on 09-17 to a + SOCKS5 proxy on esh-scale, and a 2026-09-03 comment claimed NH3's residential + address avoided CAPTCHAs. Reverting to direct NH3 egress produced a + **byte-identical** failure. Falsified. +2. **"Stale suspension timers."** A live `!ddg` bang probe on a freshly restarted + container also returned CAPTCHA. Falsified. +3. **"The version bump fixed it."** Updating `2026.9.3` → `2026.9.18` appeared to + restore all four engines — but **another session had removed the captcha'd + scrapers from the engine set four minutes before that measurement**, and I did + not check what else had moved. Two changes, one measurement, attributed to mine. + +Every one asserted causation from correlation with **no baseline measurement of the +working state**. + +## The fix + +Five keyless engines added to the default set, each bang-probed on this egress +first rather than trusted from a list: + + duckduckgo web 10 mojeek Suspended excluded + bing 10 qwant CAPTCHA excluded + yep 20 startpage Suspended excluded + yahoo 7 brave Suspended excluded + wiby 12 seznam timeout excluded + +Plus `braveapi` with a real key. **General web engines: 1 → 7.** + +Verified under the live failure condition rather than a simulated one — `google cse` +was STILL quota-suspended and three consecutive queries returned 38–41 results from +4–5 engines each. Later with braveapi: 55–63 results from six. + +## Two durable rules this settles + +⭐ **This stack tracks `:latest` ON PURPOSE — do not "improve" it by pinning.** +Upstream ships engine-handler fixes continuously because providers change bot +gating continuously; being current IS the mitigation, and a digest pin would have +frozen the breakage in place. (The althing post office is pinned for the opposite +reason — it is the message bus and must not move underfoot.) + +⭐ **The Brave API key is committed in plaintext by explicit operator decision** +(2026-09-18): *"if the service is useless, so is the key"* — free tier, rate-limited +service, not worth the machinery of keeping it out of git. It is also vaulted at +`nh3-docker/searxng-brave-api-key`. **This is scoped to one low-value credential and +is not a change to the no-secrets-in-git rule.** It cannot be un-committed; rotation +means a new key at Brave, never a history rewrite, because the repo is shared. + +⚠ There is **no env-var path into `searxng-settings.yml`** in this build. The loader +reads only `SEARXNG_SETTINGS_PATH`; the entrypoint substitutes only `ultrasecretkey`. +`SEARXNG_BRAVE_API_KEY` reaches the container and is never read — it is kept in the +compose, unused, with a comment saying so. + +## ⚠ Still open: the health script cannot see any of this + +`scripts/searxng-health.sh` printed `✓ searching` with six engines answering AND +with one. It gates on "did any results come back" and prints failed engines as +informational. Search ran on a single engine for a day with every instrument green. +**Fix it to fail on blocked engines before trusting it again.** + +Commits `9a428fd` `274d3e2` `9219942` `5b20b02`. See +[[2026-09-18-searxng-esh-egress-reverted]]. diff --git a/persistent-memory.md b/persistent-memory.md index 11adbd0..dd76a9f 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-17 ~23:20 PT (lv-mccarthy TRAINING on gx10, ~150/1380 steps. The leak gate had PASSED with five protagonist names still in every copy — corpus rebuilt, gate taught to see the class. lv-hemingway keeps its 3 by operator ruling, so the gate exits 1 on a shipped tree BY DESIGN. ESH back on the Cityside static; headscale split-DNSes nh3.phasefinal.com. Next voice seat measured, not guessed. lv-krakauer parked.)_ +_Last updated: 2026-09-18 ~15:05 PT (infrastructure day, no training work. THREE silent faults found and fixed: NH3↔Anaheim was DERP-relayed (now 6 ms direct), `.internal` DNS was failing ~10% of lookups (cross-site resolver ring + AdGuard ratelimit 0), and SearXNG had one working web engine (now seven). althing v3.6.3 deployed — hyphenated search works for the first time. FleetTools shipped and autoloaded for Codex/Grok. ⚠ lv-mccarthy's run outcome is UNVERIFIED by this session.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -115,148 +115,78 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-09-17 ~23:20 PT._ +_As of 2026-09-18 ~14:50 PT._ -### RUNNING — lv-mccarthy LoRA on pfi-gx10, launched 2026-09-17 ~23:05 PT +### ⚠ FIRST — lv-mccarthy's run outcome is UNVERIFIED by this session -`~/r49-runs/mccarthy-4b-pairs-3ep/`, launched by `~/lv-mccarthy/launch-train.sh` (committed at -`scripts/mccarthy-corpus/launch-train.sh`). At the snapshot: **~150 of 1380 steps, ~4.2 s/it, -ETA ~1h25m**, so it lands around **00:45 PT**. A deliberate mirror of the Hemingway recipe — -Qwen3-4B-Instruct, seq-len 1536, rank 32 / alpha 64, lr 1e-4, 3 epochs, batch 1 x accum 8, -seed 4919 — with pairs sha `6a04fcd21800d85c`, 3673 train / 269 val, 0 dropped. +At the previous snapshot (09-17 ~23:05 PT) the LoRA was at ~150/1380 steps with ETA ~00:45 PT. +**This session was entirely infrastructure and never touched it**; `pfi-gx10` does not resolve from +nh3-dev by that name, so the outcome was not checked rather than checked-and-found-good. Assume +nothing: look at `~/r49-runs/mccarthy-4b-pairs-3ep/` before anything else. -⚠⚠ **`--save-total-limit` defaults to 12 and that would have silently destroyed the gate.** At the -25-step save cadence over 1380 steps that keeps only epochs ~2.35-3.0 and DELETES the epoch-1 and -epoch-2 checkpoints — which is exactly what the last two gates were decided on (Hemingway's ckpt850 -at epoch 0.96 TIED his eval-loss minimum ckpt1750; on Brontë the earlier epoch-1 checkpoint won the -tiebreak on the axes that resolve). Raised to 60 so all ~55 survive, ~42 GB against 442 GB free. -Eval/save cadence is 25 rather than Hemingway's 50 because McCarthy has 460 steps/epoch against his -887 — matching EPOCH resolution (0.054 vs 0.056), which is the unit the tiebreak is expressed in. +**Next, unchanged:** checkpoint selection off the loss curve → the v2 gate (`eval-*.sh` shape) → +ship-or-park. ⚠ Before the gate, settle how the VOICE axis is read — D1 pre-registered a +punctuation-normalised secondary read and the `mccarthy` register now states the punctuation tics +explicitly, so `--system-from` hands them to the BASE control arm too. Deliberate (denies the adapter +a cheap char-bigram win) but it changes what the primary number means, and it must be settled BEFORE +a number exists. -**Next, in order:** checkpoint selection off the loss curve → the v2 gate (`eval-*.sh` shape) → -ship-or-park. ⚠ Before the gate, settle how the VOICE axis is read: D1 pre-registered a -punctuation-normalised secondary read, and the `mccarthy` register now states the punctuation tics -explicitly, so `--system-from` hands them to the BASE control arm too. That was done deliberately to -deny the adapter a cheap char-bigram win, but it changes what the primary number means and must be -settled BEFORE a number exists. +### The fleet is materially faster than it was this morning — three fixes, all verified + +Detail in the Recent decisions entries below; the operational summary is that **NH3→Anaheim went +from a throttled DERP relay to direct (6 ms, cross-site HTTP 1.2 s → 0.015 s)**, **`.internal` DNS +stopped failing ~10% of lookups and stalling 5 s on the rest**, and **SearXNG went from one working +general web engine to seven**. All three were silent — no monitor caught any of them, and two had +been degrading for months. + +⚠ **Nothing on this fleet watches DNS success rate or whether a mesh path is direct.** Today's three +faults surfaced only because tts-dev had a 1545 ms voice-loop budget and measured instead of adapting +around it. **A probe pair is proposed and UNDECIDED** — see the Recent decisions entry. + +### Still relayed: irv-ml1 + +`100.64.0.6` remains `relay "lax"` after the Anaheim fix — a different site with its own NAT +situation, untouched. Measured 24-25 ms on HTTP from nh3-dev, so it is NOT costing what Anaheim was +and needs nothing urgently. I warned tts-dev it would be slow and was wrong; they measured and +corrected me. ### OPEN LOOP — `ravenpen.com`, and a reply hamr-dev is waiting on -hamr-dev asked infra-ops to register `ravenpen.com` (the marketing name for hamr) on a relayed -operator directive. **Surfaced, not executed**, for two reasons: a peer relaying "Vuong approved it" -is not authorization for a non-refundable purchase (Miranda is the named exception, a dev session is -not), and **infra-ops holds no registrar credential at all** — the vault's only registrar-adjacent -entries are three Cloudflare tokens and this handle's is Zone/DNS/Edit scope, which cannot register -a domain. hamr-dev accepted both points and closed out. +hamr-dev asked infra-ops to register `ravenpen.com` on a relayed operator directive. **Surfaced, not +executed**: a peer relaying "Vuong approved it" is not authorization for a non-refundable purchase, +and infra-ops holds no registrar credential. hamr-dev accepted and closed out. ⚠ **If the operator registers it, POST BACK on althing thread `01M2SERDB3DR3RV7JS1J0GMF0H`** with -the registrar and expiry date — hamr-dev is explicitly waiting on that to record in hamr's own -memory, and it is the kind of cross-session obligation a context reset drops silently. +registrar and expiry — hamr-dev is explicitly waiting on that, and it is the kind of cross-session +obligation a reset drops silently. Verified 2026-09-18 ~05:20Z: `ravenpen.{com,io,app,dev,ai,net}` +all unregistered. -Verified 2026-09-18 ~05:20Z: `ravenpen.{com,io,app,dev,ai,net}` all unregistered (RDAP 404). -Trademark: **web-visible evidence only, NOT a clearance search** — USPTO's API refused the query -(405), nothing obvious surfaced, nearest neighbours are descriptive "raven pen" uses on physical -pens, a different class from software. Recommendation given: `.com` alone, or `.com` + `.ai`. +### PLANNED (operator) — move `dragonfireacoustics.com` to Namecheap, DNS to Cloudflare -### PLANNED 2026-09-18 (operator) — move `dragonfireacoustics.com` to Namecheap, DNS to Cloudflare +Unchanged from the previous snapshot and **not started**. ⚠⚠ The zone's `*` wildcard MASKS what is +really there — enumerate real records before any transfer. Expiry **2026-10-30** at eNom with no +transfer lock, and the sibling `dragonfirepro.com` was already lost exactly this way. Customer-facing; +nothing touched. Full context in the 09-17 Recent decisions entries. -Operator holds the **TAC/EPP** already. TAC is **single-use and dies 14 days after issue**, so the -clock is running; domain itself expires **2026-10-30** and a transfer adds a year, so the transfer -IS the renewal if it lands in time. +### Other standing items, unchanged -⚠ **Recommended order is DNS FIRST, transfer second** — stand the Cloudflare zone up, repoint the -nameservers at eNom (NS is a registrar function he already has access to), verify mail and the site, -THEN burn the TAC. The registry keeps the NS delegation across a transfer and Namecheap does not -overwrite it, so nothing has to be redone; and if DNS goes wrong you still have a working registrar -account to revert at instead of discovering it mid-transfer with a spent code. - -⚠⚠ **The zone's `*` wildcard MASKS what is really there.** Everything — apex, mail, webmail, admin, -ftp — answers 199.250.192.76 (dead) purely because of that one record. Cloudflare's add-a-zone scan -will happily import the dead wildcard and may not surface what is genuinely configured, so the -**7 Google Workspace MX records must be checked by hand after the import** or mail stops. The only -records that actually matter: `www A 38.120.12.45` and those MX. - -Open sub-decisions carried into the move: point the apex at 38.120.12.45 too (makes the bare domain -work for the first time and lets a replacement cert cover it), and add SPF/DMARC — the domain runs -Google Workspace with **neither**. - -### DONE: lv-mccarthy D4 pairs built — NEXT is `train_pairs_lora.py` - -`~/lv-mccarthy/pairs/` on gx10, 2026-09-17 11:55 PT via `run-pairs.sh` (committed at -`scripts/mccarthy-corpus/run-pairs.sh`). - -``` -train 3,673 pairs of 3,722 passages reject 1.32% 461,172 words 1,377 s -val 269 pairs of 276 passages reject 2.54% 33,477 words 98 s -``` - -⭐ **The val split is the largest fixture in the line** — 269 against Hemingway's 200 and -Brontë's 44 in-band, the structural cause of Brontë's underpowered voice axis. -Generator fingerprint `vllm-0.1.1.dev50+geed1f3d0c-601ad866`, **identical to Hemingway's -run and unmoved mid-run**, so the two pair sets are generator-comparable. - -Verified in the produced pairs, not assumed: **0 mid-sentence line breaks** in all 3,942 -responses, **0 chapter-argument responses**, and **0.0 quote marks per 10k words** in both -splits — McCarthy's unquoted dialogue survives rename → chunk → reflow → pair intact. -`sourcename` rejected **14 beats** that named a character the rename removed; Hemingway's -shipped pairs had no such filter at all. `narrator_retries` 101 of 3,673 (2.8%) against -Hemingway's 8.7%, which is what third-person prose looks like. - -Then, before the eval, honour D1's pre-registration: a **punctuation-normalised secondary -read** of the voice axis — and note the `mccarthy` register now states the tics, so the -base control arm gets them too. - -### SETTLED — lv-hemingway keeps its 3 hidden survivors; operator ruled "leave it" - -`Primi tivo`, `Pasionar ia`, `Chi cote`, one each per copy in a 958k-word corpus, on the -adapter serving at fv-ml1 GPU 0 `:8027`. Operator 2026-09-17: **leave it.** lv-bronte re-ran -clean. ⚠ **Consequence to remember: `leak_gate.py` now EXITS 1 on lv-hemingway's tree.** A -future session re-running that gate will see a red result on a shipped artifact — that is -expected and settled, not a regression to chase. Do not "fix" it without a fresh ruling. - -### SHIPPED — `lv-hemingway` on ckpt850, live on `vllm-voices` (fv-ml1 GPU 0 :8027) - -Beside `voices-base`, `lv-yarros`, `lv-bronte`. VOICE +0.413 delta_cb at 6.4x its floor, -closing 73.8% of the achievable span — the strongest in the line, and it clears the OLD -all-arms floor too, so it does not lean on the rule change. -⚠ MEMORISATION is the axis to read: 0.07 hit-rate against **held-out Hemingway's own 0.01**, -so ~7x the author's self-collision rate — but all 19 matched runs were READ and every one is -stock dialogue capped at 9 words, no proper noun, no plot. -→ `persistent-memory.d/2026-09-17-lv-hemingway-gate.md` - -### ⛔ PARKED — `lv-krakauer` (operator, 2026-09-17), henge id **82** - -*"he's a great writer because of his research, not because he has a strong identifiable -voice."* Corpus (126 units, 422,880 words) and builder stay committed and re-runnable. - -### OPEN for the operator - -1. **The althing route-declaring SessionStart hook is NOT installed on nh3-dev**, so any - relaunched pane silently drops to pull mode and stops receiving mail. althing's own skill - documents the hook; this box does not have it. Installing it means editing - `~/.claude/settings.json`. Raised, not answered. -2. **Hemingway's train beats: 70 of 7,094 (0.96%) name characters the rename removed.** - `audit_pairs_sourcenames.py --filter-out` yields a verified-clean 7,024-pair set in one - command; retrain ~3h17 + re-gate ~1h40. Operator said **ship stands** — recorded, not acted on. -3. **130 of 946 Hemingway entity-map surfaces are probably not names** and were renamed anyway. - Same ruling: ship stands. - -### ESH is on Verizon failover — Cityside Fiber failed twice on 2026-09-16 - -ESH egress `97.190.18.88`; crowdsec's `esh` allowlist carries it plus `23.164.40.174`, both -with 7-day expiries set 2026-09-16. ⚠ When those expire or the CGNAT egress rotates, ESH loses -colo access. Check `curl -s4 ifconfig.me` from esh-docker-vm first if ESH goes dark. - -### Older deferred set, unchanged - -AI-tab Dormant regrouping (**belayed**), `nconnect=8` on `/mnt/smithy` (**declined in scope**), -fused MoE kernel path (**parked, id 47**), TTS-stack move to fv-ml1 (**parked, id 75**). - -⚠ **Uncommitted and NOT mine:** `graphify-out/GRAPH_REPORT.md`, `scripts/seat-inventory.py`, -`stacks/searxng/{README.md,conf/searxng-settings.yml}`, `configs/esh-scale/`. Leave them alone. +- **lv-hemingway keeps its 3 separator-hidden names** (operator: "leave it"), so `leak_gate.py` exits + 1 on a shipped tree BY DESIGN. A red result there is expected, not a bug to fix. +- **`lv-krakauer` PARKED** (operator 2026-09-17), henge id **82**. +- **ESH is on the Cityside static** `128.177.138.182/30`, healthy on four axes. The two 7-day crowdsec + allowlist entries expire 2026-09-23 and are being left deliberately — Cityside failed twice in six + hours on 09-16/17. ## Recent decisions +- `[2026-09-18]` ⭐⭐⭐ **NH3↔Anaheim had been running over a throttled DERP relay, not a direct path — 78 GB of fleet traffic on someone else's free infrastructure.** Four additive objects on ana-gw gave ana-scale a stable inbound UDP 41641 endpoint; `tailscale ping` 373–522 ms → **6 ms direct**, cross-site HTTP 1.2 s → 0.015 s, STT via the ANA gateway 1.4 s → 0.25 s. ⚠ That box runs `central-nat`, so a policy `dstaddr` is the REAL internal address, not the VIP. No OOB access — back up with `show` to a local file and make additive changes ONLY. irv-ml1 still relayed. → `persistent-memory.d/2026-09-18-nh3-ana-derp-relay.md` +- `[2026-09-18]` ⭐⭐⭐ **`.internal` DNS was failing ~10% of lookups fleet-wide, from two independent causes.** A PUBLIC resolver was the fallback for a PRIVATE zone (Cloudflare answers NXDOMAIN authoritatively, so a transient miss became a hard failure) — now a cross-site ring, each site local-first with a different site as backup. Then the root cause: all three AdGuards shipped `ratelimit: 20` shared across an entire **/24**, silently dropping queries at 5 s each. Set to 0. Hard failures 3/40 → 0/40; burst timeouts 40/60 → 0/60. ⚠ `resolv.conf` is DHCP-managed — change it at the UDM/FortiGate, not the file. → `persistent-memory.d/2026-09-18-fleet-dns-ring-and-ratelimit.md` +- `[2026-09-18]` ⭐⭐ **SearXNG had ONE working general web engine and every health check said fine.** 7 of 55 were enabled-by-default and six of those are dictionary/translation engines — `inactive: false` only makes an engine SELECTABLE, `disabled: false` puts it in the DEFAULT set. Now seven. ⭐ This stack tracks `:latest` ON PURPOSE (upstream ships engine-handler fixes continuously; a pin freezes breakage). ⭐ The Brave key is committed in plaintext by explicit operator decision — scoped to one low-value credential, NOT a change to the no-secrets rule. → `persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md` +- `[2026-09-18]` ⭐⭐ **althing search returned ZERO for every hyphenated query — silently, for every handle, and on this fleet that is most of our hostnames.** FTS5 read the hyphen as a column filter; `search()` caught the error, its probe passed, and it returned `[]`. Routed to forseti (they own the code, I own rollout) → **v3.6.3** deployed, 17 s bus outage. The reported symptom, a message that never arrived, was NOT a defect: the reporter's evidence file was truncated at 8 KiB. ⚠ No CI builds the post-office image. → `persistent-memory.d/2026-09-18-althing-363-hyphen-search.md` +- `[2026-09-18]` ⭐ **FleetTools: one capability index, autoloaded by Claude, Codex and Grok from a single symlinked file.** `docs/fleettools/` + `~/FLEETTOOLS.md`; absolute detail paths because a non-Claude agent cats them. Rule zero is query-live-inventories-never-a-written-list. The global CLAUDE.md tools section went 231 lines → 39, keeping only the three rules that govern behaviour rather than lookup. → `persistent-memory.d/2026-09-18-fleettools-agent-index.md` +- `[2026-09-18]` **Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy** (operator-approved). Verified empirically — `is_dir()`/`exists()` are stat-based and follow symlinks — and the copy had ALREADY drifted (README, since 09-15). Pre-restart gate is `uv run python scripts/sync_plugin.py --check`; `hermes plugins list` is confirmation-after, never permission-before. Copy kept at `~/backups/svos_miranda-copy-20260918-1444`. ⚠ The install now tracks a working tree — an uncommitted edit in svos is what loads at the next restart. +- `[2026-09-18]` **Worldtree's `env.sh` secrets are vaulted** — 10 entries under `worldtree/` (gitea, matrix as/hs, openai, uv-index, vastblueai, wt-admin demo+personal, z-ai, zai). ⚠ `Z_AI_API_KEY` and `ZAI_API_KEY` are DIFFERENT keys despite the near-identical names (fp `26bde3fa` vs `a4884152`). `ANTHROPIC_API_KEY` was empty and skipped. + - `[2026-09-17]` ⭐⭐ **The next voice seat was MEASURED, not chosen by taste — and the corpus size ranking INVERTS the voice ranking at the top.** Our two largest authors are Stephen King (76 works, 12.1M words) and Agatha Christie (72, 5.5M); neither should get a seat, Christie being the Krakauer failure mode exactly (genius in plot architecture, prose deliberately transparent, invisible to a char-bigram Delta). Picks, in order: **Faulkner** (~15 pure novels, ~1.6M words — highest voice signal in the catalogue, AND he is McCarthy's stylistic ancestor, so training him next supplies the **hard-negative sister the gate has lacked since the Brontë record named it missing**); **Morrison** (11 novels after pruning criticism/anthology, ~818k — 11 val units, beating Hemingway's 10); **Chandler** (7 novels + a 409k short-story omnibus — fills the first-person hardboiled gap, Hemingway-class corpus size). ⚠ Faulkner's catalogue rows carry a 446k-word Snopes omnibus that duplicates novels also present individually — the Hemingway 90-96% collection-duplication trap, needs the containment pass first. → `persistent-memory.d/2026-09-17-next-voice-seats.md` - `[2026-09-17]` ⭐⭐ **Romantasy IS a real register, we already trained its most distinctive member, and the obvious next pick is its worst.** Measured on the gate's own instrument (char-bigram Burrows's Delta, ~120k words/author from mid-work), with within-author floors and cross-genre positive controls. Cluster median pair **0.537 = 1.2x the worst floor** against controls at 1.4-1.9x — tighter than cross-genre but NOT collapsed. Two findings survive either floor reading: **Yarros is the cluster OUTLIER** (4 of the 5 largest pair distances involve her), so a second romantasy seat buys measurably less than the first did; and **Maas is the centroid** (the two smallest distances in the matrix are hers), so the obvious commercial pick is the least distinctive. If the lane gets a seat it is **Kenyon** — furthest from Yarros at 0.674 and **27 works = 27 val units, the best-powered gate the line could build** (Hemingway 10, McCarthy 6, Brontë 4). ⚠ Sensitivity floor stated: one sample per pair, no repeat draws; the rank ordering is indicative, fine gaps are not resolvable. → `persistent-memory.d/2026-09-17-romantasy-register-measured.md` - `[2026-09-17]` ⚠⚠ **`dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.** RDAP: registrar **eNom, LLC** (IANA 48, Tucows; abuse@enom.com), created 2008-10-30, status `active` **only** — no `clientTransferProhibited`. Registrant redacted (Verisign RDAP is thin, eNom's own endpoint 404s). Meanwhile `dragonfirepro.com` was re-registered 2025-09-27 by a Hungarian registrar and is now **for sale on expireddomains.com** — the customer lapsed it and a drop-catcher took it. ⭐ **This also makes the cert fix mandatory rather than tidy: two of the three SANs name a domain a THIRD PARTY now owns, so that request can never validate, and our Virtualmin has been retrying it often enough to get the Let's Encrypt account PAUSED.** Nobody is minding this domain — 13-month-dead cert, 403 homepage, sibling already gone — so the six-week clock is a real risk to an 18-year-old .com with a live site. Customer-facing; nothing touched. @@ -535,6 +465,9 @@ _5 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-09-18]` **Routing SearXNG's egress through a SOCKS5 proxy on esh-scale** — one day live, reverted. It fixed nothing, and the reason I gave for reverting it was itself wrong: the rollback was the counterfactual and it falsified my own published claim. Kept reverted on its own merits (no measurable gain, added a hard ESH dependency for all fleet search). → `persistent-memory.d/2026-09-18-searxng-esh-egress-reverted.md` + +- `[2026-09-18]` **`api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings** — this build has NO `!ENV` YAML constructor, so the file was unparseable and the container crash-looped ten times with fleet search down. There is no env-var path into that file at all: the loader reads only `SEARXNG_SETTINGS_PATH` and the entrypoint substitutes only `ultrasecretkey`. Literal or nothing. - `[2026-09-15]` ⚠⚠ **Probing OPNsense API endpoints by POSTing at them — one was `/api/core/system/reboot` and it took the FV site dark for 3.5 min.** Endpoints are ACTIONS; a 200 means it ran. The call I wanted was documented in this repo's own `docs/pfi/opnsense-api-reference.md`. → `persistent-memory.d/2026-09-15-opnsense-api-reboot.md` - `[2026-09-15]` **Advertising `10.100.10.50/32` from nh3-dev** to make its LAN address mesh-reachable — black-holed it from ESH/ANA/FV/IRV while its own LAN and the internet kept working, so a one-host check passes cleanly. `lookup 52` at rule priority 5270 beats `main` at 32766. Fix belongs at the router. → `persistent-memory.d/2026-09-15-nh3-dev-ts-input-masquerade.md`