memory: snapshot — three silent fleet faults found and fixed in one afternoon
An infrastructure day with no training work, and the through-line is that every fault was invisible to monitoring. NH3↔Anaheim had been crossing a throttled DERP relay rather than a direct path for long enough to carry 78 GB; `.internal` DNS was failing roughly one lookup in ten from two independent causes; SearXNG had exactly one working general web engine. Nothing alarmed on any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget and chose to measure rather than adapt around the problem. Also landed: althing v3.6.3, which makes hyphenated search work for the first time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a capability index autoloaded by Claude, Codex and Grok from one symlinked file; Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's env.sh secrets vaulted. Six detail files. The in-flight section is rewritten and shrinks 142 lines to 64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run outcome, which was left untouched and must not be assumed good. Two foot-guns recorded, both mine: the ESH egress experiment reverted on a diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which has no constructor in that build and crash-looped the container ten times. No archival this run. 165 of 169 dated entries are under the 14-day guard and the remaining four all carry open deferred pointers, so the index stays over the soft cap at 480 lines — an over-cap file that keeps live decisions beats a scannable one that lost one.
This commit is contained in:
@@ -1,16 +1,16 @@
|
||||
# Graph Report - eshpfi-management (2026-09-18)
|
||||
|
||||
## Corpus Check
|
||||
- 623 files · ~839,853 words
|
||||
- 623 files · ~840,412 words
|
||||
- Verdict: corpus is large enough that graph structure adds value.
|
||||
|
||||
## Summary
|
||||
- 5471 nodes · 5974 edges · 631 communities (477 shown, 154 thin omitted)
|
||||
- 5472 nodes · 5975 edges · 632 communities (478 shown, 154 thin omitted)
|
||||
- Extraction: 98% EXTRACTED · 2% INFERRED · 0% AMBIGUOUS · INFERRED: 97 edges (avg confidence: 0.81)
|
||||
- Token cost: 0 input · 0 output
|
||||
|
||||
## Graph Freshness
|
||||
- Built from commit: `d812bfe9`
|
||||
- Built from commit: `ee0fa1b7`
|
||||
- Run `git rev-parse HEAD` and compare to check if the graph is stale.
|
||||
- Run `graphify update .` after code changes (no API cost).
|
||||
|
||||
@@ -152,13 +152,13 @@
|
||||
- Recent decisions (archived 2026-08-03 batch)
|
||||
- [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit
|
||||
- `[2026-09-15]` FV cross-site routing fixed — one NAT rule scoped to Anaheim only
|
||||
- category
|
||||
- stream_chunks
|
||||
- build_gallery
|
||||
- lv-mccarthy — corpus → gate → pairs
|
||||
- 7. Critical Warnings by Model
|
||||
- think-leak — reproducers for the h300 unterminated-`<think>` defect
|
||||
- services.schema.json
|
||||
- method
|
||||
- CatalogReproducibility
|
||||
- test_invocation.py
|
||||
- scriberr — self-hosted transcription + diarization (fv-ml1, GPU1)
|
||||
- elway
|
||||
@@ -431,7 +431,7 @@
|
||||
- `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression
|
||||
- searxng-health.sh
|
||||
- searxng-mcp
|
||||
- content_type
|
||||
- adapter/server.py
|
||||
- pfi-gx10 — ERP-seat SFT run 3c, staged and ready to launch
|
||||
- 2026-09-05-floor-claim-n2-retraction.md
|
||||
- FastAPI
|
||||
@@ -444,11 +444,11 @@
|
||||
- `[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against the wrong number, twice in one hour
|
||||
- launch-run-03c.sh
|
||||
- `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early
|
||||
- fields
|
||||
- .submit
|
||||
- `[2026-09-04]` esh-nas SMB account for the AudioGridder box — and the NAS is effectively open to the whole LAN
|
||||
- remote-ssh
|
||||
- verify_corpus.py
|
||||
- catalog_version
|
||||
- max
|
||||
- booth
|
||||
- tok_repro.py
|
||||
- wrap_verbatim_html
|
||||
@@ -458,7 +458,7 @@
|
||||
- `[2026-09-10]` althing 3.6.2 rolled — post office + both heralds, and it was TWO nodes not seven
|
||||
- FV to Anaheim: scoped Tailscale NAT repair
|
||||
- setup-nh3-scale-socks-egress.sh
|
||||
- generated_at
|
||||
- max_length
|
||||
- 2026-09-10-beszel-fleet-wiring.md
|
||||
- bench_lora_step.py
|
||||
- probe_carrier.py
|
||||
@@ -552,7 +552,7 @@
|
||||
- irv-ml1 dead-address sweep — EXECUTED 2026-09-15 (operator-directed)
|
||||
- fv-mesh-watchdog.sh
|
||||
- grok-token-broker
|
||||
- accepted_types
|
||||
- mime
|
||||
- kb
|
||||
- 2026-09-10-an-author-voice-adapter-transfers-subject.md
|
||||
- 2026-09-10-babybronte-h02-adapter-the-voice-transferred.md
|
||||
@@ -578,7 +578,7 @@
|
||||
- breeze-tts sizing and the fv-ml1 GPU 0 placement question (2026-09-15)
|
||||
- 2026-09-15-client-abandon-cancellation-boundary.md
|
||||
- gender_by_proximity.py
|
||||
- streamable
|
||||
- mime_from_field
|
||||
- verify_quant.py
|
||||
- [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live
|
||||
- booth/app.py
|
||||
@@ -587,28 +587,29 @@
|
||||
- description
|
||||
- score_beats.py
|
||||
- Direct inference seats + the Asset Engine control plane
|
||||
- license_warning
|
||||
- optional
|
||||
- LiteLLM gateway — the fleet's OpenAI-compatible front door
|
||||
- Fleet SSH — the infra-ops identity and host map
|
||||
- build_beat_fixture.py
|
||||
- The Booth — ephemeral media board + operator decisions
|
||||
- lifecycle
|
||||
- options
|
||||
- min
|
||||
- properties
|
||||
- $ref
|
||||
- pin_known_gender.py
|
||||
- Speech — TTS and ASR endpoints
|
||||
- output_field
|
||||
- repair_corpus_bronte.py
|
||||
- 2026-09-16-grok-broker-shelved.md
|
||||
- 2026-09-17-lv-bronte-gate.md
|
||||
- required
|
||||
- type
|
||||
- Path
|
||||
- memorization_selfsim.py
|
||||
- eval-hemingway.sh
|
||||
- memorization_dump_matches.py
|
||||
- voice-prep.py
|
||||
- althing — inter-agent message bus
|
||||
- estimated_latency
|
||||
- host
|
||||
- version
|
||||
- sanitize_tts_text
|
||||
- apply_gender_overrides.py
|
||||
- rename.py
|
||||
- elway — SSH playbook runner (CHANGING things)
|
||||
@@ -647,7 +648,7 @@
|
||||
## Import Cycles
|
||||
- None detected.
|
||||
|
||||
## Communities (631 total, 154 thin omitted)
|
||||
## Communities (632 total, 154 thin omitted)
|
||||
|
||||
### Community 0 - "quant_modelopt.py"
|
||||
Cohesion: 0.53
|
||||
@@ -670,8 +671,8 @@ Cohesion: 0.07
|
||||
Nodes (29): ana-docker (VM 10.250.50.70), ana-filebot (CT 112 on pfi-pve) — file automation, ana-nas (CT 109 on pfi-pve, 10.250.50.50), ana-wg (CT 113 on pfi-pve) — WireGuard VPN, Cross-cutting: what to check FIRST for ambiguous outages, esh-docker-vm + vm-esh-nas, esh-pve + esh-pve-nas (10.0.250.35 + 10.0.50.55), Fleet disaster-recovery runbook (+21 more)
|
||||
|
||||
### Community 5 - "properties"
|
||||
Cohesion: 0.07
|
||||
Nodes (29): anyOf, default, title, anyOf, default, title, properties, anyOf (+21 more)
|
||||
Cohesion: 0.15
|
||||
Nodes (13): anyOf, default, title, anyOf, default, title, properties, audio_field (+5 more)
|
||||
|
||||
### Community 6 - "1. Locked architectural decisions"
|
||||
Cohesion: 0.07
|
||||
@@ -706,8 +707,8 @@ Cohesion: 0.11
|
||||
Nodes (17): 1. Create rest-server-ana htpasswd entry, 2. Install secrets on esh-vm-db, 3. Initialize the repo, 4. Install prerequisites (restic, resticprofile, mongosh client), 5. Deploy profile + hook, 6. Schedule + verify, Deploy (one-time), Full host config (+9 more)
|
||||
|
||||
### Community 14 - "properties"
|
||||
Cohesion: 0.08
|
||||
Nodes (24): properties, default, title, anyOf, default, anyOf, default, title (+16 more)
|
||||
Cohesion: 0.18
|
||||
Nodes (11): anyOf, default, title, properties, default, title, title, type (+3 more)
|
||||
|
||||
### Community 15 - "Deploy (one-time setup)"
|
||||
Cohesion: 0.12
|
||||
@@ -842,8 +843,8 @@ Cohesion: 0.22
|
||||
Nodes (9): properties, anyOf, default, title, cold_start_s, warm_per_unit, anyOf, default (+1 more)
|
||||
|
||||
### Community 48 - "CatalogModel"
|
||||
Cohesion: 0.14
|
||||
Nodes (14): additionalProperties, properties, required, title, type, CatalogModel, anyOf, default (+6 more)
|
||||
Cohesion: 0.12
|
||||
Nodes (17): additionalProperties, properties, required, title, type, CatalogModel, title, type (+9 more)
|
||||
|
||||
### Community 49 - "FieldOption"
|
||||
Cohesion: 0.13
|
||||
@@ -918,8 +919,8 @@ Cohesion: 0.25
|
||||
Nodes (7): backrest, Backup scope, when wired up, Deploy, First-time configuration (in the UI), Role in the fleet, Scaling knobs, Why not `offen/docker-volume-backup`?
|
||||
|
||||
### Community 67 - "omnivoice/app.py"
|
||||
Cohesion: 0.06
|
||||
Nodes (46): Any, ChunkConfig, ChunkResult, ClockFn, GenerateFn, _base_gen_kwargs(), _chunk_config(), GenParams (+38 more)
|
||||
Cohesion: 0.09
|
||||
Nodes (26): Any, ChunkConfig, ChunkResult, _base_gen_kwargs(), _chunk_config(), GenParams, _log_chunk(), _pcm16() (+18 more)
|
||||
|
||||
### Community 68 - "calibre-web-automated"
|
||||
Cohesion: 0.25
|
||||
@@ -998,8 +999,8 @@ Cohesion: 0.25
|
||||
Nodes (7): API, Deploy, Gotchas, Switching to the 7B variant, VibeVoice, Voices, Why this stack exists
|
||||
|
||||
### Community 87 - "properties"
|
||||
Cohesion: 0.15
|
||||
Nodes (13): properties, title, type, $ref, title, type, endpoint, model (+5 more)
|
||||
Cohesion: 0.05
|
||||
Nodes (42): properties, enum, title, type, default, title, type, title (+34 more)
|
||||
|
||||
### Community 88 - "Dia / Dia2"
|
||||
Cohesion: 0.29
|
||||
@@ -1090,8 +1091,8 @@ Cohesion: 0.40
|
||||
Nodes (4): Deeper context, Fleet bootstrap — read this first, Non-negotiables, Three endpoints that answer "what exists right now"
|
||||
|
||||
### Community 110 - "CatalogSectionGroup"
|
||||
Cohesion: 0.14
|
||||
Nodes (14): additionalProperties, description, properties, required, title, type, CatalogSectionGroup, anyOf (+6 more)
|
||||
Cohesion: 0.18
|
||||
Nodes (11): additionalProperties, description, properties, required, title, type, CatalogSectionGroup, anyOf (+3 more)
|
||||
|
||||
### Community 111 - "3. Per-model breakdowns"
|
||||
Cohesion: 0.06
|
||||
@@ -1118,8 +1119,8 @@ Cohesion: 0.29
|
||||
Nodes (7): 5.1. Create Debian VM on nh3-pve, 5.2. Install PBS, 5.3. Datastore backing — Synology NFS (chosen 2026-04-22), 5.4. Create datastore, 5.5. Create a verify job on the mirror, Phase 5 done-state, Phase 5 — Provision PBS-NH3 (DR mirror)
|
||||
|
||||
### Community 117 - "Headscale overlay mesh — plan (2026-09-06)"
|
||||
Cohesion: 0.10
|
||||
Nodes (20): 06:05Z — return path built and verified (operator approved 05:50Z), 06:20Z — first roaming client verified, 06:35Z — ESH egress whitelisted in CrowdSec (operator-directed, temporary), 2026-09-06 — exit nodes advertised (multi-location egress proxy), 2026-09-06 — irv-ml1 CUTOVER COMPLETE + dante retired (done remotely, 5-day no-access), 2026-09-06 — irv-ml1 folded in ADDITIVELY (operator inaccessible 5 days — lifeline preserved), 2026-09-06 — selective masquerade applied (supersedes the SNAT-reversal note above), 2026-09-06 — throughput baseline (mesh vs the old IPsec) (+12 more)
|
||||
Cohesion: 0.09
|
||||
Nodes (21): 06:05Z — return path built and verified (operator approved 05:50Z), 06:20Z — first roaming client verified, 06:35Z — ESH egress whitelisted in CrowdSec (operator-directed, temporary), 2026-09-06 — exit nodes advertised (multi-location egress proxy), 2026-09-06 — irv-ml1 CUTOVER COMPLETE + dante retired (done remotely, 5-day no-access), 2026-09-06 — irv-ml1 folded in ADDITIVELY (operator inaccessible 5 days — lifeline preserved), 2026-09-06 — selective masquerade applied (supersedes the SNAT-reversal note above), 2026-09-06 — throughput baseline (mesh vs the old IPsec) (+13 more)
|
||||
|
||||
### Community 118 - "Phase 0 — Prerequisites (one-time, before Phase 1)"
|
||||
Cohesion: 0.33
|
||||
@@ -1193,9 +1194,9 @@ Nodes (49): [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gb
|
||||
Cohesion: 0.08
|
||||
Nodes (20): `[2026-09-15]` FV cross-site routing fixed — one NAT rule scoped to Anaheim only, Fix, Root cause, ⭐ The diagnostic signature, so the next person skips the evening, Wrong turns worth not repeating, `[2026-09-15]` Break-glass mesh path on fv-ml1 (inverted from a restore-watchdog), Design choices that matter, ⭐ INVERTED the same night, on the operator's suggestion (+12 more)
|
||||
|
||||
### Community 137 - "category"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): enum, title, type, category
|
||||
### Community 137 - "stream_chunks"
|
||||
Cohesion: 0.19
|
||||
Nodes (17): ClockFn, GenerateFn, ChunkConfig, _ema(), _est_gen_time(), plan_chunk(), protect_first_audio(), Split into sentence units, preserving punctuation. Whitespace-collapsed. (+9 more)
|
||||
|
||||
### Community 138 - "build_gallery"
|
||||
Cohesion: 0.09
|
||||
@@ -1214,12 +1215,12 @@ Cohesion: 0.22
|
||||
Nodes (8): Did our abliteration cause it? No — the base did (~83% / ~17%), Resolution — Cold-Fusion abandoned, seat rolled back to `heresy` (2026-08-21), Scripts, The fix, The trigger is TEMPERATURE, not presence_penalty, think-leak — reproducers for the h300 unterminated-`<think>` defect, Using these on any future seat, What actually happens
|
||||
|
||||
### Community 142 - "services.schema.json"
|
||||
Cohesion: 0.25
|
||||
Nodes (7): additionalProperties, description, $id, required, $schema, title, type
|
||||
Cohesion: 0.10
|
||||
Nodes (19): additionalProperties, default, title, type, description, anyOf, default, title (+11 more)
|
||||
|
||||
### Community 143 - "method"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): default, title, type, method
|
||||
### Community 143 - "CatalogReproducibility"
|
||||
Cohesion: 0.12
|
||||
Nodes (17): additionalProperties, properties, required, title, type, CatalogReproducibility, title, type (+9 more)
|
||||
|
||||
### Community 144 - "test_invocation.py"
|
||||
Cohesion: 0.25
|
||||
@@ -1338,8 +1339,8 @@ Cohesion: 0.15
|
||||
Nodes (12): 1. The MTP head is abliterated *in-band*, 2. The vision tower is preserved byte-identical, Abliteration recipe — Qwen3.8-27B (MTP-aware, vision-preserving), How this maps onto our pipeline, Measured behaviour (their numbers, for reference), Provenance, Refusal-direction selection, The attention-sink dimension — the one that bricks the model (+4 more)
|
||||
|
||||
### Community 190 - "JobManager"
|
||||
Cohesion: 0.17
|
||||
Nodes (8): Popen, Path, Job, JobManager, _pid_alive(), Singleton owner of the one active job + retained history. Thread-safe., Start a train. Raises InvalidTrainRequest (→422) or Busy (→409)., Phase 2 publish step: COPY the succeeded LoRA into ComfyUI's loras search path a
|
||||
Cohesion: 0.20
|
||||
Nodes (7): Popen, Path, Job, JobManager, _pid_alive(), Singleton owner of the one active job + retained history. Thread-safe., Phase 2 publish step: COPY the succeeded LoRA into ComfyUI's loras search path a
|
||||
|
||||
### Community 193 - "LoRA training worker"
|
||||
Cohesion: 0.22
|
||||
@@ -1755,7 +1756,7 @@ Nodes (4): ask(), classify(), load_prompts(), main()
|
||||
|
||||
### Community 327 - "worker/app.py"
|
||||
Cohesion: 0.15
|
||||
Nodes (15): RuntimeError, get_gpu_status(), post_train(), FastAPI surface for the LoRA training worker (§4.1 API — arbo is the client). E, TrainRequest, _bus_id_by_index(), _cmdline(), gpu_status() (+7 more)
|
||||
Nodes (13): get_gpu_status(), FastAPI surface for the LoRA training worker (§4.1 API — arbo is the client). E, Static configuration for the LoRA training worker. Everything load-bearing is a, _bus_id_by_index(), _cmdline(), gpu_status(), _query_compute_apps(), _query_devices() (+5 more)
|
||||
|
||||
### Community 329 - "Beszel fleet monitoring"
|
||||
Cohesion: 0.33
|
||||
@@ -1786,8 +1787,8 @@ Cohesion: 0.25
|
||||
Nodes (7): built_at, corpus, normalisation, source, total_chapters, total_words, works
|
||||
|
||||
### Community 337 - "properties"
|
||||
Cohesion: 0.06
|
||||
Nodes (35): additionalProperties, properties, required, title, type, additionalProperties, properties, required (+27 more)
|
||||
Cohesion: 0.11
|
||||
Nodes (18): additionalProperties, properties, required, title, type, CatalogAuditEntry, title, type (+10 more)
|
||||
|
||||
### Community 338 - "archival-memory.md"
|
||||
Cohesion: 0.10
|
||||
@@ -2097,9 +2098,9 @@ Nodes (6): `[2026-08-19]` Four unmanaged stacks found on live hosts — and two
|
||||
Cohesion: 0.40
|
||||
Nodes (4): `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression, The lora_B signal worth carrying forward, The verdict (brokkr-smithy-dev, run03-gate/RESULT-run03c-gate.md), What the port proved about the box
|
||||
|
||||
### Community 440 - "content_type"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): default, title, type, content_type
|
||||
### Community 440 - "adapter/server.py"
|
||||
Cohesion: 0.29
|
||||
Nodes (7): _get_model(), OpenAI-ish /v1/audio/speech adapter in front of the Zonos Python SDK. Why this, _speaker_embedding(), speech(), SpeechRequest, JSONResponse, Zonos
|
||||
|
||||
### Community 441 - "pfi-gx10 — ERP-seat SFT run 3c, staged and ready to launch"
|
||||
Cohesion: 0.05
|
||||
@@ -2110,8 +2111,8 @@ Cohesion: 0.40
|
||||
Nodes (3): `[2026-09-05]` A peer's "2.7x stack effect" was a coin flip — the operator caught it and the arithmetic is worth keeping, `[2026-09-04/05]` vLLM RUNS on sm_121 — and the blocker was `ninja` not being on PATH, not the silicon, Run 4 — the corpus arm
|
||||
|
||||
### Community 443 - "FastAPI"
|
||||
Cohesion: 0.12
|
||||
Nodes (18): ACEStepInput, ACEStepOutput, generate_audio(), initialize_pipeline(), Generate music; respond with the audio bytes inline. Pre-2026-05-11 this re, ACEStepPipeline, _get_model(), OpenAI-ish /v1/audio/speech adapter in front of the Zonos Python SDK. Why this (+10 more)
|
||||
Cohesion: 0.16
|
||||
Nodes (13): ACEStepInput, ACEStepOutput, generate_audio(), initialize_pipeline(), Generate music; respond with the audio bytes inline. Pre-2026-05-11 this re, ACEStepPipeline, BaseModel, FastAPI (+5 more)
|
||||
|
||||
### Community 445 - "`[2026-09-15]` ESPHome modernised for ha-dev; `kb` search tool for the personal Worldtree KB"
|
||||
Cohesion: 0.50
|
||||
@@ -2137,17 +2138,17 @@ Nodes (3): `[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against
|
||||
Cohesion: 0.50
|
||||
Nodes (3): `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early, ⚠ IT DEGRADED, AND MY "PLATEAU" READ WAS WRONG, Method notes worth keeping
|
||||
|
||||
### Community 454 - "fields"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): items, title, type, fields
|
||||
### Community 454 - ".submit"
|
||||
Cohesion: 0.33
|
||||
Nodes (4): RuntimeError, Busy, Start a train. Raises InvalidTrainRequest (→422) or Busy (→409)., A second train was dispatched while one is active → HTTP 409.
|
||||
|
||||
### Community 456 - "remote-ssh"
|
||||
Cohesion: 0.50
|
||||
Nodes (3): SSH_MCP_STRICT_ALLOWLIST, node, remote-ssh
|
||||
|
||||
### Community 458 - "catalog_version"
|
||||
### Community 458 - "max"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): default, title, type, catalog_version
|
||||
Nodes (4): anyOf, default, title, max
|
||||
|
||||
### Community 462 - "wrap_verbatim_html"
|
||||
Cohesion: 0.22
|
||||
@@ -2173,13 +2174,13 @@ Nodes (6): `[2026-09-10]` althing 3.6.2 rolled — post office + both heralds, a
|
||||
Cohesion: 0.25
|
||||
Nodes (7): Backup and rollback, Broadened to the whole FV LAN, same session, Evidence and verification, ⭐ EXTENDED 2026-09-15 — NH3, ESH and Irvine now have the same rule, FV to Anaheim: scoped Tailscale NAT repair, Persistent configuration, What this looks like when it breaks, so the next person skips the evening
|
||||
|
||||
### Community 469 - "generated_at"
|
||||
### Community 469 - "max_length"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): anyOf, default, title, generated_at
|
||||
Nodes (4): anyOf, default, title, max_length
|
||||
|
||||
### Community 474 - "invocation.py"
|
||||
Cohesion: 0.14
|
||||
Nodes (17): Path, test_published_relative_path(), test_published_relative_path_explicit_train_id_wins(), ValueError, Static configuration for the LoRA training worker. Everything load-bearing is a, LoRA training worker — a host service (runs as llmuser on irv-ml1) that runs sd-, InvalidTrainRequest, published_relative_path() (+9 more)
|
||||
Cohesion: 0.19
|
||||
Nodes (14): Path, test_published_relative_path(), test_published_relative_path_explicit_train_id_wins(), ValueError, InvalidTrainRequest, published_relative_path(), Fixed-invocation command builder — the enforcement point for INV-T7. arbo hands, The ComfyUI-relative loras path for a succeeded LoRA (Phase 2 publish step). (+6 more)
|
||||
|
||||
### Community 483 - "recover_direction"
|
||||
Cohesion: 0.52
|
||||
@@ -2329,9 +2330,9 @@ Nodes (6): ⚠⚠ FOUR MORE LIVE BREAKAGES on OTHER hosts — the sweep's real f
|
||||
Cohesion: 0.29
|
||||
Nodes (6): Current state — ⛔ SHELVED, deliberately, 2026-09-16, For a consumer, grok-token-broker, Invariants, The problem, ⛔ Why the loop is disarmed until measured
|
||||
|
||||
### Community 564 - "accepted_types"
|
||||
### Community 564 - "mime"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): anyOf, default, title, accepted_types
|
||||
Nodes (4): anyOf, default, title, mime
|
||||
|
||||
### Community 582 - "build_corpus_hemingway.py"
|
||||
Cohesion: 0.20
|
||||
@@ -2353,9 +2354,9 @@ Nodes (8): breeze-tts sizing and the fv-ml1 GPU 0 placement question (2026-09-15
|
||||
Cohesion: 0.31
|
||||
Nodes (9): decide(), load_text(), main(), Path, D2b for BabyHemingway: resolve entity gender by MAJORITY VOTE over nearby…, One pass over the corpus for EVERY surface at once. The obvious shape -- rescan…, Score a name's local pronoun mix AGAINST THE CORPUS BASE RATE, not against…, score() (+1 more)
|
||||
|
||||
### Community 590 - "streamable"
|
||||
### Community 590 - "mime_from_field"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): streamable, default, title, type
|
||||
Nodes (4): anyOf, default, title, mime_from_field
|
||||
|
||||
### Community 591 - "verify_quant.py"
|
||||
Cohesion: 0.40
|
||||
@@ -2381,9 +2382,9 @@ Nodes (7): load(), main(), per_seed(), Score beat arms against a decision rule f
|
||||
Cohesion: 0.33
|
||||
Nodes (5): Asset Engine — `http://10.250.50.70:8200`, Direct inference seats + the Asset Engine control plane, GPU hosts, ⚠ Query the live inventory, never a written list, Traps
|
||||
|
||||
### Community 599 - "license_warning"
|
||||
### Community 599 - "optional"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): anyOf, default, title, license_warning
|
||||
Nodes (4): default, title, type, optional
|
||||
|
||||
### Community 600 - "LiteLLM gateway — the fleet's OpenAI-compatible front door"
|
||||
Cohesion: 0.33
|
||||
@@ -2397,41 +2398,45 @@ Nodes (5): Fleet SSH — the infra-ops identity and host map, Identity, Mesh, Si
|
||||
Cohesion: 0.40
|
||||
Nodes (4): Ask the operator to decide (no chat round-trip), Show the operator something, Standing convention — the link board, The Booth — ephemeral media board + operator decisions
|
||||
|
||||
### Community 604 - "lifecycle"
|
||||
Cohesion: 0.67
|
||||
Nodes (3): anyOf, default, lifecycle
|
||||
### Community 604 - "options"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): anyOf, default, title, options
|
||||
|
||||
### Community 605 - "min"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): anyOf, default, title, min
|
||||
|
||||
### Community 606 - "properties"
|
||||
Cohesion: 0.15
|
||||
Nodes (14): $ref, properties, reproducibility_audit, section_groups, services, items, title, type (+6 more)
|
||||
### Community 606 - "$ref"
|
||||
Cohesion: 0.14
|
||||
Nodes (14): items, title, type, $ref, fields, section_groups, services, items (+6 more)
|
||||
|
||||
### Community 608 - "Speech — TTS and ASR endpoints"
|
||||
Cohesion: 0.40
|
||||
Nodes (4): ASR, Speech — TTS and ASR endpoints, Traps, TTS
|
||||
|
||||
### Community 609 - "output_field"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): anyOf, default, title, output_field
|
||||
|
||||
### Community 612 - "2026-09-17-lv-bronte-gate.md"
|
||||
Cohesion: 0.05
|
||||
Nodes (41): `[2026-09-16]` lv-hemingway corpus: half the work was EXCLUSION, and the gate found what a hand count would not, `[2026-09-16]` The lv-* voice line: Option C proved, lv-yarros shipped, lv-hemingway training, `[2026-09-16]` voices-seat: LoRA over merge, measured — and GPU 0 is now full, `[2026-09-17]` The beat-writing model restores the author's real names — a leak the corpus gate structurally cannot see, `[2026-09-17]` lv-bronte: corpus gated for real, adapter trained, SHIPPED with a FAILED voice axis on the record, ⚠ A DEFECT IN THE v2 RULE ITSELF, worth fixing before the next author, ⚠⚠ AMENDED 2026-09-17 — THE VOICE AXIS PASSES UNDER THE CORRECTED FLOOR RULE, Artefacts (+33 more)
|
||||
|
||||
### Community 613 - "required"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): required, default, title, type
|
||||
|
||||
### Community 614 - "type"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): type, enum, title, type
|
||||
|
||||
### Community 620 - "althing — inter-agent message bus"
|
||||
Cohesion: 0.50
|
||||
Nodes (3): althing — inter-agent message bus, Commands, Traps
|
||||
|
||||
### Community 621 - "estimated_latency"
|
||||
Cohesion: 0.67
|
||||
Nodes (3): anyOf, default, estimated_latency
|
||||
|
||||
### Community 622 - "host"
|
||||
Cohesion: 0.67
|
||||
Nodes (3): title, type, host
|
||||
|
||||
### Community 623 - "version"
|
||||
Cohesion: 0.67
|
||||
Nodes (3): version, title, type
|
||||
### Community 621 - "sanitize_tts_text"
|
||||
Cohesion: 0.50
|
||||
Nodes (3): Language-safe text sanitizer for the OmniVoice TTS wrapper. Strips artifacts th, Return ``text`` cleaned of artifacts that shouldn't be spoken, language-safe., sanitize_tts_text()
|
||||
|
||||
### Community 625 - "rename.py"
|
||||
Cohesion: 0.60
|
||||
@@ -2450,7 +2455,7 @@ Cohesion: 0.25
|
||||
Nodes (8): _cli(), Regression: the branch ended with `[ "$n" -eq 0 ] && echo ...`, so it returned…, Regression: ids are 8 hex chars and roughly one in forty is all digits. Those…, test_cli_links_exits_zero_on_a_NON_empty_board(), test_cli_unlink_by_id_even_when_the_id_is_all_digits(), test_cli_unlink_by_index(), test_cli_unlink_of_a_stale_id_leaves_the_board_alone(), test_cli_unlink_rejects_a_non_id_non_index()
|
||||
|
||||
## Knowledge Gaps
|
||||
- **3021 isolated node(s):** `⚠ YOU ARE `infra-ops` — the fleet operations agent`, `Persistent memory`, `Model quantization`, `Training runs`, `Purpose` (+3016 more)
|
||||
- **3022 isolated node(s):** `Why Headscale (decided 2026-09-06)`, `Site facts that drive placement`, `Topology`, `Access & credentials — all in hand (verified 2026-09-06)`, `Phases` (+3017 more)
|
||||
These have ≤1 connection - possible missing edges or undocumented components.
|
||||
- **154 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
|
||||
|
||||
@@ -2458,16 +2463,16 @@ Nodes (8): _cli(), Regression: the branch ended with `[ "$n" -eq 0 ] && echo ...
|
||||
_Questions this graph is uniquely positioned to answer:_
|
||||
|
||||
- **Why does `create_app()` connect `booth/app.py` to `FastAPI`, `test_booth.py`, `Path`?**
|
||||
_High betweenness centrality (0.003) - this node is a cross-community bridge._
|
||||
- **Why does `main()` connect `abliterate.py` to `convert_hf_to_native.py`, `Path`?**
|
||||
_High betweenness centrality (0.002) - this node is a cross-community bridge._
|
||||
_High betweenness centrality (0.004) - this node is a cross-community bridge._
|
||||
- **Why does ``[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c` connect ``[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c` to `archival-memory.md`?**
|
||||
_High betweenness centrality (0.002) - this node is a cross-community bridge._
|
||||
- **What connects `⚠ YOU ARE `infra-ops` — the fleet operations agent`, `Persistent memory`, `Model quantization` to the rest of the system?**
|
||||
_3021 weakly-connected nodes found - possible documentation gaps or missing edges._
|
||||
- **Why does `CatalogService` connect `$defs` to `properties`?**
|
||||
_High betweenness centrality (0.001) - this node is a cross-community bridge._
|
||||
- **What connects `Why Headscale (decided 2026-09-06)`, `Site facts that drive placement`, `Topology` to the rest of the system?**
|
||||
_3022 weakly-connected nodes found - possible documentation gaps or missing edges._
|
||||
- **Should `Status + Open Issues` be split into smaller, more focused modules?**
|
||||
_Cohesion score 0.05555555555555555 - nodes in this community are weakly interconnected._
|
||||
- **Should `Fleet disaster-recovery runbook` be split into smaller, more focused modules?**
|
||||
_Cohesion score 0.06666666666666667 - nodes in this community are weakly interconnected._
|
||||
- **Should `properties` be split into smaller, more focused modules?**
|
||||
_Cohesion score 0.06896551724137931 - nodes in this community are weakly interconnected._
|
||||
- **Should `1. Locked architectural decisions` be split into smaller, more focused modules?**
|
||||
_Cohesion score 0.07407407407407407 - nodes in this community are weakly interconnected._
|
||||
@@ -0,0 +1,80 @@
|
||||
# [2026-09-18] althing search returned ZERO for every hyphenated query — silently, for everyone
|
||||
|
||||
Routed in by heid on behalf of peedlar-dev as "a message reached storage but never
|
||||
entered the inbox unread-set or search index". Two symptoms, and **neither was what
|
||||
was reported**.
|
||||
|
||||
## Symptom 1 — "never delivered": NOT A DEFECT
|
||||
|
||||
Message 2571 was fully intact: recipients row present, in the FTS index, `read_at`
|
||||
set. **Zero messages fleet-wide have no recipient row — 0 of 2583.**
|
||||
|
||||
Grouping the reader's `read_at` values gives one group per `postbox read`
|
||||
transaction, and the second consumed **2570 AND 2571 together**. `postbox read`
|
||||
advances over exactly what it printed, so that read printed both.
|
||||
|
||||
⚠ **The evidence was truncated.** Three of four tee'd files sat at exact 4 KiB
|
||||
multiples ending mid-sentence; `read-2-of-3.md` was 8,192 bytes while message 2570's
|
||||
body ALONE is 64,975. It held under 13% of 2570 and never reached 2571. That read
|
||||
emitted ≥127 KB and 8 KB survived. The reasoning — "a read that included 2571 would
|
||||
have printed it into one of these files" — was sound on a premise that failed.
|
||||
|
||||
## Symptom 2 — A REAL DEFECT, and far wider than one message
|
||||
|
||||
**Any query containing a hyphen returned zero hits, silently, for every handle.**
|
||||
|
||||
'nh3-docker' -> OperationalError: no such column: docker
|
||||
'"nh3-docker"' -> 20 rows
|
||||
'tts-dev' -> OperationalError: no such column: dev
|
||||
'"tts-dev"' -> 47 rows
|
||||
|
||||
FTS5 reads the hyphen as a column filter and tries to resolve the trailing token as
|
||||
a column name. `store.py` passed the caller's query to `MATCH` unescaped; `search()`
|
||||
caught the `OperationalError`, ran its probe (which succeeded — the store was
|
||||
healthy), and returned `[]`, indistinguishable from "no results".
|
||||
|
||||
On this fleet that is most proper nouns we own: `nh3-docker`, `irv-ml1`,
|
||||
`esh-docker-vm`, `fv-ml1`, `tts-dev`, `peedlar-dev`.
|
||||
|
||||
⚠ **The discriminator was working as designed.** It correctly separated
|
||||
store-broken from caller-typo. The defect was that it classified an ordinary
|
||||
hyphenated query as a typo.
|
||||
|
||||
## The fix, and the boundary that held
|
||||
|
||||
Routed to forseti (althing's owner) rather than patched — infra-ops owns the
|
||||
**rollout**, not the codebase. forseti implemented on operator instruction and
|
||||
shipped **v3.6.3** (`5de41b7`): valid FTS syntax intact, `OperationalError` retries
|
||||
once as an escaped literal phrase before the health probe. Plus a read-summary line
|
||||
(`consumed N messages: 2570, 2571`) that makes the truncation case detectable.
|
||||
|
||||
⚠ **I twice declined to act on relayed authorization** — "Vuong requested deployment"
|
||||
arriving through a peer is not the operator instructing me, and a post-office bounce
|
||||
takes the bus down for every handle. Got it directly, then executed.
|
||||
|
||||
## Rollout mechanics worth keeping
|
||||
|
||||
- **No CI builds the post-office image.** `deploy.yaml` ships althing-*chamber* to
|
||||
ana-docker; `ci.yaml` runs tests and the frontend. The root `Dockerfile` IS the
|
||||
post office and nothing publishes it — a build-and-push step, not a pull.
|
||||
- Built from `git archive v3.6.3`, **not a working tree**; archive hash verified
|
||||
identical after transfer. My wheel came out byte-identical to forseti's.
|
||||
- Bus outage **19:13:40 → 19:13:57 UTC, 17 seconds**.
|
||||
- Deployed digest `sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4`;
|
||||
rollback pin kept as `compose.yaml.bak-3.6.2`.
|
||||
- ⚠ **Canonical `stacks/althing-post-office/compose.yaml` had to be re-pinned too.**
|
||||
forseti caught it: hand-deploying leaves canonical intent stale, so the next routine
|
||||
`deploy-stack.sh` would have silently ROLLED THE BUS BACK. Commit `670ac9e`.
|
||||
|
||||
## Still open — two postbox fixes proposed to forseti
|
||||
|
||||
1. **`--body -` is taken as a literal hyphen, not stdin.** Four real messages lost to
|
||||
it today (tts-dev ×3, ledger-dev earlier). Store audit: 5 of 2,637 messages have a
|
||||
one-character body and **every accidental one is a literal `-`**; the fifth is a
|
||||
deliberate `'x'` probe. forseti's scope: reserve `--body -` for stdin in both verbs.
|
||||
**Awaiting operator.** ⚠ Empty bodies are already rejected (PRE-306, verified) — no
|
||||
minimum-length rule is needed.
|
||||
2. A bounded consuming read — **withdrawn**, and rightly. A message-count limit does
|
||||
not bound one very large body, so it would not have saved peedlar-dev.
|
||||
|
||||
Until it ships, the mitigation is entirely sender-side: pipe the body, never `--body`.
|
||||
@@ -0,0 +1,94 @@
|
||||
# [2026-09-18] `.internal` DNS was failing ~10% of lookups — two causes, both fleet-wide
|
||||
|
||||
Two independent faults, fixed in order. Together they were costing roughly one
|
||||
`.internal` lookup in ten either a hard failure or a five-second stall, on every
|
||||
DHCP client at every site.
|
||||
|
||||
## Fault 1 — a PUBLIC resolver was the fallback for a PRIVATE zone
|
||||
|
||||
DHCP handed out `10.100.50.40, 1.1.1.1` on both NH3 VLANs. When the internal
|
||||
resolver missed a packet, the resolver waited out its timeout, fell through to
|
||||
Cloudflare, and got an **authoritative NXDOMAIN** — so a transient miss became a
|
||||
definitive "no such host" rather than a retry. Binary failure: instant, or five
|
||||
seconds then `gaierror`.
|
||||
|
||||
**Anaheim was worse.** Its FortiGate handed clients *itself* as resolver and
|
||||
forwarded to `1.1.1.1`/`1.0.0.1`, so **no ANA host could resolve `.internal` at
|
||||
all** — `ana-docker`, which HOSTS the Anaheim AdGuard, had `1.1.1.1` in its own
|
||||
`resolv.conf`.
|
||||
|
||||
⚠ **`/etc/resolv.conf` on these hosts is DHCP-managed** (dhclient on ens18). Hand
|
||||
editing it survives exactly until the next lease renewal — tts-dev's original
|
||||
framing was "fix nh3-dev's resolv.conf", which would have worked, been verified,
|
||||
and then silently reverted. Change it at the source:
|
||||
|
||||
- **UniFi**: `dhcpd_dns_1` / `dhcpd_dns_2` per network object.
|
||||
- **FortiGate**: `config system dhcp server`, `set dns-service specify` +
|
||||
`dns-server1` / `dns-server2`.
|
||||
- Then `sudo dhclient -1 -v ens18` to pick it up **without releasing** the lease.
|
||||
|
||||
**Operator's ruling was broader than the proposal**: each site's backup resolver
|
||||
should be *another site's* resolver. ESH already worked this way; the other two
|
||||
were set, and the ring closes.
|
||||
|
||||
ESH primary 10.0.50.45 backup 10.100.50.40 (NH3) [pre-existing]
|
||||
NH3 primary 10.100.50.40 backup 10.250.50.70 (ANA) [changed]
|
||||
ANA primary 10.250.50.70 backup 10.0.50.45 (ESH) [changed]
|
||||
|
||||
All six members verified to answer the private zone. ⚠ The SureFire tenant DHCP
|
||||
server (`sfsrv`) and the fortilink switch-management server were deliberately left
|
||||
alone — tenant property.
|
||||
|
||||
**Result: hard failures 3/40 → 0/40. The 5 s stalls remained.**
|
||||
|
||||
## Fault 2 — THE ROOT CAUSE: AdGuard rate-limiting, identical on all three
|
||||
|
||||
ratelimit: 20
|
||||
ratelimit_subnet_len_ipv4: 24
|
||||
ratelimit_whitelist: []
|
||||
|
||||
**20 queries per second shared across an entire /24 — per subnet, not per client.**
|
||||
Every host on a VLAN draws from one bucket, so one busy container starves the rest,
|
||||
and anything over the line is **silently dropped**, costing the client its full 5 s
|
||||
resolver timeout.
|
||||
|
||||
Demonstrated rather than inferred — 60 concurrent queries at one resolver:
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| median | 5044 ms | 72 ms |
|
||||
| timed out (>4 s) | **40 of 60** | **0 of 60** |
|
||||
|
||||
Set to `ratelimit: 0` on all three. These are LAN-only resolvers behind three
|
||||
firewalls; that setting exists to blunt internet-side DNS amplification, which is
|
||||
not this.
|
||||
|
||||
⚠ **Restarting AdGuard is required for a config change and briefly drops that site's
|
||||
primary resolver — do the three ONE AT A TIME** so the cross-site ring always has a
|
||||
live member. That the ring made this rollout safe is the point of having built it
|
||||
first, an hour earlier.
|
||||
|
||||
## End-to-end, the original symptom
|
||||
|
||||
40 TCP connects to a service by name vs by IP; the by-IP arm is the control.
|
||||
|
||||
| | original | after ring only | after ratelimit 0 |
|
||||
|---|---|---|---|
|
||||
| by NAME p90 | 120 ms | 5017 ms | **13.5 ms** |
|
||||
| by NAME >1 s | 3/40 | 5/40 | **0/40** |
|
||||
| by NAME failed | 2/40 | 0/40 | **0/40** |
|
||||
| by IP p90 (control) | 25 ms | — | 11.7 ms |
|
||||
|
||||
tts-dev confirmed from talk's own request path: name and IP **1.5 ms apart at the
|
||||
median**. Resolution has stopped being a cost rather than become a smaller one.
|
||||
|
||||
## What this says about monitoring
|
||||
|
||||
Nothing on this fleet watches DNS success rate. The fleet's actual DNS monitor
|
||||
today was a voice loop with a 1545 ms budget whose owner chose to measure instead
|
||||
of adapting around the problem. Both faults were invisible to Beszel and Uptime
|
||||
Kuma — every host was up, every service answered.
|
||||
|
||||
Related: [[2026-09-18-nh3-ana-derp-relay]] — the ANA AdGuard is only a sensible
|
||||
cross-site backup because that fix took NH3→ANA from ~1.2 s to ~15 ms the same
|
||||
afternoon. The two changes compose.
|
||||
@@ -0,0 +1,57 @@
|
||||
# [2026-09-18] FleetTools — one capability index, autoloaded by three agent families
|
||||
|
||||
Every agent on nh3-dev needs the same answers — what runs here, how do I call it,
|
||||
what will bite me — but that knowledge lived in `~/.claude/CLAUDE.md`, which only
|
||||
Claude sessions load, interleaved with operator preferences other families have no
|
||||
use for.
|
||||
|
||||
## Shape
|
||||
|
||||
**Two-tier, matching the persistent-memory split.** `FLEETTOOLS.md` is a ~135-line
|
||||
index an agent reads whole; each entry links to a detail file it opens only when it
|
||||
needs that tool. Reading the index costs about a fifth of reading the tree.
|
||||
|
||||
- Canonical: `docs/fleettools/` in this repo (git-tracked)
|
||||
- Discovery: `~/FLEETTOOLS.md` → symlink to the index
|
||||
- **Detail links are ABSOLUTE paths**, not relative markdown — a non-Claude agent
|
||||
`cat`s the path rather than following a link, and may be anywhere on the filesystem.
|
||||
|
||||
Covers: althing, the Booth, `secret`, LiteLLM, inference seats + Asset Engine,
|
||||
speech, Arbo, elway, fleet SSH, observability, graphify, Playwright, the Henge, and
|
||||
a segregated Claude-only file (ratecheck, remote-ssh MCP, task-board, skills).
|
||||
|
||||
## Autoload — verified per family, not assumed
|
||||
|
||||
- **Codex** reads `$CODEX_HOME/AGENTS.md` — the binary carries the string
|
||||
`"Failed to read global AGENTS.md instructions from"`.
|
||||
- **Grok** always scans `$GROK_HOME/rules/` and loads **every `*.md` in it regardless
|
||||
of name** — from its own embedded docs.
|
||||
|
||||
Both locations were empty. `AGENT-BOOTSTRAP.md` is **symlinked** into each rather than
|
||||
copied — one file, three families, no drift surface.
|
||||
|
||||
⚠ Grok also scans `~/.claude/rules/` and recognises `CLAUDE.md` via harness-compat
|
||||
settings. That directory is empty today, but Grok can see Claude's config.
|
||||
|
||||
## Rule zero, which is the load-bearing part
|
||||
|
||||
**Query live inventories, never a written list.** The index points at
|
||||
Homepage `/api/services`, asset-engine `/api/v1/services`, LiteLLM `/v1/models`, and
|
||||
every FastAPI seat's `/openapi.json`. A copied service table would be stale within a
|
||||
month — the LiteLLM roster was already at 40 models where the old CLAUDE.md note said
|
||||
~30.
|
||||
|
||||
## The CLAUDE.md swap
|
||||
|
||||
`~/.claude/CLAUDE.md`'s "Global tools available" section went from **231 lines to 39**
|
||||
(file 1218 → 1026, a 16% cut to what every Claude session loads). Three things stayed
|
||||
inline deliberately because they govern BEHAVIOUR rather than lookup: run `ratecheck`
|
||||
before expensive work, post operator-facing links to the Booth board, and the vault is
|
||||
the credential source of truth. **A rule behind a file read is a rule that stops
|
||||
firing.** Backup at `~/.claude/CLAUDE.md.bak-20260918-074048`.
|
||||
|
||||
The shared all-agents LiteLLM key stays inline there too — every session needs it and a
|
||||
vault round-trip measured over two minutes — but it is now ALSO vaulted at
|
||||
`litellm/all-agents-shared-key`, since it had been single-copy.
|
||||
|
||||
Commits `53c3e80` `21d24c5`.
|
||||
@@ -0,0 +1,91 @@
|
||||
# [2026-09-18] NH3↔Anaheim was DERP-relayed, not direct — a UDP 41641 port-forward on ana-gw fixed it
|
||||
|
||||
Every NH3→Anaheim flow had been crossing Tailscale's LA DERP relay rather than a
|
||||
direct path, long enough to have carried **78 GB tx on the NH3 side alone**. DERP
|
||||
is a deliberately throttled fallback — a reachability mechanism, not a data plane —
|
||||
so this imposed both a fixed round-trip penalty and a bandwidth ceiling on LiteLLM,
|
||||
Beszel, task-board, vor and the Henge alike.
|
||||
|
||||
## How it surfaced, and how it did NOT
|
||||
|
||||
tts-dev reported ext-stt transcriptions 4–18× slower through the LiteLLM gateway
|
||||
than straight at the Parakeet host, with a ~1.9 s fixed floor plus a per-byte cost.
|
||||
They had already cleared GPU contention, the ASR itself and output quality with
|
||||
real controls.
|
||||
|
||||
**No monitor caught it.** Beszel had every host green. Nothing on this fleet checks
|
||||
whether a mesh path is direct.
|
||||
|
||||
## The diagnosis, and two wrong causes published on the way
|
||||
|
||||
Their hypothesis was a dead first target in a LiteLLM fallback list. Checked: `ext-stt`
|
||||
has exactly ONE deployment, no fallback, nothing to retry against. Langfuse looked like
|
||||
a great candidate for a fixed floor — env vars still set, container gone — but it was
|
||||
retired from the callback list 2026-06-20 and nothing calls it per-request.
|
||||
|
||||
The decisive arm was one neither of us had run: **measure from ana-docker itself**, so
|
||||
both arms share the identical onward path to FV.
|
||||
|
||||
ana-docker -> fv-ml1 DIRECT 0.289 / 0.218 / 0.273 / 0.220 s
|
||||
ana-docker -> gateway -> fv-ml1 0.463 / 0.211 / 0.222 / 0.222 s
|
||||
|
||||
The gateway adds ~0. tts-dev measured 2618 ms for that clip; from the gateway's own
|
||||
host it is 220 ms. Their two arms differed in TWO things — whether LiteLLM was in the
|
||||
path AND which destination the packets crossed the network to reach — and the entire
|
||||
delta was attributed to the variable of interest.
|
||||
|
||||
nh3-dev -> fv-ml1 direct 0.269 / 0.219 / 0.216 s
|
||||
nh3-dev -> ANA gateway 1.449 / 1.399 / 1.404 s
|
||||
|
||||
nh3-scale$ tailscale ping 100.64.0.3
|
||||
pong via DERP(lax) in 507ms, 518, 521, 522, 373, 423, 464, 493
|
||||
direct connection not established
|
||||
|
||||
## Why hole-punching failed
|
||||
|
||||
ana-scale advertised `38.120.12.42:41641`, but its netcheck mapped to `:60798` — the
|
||||
Anaheim NAT was not preserving the port — and `PortMapping` was empty, so no UPnP or
|
||||
NAT-PMP was establishing one. `MappingVariesByDestIP: false`, so not symmetric NAT;
|
||||
simply no reachable inbound endpoint. ESH↔Anaheim and NH3↔ESH were already direct;
|
||||
only this pair failed.
|
||||
|
||||
## The fix — four ADDITIVE objects on ana-gw (10.250.0.1)
|
||||
|
||||
⚠ **This box runs `central-nat enable`, so a policy's `dstaddr` is the REAL internal
|
||||
address and not the VIP.** The existing `wg-to-ana-wg` VIP+policy pair is the house
|
||||
template; writing what a FortiGate normally wants would have failed.
|
||||
|
||||
| object | value |
|
||||
|---|---|
|
||||
| `firewall address` | `ana-scale-ip` → 10.250.50.45/32, iface `servers` |
|
||||
| `firewall service custom` | `Tailscale-41641` → `udp-portrange 41641` |
|
||||
| `firewall vip` | `tailscale-to-ana-scale` → 38.120.12.42:41641 udp → 10.250.50.45:41641, extintf wan1 |
|
||||
| `firewall policy` id 75 | wan1→servers, all→`ana-scale-ip`, `Tailscale-41641`, accept |
|
||||
|
||||
## Measured before → after
|
||||
|
||||
| path | before | after |
|
||||
|---|---|---|
|
||||
| `tailscale ping` nh3-scale→ana-scale | 373–522 ms via DERP(lax) | **6 ms direct** |
|
||||
| STT via the ANA gateway (96 kB clip) | 1.399–1.449 s | **0.237–0.270 s** |
|
||||
| Beszel HTTP nh3-dev→ana-docker | 0.94–1.29 s | **0.014–0.016 s** |
|
||||
|
||||
tts-dev confirmed independently from talk's own vantage: 2618 → 249 ms and 4119 →
|
||||
548 ms, converged with their direct arm.
|
||||
|
||||
## Operational notes for next time
|
||||
|
||||
⚠ **This edge has NO out-of-band access.** Back up first — `ssh infra-ops@10.250.0.1
|
||||
'show' > file` captures the full non-default config (11,320 lines) without needing a
|
||||
tftp server; kept at `~/backups/ana-gw-config-BEFORE-<stamp>.txt`. Credentials vaulted
|
||||
at `fortigate/ana-gw-infra-ops-password`, reached with
|
||||
`sshpass -e ssh -o PubkeyAuthentication=no`. **Additive objects only** — never edit an
|
||||
existing policy or interface over a 327 ms link you cannot recover.
|
||||
|
||||
⚠ **irv-ml1 (100.64.0.6) remains `relay "lax"`.** Same class, different site, not fixed
|
||||
by this. Measured 24-25 ms on HTTP from nh3-dev, so it is not costing what Anaheim was.
|
||||
I warned tts-dev their TTS stack was paying for it; they measured and I was wrong.
|
||||
|
||||
Documented in `docs/pfi/headscale-mesh-plan.md`; commit `5a9fad8`. See
|
||||
[[2026-09-18-fleet-dns-ring-and-ratelimit]] — the ANA AdGuard only became a viable
|
||||
cross-site DNS backup because of this fix.
|
||||
@@ -0,0 +1,47 @@
|
||||
# [2026-09-18] SearXNG's ESH SOCKS5 egress — one day live, reverted, and it fixed nothing
|
||||
|
||||
On 2026-09-17, at operator request, SearXNG's search egress was moved from
|
||||
nh3-docker's direct residential path to `socks5h://10.0.50.65:1080` on **esh-scale**
|
||||
(CT 108 on esh-pve) — an application-level proxy, no host route or exit-node change.
|
||||
Reverted 2026-09-18.
|
||||
|
||||
## Why it was reverted, and why the STATED reason was wrong
|
||||
|
||||
It was reverted on a diagnosis that turned out to be false: that the move had cost
|
||||
three of four search engines to CAPTCHAs. Reverting to direct NH3 egress produced a
|
||||
**byte-identical** result — same three engines down — which falsified it. A live
|
||||
`!ddg` bang probe on a freshly restarted container also CAPTCHA'd, ruling out stale
|
||||
suspension timers too.
|
||||
|
||||
The real cause was elsewhere (see [[2026-09-18-searxng-one-engine-to-seven]]).
|
||||
|
||||
**The revert still stands on its own merits**: ESH egress bought no measurable
|
||||
improvement while making all fleet search depend on ESH WAN and mesh availability.
|
||||
The simpler configuration is the better one. It is simply not the fix for the engines.
|
||||
|
||||
## What was built, because it was built correctly
|
||||
|
||||
The proxy side was not at fault and is worth keeping as a pattern. `microsocks` ran
|
||||
as `nobody` under `searxng-egress.service`, bound `10.0.50.65:1080` only, and allowed
|
||||
source `10.100.50.40` alone — everything else had to supply a password regenerated at
|
||||
each start and never distributed. Allowed-host egress and denied-host rejection were
|
||||
both tested. The unit is kept at `configs/esh-scale/searxng-egress.service`; the
|
||||
service is **stopped and disabled** on esh-scale, and `tailscaled` there was not
|
||||
touched (CT 108 is ESH's whole-site mesh SPOF).
|
||||
|
||||
Measured egress was `128.177.138.182` in use and `154.50.58.126` at cutover — ESH's
|
||||
WAN address moves and nothing pins it, so the number in the stack README is historical.
|
||||
|
||||
## The transferable lesson
|
||||
|
||||
⚠ **I asserted causation from correlation with no baseline.** The only evidence that
|
||||
residential egress avoided CAPTCHAs was a config comment dated 2026-09-03, which was
|
||||
no longer true of that address. Five samples of the post-change state and zero of the
|
||||
working state is not a comparison. The rollback WAS the counterfactual, and it
|
||||
falsified the claim I had already published in a commit message.
|
||||
|
||||
This was the first of three wrong causal attributions in a single afternoon. The
|
||||
common shape: measure after a change, attribute the delta to *my* change, never check
|
||||
what else moved.
|
||||
|
||||
Commits `156e126` (applied), `1a35181` (reverted).
|
||||
@@ -0,0 +1,81 @@
|
||||
# [2026-09-18] SearXNG had ONE working general web engine — now seven, and the health script never noticed
|
||||
|
||||
The fleet `web_search` MCP tool every nh3-dev session uses was answering from
|
||||
`google cse` alone, and had been for at least a day, with every instrument green.
|
||||
|
||||
## The root cause, which is subtler than "engines were disabled"
|
||||
|
||||
Of **55 general-category engines, 7 were enabled by default** — and six of those
|
||||
seven are dictionary, translation, currency and encyclopedia engines that return
|
||||
nothing for an ordinary web query. `google cse` was the only general WEB engine on
|
||||
the instance. When its quota exhausted, a query returned **HTTP 200 with an empty
|
||||
`results` array**.
|
||||
|
||||
⚠ **The distinction that hid it: `inactive: false` only makes an engine
|
||||
SELECTABLE; `disabled: false` is what puts it in the DEFAULT set.** The other 48
|
||||
were selectable-but-off, which an API client has no way to change.
|
||||
|
||||
## Three wrong causes published before the right one
|
||||
|
||||
Recorded because the pattern repeated all day and each was reasoned, not careless.
|
||||
|
||||
1. **"The ESH egress move broke it."** Search egress had been moved on 09-17 to a
|
||||
SOCKS5 proxy on esh-scale, and a 2026-09-03 comment claimed NH3's residential
|
||||
address avoided CAPTCHAs. Reverting to direct NH3 egress produced a
|
||||
**byte-identical** failure. Falsified.
|
||||
2. **"Stale suspension timers."** A live `!ddg` bang probe on a freshly restarted
|
||||
container also returned CAPTCHA. Falsified.
|
||||
3. **"The version bump fixed it."** Updating `2026.9.3` → `2026.9.18` appeared to
|
||||
restore all four engines — but **another session had removed the captcha'd
|
||||
scrapers from the engine set four minutes before that measurement**, and I did
|
||||
not check what else had moved. Two changes, one measurement, attributed to mine.
|
||||
|
||||
Every one asserted causation from correlation with **no baseline measurement of the
|
||||
working state**.
|
||||
|
||||
## The fix
|
||||
|
||||
Five keyless engines added to the default set, each bang-probed on this egress
|
||||
first rather than trusted from a list:
|
||||
|
||||
duckduckgo web 10 mojeek Suspended excluded
|
||||
bing 10 qwant CAPTCHA excluded
|
||||
yep 20 startpage Suspended excluded
|
||||
yahoo 7 brave Suspended excluded
|
||||
wiby 12 seznam timeout excluded
|
||||
|
||||
Plus `braveapi` with a real key. **General web engines: 1 → 7.**
|
||||
|
||||
Verified under the live failure condition rather than a simulated one — `google cse`
|
||||
was STILL quota-suspended and three consecutive queries returned 38–41 results from
|
||||
4–5 engines each. Later with braveapi: 55–63 results from six.
|
||||
|
||||
## Two durable rules this settles
|
||||
|
||||
⭐ **This stack tracks `:latest` ON PURPOSE — do not "improve" it by pinning.**
|
||||
Upstream ships engine-handler fixes continuously because providers change bot
|
||||
gating continuously; being current IS the mitigation, and a digest pin would have
|
||||
frozen the breakage in place. (The althing post office is pinned for the opposite
|
||||
reason — it is the message bus and must not move underfoot.)
|
||||
|
||||
⭐ **The Brave API key is committed in plaintext by explicit operator decision**
|
||||
(2026-09-18): *"if the service is useless, so is the key"* — free tier, rate-limited
|
||||
service, not worth the machinery of keeping it out of git. It is also vaulted at
|
||||
`nh3-docker/searxng-brave-api-key`. **This is scoped to one low-value credential and
|
||||
is not a change to the no-secrets-in-git rule.** It cannot be un-committed; rotation
|
||||
means a new key at Brave, never a history rewrite, because the repo is shared.
|
||||
|
||||
⚠ There is **no env-var path into `searxng-settings.yml`** in this build. The loader
|
||||
reads only `SEARXNG_SETTINGS_PATH`; the entrypoint substitutes only `ultrasecretkey`.
|
||||
`SEARXNG_BRAVE_API_KEY` reaches the container and is never read — it is kept in the
|
||||
compose, unused, with a comment saying so.
|
||||
|
||||
## ⚠ Still open: the health script cannot see any of this
|
||||
|
||||
`scripts/searxng-health.sh` printed `✓ searching` with six engines answering AND
|
||||
with one. It gates on "did any results come back" and prints failed engines as
|
||||
informational. Search ran on a single engine for a day with every instrument green.
|
||||
**Fix it to fail on blocked engines before trusting it again.**
|
||||
|
||||
Commits `9a428fd` `274d3e2` `9219942` `5b20b02`. See
|
||||
[[2026-09-18-searxng-esh-egress-reverted]].
|
||||
+60
-127
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-17 ~23:20 PT (lv-mccarthy TRAINING on gx10, ~150/1380 steps. The leak gate had PASSED with five protagonist names still in every copy — corpus rebuilt, gate taught to see the class. lv-hemingway keeps its 3 by operator ruling, so the gate exits 1 on a shipped tree BY DESIGN. ESH back on the Cityside static; headscale split-DNSes nh3.phasefinal.com. Next voice seat measured, not guessed. lv-krakauer parked.)_
|
||||
_Last updated: 2026-09-18 ~15:05 PT (infrastructure day, no training work. THREE silent faults found and fixed: NH3↔Anaheim was DERP-relayed (now 6 ms direct), `.internal` DNS was failing ~10% of lookups (cross-site resolver ring + AdGuard ratelimit 0), and SearXNG had one working web engine (now seven). althing v3.6.3 deployed — hyphenated search works for the first time. FleetTools shipped and autoloaded for Codex/Grok. ⚠ lv-mccarthy's run outcome is UNVERIFIED by this session.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -115,148 +115,78 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-09-17 ~23:20 PT._
|
||||
_As of 2026-09-18 ~14:50 PT._
|
||||
|
||||
### RUNNING — lv-mccarthy LoRA on pfi-gx10, launched 2026-09-17 ~23:05 PT
|
||||
### ⚠ FIRST — lv-mccarthy's run outcome is UNVERIFIED by this session
|
||||
|
||||
`~/r49-runs/mccarthy-4b-pairs-3ep/`, launched by `~/lv-mccarthy/launch-train.sh` (committed at
|
||||
`scripts/mccarthy-corpus/launch-train.sh`). At the snapshot: **~150 of 1380 steps, ~4.2 s/it,
|
||||
ETA ~1h25m**, so it lands around **00:45 PT**. A deliberate mirror of the Hemingway recipe —
|
||||
Qwen3-4B-Instruct, seq-len 1536, rank 32 / alpha 64, lr 1e-4, 3 epochs, batch 1 x accum 8,
|
||||
seed 4919 — with pairs sha `6a04fcd21800d85c`, 3673 train / 269 val, 0 dropped.
|
||||
At the previous snapshot (09-17 ~23:05 PT) the LoRA was at ~150/1380 steps with ETA ~00:45 PT.
|
||||
**This session was entirely infrastructure and never touched it**; `pfi-gx10` does not resolve from
|
||||
nh3-dev by that name, so the outcome was not checked rather than checked-and-found-good. Assume
|
||||
nothing: look at `~/r49-runs/mccarthy-4b-pairs-3ep/` before anything else.
|
||||
|
||||
⚠⚠ **`--save-total-limit` defaults to 12 and that would have silently destroyed the gate.** At the
|
||||
25-step save cadence over 1380 steps that keeps only epochs ~2.35-3.0 and DELETES the epoch-1 and
|
||||
epoch-2 checkpoints — which is exactly what the last two gates were decided on (Hemingway's ckpt850
|
||||
at epoch 0.96 TIED his eval-loss minimum ckpt1750; on Brontë the earlier epoch-1 checkpoint won the
|
||||
tiebreak on the axes that resolve). Raised to 60 so all ~55 survive, ~42 GB against 442 GB free.
|
||||
Eval/save cadence is 25 rather than Hemingway's 50 because McCarthy has 460 steps/epoch against his
|
||||
887 — matching EPOCH resolution (0.054 vs 0.056), which is the unit the tiebreak is expressed in.
|
||||
**Next, unchanged:** checkpoint selection off the loss curve → the v2 gate (`eval-*.sh` shape) →
|
||||
ship-or-park. ⚠ Before the gate, settle how the VOICE axis is read — D1 pre-registered a
|
||||
punctuation-normalised secondary read and the `mccarthy` register now states the punctuation tics
|
||||
explicitly, so `--system-from` hands them to the BASE control arm too. Deliberate (denies the adapter
|
||||
a cheap char-bigram win) but it changes what the primary number means, and it must be settled BEFORE
|
||||
a number exists.
|
||||
|
||||
**Next, in order:** checkpoint selection off the loss curve → the v2 gate (`eval-*.sh` shape) →
|
||||
ship-or-park. ⚠ Before the gate, settle how the VOICE axis is read: D1 pre-registered a
|
||||
punctuation-normalised secondary read, and the `mccarthy` register now states the punctuation tics
|
||||
explicitly, so `--system-from` hands them to the BASE control arm too. That was done deliberately to
|
||||
deny the adapter a cheap char-bigram win, but it changes what the primary number means and must be
|
||||
settled BEFORE a number exists.
|
||||
### The fleet is materially faster than it was this morning — three fixes, all verified
|
||||
|
||||
Detail in the Recent decisions entries below; the operational summary is that **NH3→Anaheim went
|
||||
from a throttled DERP relay to direct (6 ms, cross-site HTTP 1.2 s → 0.015 s)**, **`.internal` DNS
|
||||
stopped failing ~10% of lookups and stalling 5 s on the rest**, and **SearXNG went from one working
|
||||
general web engine to seven**. All three were silent — no monitor caught any of them, and two had
|
||||
been degrading for months.
|
||||
|
||||
⚠ **Nothing on this fleet watches DNS success rate or whether a mesh path is direct.** Today's three
|
||||
faults surfaced only because tts-dev had a 1545 ms voice-loop budget and measured instead of adapting
|
||||
around it. **A probe pair is proposed and UNDECIDED** — see the Recent decisions entry.
|
||||
|
||||
### Still relayed: irv-ml1
|
||||
|
||||
`100.64.0.6` remains `relay "lax"` after the Anaheim fix — a different site with its own NAT
|
||||
situation, untouched. Measured 24-25 ms on HTTP from nh3-dev, so it is NOT costing what Anaheim was
|
||||
and needs nothing urgently. I warned tts-dev it would be slow and was wrong; they measured and
|
||||
corrected me.
|
||||
|
||||
### OPEN LOOP — `ravenpen.com`, and a reply hamr-dev is waiting on
|
||||
|
||||
hamr-dev asked infra-ops to register `ravenpen.com` (the marketing name for hamr) on a relayed
|
||||
operator directive. **Surfaced, not executed**, for two reasons: a peer relaying "Vuong approved it"
|
||||
is not authorization for a non-refundable purchase (Miranda is the named exception, a dev session is
|
||||
not), and **infra-ops holds no registrar credential at all** — the vault's only registrar-adjacent
|
||||
entries are three Cloudflare tokens and this handle's is Zone/DNS/Edit scope, which cannot register
|
||||
a domain. hamr-dev accepted both points and closed out.
|
||||
hamr-dev asked infra-ops to register `ravenpen.com` on a relayed operator directive. **Surfaced, not
|
||||
executed**: a peer relaying "Vuong approved it" is not authorization for a non-refundable purchase,
|
||||
and infra-ops holds no registrar credential. hamr-dev accepted and closed out.
|
||||
|
||||
⚠ **If the operator registers it, POST BACK on althing thread `01M2SERDB3DR3RV7JS1J0GMF0H`** with
|
||||
the registrar and expiry date — hamr-dev is explicitly waiting on that to record in hamr's own
|
||||
memory, and it is the kind of cross-session obligation a context reset drops silently.
|
||||
registrar and expiry — hamr-dev is explicitly waiting on that, and it is the kind of cross-session
|
||||
obligation a reset drops silently. Verified 2026-09-18 ~05:20Z: `ravenpen.{com,io,app,dev,ai,net}`
|
||||
all unregistered.
|
||||
|
||||
Verified 2026-09-18 ~05:20Z: `ravenpen.{com,io,app,dev,ai,net}` all unregistered (RDAP 404).
|
||||
Trademark: **web-visible evidence only, NOT a clearance search** — USPTO's API refused the query
|
||||
(405), nothing obvious surfaced, nearest neighbours are descriptive "raven pen" uses on physical
|
||||
pens, a different class from software. Recommendation given: `.com` alone, or `.com` + `.ai`.
|
||||
### PLANNED (operator) — move `dragonfireacoustics.com` to Namecheap, DNS to Cloudflare
|
||||
|
||||
### PLANNED 2026-09-18 (operator) — move `dragonfireacoustics.com` to Namecheap, DNS to Cloudflare
|
||||
Unchanged from the previous snapshot and **not started**. ⚠⚠ The zone's `*` wildcard MASKS what is
|
||||
really there — enumerate real records before any transfer. Expiry **2026-10-30** at eNom with no
|
||||
transfer lock, and the sibling `dragonfirepro.com` was already lost exactly this way. Customer-facing;
|
||||
nothing touched. Full context in the 09-17 Recent decisions entries.
|
||||
|
||||
Operator holds the **TAC/EPP** already. TAC is **single-use and dies 14 days after issue**, so the
|
||||
clock is running; domain itself expires **2026-10-30** and a transfer adds a year, so the transfer
|
||||
IS the renewal if it lands in time.
|
||||
### Other standing items, unchanged
|
||||
|
||||
⚠ **Recommended order is DNS FIRST, transfer second** — stand the Cloudflare zone up, repoint the
|
||||
nameservers at eNom (NS is a registrar function he already has access to), verify mail and the site,
|
||||
THEN burn the TAC. The registry keeps the NS delegation across a transfer and Namecheap does not
|
||||
overwrite it, so nothing has to be redone; and if DNS goes wrong you still have a working registrar
|
||||
account to revert at instead of discovering it mid-transfer with a spent code.
|
||||
|
||||
⚠⚠ **The zone's `*` wildcard MASKS what is really there.** Everything — apex, mail, webmail, admin,
|
||||
ftp — answers 199.250.192.76 (dead) purely because of that one record. Cloudflare's add-a-zone scan
|
||||
will happily import the dead wildcard and may not surface what is genuinely configured, so the
|
||||
**7 Google Workspace MX records must be checked by hand after the import** or mail stops. The only
|
||||
records that actually matter: `www A 38.120.12.45` and those MX.
|
||||
|
||||
Open sub-decisions carried into the move: point the apex at 38.120.12.45 too (makes the bare domain
|
||||
work for the first time and lets a replacement cert cover it), and add SPF/DMARC — the domain runs
|
||||
Google Workspace with **neither**.
|
||||
|
||||
### DONE: lv-mccarthy D4 pairs built — NEXT is `train_pairs_lora.py`
|
||||
|
||||
`~/lv-mccarthy/pairs/` on gx10, 2026-09-17 11:55 PT via `run-pairs.sh` (committed at
|
||||
`scripts/mccarthy-corpus/run-pairs.sh`).
|
||||
|
||||
```
|
||||
train 3,673 pairs of 3,722 passages reject 1.32% 461,172 words 1,377 s
|
||||
val 269 pairs of 276 passages reject 2.54% 33,477 words 98 s
|
||||
```
|
||||
|
||||
⭐ **The val split is the largest fixture in the line** — 269 against Hemingway's 200 and
|
||||
Brontë's 44 in-band, the structural cause of Brontë's underpowered voice axis.
|
||||
Generator fingerprint `vllm-0.1.1.dev50+geed1f3d0c-601ad866`, **identical to Hemingway's
|
||||
run and unmoved mid-run**, so the two pair sets are generator-comparable.
|
||||
|
||||
Verified in the produced pairs, not assumed: **0 mid-sentence line breaks** in all 3,942
|
||||
responses, **0 chapter-argument responses**, and **0.0 quote marks per 10k words** in both
|
||||
splits — McCarthy's unquoted dialogue survives rename → chunk → reflow → pair intact.
|
||||
`sourcename` rejected **14 beats** that named a character the rename removed; Hemingway's
|
||||
shipped pairs had no such filter at all. `narrator_retries` 101 of 3,673 (2.8%) against
|
||||
Hemingway's 8.7%, which is what third-person prose looks like.
|
||||
|
||||
Then, before the eval, honour D1's pre-registration: a **punctuation-normalised secondary
|
||||
read** of the voice axis — and note the `mccarthy` register now states the tics, so the
|
||||
base control arm gets them too.
|
||||
|
||||
### SETTLED — lv-hemingway keeps its 3 hidden survivors; operator ruled "leave it"
|
||||
|
||||
`Primi tivo`, `Pasionar ia`, `Chi cote`, one each per copy in a 958k-word corpus, on the
|
||||
adapter serving at fv-ml1 GPU 0 `:8027`. Operator 2026-09-17: **leave it.** lv-bronte re-ran
|
||||
clean. ⚠ **Consequence to remember: `leak_gate.py` now EXITS 1 on lv-hemingway's tree.** A
|
||||
future session re-running that gate will see a red result on a shipped artifact — that is
|
||||
expected and settled, not a regression to chase. Do not "fix" it without a fresh ruling.
|
||||
|
||||
### SHIPPED — `lv-hemingway` on ckpt850, live on `vllm-voices` (fv-ml1 GPU 0 :8027)
|
||||
|
||||
Beside `voices-base`, `lv-yarros`, `lv-bronte`. VOICE +0.413 delta_cb at 6.4x its floor,
|
||||
closing 73.8% of the achievable span — the strongest in the line, and it clears the OLD
|
||||
all-arms floor too, so it does not lean on the rule change.
|
||||
⚠ MEMORISATION is the axis to read: 0.07 hit-rate against **held-out Hemingway's own 0.01**,
|
||||
so ~7x the author's self-collision rate — but all 19 matched runs were READ and every one is
|
||||
stock dialogue capped at 9 words, no proper noun, no plot.
|
||||
→ `persistent-memory.d/2026-09-17-lv-hemingway-gate.md`
|
||||
|
||||
### ⛔ PARKED — `lv-krakauer` (operator, 2026-09-17), henge id **82**
|
||||
|
||||
*"he's a great writer because of his research, not because he has a strong identifiable
|
||||
voice."* Corpus (126 units, 422,880 words) and builder stay committed and re-runnable.
|
||||
|
||||
### OPEN for the operator
|
||||
|
||||
1. **The althing route-declaring SessionStart hook is NOT installed on nh3-dev**, so any
|
||||
relaunched pane silently drops to pull mode and stops receiving mail. althing's own skill
|
||||
documents the hook; this box does not have it. Installing it means editing
|
||||
`~/.claude/settings.json`. Raised, not answered.
|
||||
2. **Hemingway's train beats: 70 of 7,094 (0.96%) name characters the rename removed.**
|
||||
`audit_pairs_sourcenames.py --filter-out` yields a verified-clean 7,024-pair set in one
|
||||
command; retrain ~3h17 + re-gate ~1h40. Operator said **ship stands** — recorded, not acted on.
|
||||
3. **130 of 946 Hemingway entity-map surfaces are probably not names** and were renamed anyway.
|
||||
Same ruling: ship stands.
|
||||
|
||||
### ESH is on Verizon failover — Cityside Fiber failed twice on 2026-09-16
|
||||
|
||||
ESH egress `97.190.18.88`; crowdsec's `esh` allowlist carries it plus `23.164.40.174`, both
|
||||
with 7-day expiries set 2026-09-16. ⚠ When those expire or the CGNAT egress rotates, ESH loses
|
||||
colo access. Check `curl -s4 ifconfig.me` from esh-docker-vm first if ESH goes dark.
|
||||
|
||||
### Older deferred set, unchanged
|
||||
|
||||
AI-tab Dormant regrouping (**belayed**), `nconnect=8` on `/mnt/smithy` (**declined in scope**),
|
||||
fused MoE kernel path (**parked, id 47**), TTS-stack move to fv-ml1 (**parked, id 75**).
|
||||
|
||||
⚠ **Uncommitted and NOT mine:** `graphify-out/GRAPH_REPORT.md`, `scripts/seat-inventory.py`,
|
||||
`stacks/searxng/{README.md,conf/searxng-settings.yml}`, `configs/esh-scale/`. Leave them alone.
|
||||
- **lv-hemingway keeps its 3 separator-hidden names** (operator: "leave it"), so `leak_gate.py` exits
|
||||
1 on a shipped tree BY DESIGN. A red result there is expected, not a bug to fix.
|
||||
- **`lv-krakauer` PARKED** (operator 2026-09-17), henge id **82**.
|
||||
- **ESH is on the Cityside static** `128.177.138.182/30`, healthy on four axes. The two 7-day crowdsec
|
||||
allowlist entries expire 2026-09-23 and are being left deliberately — Cityside failed twice in six
|
||||
hours on 09-16/17.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-18]` ⭐⭐⭐ **NH3↔Anaheim had been running over a throttled DERP relay, not a direct path — 78 GB of fleet traffic on someone else's free infrastructure.** Four additive objects on ana-gw gave ana-scale a stable inbound UDP 41641 endpoint; `tailscale ping` 373–522 ms → **6 ms direct**, cross-site HTTP 1.2 s → 0.015 s, STT via the ANA gateway 1.4 s → 0.25 s. ⚠ That box runs `central-nat`, so a policy `dstaddr` is the REAL internal address, not the VIP. No OOB access — back up with `show` to a local file and make additive changes ONLY. irv-ml1 still relayed. → `persistent-memory.d/2026-09-18-nh3-ana-derp-relay.md`
|
||||
- `[2026-09-18]` ⭐⭐⭐ **`.internal` DNS was failing ~10% of lookups fleet-wide, from two independent causes.** A PUBLIC resolver was the fallback for a PRIVATE zone (Cloudflare answers NXDOMAIN authoritatively, so a transient miss became a hard failure) — now a cross-site ring, each site local-first with a different site as backup. Then the root cause: all three AdGuards shipped `ratelimit: 20` shared across an entire **/24**, silently dropping queries at 5 s each. Set to 0. Hard failures 3/40 → 0/40; burst timeouts 40/60 → 0/60. ⚠ `resolv.conf` is DHCP-managed — change it at the UDM/FortiGate, not the file. → `persistent-memory.d/2026-09-18-fleet-dns-ring-and-ratelimit.md`
|
||||
- `[2026-09-18]` ⭐⭐ **SearXNG had ONE working general web engine and every health check said fine.** 7 of 55 were enabled-by-default and six of those are dictionary/translation engines — `inactive: false` only makes an engine SELECTABLE, `disabled: false` puts it in the DEFAULT set. Now seven. ⭐ This stack tracks `:latest` ON PURPOSE (upstream ships engine-handler fixes continuously; a pin freezes breakage). ⭐ The Brave key is committed in plaintext by explicit operator decision — scoped to one low-value credential, NOT a change to the no-secrets rule. → `persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md`
|
||||
- `[2026-09-18]` ⭐⭐ **althing search returned ZERO for every hyphenated query — silently, for every handle, and on this fleet that is most of our hostnames.** FTS5 read the hyphen as a column filter; `search()` caught the error, its probe passed, and it returned `[]`. Routed to forseti (they own the code, I own rollout) → **v3.6.3** deployed, 17 s bus outage. The reported symptom, a message that never arrived, was NOT a defect: the reporter's evidence file was truncated at 8 KiB. ⚠ No CI builds the post-office image. → `persistent-memory.d/2026-09-18-althing-363-hyphen-search.md`
|
||||
- `[2026-09-18]` ⭐ **FleetTools: one capability index, autoloaded by Claude, Codex and Grok from a single symlinked file.** `docs/fleettools/` + `~/FLEETTOOLS.md`; absolute detail paths because a non-Claude agent cats them. Rule zero is query-live-inventories-never-a-written-list. The global CLAUDE.md tools section went 231 lines → 39, keeping only the three rules that govern behaviour rather than lookup. → `persistent-memory.d/2026-09-18-fleettools-agent-index.md`
|
||||
- `[2026-09-18]` **Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy** (operator-approved). Verified empirically — `is_dir()`/`exists()` are stat-based and follow symlinks — and the copy had ALREADY drifted (README, since 09-15). Pre-restart gate is `uv run python scripts/sync_plugin.py --check`; `hermes plugins list` is confirmation-after, never permission-before. Copy kept at `~/backups/svos_miranda-copy-20260918-1444`. ⚠ The install now tracks a working tree — an uncommitted edit in svos is what loads at the next restart.
|
||||
- `[2026-09-18]` **Worldtree's `env.sh` secrets are vaulted** — 10 entries under `worldtree/` (gitea, matrix as/hs, openai, uv-index, vastblueai, wt-admin demo+personal, z-ai, zai). ⚠ `Z_AI_API_KEY` and `ZAI_API_KEY` are DIFFERENT keys despite the near-identical names (fp `26bde3fa` vs `a4884152`). `ANTHROPIC_API_KEY` was empty and skipped.
|
||||
|
||||
- `[2026-09-17]` ⭐⭐ **The next voice seat was MEASURED, not chosen by taste — and the corpus size ranking INVERTS the voice ranking at the top.** Our two largest authors are Stephen King (76 works, 12.1M words) and Agatha Christie (72, 5.5M); neither should get a seat, Christie being the Krakauer failure mode exactly (genius in plot architecture, prose deliberately transparent, invisible to a char-bigram Delta). Picks, in order: **Faulkner** (~15 pure novels, ~1.6M words — highest voice signal in the catalogue, AND he is McCarthy's stylistic ancestor, so training him next supplies the **hard-negative sister the gate has lacked since the Brontë record named it missing**); **Morrison** (11 novels after pruning criticism/anthology, ~818k — 11 val units, beating Hemingway's 10); **Chandler** (7 novels + a 409k short-story omnibus — fills the first-person hardboiled gap, Hemingway-class corpus size). ⚠ Faulkner's catalogue rows carry a 446k-word Snopes omnibus that duplicates novels also present individually — the Hemingway 90-96% collection-duplication trap, needs the containment pass first. → `persistent-memory.d/2026-09-17-next-voice-seats.md`
|
||||
- `[2026-09-17]` ⭐⭐ **Romantasy IS a real register, we already trained its most distinctive member, and the obvious next pick is its worst.** Measured on the gate's own instrument (char-bigram Burrows's Delta, ~120k words/author from mid-work), with within-author floors and cross-genre positive controls. Cluster median pair **0.537 = 1.2x the worst floor** against controls at 1.4-1.9x — tighter than cross-genre but NOT collapsed. Two findings survive either floor reading: **Yarros is the cluster OUTLIER** (4 of the 5 largest pair distances involve her), so a second romantasy seat buys measurably less than the first did; and **Maas is the centroid** (the two smallest distances in the matrix are hers), so the obvious commercial pick is the least distinctive. If the lane gets a seat it is **Kenyon** — furthest from Yarros at 0.674 and **27 works = 27 val units, the best-powered gate the line could build** (Hemingway 10, McCarthy 6, Brontë 4). ⚠ Sensitivity floor stated: one sample per pair, no repeat draws; the rank ordering is indicative, fine gaps are not resolvable. → `persistent-memory.d/2026-09-17-romantasy-register-measured.md`
|
||||
- `[2026-09-17]` ⚠⚠ **`dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.** RDAP: registrar **eNom, LLC** (IANA 48, Tucows; abuse@enom.com), created 2008-10-30, status `active` **only** — no `clientTransferProhibited`. Registrant redacted (Verisign RDAP is thin, eNom's own endpoint 404s). Meanwhile `dragonfirepro.com` was re-registered 2025-09-27 by a Hungarian registrar and is now **for sale on expireddomains.com** — the customer lapsed it and a drop-catcher took it. ⭐ **This also makes the cert fix mandatory rather than tidy: two of the three SANs name a domain a THIRD PARTY now owns, so that request can never validate, and our Virtualmin has been retrying it often enough to get the Let's Encrypt account PAUSED.** Nobody is minding this domain — 13-month-dead cert, 403 homepage, sibling already gone — so the six-week clock is a real risk to an 18-year-old .com with a live site. Customer-facing; nothing touched.
|
||||
@@ -535,6 +465,9 @@ _5 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-09-18]` **Routing SearXNG's egress through a SOCKS5 proxy on esh-scale** — one day live, reverted. It fixed nothing, and the reason I gave for reverting it was itself wrong: the rollback was the counterfactual and it falsified my own published claim. Kept reverted on its own merits (no measurable gain, added a hard ESH dependency for all fleet search). → `persistent-memory.d/2026-09-18-searxng-esh-egress-reverted.md`
|
||||
|
||||
- `[2026-09-18]` **`api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings** — this build has NO `!ENV` YAML constructor, so the file was unparseable and the container crash-looped ten times with fleet search down. There is no env-var path into that file at all: the loader reads only `SEARXNG_SETTINGS_PATH` and the entrypoint substitutes only `ultrasecretkey`. Literal or nothing.
|
||||
- `[2026-09-15]` ⚠⚠ **Probing OPNsense API endpoints by POSTing at them — one was `/api/core/system/reboot` and it took the FV site dark for 3.5 min.** Endpoints are ACTIONS; a 200 means it ran. The call I wanted was documented in this repo's own `docs/pfi/opnsense-api-reference.md`. → `persistent-memory.d/2026-09-15-opnsense-api-reboot.md`
|
||||
|
||||
- `[2026-09-15]` **Advertising `10.100.10.50/32` from nh3-dev** to make its LAN address mesh-reachable — black-holed it from ESH/ANA/FV/IRV while its own LAN and the internet kept working, so a one-host check passes cleanly. `lookup 52` at rule priority 5270 beats `main` at 32766. Fix belongs at the router. → `persistent-memory.d/2026-09-15-nh3-dev-ts-input-masquerade.md`
|
||||
|
||||
Reference in New Issue
Block a user