memory: snapshot — three silent fleet faults found and fixed in one afternoon
An infrastructure day with no training work, and the through-line is that every fault was invisible to monitoring. NH3↔Anaheim had been crossing a throttled DERP relay rather than a direct path for long enough to carry 78 GB; `.internal` DNS was failing roughly one lookup in ten from two independent causes; SearXNG had exactly one working general web engine. Nothing alarmed on any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget and chose to measure rather than adapt around the problem. Also landed: althing v3.6.3, which makes hyphenated search work for the first time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a capability index autoloaded by Claude, Codex and Grok from one symlinked file; Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's env.sh secrets vaulted. Six detail files. The in-flight section is rewritten and shrinks 142 lines to 64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run outcome, which was left untouched and must not be assumed good. Two foot-guns recorded, both mine: the ESH egress experiment reverted on a diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which has no constructor in that build and crash-looped the container ten times. No archival this run. 165 of 169 dated entries are under the 14-day guard and the remaining four all carry open deferred pointers, so the index stays over the soft cap at 480 lines — an over-cap file that keeps live decisions beats a scannable one that lost one.
This commit is contained in:
+60
-127
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-17 ~23:20 PT (lv-mccarthy TRAINING on gx10, ~150/1380 steps. The leak gate had PASSED with five protagonist names still in every copy — corpus rebuilt, gate taught to see the class. lv-hemingway keeps its 3 by operator ruling, so the gate exits 1 on a shipped tree BY DESIGN. ESH back on the Cityside static; headscale split-DNSes nh3.phasefinal.com. Next voice seat measured, not guessed. lv-krakauer parked.)_
|
||||
_Last updated: 2026-09-18 ~15:05 PT (infrastructure day, no training work. THREE silent faults found and fixed: NH3↔Anaheim was DERP-relayed (now 6 ms direct), `.internal` DNS was failing ~10% of lookups (cross-site resolver ring + AdGuard ratelimit 0), and SearXNG had one working web engine (now seven). althing v3.6.3 deployed — hyphenated search works for the first time. FleetTools shipped and autoloaded for Codex/Grok. ⚠ lv-mccarthy's run outcome is UNVERIFIED by this session.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -115,148 +115,78 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-09-17 ~23:20 PT._
|
||||
_As of 2026-09-18 ~14:50 PT._
|
||||
|
||||
### RUNNING — lv-mccarthy LoRA on pfi-gx10, launched 2026-09-17 ~23:05 PT
|
||||
### ⚠ FIRST — lv-mccarthy's run outcome is UNVERIFIED by this session
|
||||
|
||||
`~/r49-runs/mccarthy-4b-pairs-3ep/`, launched by `~/lv-mccarthy/launch-train.sh` (committed at
|
||||
`scripts/mccarthy-corpus/launch-train.sh`). At the snapshot: **~150 of 1380 steps, ~4.2 s/it,
|
||||
ETA ~1h25m**, so it lands around **00:45 PT**. A deliberate mirror of the Hemingway recipe —
|
||||
Qwen3-4B-Instruct, seq-len 1536, rank 32 / alpha 64, lr 1e-4, 3 epochs, batch 1 x accum 8,
|
||||
seed 4919 — with pairs sha `6a04fcd21800d85c`, 3673 train / 269 val, 0 dropped.
|
||||
At the previous snapshot (09-17 ~23:05 PT) the LoRA was at ~150/1380 steps with ETA ~00:45 PT.
|
||||
**This session was entirely infrastructure and never touched it**; `pfi-gx10` does not resolve from
|
||||
nh3-dev by that name, so the outcome was not checked rather than checked-and-found-good. Assume
|
||||
nothing: look at `~/r49-runs/mccarthy-4b-pairs-3ep/` before anything else.
|
||||
|
||||
⚠⚠ **`--save-total-limit` defaults to 12 and that would have silently destroyed the gate.** At the
|
||||
25-step save cadence over 1380 steps that keeps only epochs ~2.35-3.0 and DELETES the epoch-1 and
|
||||
epoch-2 checkpoints — which is exactly what the last two gates were decided on (Hemingway's ckpt850
|
||||
at epoch 0.96 TIED his eval-loss minimum ckpt1750; on Brontë the earlier epoch-1 checkpoint won the
|
||||
tiebreak on the axes that resolve). Raised to 60 so all ~55 survive, ~42 GB against 442 GB free.
|
||||
Eval/save cadence is 25 rather than Hemingway's 50 because McCarthy has 460 steps/epoch against his
|
||||
887 — matching EPOCH resolution (0.054 vs 0.056), which is the unit the tiebreak is expressed in.
|
||||
**Next, unchanged:** checkpoint selection off the loss curve → the v2 gate (`eval-*.sh` shape) →
|
||||
ship-or-park. ⚠ Before the gate, settle how the VOICE axis is read — D1 pre-registered a
|
||||
punctuation-normalised secondary read and the `mccarthy` register now states the punctuation tics
|
||||
explicitly, so `--system-from` hands them to the BASE control arm too. Deliberate (denies the adapter
|
||||
a cheap char-bigram win) but it changes what the primary number means, and it must be settled BEFORE
|
||||
a number exists.
|
||||
|
||||
**Next, in order:** checkpoint selection off the loss curve → the v2 gate (`eval-*.sh` shape) →
|
||||
ship-or-park. ⚠ Before the gate, settle how the VOICE axis is read: D1 pre-registered a
|
||||
punctuation-normalised secondary read, and the `mccarthy` register now states the punctuation tics
|
||||
explicitly, so `--system-from` hands them to the BASE control arm too. That was done deliberately to
|
||||
deny the adapter a cheap char-bigram win, but it changes what the primary number means and must be
|
||||
settled BEFORE a number exists.
|
||||
### The fleet is materially faster than it was this morning — three fixes, all verified
|
||||
|
||||
Detail in the Recent decisions entries below; the operational summary is that **NH3→Anaheim went
|
||||
from a throttled DERP relay to direct (6 ms, cross-site HTTP 1.2 s → 0.015 s)**, **`.internal` DNS
|
||||
stopped failing ~10% of lookups and stalling 5 s on the rest**, and **SearXNG went from one working
|
||||
general web engine to seven**. All three were silent — no monitor caught any of them, and two had
|
||||
been degrading for months.
|
||||
|
||||
⚠ **Nothing on this fleet watches DNS success rate or whether a mesh path is direct.** Today's three
|
||||
faults surfaced only because tts-dev had a 1545 ms voice-loop budget and measured instead of adapting
|
||||
around it. **A probe pair is proposed and UNDECIDED** — see the Recent decisions entry.
|
||||
|
||||
### Still relayed: irv-ml1
|
||||
|
||||
`100.64.0.6` remains `relay "lax"` after the Anaheim fix — a different site with its own NAT
|
||||
situation, untouched. Measured 24-25 ms on HTTP from nh3-dev, so it is NOT costing what Anaheim was
|
||||
and needs nothing urgently. I warned tts-dev it would be slow and was wrong; they measured and
|
||||
corrected me.
|
||||
|
||||
### OPEN LOOP — `ravenpen.com`, and a reply hamr-dev is waiting on
|
||||
|
||||
hamr-dev asked infra-ops to register `ravenpen.com` (the marketing name for hamr) on a relayed
|
||||
operator directive. **Surfaced, not executed**, for two reasons: a peer relaying "Vuong approved it"
|
||||
is not authorization for a non-refundable purchase (Miranda is the named exception, a dev session is
|
||||
not), and **infra-ops holds no registrar credential at all** — the vault's only registrar-adjacent
|
||||
entries are three Cloudflare tokens and this handle's is Zone/DNS/Edit scope, which cannot register
|
||||
a domain. hamr-dev accepted both points and closed out.
|
||||
hamr-dev asked infra-ops to register `ravenpen.com` on a relayed operator directive. **Surfaced, not
|
||||
executed**: a peer relaying "Vuong approved it" is not authorization for a non-refundable purchase,
|
||||
and infra-ops holds no registrar credential. hamr-dev accepted and closed out.
|
||||
|
||||
⚠ **If the operator registers it, POST BACK on althing thread `01M2SERDB3DR3RV7JS1J0GMF0H`** with
|
||||
the registrar and expiry date — hamr-dev is explicitly waiting on that to record in hamr's own
|
||||
memory, and it is the kind of cross-session obligation a context reset drops silently.
|
||||
registrar and expiry — hamr-dev is explicitly waiting on that, and it is the kind of cross-session
|
||||
obligation a reset drops silently. Verified 2026-09-18 ~05:20Z: `ravenpen.{com,io,app,dev,ai,net}`
|
||||
all unregistered.
|
||||
|
||||
Verified 2026-09-18 ~05:20Z: `ravenpen.{com,io,app,dev,ai,net}` all unregistered (RDAP 404).
|
||||
Trademark: **web-visible evidence only, NOT a clearance search** — USPTO's API refused the query
|
||||
(405), nothing obvious surfaced, nearest neighbours are descriptive "raven pen" uses on physical
|
||||
pens, a different class from software. Recommendation given: `.com` alone, or `.com` + `.ai`.
|
||||
### PLANNED (operator) — move `dragonfireacoustics.com` to Namecheap, DNS to Cloudflare
|
||||
|
||||
### PLANNED 2026-09-18 (operator) — move `dragonfireacoustics.com` to Namecheap, DNS to Cloudflare
|
||||
Unchanged from the previous snapshot and **not started**. ⚠⚠ The zone's `*` wildcard MASKS what is
|
||||
really there — enumerate real records before any transfer. Expiry **2026-10-30** at eNom with no
|
||||
transfer lock, and the sibling `dragonfirepro.com` was already lost exactly this way. Customer-facing;
|
||||
nothing touched. Full context in the 09-17 Recent decisions entries.
|
||||
|
||||
Operator holds the **TAC/EPP** already. TAC is **single-use and dies 14 days after issue**, so the
|
||||
clock is running; domain itself expires **2026-10-30** and a transfer adds a year, so the transfer
|
||||
IS the renewal if it lands in time.
|
||||
### Other standing items, unchanged
|
||||
|
||||
⚠ **Recommended order is DNS FIRST, transfer second** — stand the Cloudflare zone up, repoint the
|
||||
nameservers at eNom (NS is a registrar function he already has access to), verify mail and the site,
|
||||
THEN burn the TAC. The registry keeps the NS delegation across a transfer and Namecheap does not
|
||||
overwrite it, so nothing has to be redone; and if DNS goes wrong you still have a working registrar
|
||||
account to revert at instead of discovering it mid-transfer with a spent code.
|
||||
|
||||
⚠⚠ **The zone's `*` wildcard MASKS what is really there.** Everything — apex, mail, webmail, admin,
|
||||
ftp — answers 199.250.192.76 (dead) purely because of that one record. Cloudflare's add-a-zone scan
|
||||
will happily import the dead wildcard and may not surface what is genuinely configured, so the
|
||||
**7 Google Workspace MX records must be checked by hand after the import** or mail stops. The only
|
||||
records that actually matter: `www A 38.120.12.45` and those MX.
|
||||
|
||||
Open sub-decisions carried into the move: point the apex at 38.120.12.45 too (makes the bare domain
|
||||
work for the first time and lets a replacement cert cover it), and add SPF/DMARC — the domain runs
|
||||
Google Workspace with **neither**.
|
||||
|
||||
### DONE: lv-mccarthy D4 pairs built — NEXT is `train_pairs_lora.py`
|
||||
|
||||
`~/lv-mccarthy/pairs/` on gx10, 2026-09-17 11:55 PT via `run-pairs.sh` (committed at
|
||||
`scripts/mccarthy-corpus/run-pairs.sh`).
|
||||
|
||||
```
|
||||
train 3,673 pairs of 3,722 passages reject 1.32% 461,172 words 1,377 s
|
||||
val 269 pairs of 276 passages reject 2.54% 33,477 words 98 s
|
||||
```
|
||||
|
||||
⭐ **The val split is the largest fixture in the line** — 269 against Hemingway's 200 and
|
||||
Brontë's 44 in-band, the structural cause of Brontë's underpowered voice axis.
|
||||
Generator fingerprint `vllm-0.1.1.dev50+geed1f3d0c-601ad866`, **identical to Hemingway's
|
||||
run and unmoved mid-run**, so the two pair sets are generator-comparable.
|
||||
|
||||
Verified in the produced pairs, not assumed: **0 mid-sentence line breaks** in all 3,942
|
||||
responses, **0 chapter-argument responses**, and **0.0 quote marks per 10k words** in both
|
||||
splits — McCarthy's unquoted dialogue survives rename → chunk → reflow → pair intact.
|
||||
`sourcename` rejected **14 beats** that named a character the rename removed; Hemingway's
|
||||
shipped pairs had no such filter at all. `narrator_retries` 101 of 3,673 (2.8%) against
|
||||
Hemingway's 8.7%, which is what third-person prose looks like.
|
||||
|
||||
Then, before the eval, honour D1's pre-registration: a **punctuation-normalised secondary
|
||||
read** of the voice axis — and note the `mccarthy` register now states the tics, so the
|
||||
base control arm gets them too.
|
||||
|
||||
### SETTLED — lv-hemingway keeps its 3 hidden survivors; operator ruled "leave it"
|
||||
|
||||
`Primi tivo`, `Pasionar ia`, `Chi cote`, one each per copy in a 958k-word corpus, on the
|
||||
adapter serving at fv-ml1 GPU 0 `:8027`. Operator 2026-09-17: **leave it.** lv-bronte re-ran
|
||||
clean. ⚠ **Consequence to remember: `leak_gate.py` now EXITS 1 on lv-hemingway's tree.** A
|
||||
future session re-running that gate will see a red result on a shipped artifact — that is
|
||||
expected and settled, not a regression to chase. Do not "fix" it without a fresh ruling.
|
||||
|
||||
### SHIPPED — `lv-hemingway` on ckpt850, live on `vllm-voices` (fv-ml1 GPU 0 :8027)
|
||||
|
||||
Beside `voices-base`, `lv-yarros`, `lv-bronte`. VOICE +0.413 delta_cb at 6.4x its floor,
|
||||
closing 73.8% of the achievable span — the strongest in the line, and it clears the OLD
|
||||
all-arms floor too, so it does not lean on the rule change.
|
||||
⚠ MEMORISATION is the axis to read: 0.07 hit-rate against **held-out Hemingway's own 0.01**,
|
||||
so ~7x the author's self-collision rate — but all 19 matched runs were READ and every one is
|
||||
stock dialogue capped at 9 words, no proper noun, no plot.
|
||||
→ `persistent-memory.d/2026-09-17-lv-hemingway-gate.md`
|
||||
|
||||
### ⛔ PARKED — `lv-krakauer` (operator, 2026-09-17), henge id **82**
|
||||
|
||||
*"he's a great writer because of his research, not because he has a strong identifiable
|
||||
voice."* Corpus (126 units, 422,880 words) and builder stay committed and re-runnable.
|
||||
|
||||
### OPEN for the operator
|
||||
|
||||
1. **The althing route-declaring SessionStart hook is NOT installed on nh3-dev**, so any
|
||||
relaunched pane silently drops to pull mode and stops receiving mail. althing's own skill
|
||||
documents the hook; this box does not have it. Installing it means editing
|
||||
`~/.claude/settings.json`. Raised, not answered.
|
||||
2. **Hemingway's train beats: 70 of 7,094 (0.96%) name characters the rename removed.**
|
||||
`audit_pairs_sourcenames.py --filter-out` yields a verified-clean 7,024-pair set in one
|
||||
command; retrain ~3h17 + re-gate ~1h40. Operator said **ship stands** — recorded, not acted on.
|
||||
3. **130 of 946 Hemingway entity-map surfaces are probably not names** and were renamed anyway.
|
||||
Same ruling: ship stands.
|
||||
|
||||
### ESH is on Verizon failover — Cityside Fiber failed twice on 2026-09-16
|
||||
|
||||
ESH egress `97.190.18.88`; crowdsec's `esh` allowlist carries it plus `23.164.40.174`, both
|
||||
with 7-day expiries set 2026-09-16. ⚠ When those expire or the CGNAT egress rotates, ESH loses
|
||||
colo access. Check `curl -s4 ifconfig.me` from esh-docker-vm first if ESH goes dark.
|
||||
|
||||
### Older deferred set, unchanged
|
||||
|
||||
AI-tab Dormant regrouping (**belayed**), `nconnect=8` on `/mnt/smithy` (**declined in scope**),
|
||||
fused MoE kernel path (**parked, id 47**), TTS-stack move to fv-ml1 (**parked, id 75**).
|
||||
|
||||
⚠ **Uncommitted and NOT mine:** `graphify-out/GRAPH_REPORT.md`, `scripts/seat-inventory.py`,
|
||||
`stacks/searxng/{README.md,conf/searxng-settings.yml}`, `configs/esh-scale/`. Leave them alone.
|
||||
- **lv-hemingway keeps its 3 separator-hidden names** (operator: "leave it"), so `leak_gate.py` exits
|
||||
1 on a shipped tree BY DESIGN. A red result there is expected, not a bug to fix.
|
||||
- **`lv-krakauer` PARKED** (operator 2026-09-17), henge id **82**.
|
||||
- **ESH is on the Cityside static** `128.177.138.182/30`, healthy on four axes. The two 7-day crowdsec
|
||||
allowlist entries expire 2026-09-23 and are being left deliberately — Cityside failed twice in six
|
||||
hours on 09-16/17.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-18]` ⭐⭐⭐ **NH3↔Anaheim had been running over a throttled DERP relay, not a direct path — 78 GB of fleet traffic on someone else's free infrastructure.** Four additive objects on ana-gw gave ana-scale a stable inbound UDP 41641 endpoint; `tailscale ping` 373–522 ms → **6 ms direct**, cross-site HTTP 1.2 s → 0.015 s, STT via the ANA gateway 1.4 s → 0.25 s. ⚠ That box runs `central-nat`, so a policy `dstaddr` is the REAL internal address, not the VIP. No OOB access — back up with `show` to a local file and make additive changes ONLY. irv-ml1 still relayed. → `persistent-memory.d/2026-09-18-nh3-ana-derp-relay.md`
|
||||
- `[2026-09-18]` ⭐⭐⭐ **`.internal` DNS was failing ~10% of lookups fleet-wide, from two independent causes.** A PUBLIC resolver was the fallback for a PRIVATE zone (Cloudflare answers NXDOMAIN authoritatively, so a transient miss became a hard failure) — now a cross-site ring, each site local-first with a different site as backup. Then the root cause: all three AdGuards shipped `ratelimit: 20` shared across an entire **/24**, silently dropping queries at 5 s each. Set to 0. Hard failures 3/40 → 0/40; burst timeouts 40/60 → 0/60. ⚠ `resolv.conf` is DHCP-managed — change it at the UDM/FortiGate, not the file. → `persistent-memory.d/2026-09-18-fleet-dns-ring-and-ratelimit.md`
|
||||
- `[2026-09-18]` ⭐⭐ **SearXNG had ONE working general web engine and every health check said fine.** 7 of 55 were enabled-by-default and six of those are dictionary/translation engines — `inactive: false` only makes an engine SELECTABLE, `disabled: false` puts it in the DEFAULT set. Now seven. ⭐ This stack tracks `:latest` ON PURPOSE (upstream ships engine-handler fixes continuously; a pin freezes breakage). ⭐ The Brave key is committed in plaintext by explicit operator decision — scoped to one low-value credential, NOT a change to the no-secrets rule. → `persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md`
|
||||
- `[2026-09-18]` ⭐⭐ **althing search returned ZERO for every hyphenated query — silently, for every handle, and on this fleet that is most of our hostnames.** FTS5 read the hyphen as a column filter; `search()` caught the error, its probe passed, and it returned `[]`. Routed to forseti (they own the code, I own rollout) → **v3.6.3** deployed, 17 s bus outage. The reported symptom, a message that never arrived, was NOT a defect: the reporter's evidence file was truncated at 8 KiB. ⚠ No CI builds the post-office image. → `persistent-memory.d/2026-09-18-althing-363-hyphen-search.md`
|
||||
- `[2026-09-18]` ⭐ **FleetTools: one capability index, autoloaded by Claude, Codex and Grok from a single symlinked file.** `docs/fleettools/` + `~/FLEETTOOLS.md`; absolute detail paths because a non-Claude agent cats them. Rule zero is query-live-inventories-never-a-written-list. The global CLAUDE.md tools section went 231 lines → 39, keeping only the three rules that govern behaviour rather than lookup. → `persistent-memory.d/2026-09-18-fleettools-agent-index.md`
|
||||
- `[2026-09-18]` **Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy** (operator-approved). Verified empirically — `is_dir()`/`exists()` are stat-based and follow symlinks — and the copy had ALREADY drifted (README, since 09-15). Pre-restart gate is `uv run python scripts/sync_plugin.py --check`; `hermes plugins list` is confirmation-after, never permission-before. Copy kept at `~/backups/svos_miranda-copy-20260918-1444`. ⚠ The install now tracks a working tree — an uncommitted edit in svos is what loads at the next restart.
|
||||
- `[2026-09-18]` **Worldtree's `env.sh` secrets are vaulted** — 10 entries under `worldtree/` (gitea, matrix as/hs, openai, uv-index, vastblueai, wt-admin demo+personal, z-ai, zai). ⚠ `Z_AI_API_KEY` and `ZAI_API_KEY` are DIFFERENT keys despite the near-identical names (fp `26bde3fa` vs `a4884152`). `ANTHROPIC_API_KEY` was empty and skipped.
|
||||
|
||||
- `[2026-09-17]` ⭐⭐ **The next voice seat was MEASURED, not chosen by taste — and the corpus size ranking INVERTS the voice ranking at the top.** Our two largest authors are Stephen King (76 works, 12.1M words) and Agatha Christie (72, 5.5M); neither should get a seat, Christie being the Krakauer failure mode exactly (genius in plot architecture, prose deliberately transparent, invisible to a char-bigram Delta). Picks, in order: **Faulkner** (~15 pure novels, ~1.6M words — highest voice signal in the catalogue, AND he is McCarthy's stylistic ancestor, so training him next supplies the **hard-negative sister the gate has lacked since the Brontë record named it missing**); **Morrison** (11 novels after pruning criticism/anthology, ~818k — 11 val units, beating Hemingway's 10); **Chandler** (7 novels + a 409k short-story omnibus — fills the first-person hardboiled gap, Hemingway-class corpus size). ⚠ Faulkner's catalogue rows carry a 446k-word Snopes omnibus that duplicates novels also present individually — the Hemingway 90-96% collection-duplication trap, needs the containment pass first. → `persistent-memory.d/2026-09-17-next-voice-seats.md`
|
||||
- `[2026-09-17]` ⭐⭐ **Romantasy IS a real register, we already trained its most distinctive member, and the obvious next pick is its worst.** Measured on the gate's own instrument (char-bigram Burrows's Delta, ~120k words/author from mid-work), with within-author floors and cross-genre positive controls. Cluster median pair **0.537 = 1.2x the worst floor** against controls at 1.4-1.9x — tighter than cross-genre but NOT collapsed. Two findings survive either floor reading: **Yarros is the cluster OUTLIER** (4 of the 5 largest pair distances involve her), so a second romantasy seat buys measurably less than the first did; and **Maas is the centroid** (the two smallest distances in the matrix are hers), so the obvious commercial pick is the least distinctive. If the lane gets a seat it is **Kenyon** — furthest from Yarros at 0.674 and **27 works = 27 val units, the best-powered gate the line could build** (Hemingway 10, McCarthy 6, Brontë 4). ⚠ Sensitivity floor stated: one sample per pair, no repeat draws; the rank ordering is indicative, fine gaps are not resolvable. → `persistent-memory.d/2026-09-17-romantasy-register-measured.md`
|
||||
- `[2026-09-17]` ⚠⚠ **`dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.** RDAP: registrar **eNom, LLC** (IANA 48, Tucows; abuse@enom.com), created 2008-10-30, status `active` **only** — no `clientTransferProhibited`. Registrant redacted (Verisign RDAP is thin, eNom's own endpoint 404s). Meanwhile `dragonfirepro.com` was re-registered 2025-09-27 by a Hungarian registrar and is now **for sale on expireddomains.com** — the customer lapsed it and a drop-catcher took it. ⭐ **This also makes the cert fix mandatory rather than tidy: two of the three SANs name a domain a THIRD PARTY now owns, so that request can never validate, and our Virtualmin has been retrying it often enough to get the Let's Encrypt account PAUSED.** Nobody is minding this domain — 13-month-dead cert, 403 homepage, sibling already gone — so the six-week clock is a real risk to an 18-year-old .com with a live site. Customer-facing; nothing touched.
|
||||
@@ -535,6 +465,9 @@ _5 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-09-18]` **Routing SearXNG's egress through a SOCKS5 proxy on esh-scale** — one day live, reverted. It fixed nothing, and the reason I gave for reverting it was itself wrong: the rollback was the counterfactual and it falsified my own published claim. Kept reverted on its own merits (no measurable gain, added a hard ESH dependency for all fleet search). → `persistent-memory.d/2026-09-18-searxng-esh-egress-reverted.md`
|
||||
|
||||
- `[2026-09-18]` **`api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings** — this build has NO `!ENV` YAML constructor, so the file was unparseable and the container crash-looped ten times with fleet search down. There is no env-var path into that file at all: the loader reads only `SEARXNG_SETTINGS_PATH` and the entrypoint substitutes only `ultrasecretkey`. Literal or nothing.
|
||||
- `[2026-09-15]` ⚠⚠ **Probing OPNsense API endpoints by POSTing at them — one was `/api/core/system/reboot` and it took the FV site dark for 3.5 min.** Endpoints are ACTIONS; a 200 means it ran. The call I wanted was documented in this repo's own `docs/pfi/opnsense-api-reference.md`. → `persistent-memory.d/2026-09-15-opnsense-api-reboot.md`
|
||||
|
||||
- `[2026-09-15]` **Advertising `10.100.10.50/32` from nh3-dev** to make its LAN address mesh-reachable — black-holed it from ESH/ANA/FV/IRV while its own LAN and the internet kept working, so a one-host check passes cleanly. `lookup 52` at rule priority 5270 beats `main` at 32766. Fix belongs at the router. → `persistent-memory.d/2026-09-15-nh3-dev-ts-input-masquerade.md`
|
||||
|
||||
Reference in New Issue
Block a user