memory: ESH back on the Cityside static (confirmed four ways); lv-hemingway left as-is per operator

This commit is contained in:
2026-09-17 15:53:11 -07:00
parent 3ac13c5351
commit 9a1c028b9f
2 changed files with 36 additions and 7 deletions
@@ -38,4 +38,31 @@ trusthost3 = `128.177.138.182`, so logins to ana-gw from ESH are refused. The do
(`direct 23.164.40.174:41641`), which is why latency read 9 ms rather than a DERP figure. An
expectation of relay-on-CGNAT was wrong.
## RESOLVED 2026-09-17 ~12:30 PT — the static is back, confirmed on four axes
Not one check, because egress alone cannot tell a static WAN from a carrier NAT that happens
to answer (see auto-memory `feedback_egress_ip_cannot_detect_cgnat`):
```
config UDM WAN1 `wan_type = static`, ip 128.177.138.182, mask /30, gw 128.177.138.181
— switched BACK from the DHCP the operator set at 20:11 during the outage
active stat/health: isp_name "Cityside Fiber", ASN 18731, num_disconnected 0.
WAN2 Verizon-5G is failover-only at priority 2 and idle.
egress esh-docker-vm sees 128.177.138.182 — EQUAL to the WAN ip, so not behind CGNAT
perf 2005/2142 Mbps symmetric; colo -> ESH 5.0 ms, 0% loss over 4 hosts-worth of pings
(Cityside CGNAT read 9 ms, Verizon failover 33-37 ms)
```
**The FortiGate pin un-broke itself and that was verified, not inferred.** `infra-ops`
trusthost3 is `128.177.138.182`; from esh-docker-vm, ana-gw `tcp/22` is OPEN and the
FortiGate offers a password prompt rather than dropping the connection — a trusthost
mismatch refuses outright, so reaching auth *is* the trusthost passing. The dormant
`esh-ana` IPsec bind to wan1/static is correct again (still disabled, still no impact).
**The crowdsec temporary allowlist entries are being LEFT to expire on their own**
(2026-09-23): `97.190.18.88` Verizon and `23.164.40.174` Cityside CGNAT. Cityside failed
twice in six hours on 09-17, so until the line has earned some confidence those two are
cheap insurance against the exact false-ban blackout this rotation-fragility caused before.
`128.177.138.182` stays never-expiry.
Related: [[2026-09-06-headscale-cutover]], [[2026-09-08-esh-static-wan-followups-and-ytvc]].
+9 -7
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-17 ~12:15 PT (lv-mccarthy: the leak gate had PASSED with five protagonist names still in every copy — hidden from `\bName\b` by a drop-cap space or a line-break hyphen. Corpus rebuilt, gate taught to see the class, D4 pairs generating on gx10. lv-hemingway carries 3 of the same and is LIVE — operator call. lv-krakauer parked.)_
_Last updated: 2026-09-17 ~12:15 PT (lv-mccarthy: the leak gate had PASSED with five protagonist names still in every copy — hidden from `\bName\b` by a drop-cap space or a line-break hyphen. Corpus rebuilt, gate taught to see the class, D4 pairs generating on gx10. lv-hemingway keeps its 3 — operator ruled leave it, so the gate now exits 1 on a shipped tree BY DESIGN. ESH back on the Cityside static. lv-krakauer parked.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -143,13 +143,13 @@ Then, before the eval, honour D1's pre-registration: a **punctuation-normalised
read** of the voice axis — and note the `mccarthy` register now states the tics, so the
base control arm gets them too.
### OPERATOR CALL — lv-hemingway carries 3 of the same hidden survivors and is LIVE
### SETTLED — lv-hemingway keeps its 3 hidden survivors; operator ruled "leave it"
`Primi tivo`, `Pasionar ia`, `Chi cote` one occurrence each per copy, in all 6 copies of a
958k-word corpus, on the adapter serving at fv-ml1 GPU 0 `:8027`. Confirmed by running the
committed gate, which now exits 1 on it (34.6 s). lv-bronte re-run clean (0, 9.7 s).
Re-gating means a corpus rebuild and a retrain; three names may not be worth it.
**Not acted on — this one is yours.**
`Primi tivo`, `Pasionar ia`, `Chi cote`, one each per copy in a 958k-word corpus, on the
adapter serving at fv-ml1 GPU 0 `:8027`. Operator 2026-09-17: **leave it.** lv-bronte re-ran
clean. ⚠ **Consequence to remember: `leak_gate.py` now EXITS 1 on lv-hemingway's tree.** A
future session re-running that gate will see a red result on a shipped artifact — that is
expected and settled, not a regression to chase. Do not "fix" it without a fresh ruling.
### SHIPPED — `lv-hemingway` on ckpt850, live on `vllm-voices` (fv-ml1 GPU 0 :8027)
@@ -194,6 +194,8 @@ fused MoE kernel path (**parked, id 47**), TTS-stack move to fv-ml1 (**parked, i
## Recent decisions
- `[2026-09-17]` **ESH is back on the Cityside static `128.177.138.182/30` and the site is healthy — confirmed on four axes, not one.** UDM WAN1 `wan_type` is `static` again (switched back from the DHCP set during the 09-17 outage), `stat/health` names Cityside Fiber with 0 disconnected and Verizon-5G idle at failover priority 2, esh-docker-vm's egress EQUALS the WAN ip so nothing is behind CGNAT, and colo→ESH reads **5.0 ms / 0% loss** at 2005/2142 Mbps (Cityside CGNAT was 9 ms, Verizon failover 3337 ms). ⭐ The FortiGate `infra-ops` trusthost3 pin un-broke itself and that was VERIFIED: from ESH, ana-gw tcp/22 is open and offers a password prompt, which a trusthost mismatch would never do. ⚠ The two 7-day crowdsec entries are being left to expire 2026-09-23 on purpose — Cityside failed twice in six hours, so they are cheap insurance. → `persistent-memory.d/2026-09-17-esh-fiber-outages.md`
- `[2026-09-17]` **Operator ruled "leave it" on lv-hemingway's 3 separator-hidden names.** So `leak_gate.py` exits 1 on a SHIPPED tree by design; a future session seeing that red result should read this line, not start fixing. lv-bronte re-ran clean.
- `[2026-09-17]` ⭐⭐⭐ **The leak gate PASSED lv-mccarthy while five protagonist names sat in all six copies, and the blind spot generalises to every corpus in the line.** `\b(Surface)\b` cannot match a name with a character inserted in it, so a mangled occurrence is unrenameable AND unreportable: `B ell`, `C higurh`, `M oss`, `T oadvine` (a small-caps drop cap kept as its own token) and `Toad-vine`, `Glan-ton` (a print line-break hyphen). Every VISIBLE occurrence had been renamed, which is what made it invisible. Same family as lv-bronte's `_Antigua_`, now generalised: **any separator inside a name blinds a word-boundary scan.** Fixed in the corpus builder (rules 4+5, counted), and `leak_gate.py` now runs a separator-tolerant pass with its own controls that FAILS the gate — validated against the pre-fix tree. ⚠ Its fragment filter is load-bearing: a naive scan returns 18 false positives on Hemingway (`God damn`, `I run`) against 3 real. Whole D1→D3 chain reproduced byte-identically before and after. Commit `c559664`. → `persistent-memory.d/2026-09-17-mccarthy-split-name-leak.md`
- `[2026-09-17]` ⭐⭐ **The SHIPPED lv-bronte adapter emits mid-sentence line breaks at 12.46 per 1k chars, and nothing downstream looks for it.** Its corpus is 100% hard-wrapped at ~68 chars (Gutenberg plain text) and the wrap transfers: base control 0.00, ckpt475 (shipped) 12.46, ckpt925 11.79, every Hemingway arm 0.00 on a 0%-wrapped corpus. Both controls fire. `score_beats.py` passed Brontë's damage axis anyway. McCarthy is the MIXED case — The Road wrapped, the other five works not — which is worse to learn than either pure one, so `build_sft_pairs.py --reflow-hard-wraps` (DEFECT 4) fixes it at pair time, off by default. ⚠ The obvious fix, joining every interior newline, CORRUPTS 46 two-speaker exchanges whose blank line was lost — and unmarked dialogue is the one thing this adapter exists to learn. The rule splits on sentence-final punctuation and takes the cheaper error deliberately.
- `[2026-09-17]`**The `mccarthy` register names the punctuation ON PURPOSE, and that is a gate-design call made before any McCarthy number existed.** `eval-*.sh` drives the base control arm with the SAME system prompt via `--system-from`, and `voice_distance.py` is Burrows's Delta over character bigrams — so a tic left OUT of the register is a cheap win only the adapter can take, on a corpus measuring 0.0 quote marks per 10k against Hemingway's 838. Stating them hands them to the control too. Cost stated up front: the voice axis gets harder, and McCarthy's 276-passage val split (against Brontë's 44) is why that trade is affordable here and was not there.