vh
c445ce9e93
In-flight rewritten for the next goal. lv-hemingway is TRAINED and nothing else has been done to it: ship candidate is checkpoint-1750 (ep 1.97, eval 2.2783), the end-of-run adapter is 0.0763 worse, and the v2 gate has not been run. Every instrument it needs was parameterised during the lv-bronte run tonight and the in-flight section names all four with their traps. New detail files: 2026-09-17-lv-bronte-gate.md shipped, voice axis failed, why anyway 2026-09-17-beat-contamination-leak.md the leak the corpus gate cannot see 2026-09-17-esh-fiber-outages.md two Cityside failures, rotation fragility Also commits the memorization_check.py parameterisation, which was left uncommitted: its hardcoded Yarros defaults would have compared a Hemingway arm against the Yarros corpus and reported a meaningless clean zero. Auto-archival: index was 415 lines pre-run, over the 300 cap. Only five entries cleared the 14-day age guard, and three of those carry open deferred pointers (fused MoE park 47, nconnect=8, AI-tab belayed) and are referenced by in-flight. A fourth — every CI job on pfi-fleet runs as root on ana-docker — is a live security property rather than settled history, so it is held back deliberately. One entry archived. The file stays over cap, which is the guard working: an over-cap file that keeps live decisions beats a scannable one that lost them.
2.4 KiB
[2026-09-17] ESH: Cityside Fiber failed twice in six hours; site ran on Verizon failover
Timeline (PDT).
19:09:07 Cityside dies. UDM fails over to Verizon 5G (WAN2). Site stays up at ~33 ms.
19:51 Verified healthy on failover.
20:01:37 esh-scale drops off the headscale mesh; 10.0.0.0/16 withdrawn; whole site dark
from the colo. Beszel fires on all five ESH hosts.
20:11-15 Service restored. Operator had switched WAN1 to DHCP to get Cityside working at
all; it came back on CGNAT, not the static. Latency back to 9 ms.
01:06:23 Cityside fails AGAIN. Failover to Verizon. Site up, ~37 ms.
⭐ The 20:01 blackout was most likely the operator's own WAN reconfiguration, not ISP instability — switching the WAN type bounces the interface, esh-scale loses its path, headscale withdraws the route, and the site vanishes from the colo's view until it settles. An earlier session theory ("Cityside came back half-provisioned and the UDM failed back into an unstable session") is retired.
⚠ The diagnostic that mattered: physical link stayed UP at 2.5 GE with zero errors
throughout, while the ISP's next-hop 128.177.138.181 was unresponsive. So "the ONT is
fine, it is upstream of the ONT" — the line to give Cityside. Traceroute from NH3 reached
209.249.146.170 (one hop short) before dying, so the prefix was still routed.
⚠ CROWDSEC ROTATION FRAGILITY IS LIVE. The esh allowlist on ana-docker carries the now-
dark static 128.177.138.182 (never-expiry), plus 97.190.18.88 (Verizon failover) and
23.164.40.174 (Cityside CGNAT), both 7-day expiry. ESH is on a rotating carrier-NAT
egress until the static is restored — the exact regime the 09-08 static purchase was meant to
end, and the class that once blackholed the whole site via a false ban. If ESH loses colo
access, check curl -s4 ifconfig.me from esh-docker-vm FIRST and allowlist the new address.
Still pinned to the dark static and broken until it returns: FortiGate infra-ops
trusthost3 = 128.177.138.182, so logins to ana-gw from ESH are refused. The dormant
esh-ana IPsec is bound to wan1/static (disabled, so no impact).
⭐ The mesh was NOT degraded on CGNAT — tailscale hole-punched straight through
(direct 23.164.40.174:41641), which is why latency read 9 ms rather than a DERP figure. An
expectation of relay-on-CGNAT was wrong.
Related: 2026-09-06-headscale-cutover, 2026-09-08-esh-static-wan-followups-and-ytvc.