4.1 KiB
[2026-09-17] ESH: Cityside Fiber failed twice in six hours; site ran on Verizon failover
Timeline (PDT).
19:09:07 Cityside dies. UDM fails over to Verizon 5G (WAN2). Site stays up at ~33 ms.
19:51 Verified healthy on failover.
20:01:37 esh-scale drops off the headscale mesh; 10.0.0.0/16 withdrawn; whole site dark
from the colo. Beszel fires on all five ESH hosts.
20:11-15 Service restored. Operator had switched WAN1 to DHCP to get Cityside working at
all; it came back on CGNAT, not the static. Latency back to 9 ms.
01:06:23 Cityside fails AGAIN. Failover to Verizon. Site up, ~37 ms.
⭐ The 20:01 blackout was most likely the operator's own WAN reconfiguration, not ISP instability — switching the WAN type bounces the interface, esh-scale loses its path, headscale withdraws the route, and the site vanishes from the colo's view until it settles. An earlier session theory ("Cityside came back half-provisioned and the UDM failed back into an unstable session") is retired.
⚠ The diagnostic that mattered: physical link stayed UP at 2.5 GE with zero errors
throughout, while the ISP's next-hop 128.177.138.181 was unresponsive. So "the ONT is
fine, it is upstream of the ONT" — the line to give Cityside. Traceroute from NH3 reached
209.249.146.170 (one hop short) before dying, so the prefix was still routed.
⚠ CROWDSEC ROTATION FRAGILITY IS LIVE. The esh allowlist on ana-docker carries the now-
dark static 128.177.138.182 (never-expiry), plus 97.190.18.88 (Verizon failover) and
23.164.40.174 (Cityside CGNAT), both 7-day expiry. ESH is on a rotating carrier-NAT
egress until the static is restored — the exact regime the 09-08 static purchase was meant to
end, and the class that once blackholed the whole site via a false ban. If ESH loses colo
access, check curl -s4 ifconfig.me from esh-docker-vm FIRST and allowlist the new address.
Still pinned to the dark static and broken until it returns: FortiGate infra-ops
trusthost3 = 128.177.138.182, so logins to ana-gw from ESH are refused. The dormant
esh-ana IPsec is bound to wan1/static (disabled, so no impact).
⭐ The mesh was NOT degraded on CGNAT — tailscale hole-punched straight through
(direct 23.164.40.174:41641), which is why latency read 9 ms rather than a DERP figure. An
expectation of relay-on-CGNAT was wrong.
RESOLVED 2026-09-17 ~12:30 PT — the static is back, confirmed on four axes
Not one check, because egress alone cannot tell a static WAN from a carrier NAT that happens
to answer (see auto-memory feedback_egress_ip_cannot_detect_cgnat):
config UDM WAN1 `wan_type = static`, ip 128.177.138.182, mask /30, gw 128.177.138.181
— switched BACK from the DHCP the operator set at 20:11 during the outage
active stat/health: isp_name "Cityside Fiber", ASN 18731, num_disconnected 0.
WAN2 Verizon-5G is failover-only at priority 2 and idle.
egress esh-docker-vm sees 128.177.138.182 — EQUAL to the WAN ip, so not behind CGNAT
perf 2005/2142 Mbps symmetric; colo -> ESH 5.0 ms, 0% loss over 4 hosts-worth of pings
(Cityside CGNAT read 9 ms, Verizon failover 33-37 ms)
⭐ The FortiGate pin un-broke itself and that was verified, not inferred. infra-ops
trusthost3 is 128.177.138.182; from esh-docker-vm, ana-gw tcp/22 is OPEN and the
FortiGate offers a password prompt rather than dropping the connection — a trusthost
mismatch refuses outright, so reaching auth is the trusthost passing. The dormant
esh-ana IPsec bind to wan1/static is correct again (still disabled, still no impact).
⚠ The crowdsec temporary allowlist entries are being LEFT to expire on their own
(2026-09-23): 97.190.18.88 Verizon and 23.164.40.174 Cityside CGNAT. Cityside failed
twice in six hours on 09-17, so until the line has earned some confidence those two are
cheap insurance against the exact false-ban blackout this rotation-fragility caused before.
128.177.138.182 stays never-expiry.
Related: 2026-09-06-headscale-cutover, 2026-09-08-esh-static-wan-followups-and-ytvc.