# `[2026-09-17]` ESH: Cityside Fiber failed twice in six hours; site ran on Verizon failover

**Timeline (PDT).**

```
19:09:07  Cityside dies. UDM fails over to Verizon 5G (WAN2). Site stays up at ~33 ms.
19:51     Verified healthy on failover.
20:01:37  esh-scale drops off the headscale mesh; 10.0.0.0/16 withdrawn; whole site dark
          from the colo. Beszel fires on all five ESH hosts.
20:11-15  Service restored. Operator had switched WAN1 to DHCP to get Cityside working at
          all; it came back on CGNAT, not the static. Latency back to 9 ms.
01:06:23  Cityside fails AGAIN. Failover to Verizon. Site up, ~37 ms.
```

⭐ **The 20:01 blackout was most likely the operator's own WAN reconfiguration**, not ISP
instability — switching the WAN type bounces the interface, esh-scale loses its path,
headscale withdraws the route, and the site vanishes from the colo's view until it settles.
An earlier session theory ("Cityside came back half-provisioned and the UDM failed back into
an unstable session") is retired.

⚠ **The diagnostic that mattered: physical link stayed UP at 2.5 GE with zero errors
throughout, while the ISP's next-hop `128.177.138.181` was unresponsive.** So "the ONT is
fine, it is upstream of the ONT" — the line to give Cityside. Traceroute from NH3 reached
`209.249.146.170` (one hop short) before dying, so the prefix was still routed.

⚠ **CROWDSEC ROTATION FRAGILITY IS LIVE.** The `esh` allowlist on ana-docker carries the now-
dark static `128.177.138.182` (never-expiry), plus `97.190.18.88` (Verizon failover) and
`23.164.40.174` (Cityside CGNAT), both **7-day expiry**. ESH is on a rotating carrier-NAT
egress until the static is restored — the exact regime the 09-08 static purchase was meant to
end, and the class that once blackholed the whole site via a false ban. **If ESH loses colo
access, check `curl -s4 ifconfig.me` from esh-docker-vm FIRST** and allowlist the new address.

**Still pinned to the dark static and broken until it returns:** FortiGate `infra-ops`
trusthost3 = `128.177.138.182`, so logins to ana-gw from ESH are refused. The dormant
`esh-ana` IPsec is bound to wan1/static (disabled, so no impact).

⭐ **The mesh was NOT degraded on CGNAT** — tailscale hole-punched straight through
(`direct 23.164.40.174:41641`), which is why latency read 9 ms rather than a DERP figure. An
expectation of relay-on-CGNAT was wrong.

Related: [[2026-09-06-headscale-cutover]], [[2026-09-08-esh-static-wan-followups-and-ytvc]].