# `[2026-09-17]` ESH: Cityside Fiber failed twice in six hours; site ran on Verizon failover

**Timeline (PDT).**

```
19:09:07  Cityside dies. UDM fails over to Verizon 5G (WAN2). Site stays up at ~33 ms.
19:51     Verified healthy on failover.
20:01:37  esh-scale drops off the headscale mesh; 10.0.0.0/16 withdrawn; whole site dark
          from the colo. Beszel fires on all five ESH hosts.
20:11-15  Service restored. Operator had switched WAN1 to DHCP to get Cityside working at
          all; it came back on CGNAT, not the static. Latency back to 9 ms.
01:06:23  Cityside fails AGAIN. Failover to Verizon. Site up, ~37 ms.
```

⭐ **The 20:01 blackout was most likely the operator's own WAN reconfiguration**, not ISP
instability — switching the WAN type bounces the interface, esh-scale loses its path,
headscale withdraws the route, and the site vanishes from the colo's view until it settles.
An earlier session theory ("Cityside came back half-provisioned and the UDM failed back into
an unstable session") is retired.

⚠ **The diagnostic that mattered: physical link stayed UP at 2.5 GE with zero errors
throughout, while the ISP's next-hop `128.177.138.181` was unresponsive.** So "the ONT is
fine, it is upstream of the ONT" — the line to give Cityside. Traceroute from NH3 reached
`209.249.146.170` (one hop short) before dying, so the prefix was still routed.

⚠ **CROWDSEC ROTATION FRAGILITY IS LIVE.** The `esh` allowlist on ana-docker carries the now-
dark static `128.177.138.182` (never-expiry), plus `97.190.18.88` (Verizon failover) and
`23.164.40.174` (Cityside CGNAT), both **7-day expiry**. ESH is on a rotating carrier-NAT
egress until the static is restored — the exact regime the 09-08 static purchase was meant to
end, and the class that once blackholed the whole site via a false ban. **If ESH loses colo
access, check `curl -s4 ifconfig.me` from esh-docker-vm FIRST** and allowlist the new address.

**Still pinned to the dark static and broken until it returns:** FortiGate `infra-ops`
trusthost3 = `128.177.138.182`, so logins to ana-gw from ESH are refused. The dormant
`esh-ana` IPsec is bound to wan1/static (disabled, so no impact).

⭐ **The mesh was NOT degraded on CGNAT** — tailscale hole-punched straight through
(`direct 23.164.40.174:41641`), which is why latency read 9 ms rather than a DERP figure. An
expectation of relay-on-CGNAT was wrong.

## RESOLVED 2026-09-17 ~12:30 PT — the static is back, confirmed on four axes

Not one check, because egress alone cannot tell a static WAN from a carrier NAT that happens
to answer (see auto-memory `feedback_egress_ip_cannot_detect_cgnat`):

```
config     UDM WAN1 `wan_type = static`, ip 128.177.138.182, mask /30, gw 128.177.138.181
           — switched BACK from the DHCP the operator set at 20:11 during the outage
active     stat/health: isp_name "Cityside Fiber", ASN 18731, num_disconnected 0.
           WAN2 Verizon-5G is failover-only at priority 2 and idle.
egress     esh-docker-vm sees 128.177.138.182 — EQUAL to the WAN ip, so not behind CGNAT
perf       2005/2142 Mbps symmetric; colo -> ESH 5.0 ms, 0% loss over 4 hosts-worth of pings
           (Cityside CGNAT read 9 ms, Verizon failover 33-37 ms)
```

⭐ **The FortiGate pin un-broke itself and that was verified, not inferred.** `infra-ops`
trusthost3 is `128.177.138.182`; from esh-docker-vm, ana-gw `tcp/22` is OPEN and the
FortiGate offers a password prompt rather than dropping the connection — a trusthost
mismatch refuses outright, so reaching auth *is* the trusthost passing. The dormant
`esh-ana` IPsec bind to wan1/static is correct again (still disabled, still no impact).

⚠ **The crowdsec temporary allowlist entries are being LEFT to expire on their own**
(2026-09-23): `97.190.18.88` Verizon and `23.164.40.174` Cityside CGNAT. Cityside failed
twice in six hours on 09-17, so until the line has earned some confidence those two are
cheap insurance against the exact false-ban blackout this rotation-fragility caused before.
`128.177.138.182` stays never-expiry.

Related: [[2026-09-06-headscale-cutover]], [[2026-09-08-esh-static-wan-followups-and-ytvc]].