ops(ana-gw): establish the per-flow cap is IPsec-specific, not capacity

The operator asked whether the 80F is simply out of capacity. It is
not. Routing a single flow between two Anaheim VLANs through the same
box, with no tunnel involved, sustains 940 Mbit/s, which saturates the
link, and eight flows over that path return the same figure. There is
therefore no per-session ceiling in the plain forwarding path and the
roughly 250 Mbit/s per-flow limit belongs to the IPsec datapath alone.

Measuring the second tunnel with the same probe supports this. The ESH
side, terminating on different gateway hardware at nearly half the
round-trip time, returns 268 Mbit/s on one stream against NH3's 245,
and 715 against 692 across eight. A window-limited path would have
been substantially faster at the shorter round-trip, so the agreement
between two dissimilar far ends is further evidence of a rate cap.

This also corrects the previous note, which named the FortiGate on
reasoning that could not separate the box from the protocol, since
every slow path was both. That separation is still not established:
both tunnels terminate on UniFi gateways running common firmware, so
the cap could belong to either side. The note records the tunnel
topology that would decide it and flags the bearing on the pending
replacement.
This commit is contained in:
2026-08-23 14:48:34 -07:00
parent cf0cb2cbb3
commit d42e9d8712
@@ -228,3 +228,68 @@ a non-IPsec single stream between two Anaheim VLANs at low RTT.**
FortiGate's IPsec path, replacing the box plausibly lifts single-stream
throughput toward the WireGuard figure. That is a point in favour of the
cutover, and it is cheap to verify afterwards by re-running the sweep.
---
## FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec
Operator asked directly whether the 80F "can't handle the traffic". It can.
Two new measurements settle the shape of this, and correct an overstatement in
FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded —
every slow path was *both* IPsec *and* FortiGate, so protocol and box could not
be separated by that argument).
### The 80F routes a single flow at line rate when IPsec is not involved
`ana-ml2 → pfi-pve`, inter-VLAN **through** ana-gw (traceroute hop 1 =
`10.250.50.1`), 0.36 ms RTT, no tunnel:
| streams | throughput |
|---|---|
| 1 | **940.2 Mbit/s** |
| 8 | 939.3 Mbit/s |
Single stream saturates 1 GbE. So the box does **not** cap single sessions in
general, and there is no per-session ceiling in its plain forwarding path. The
~250 Mbit/s per-flow cap is **specific to the IPsec datapath**.
### Both IPsec tunnels converge on the same numbers despite different far ends
Measured today with the same probe:
| tunnel | far-end gateway | RTT | 1 stream | 8 streams |
|---|---|---|---|---|
| NH3 ↔ ANA | UDM Pro **SE** | 6.7 ms | 245 Mbit/s | 692 Mbit/s |
| ESH ↔ ANA | UDM Pro **Max** | 3.9 ms | **268 Mbit/s** | **715 Mbit/s** |
Different gateway hardware, different sites, different uplinks, and RTT
differing by 1.7× — yet single-stream differs by only 9%. **If this were
window-limited the 3.9 ms path would be ~1.7× faster.** It is not, which is
independent confirmation of a rate cap rather than a BDP effect.
### Capacity summary — the box has headroom it will not give one flow
- plain routing, 1 stream: **940 Mbit/s** (line rate)
- plain routing to internet, 8 streams: **2,153 Mbit/s**
- IPsec, 8 streams: **692715 Mbit/s**
- IPsec, 1 stream: **245268 Mbit/s**
- CPU **100% idle** throughout; IPsec NPU-offloaded (`npu_flag=03`)
Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath
distributes work **by inner session** — consistent with IPsec offload binding a
session to one crypto engine.
### What is still NOT separated
Whether the cap belongs to **the 80F's IPsec offload** or to **UniFi's IPsec
implementation**. Both tunnels have a UDM at the far end, and both UDMs run the
same UniFi firmware, so identical caps are explainable either way. The Pro Max
being only 9% faster than the Pro SE argues against the UniFi side (a beefier
CPU should show more), but that is suggestive, not conclusive.
**The test that closes it:** an IPsec tunnel whose endpoints do not include the
80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured
single-stream. If it also caps ~250, the FortiGate is exonerated and replacing
it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM
WireGuard achieves, the 80F is the limiter. **Bears directly on the pending
FortiGate cutover** — worth running before that decision, not after.