diff --git a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md index 8c1b1a3..e137c08 100644 --- a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md +++ b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md @@ -228,3 +228,68 @@ a non-IPsec single stream between two Anaheim VLANs at low RTT.** FortiGate's IPsec path, replacing the box plausibly lifts single-stream throughput toward the WireGuard figure. That is a point in favour of the cutover, and it is cheap to verify afterwards by re-running the sweep. + +--- + +## FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec + +Operator asked directly whether the 80F "can't handle the traffic". It can. +Two new measurements settle the shape of this, and correct an overstatement in +FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded — +every slow path was *both* IPsec *and* FortiGate, so protocol and box could not +be separated by that argument). + +### The 80F routes a single flow at line rate when IPsec is not involved + +`ana-ml2 → pfi-pve`, inter-VLAN **through** ana-gw (traceroute hop 1 = +`10.250.50.1`), 0.36 ms RTT, no tunnel: + +| streams | throughput | +|---|---| +| 1 | **940.2 Mbit/s** | +| 8 | 939.3 Mbit/s | + +Single stream saturates 1 GbE. So the box does **not** cap single sessions in +general, and there is no per-session ceiling in its plain forwarding path. The +~250 Mbit/s per-flow cap is **specific to the IPsec datapath**. + +### Both IPsec tunnels converge on the same numbers despite different far ends + +Measured today with the same probe: + +| tunnel | far-end gateway | RTT | 1 stream | 8 streams | +|---|---|---|---|---| +| NH3 ↔ ANA | UDM Pro **SE** | 6.7 ms | 245 Mbit/s | 692 Mbit/s | +| ESH ↔ ANA | UDM Pro **Max** | 3.9 ms | **268 Mbit/s** | **715 Mbit/s** | + +Different gateway hardware, different sites, different uplinks, and RTT +differing by 1.7× — yet single-stream differs by only 9%. **If this were +window-limited the 3.9 ms path would be ~1.7× faster.** It is not, which is +independent confirmation of a rate cap rather than a BDP effect. + +### Capacity summary — the box has headroom it will not give one flow + +- plain routing, 1 stream: **940 Mbit/s** (line rate) +- plain routing to internet, 8 streams: **2,153 Mbit/s** +- IPsec, 8 streams: **692–715 Mbit/s** +- IPsec, 1 stream: **245–268 Mbit/s** +- CPU **100% idle** throughout; IPsec NPU-offloaded (`npu_flag=03`) + +Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath +distributes work **by inner session** — consistent with IPsec offload binding a +session to one crypto engine. + +### What is still NOT separated + +Whether the cap belongs to **the 80F's IPsec offload** or to **UniFi's IPsec +implementation**. Both tunnels have a UDM at the far end, and both UDMs run the +same UniFi firmware, so identical caps are explainable either way. The Pro Max +being only 9% faster than the Pro SE argues against the UniFi side (a beefier +CPU should show more), but that is suggestive, not conclusive. + +**The test that closes it:** an IPsec tunnel whose endpoints do not include the +80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured +single-stream. If it also caps ~250, the FortiGate is exonerated and replacing +it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM +WireGuard achieves, the 80F is the limiter. **Bears directly on the pending +FortiGate cutover** — worth running before that decision, not after.