vh
d42e9d8712
The operator asked whether the 80F is simply out of capacity. It is not. Routing a single flow between two Anaheim VLANs through the same box, with no tunnel involved, sustains 940 Mbit/s, which saturates the link, and eight flows over that path return the same figure. There is therefore no per-session ceiling in the plain forwarding path and the roughly 250 Mbit/s per-flow limit belongs to the IPsec datapath alone. Measuring the second tunnel with the same probe supports this. The ESH side, terminating on different gateway hardware at nearly half the round-trip time, returns 268 Mbit/s on one stream against NH3's 245, and 715 against 692 across eight. A window-limited path would have been substantially faster at the shorter round-trip, so the agreement between two dissimilar far ends is further evidence of a rate cap. This also corrects the previous note, which named the FortiGate on reasoning that could not separate the box from the protocol, since every slow path was both. That separation is still not established: both tunnels terminate on UniFi gateways running common firmware, so the cap could belong to either side. The note records the tunnel topology that would decide it and flags the bearing on the pending replacement.
14 KiB
[2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit
⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at the bottom before acting on anything here. The headline is wrong (the relevant ceiling is NH3's 1 Gbps uplink, not Anaheim's 2 Gbps), the aggregate number is wrong (692 Mbit/s at 8 streams, not ~550 — the original stopped measuring at 4), and the proposed remedy is impossible: UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The per-stream observation and the parallelise-your-transfers mitigation are the parts that survive.
The operator noticed site-to-site transfers were slow for a datacenter fiber handoff and asked whether WireGuard was the limit. It is not WireGuard, and the circuit is fine.
Measured
ana-docker -> internet, 8 parallel 2,153 Mbit/s <- the 2 Gbps handoff, delivering
ANA <-> NH3 through the tunnel, 4 par. 460 Mbit/s
FortiGate's own recorded peak 554 Mbit/s
ANA <-> NH3, single stream 227 Mbit/s
ANA <-> ESH, single stream 249-265 Mbit/s
ESH <-> NH3 (never touches ana-gw) 545-557 Mbit/s on a SINGLE stream
Method: stdlib TCP probe (no ssh, no crypto, no compression) between site endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling.
What it is not
- Not WireGuard. Both Anaheim tunnels are IPsec on ana-gw
(
pfi-ana-nh3-> 70.230.226.88,ana-eshudm-dyn-> the ESH UDM). WireGuard on ana-wg is remote-access only and is not in this path. Traceroute confirms: both slow paths have hop 1 =10.250.50.1(the FortiGate); the fast ESH<->NH3 path rides a192.168.xSite Magic overlay and never touches it. - Not CPU or crypto exhaustion. FortiGate CPU was 100% idle across all
8 cores during the tests, and both live tunnels report
npu_flag=03withdec_npuid=1 enc_npuid=1— encrypt and decrypt are hardware-offloaded. - Not a 250 Mbit/s cap. That was the first number and it is misleading — single-stream TCP. Four parallel streams doubled it. Quote the aggregate.
- Not the interface. wan1:
rxe=0 txe=0 rxd=0 txd=0, no collisions.
Most likely cause
Both tunnels negotiate aes256-sha1 in phase 1 and phase 2 (dhgrp 14,
IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly
faster on AES-GCM, which combines encryption and authentication in one
pass. The datasheet IPsec headline for an 80F assumes GCM with large packets,
not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x
shortfall is consistent with that.
Not executed
Changing the proposal is a production-edge change requiring a matching change at the far end (NH3 UDM and the ESH UDM), and each tunnel drops while it renegotiates. Left for the operator. See the index entry for authorization state.
Immediate mitigation, no config change
Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a single stream through IPsec gets 227. Anything moving bulk data across the Anaheim link should parallelise — that alone roughly doubles throughput today.
Practical consequence already observed
/mnt/smithy mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s
from nh3-dev (same file, same mount) — that gap is this tunnel, not NFS and
not the NAS. See 2026-08-23-smithy-mount-ana-ml2.
Access note
ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. sshpass is absent
on nh3-dev; connect with paramiko via uv run --with paramiko. Password is
vaulted at fortigate/ana-gw-infra-ops-password. diagnose vpn tunnel list
prints live ESP session keys — never paste its output into althing, a
booth, or a commit.
CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist
The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done, and the measurements taken while trying show there is very little left to win.
AES-GCM is unavailable on the far end — not a naming problem
The NH3 edge is a UDM Pro SE terminating pfi-nh3-ana (networkconf
_id 697d64414c85dd2b6669b00a, ifname vti64). Its UniFi API validates the
crypto enum and rejected every GCM spelling tried — aes256gcm, aes256gcm128,
aes256gcm16, aes-256-gcm, aes256-gcm, aes256gcm12, gcm, aes128gcm128
— all HTTP 400 api.err.InvalidPayload, nothing applied.
The control that makes this conclusive: the identical request body with
ipsec_esp_encryption: "aes256" returns HTTP 200 rc:ok. So the 400s are the
enum rejecting the value, not a malformed body. Corroborating: zero
case-insensitive gcm matches across 7.3 MB of UniFi OS UI bundles.
Accepted enum (probed): aes128, aes192, aes256, 3des → 200; des,
chacha20poly1305 → 400. There is no AEAD option. Both Anaheim tunnels land on
UniFi far ends, so this blocks the ESH tunnel too.
The FortiGate side was widened and is GCM-capable: phase2 pfi-ana-nh3 now
reads set proposal aes256-sha1 aes256gcm. Left in place deliberately — it is
functionally identical while the peer only offers CBC, and reverting it would
cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched;
IKE protects the control channel only and has no bearing on data throughput.
The numbers that retire this as a problem
Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on the same UDM and the same uplink, with the same stdlib TCP probe:
| streams | IPsec NH3→ANA | WireGuard NH3→ESH |
|---|---|---|
| 1 | 245 Mbit/s | 557 Mbit/s |
| 4 | 471 Mbit/s | 767 Mbit/s |
| 8 | 692 Mbit/s | 795 Mbit/s |
NH3's WAN is a 1 Gbps link (uplink.speed = 1000, port capable of 10G) —
that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel.
So the tunnel does ~69% of the achievable uplink at 8 streams, and the
IPsec-vs-WireGuard gap collapses from 2.3× at one stream to 15% at eight.
Re-architecting the transport (site-to-site WireGuard via ana-wg, since
FortiOS has no WireGuard) would chase that last 15%. Not worth it.
What the constraint actually is
A per-stream limit (~245 Mbit/s), not an aggregate crypto ceiling. Both
endpoints are idle at load — FortiGate CPU 100% idle with npu_flag=03
(offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The
shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from
a higher floor (557 → 795 is only 1.43× scaling).
Actionable consequence
Anything moving bulk data across this link should parallelise — 245 → 692
Mbit/s, a 2.8× win with no config change. For single-stream workloads that
cannot be parallelised at the application layer, NFS nconnect=N is the
lever: it opens N TCP connections per mount, converting a single-stream
workload into a parallel one. The /mnt/smithy mount on ana-ml2 reading at
24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live
example — remounting with nconnect=8 is the obvious test.
Foot-gun recorded
Probing the enum by PUTting candidate values applies the accepted ones. A
probe loop here timed out with 3des briefly live on the NH3 side, which the
FortiGate would not accept — a short tunnel outage until aes256 was restored
(~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi
config enum this way, restore the known-good value after every 200, not at
the end of the loop. Post-change verification: the UDM object was diffed
field-by-field against its pre-change snapshot and is byte-identical.
FOLLOW-UP (2026-08-23): what the per-stream limit actually is
The correction above called the constraint "per-SA/per-flow serialisation". That was a hand-wave. Measured properly, it is a hard per-flow rate cap of ~230–245 Mbit/s with a very deep buffer in front of it — not a tuning problem, not loss, not window size.
The evidence: pin the send buffer and sweep it
Single stream NH3 → ana-docker, SO_SNDBUF pinned, ss -ti sampled in flight:
| in-flight cap | throughput | RTT in flight | minRTT | retrans |
|---|---|---|---|---|
| 256 KB | 224 Mbit/s | 7.8 ms | 5.3 ms | 0 |
| 416 KB | 225 Mbit/s | 11.8 ms | 6.6 ms | 0 |
| 416 KB | 245–247 Mbit/s | 12.0 ms | 5.6 ms | 0 |
| ~3.3 MB (autotuned) | 245 Mbit/s | 107 ms | 5.5 ms | 0 |
Throughput is flat across a 13× range of in-flight data while RTT scales with
it. That is the signature of a fixed service rate with a standing queue: the
window controls only how much queue you build, never how fast you go. Had this
been window-limited, throughput would have risen with the buffer. Had it been
congestion, there would be retransmits — there are essentially none
(retrans:0, 0% ping loss).
So net.ipv4.tcp_* tuning, window scaling and congestion-control choice are all
red herrings here. Do not go there.
Bufferbloat: one bulk stream wrecks latency for everything else
Measured on the same tunnel, ping to ana-docker:
- idle: 6.9 ms avg
- during a single bulk TCP stream: 102 ms avg, 136 ms max, 0% loss
15× latency inflation from one transfer. This is the operationally important finding — any interactive traffic sharing the Anaheim link (ssh, RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it takes only one stream to do it. Parallelising transfers makes throughput better and this worse. If it starts biting, the fix is an AQM/shaper on the tunnel (or rate-limiting bulk jobs), not more buffer.
Where the cap lives — strong inference, not proof
Three paths, and the FortiGate is the only variable:
| path | single-stream |
|---|---|
| FortiGate ↔ NH3 UDM (IPsec) | 245 Mbit/s |
| FortiGate ↔ ESH UDM (IPsec) | 249–265 Mbit/s |
| NH3 UDM ↔ ESH UDM (WireGuard, no FortiGate) | 557 Mbit/s |
Present in both slow paths, absent from the fast one. Aggregate over the same SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole — many flows spread out fine, one flow does not.
The mechanism that fits is FortiGate NPU IPsec offload being per-session:
each firewall session is bound to one crypto engine, so a single TCP flow is
capped at one engine's rate while many sessions spread across engines. This
is inference from the throughput shape, not something confirmed on the box —
diagnose sys session list was not captured for a TCP flow (the filter caught
only traceroute UDP probes). A single-stream control through ana-gw without
IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is
window-limited and does not discriminate. If this matters, the clean test is
a non-IPsec single stream between two Anaheim VLANs at low RTT.
Relevant to the FortiGate cutover decision: if the per-flow cap is the FortiGate's IPsec path, replacing the box plausibly lifts single-stream throughput toward the WireGuard figure. That is a point in favour of the cutover, and it is cheap to verify afterwards by re-running the sweep.
FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec
Operator asked directly whether the 80F "can't handle the traffic". It can. Two new measurements settle the shape of this, and correct an overstatement in FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded — every slow path was both IPsec and FortiGate, so protocol and box could not be separated by that argument).
The 80F routes a single flow at line rate when IPsec is not involved
ana-ml2 → pfi-pve, inter-VLAN through ana-gw (traceroute hop 1 =
10.250.50.1), 0.36 ms RTT, no tunnel:
| streams | throughput |
|---|---|
| 1 | 940.2 Mbit/s |
| 8 | 939.3 Mbit/s |
Single stream saturates 1 GbE. So the box does not cap single sessions in general, and there is no per-session ceiling in its plain forwarding path. The ~250 Mbit/s per-flow cap is specific to the IPsec datapath.
Both IPsec tunnels converge on the same numbers despite different far ends
Measured today with the same probe:
| tunnel | far-end gateway | RTT | 1 stream | 8 streams |
|---|---|---|---|---|
| NH3 ↔ ANA | UDM Pro SE | 6.7 ms | 245 Mbit/s | 692 Mbit/s |
| ESH ↔ ANA | UDM Pro Max | 3.9 ms | 268 Mbit/s | 715 Mbit/s |
Different gateway hardware, different sites, different uplinks, and RTT differing by 1.7× — yet single-stream differs by only 9%. If this were window-limited the 3.9 ms path would be ~1.7× faster. It is not, which is independent confirmation of a rate cap rather than a BDP effect.
Capacity summary — the box has headroom it will not give one flow
- plain routing, 1 stream: 940 Mbit/s (line rate)
- plain routing to internet, 8 streams: 2,153 Mbit/s
- IPsec, 8 streams: 692–715 Mbit/s
- IPsec, 1 stream: 245–268 Mbit/s
- CPU 100% idle throughout; IPsec NPU-offloaded (
npu_flag=03)
Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath distributes work by inner session — consistent with IPsec offload binding a session to one crypto engine.
What is still NOT separated
Whether the cap belongs to the 80F's IPsec offload or to UniFi's IPsec implementation. Both tunnels have a UDM at the far end, and both UDMs run the same UniFi firmware, so identical caps are explainable either way. The Pro Max being only 9% faster than the Pro SE argues against the UniFi side (a beefier CPU should show more), but that is suggestive, not conclusive.
The test that closes it: an IPsec tunnel whose endpoints do not include the 80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured single-stream. If it also caps ~250, the FortiGate is exonerated and replacing it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM WireGuard achieves, the 80F is the limiter. Bears directly on the pending FortiGate cutover — worth running before that decision, not after.