The earlier note described the constraint as per-flow serialisation
without evidence. Pinning SO_SNDBUF and sweeping it shows what it
actually is: throughput holds flat between 224 and 247 Mbit/s across
a thirteenfold range of in-flight data while round-trip time scales
with the buffer instead, from 7.8 ms up to 107 ms. Retransmissions
are absent throughout. That is a fixed service rate with a standing
queue ahead of it, so socket buffer, window scaling and congestion
control are not the lever and should not be pursued.
The same measurements surface something with wider consequences than
throughput: a single bulk stream lifts tunnel latency from 6.9 ms to
102 ms average, 136 ms peak, with no loss. Interactive traffic
sharing the Anaheim link degrades sharply whenever anything moves
bulk data, and parallelising transfers makes that worse rather than
better.
Localisation rests on the FortiGate appearing in both slow paths and
in neither fast one, with aggregate throughput over the same security
association reaching 692 Mbit/s. Per-session NPU offload fits the
shape but was not confirmed on the device; the note records the test
that would settle it and flags the bearing this has on the pending
FortiGate replacement.
vh
cf0cb2cbb3