vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

ops: WireGuard over the same path carries one stream at 767 Mbit/s

Browse Source 
The operator asked for a WireGuard test between ana-wg and NH3 across
the public internet, which is the arrangement that separates the path
from the crypto since neither gateway performs encryption in it. A
single stream returns 767 Mbit/s against 245 for IPsec over the same
wire, and eight streams return 763, so one flow already saturates the
path and there is no per-flow penalty whatsoever.

That settles several things at once. The limit is not the circuit,
the NH3 uplink, the ISP or the physical path, all of which carry 767
on one flow. Ranking the implementations, Linux WireGuard shows no
per-flow penalty, UniFi's WireGuard shows roughly 1.4x, and IPsec on
this pair shows 2.8x. Latency under a single bulk stream tells the
same story, rising to 12.7 ms on WireGuard against 102 ms on IPsec.

This overturns the earlier conclusion that changing transport was not
worth pursuing, which compared eight-stream figures and so understated
the gap for single-stream work by a factor of three. A WireGuard
site-to-site terminated on ana-wg now looks worth considering, and the
note records the open questions around failover and policy.

Attribution between the FortiGate and the UniFi gateway remains
unresolved, since both perform IPsec in the slow measurements, but the
remedy does not depend on which one is responsible.

The test tunnel was removed, ana-wg is back to its original three
peers and the generated keys were shredded.
This commit is contained in:
Vuong Hoang
2026-08-23 14:57:29 -07:00
parent d42e9d8712
commit 8ecffa1fab
1 changed files with 78 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md
+78
View File
@@ -293,3 +293,81 @@ single-stream. If it also caps ~250, the FortiGate is exonerated and replacing
it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM
WireGuard achieves, the 80F is the limiter. **Bears directly on the pending
FortiGate cutover** — worth running before that decision, not after.
---
## FOLLOW-UP 3 (2026-08-23): WireGuard over the same internet path does 767 Mbit/s on ONE stream
Operator asked for a WireGuard test from `ana-wg` to NH3 over the public
internet. It is the test that separates the *path* from the *crypto*, and the
answer is unambiguous. **It also overturns FOLLOW-UP 1's "re-architecting the
transport is not worth it" — that conclusion compared 8-stream numbers and was
wrong for single-stream workloads.**
### Setup (fully torn down afterwards)
`ana-wg` (10.250.50.252, Debian 12 LXC, 4 cores) already has an
internet-reachable WireGuard endpoint: wg0 on **UDP 31337**, published by
FortiGate VIP `wg-to-ana-wg` (extip **38.120.12.42** → 10.250.50.252:31337,
policy 46, service `WireGuard-LEET`). **No FortiGate change was needed.** A
temporary `wgt0` was created on nh3-dev (10.30.10.200/32) as a fourth peer on
wg0, measured, then removed — ana-wg is back to its original 3 peers and the
keys were shredded. `wireguard-tools` was installed on nh3-dev and **left in
place** (benign, and wanted if this becomes permanent).
In this topology **neither gateway does crypto**: the FortiGate and the NH3 UDM
only NAT/forward UDP, and Linux does WireGuard at both ends.
### The full comparison
| path | crypto performed by | 1 stream | 8 streams |
|---|---|---|---|
| IPsec NH3↔ANA | FortiGate + UDM | 245 Mbit/s | 692 Mbit/s |
| IPsec ESH↔ANA | FortiGate + UDM | 268 Mbit/s | 715 Mbit/s |
| **WireGuard NH3→ana-wg** (same internet path) | **Linux + Linux** | **767 Mbit/s** | 763 Mbit/s |
| WireGuard NH3↔ESH (Site Magic) | UDM + UDM | 557 Mbit/s | 795 Mbit/s |
| plain routing through the 80F (inter-VLAN) | none | 940 Mbit/s | 939 Mbit/s |
**One stream equals eight streams over Linux WireGuard (767 ≈ 763).** There is
no per-flow penalty at all, and a single flow already saturates the path. So
the ~245 Mbit/s per-flow cap is **not** the ISP, not the circuit, not the NH3
uplink and not the physical path — all of which sustain 767 on one flow.
Per-flow penalty ranks by implementation:
- **Linux WireGuard — none** (767 → 763, flat)
- **UDM WireGuard — mild**, ~1.4× (557 → 795)
- **IPsec on this pair — severe**, ~2.8× (245 → 692)
### Latency under load — the same story
| path | idle | during ONE bulk stream |
|---|---|---|
| IPsec NH3↔ANA | 6.9 ms | **102 ms** avg, 136 ms max |
| WireGuard NH3→ana-wg | 6.2 ms | **12.7 ms** avg, 23 ms max |
WireGuard carries **3.1× the single-stream throughput with 8× less latency
inflation** on the same wire.
### Attribution — still not fully separated, and it no longer matters much
Both IPsec measurements have a FortiGate *and* a UDM doing IPsec, so this still
does not isolate which one imposes the 2.8× penalty. Closing that would need
Linux↔Linux IPsec or UDM↔UDM IPsec on the same path. **But the practical
decision no longer depends on the answer**, because the fix is the same either
way and it is already demonstrated.
### Recommendation (supersedes FOLLOW-UP 1)
A **WireGuard site-to-site between NH3 and Anaheim, terminated on `ana-wg`**, is
worth real consideration: 3.1× single-stream, flat scaling, far better latency
under load, and it reuses infrastructure that already exists and is already
internet-reachable. It is also the architecture already proven for NH3↔ESH.
Open questions before committing: routing/failover if ana-wg (an LXC) is down,
whether it replaces or parallels the IPsec tunnel, and firewall policy for the
new transit. ana-wg CPU was only ~40% busy across 4 cores at 767 Mbit/s, so it
has headroom.
**AND: `nconnect=8` on /mnt/smithy remains worth doing regardless** — it is the
same lever (turn one flow into many) and brokkr-smithy-dev has given standing
approval to apply it once the FortiGate work settles, with no need to ask again.