vh
b6340519bc
Session captured for a context reset. Six new detail entries. THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~550 Mbit/s aggregate against a circuit measured at 2,153 Mbit/s. Not WireGuard (it is IPsec on ana-gw), not CPU (idle), not crypto exhaustion (NPU-offloaded), not the fibre. Both tunnels negotiate aes256-sha1; AES-GCM proposed. Operator signalled authorization; execution pending, untracked by operator choice. Also recorded: selene retired after losing a head-to-head on its own job with chat-judge moved to gen and the model name left to 404; the 7-alias collision on the gen seat that makes cross-alias corroboration an echo; hrafn adopted and its CI found to have been reporting green while deploying nothing for its whole life; all three Worldtree instances de-armed from a 69-day-stale :latest and the Matrix homeserver re-plumbed to personal; every secret-bearing .env on ana-docker tightened to 0600; the pfi org closing the repo-creation half of the credential-migration directive; booth kept-board deletion and link pruning. Two entries under Tried and abandoned: the CI checkout assertion that broke the pipeline twice and was removed, and my proposal to alias a retired model name at a different model, which the operator correctly overruled. Index 271 -> 282 lines, under the 300 cap, so no archival fired.
72 lines
3.2 KiB
Markdown
72 lines
3.2 KiB
Markdown
# [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit
|
|
|
|
The operator noticed site-to-site transfers were slow for a datacenter fiber
|
|
handoff and asked whether WireGuard was the limit. It is not WireGuard, and the
|
|
circuit is fine.
|
|
|
|
## Measured
|
|
|
|
```
|
|
ana-docker -> internet, 8 parallel 2,153 Mbit/s <- the 2 Gbps handoff, delivering
|
|
ANA <-> NH3 through the tunnel, 4 par. 460 Mbit/s
|
|
FortiGate's own recorded peak 554 Mbit/s
|
|
ANA <-> NH3, single stream 227 Mbit/s
|
|
ANA <-> ESH, single stream 249-265 Mbit/s
|
|
ESH <-> NH3 (never touches ana-gw) 545-557 Mbit/s on a SINGLE stream
|
|
```
|
|
|
|
Method: stdlib TCP probe (no ssh, no crypto, no compression) between site
|
|
endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner
|
|
Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling.
|
|
|
|
## What it is not
|
|
|
|
- **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw
|
|
(`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard
|
|
on ana-wg is remote-access only and is not in this path. Traceroute confirms:
|
|
both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast
|
|
ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it.
|
|
- **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all
|
|
8 cores** during the tests, and both live tunnels report `npu_flag=03` with
|
|
`dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded.
|
|
- **Not a 250 Mbit/s cap.** That was the first number and it is misleading —
|
|
single-stream TCP. Four parallel streams doubled it. Quote the aggregate.
|
|
- **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions.
|
|
|
|
## Most likely cause
|
|
|
|
Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14,
|
|
IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly
|
|
faster on **AES-GCM**, which combines encryption and authentication in one
|
|
pass. The datasheet IPsec headline for an 80F assumes GCM with large packets,
|
|
not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x
|
|
shortfall is consistent with that.
|
|
|
|
## Not executed
|
|
|
|
Changing the proposal is a **production-edge change requiring a matching
|
|
change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while
|
|
it renegotiates. Left for the operator. See the index entry for authorization
|
|
state.
|
|
|
|
## Immediate mitigation, no config change
|
|
|
|
Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a
|
|
single stream through IPsec gets 227. **Anything moving bulk data across the
|
|
Anaheim link should parallelise** — that alone roughly doubles throughput
|
|
today.
|
|
|
|
## Practical consequence already observed
|
|
|
|
`/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s
|
|
from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and
|
|
not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]].
|
|
|
|
## Access note
|
|
|
|
ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent
|
|
on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is
|
|
vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list`
|
|
prints live ESP session keys** — never paste its output into althing, a
|
|
booth, or a commit.
|