# [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit

The operator noticed site-to-site transfers were slow for a datacenter fiber
handoff and asked whether WireGuard was the limit. It is not WireGuard, and the
circuit is fine.

## Measured

```
ana-docker -> internet, 8 parallel     2,153 Mbit/s   <- the 2 Gbps handoff, delivering
ANA <-> NH3 through the tunnel, 4 par.   460 Mbit/s
FortiGate's own recorded peak            554 Mbit/s
ANA <-> NH3, single stream               227 Mbit/s
ANA <-> ESH, single stream               249-265 Mbit/s
ESH <-> NH3 (never touches ana-gw)       545-557 Mbit/s  on a SINGLE stream
```

Method: stdlib TCP probe (no ssh, no crypto, no compression) between site
endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner
Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling.

## What it is not

- **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw
  (`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard
  on ana-wg is remote-access only and is not in this path. Traceroute confirms:
  both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast
  ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it.
- **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all
  8 cores** during the tests, and both live tunnels report `npu_flag=03` with
  `dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded.
- **Not a 250 Mbit/s cap.** That was the first number and it is misleading —
  single-stream TCP. Four parallel streams doubled it. Quote the aggregate.
- **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions.

## Most likely cause

Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14,
IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly
faster on **AES-GCM**, which combines encryption and authentication in one
pass. The datasheet IPsec headline for an 80F assumes GCM with large packets,
not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x
shortfall is consistent with that.

## Not executed

Changing the proposal is a **production-edge change requiring a matching
change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while
it renegotiates. Left for the operator. See the index entry for authorization
state.

## Immediate mitigation, no config change

Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a
single stream through IPsec gets 227. **Anything moving bulk data across the
Anaheim link should parallelise** — that alone roughly doubles throughput
today.

## Practical consequence already observed

`/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s
from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and
not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]].

## Access note

ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent
on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is
vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list`
prints live ESP session keys** — never paste its output into althing, a
booth, or a commit.