Files

T

vh b6340519bc memory: snapshot — Anaheim tunnel at 25% of circuit; selene retired; hrafn CI fixed

Session captured for a context reset. Six new detail entries.

THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~550 Mbit/s aggregate against a
circuit measured at 2,153 Mbit/s. Not WireGuard (it is IPsec on ana-gw), not
CPU (idle), not crypto exhaustion (NPU-offloaded), not the fibre. Both tunnels
negotiate aes256-sha1; AES-GCM proposed. Operator signalled authorization;
execution pending, untracked by operator choice.

Also recorded: selene retired after losing a head-to-head on its own job with
chat-judge moved to gen and the model name left to 404; the 7-alias collision
on the gen seat that makes cross-alias corroboration an echo; hrafn adopted and
its CI found to have been reporting green while deploying nothing for its whole
life; all three Worldtree instances de-armed from a 69-day-stale :latest and
the Matrix homeserver re-plumbed to personal; every secret-bearing .env on
ana-docker tightened to 0600; the pfi org closing the repo-creation half of the
credential-migration directive; booth kept-board deletion and link pruning.

Two entries under Tried and abandoned: the CI checkout assertion that broke the
pipeline twice and was removed, and my proposal to alias a retired model name
at a different model, which the operator correctly overruled.

Index 271 -> 282 lines, under the 300 cap, so no archival fired.

2026-08-23 13:35:48 -07:00

3.2 KiB

Raw Blame History

[2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit

The operator noticed site-to-site transfers were slow for a datacenter fiber handoff and asked whether WireGuard was the limit. It is not WireGuard, and the circuit is fine.

Measured

ana-docker -> internet, 8 parallel     2,153 Mbit/s   <- the 2 Gbps handoff, delivering
ANA <-> NH3 through the tunnel, 4 par.   460 Mbit/s
FortiGate's own recorded peak            554 Mbit/s
ANA <-> NH3, single stream               227 Mbit/s
ANA <-> ESH, single stream               249-265 Mbit/s
ESH <-> NH3 (never touches ana-gw)       545-557 Mbit/s  on a SINGLE stream

Method: stdlib TCP probe (no ssh, no crypto, no compression) between site endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling.

What it is not

Not WireGuard. Both Anaheim tunnels are IPsec on ana-gw (pfi-ana-nh3 -> 70.230.226.88, ana-eshudm-dyn -> the ESH UDM). WireGuard on ana-wg is remote-access only and is not in this path. Traceroute confirms: both slow paths have hop 1 = 10.250.50.1 (the FortiGate); the fast ESH<->NH3 path rides a 192.168.x Site Magic overlay and never touches it.
Not CPU or crypto exhaustion. FortiGate CPU was 100% idle across all 8 cores during the tests, and both live tunnels report npu_flag=03 with dec_npuid=1 enc_npuid=1 — encrypt and decrypt are hardware-offloaded.
Not a 250 Mbit/s cap. That was the first number and it is misleading — single-stream TCP. Four parallel streams doubled it. Quote the aggregate.
Not the interface. wan1: rxe=0 txe=0 rxd=0 txd=0, no collisions.

Most likely cause

Both tunnels negotiate aes256-sha1 in phase 1 and phase 2 (dhgrp 14, IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly faster on AES-GCM, which combines encryption and authentication in one pass. The datasheet IPsec headline for an 80F assumes GCM with large packets, not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x shortfall is consistent with that.

Not executed

Changing the proposal is a production-edge change requiring a matching change at the far end (NH3 UDM and the ESH UDM), and each tunnel drops while it renegotiates. Left for the operator. See the index entry for authorization state.

Immediate mitigation, no config change

Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a single stream through IPsec gets 227. Anything moving bulk data across the Anaheim link should parallelise — that alone roughly doubles throughput today.

Practical consequence already observed

/mnt/smithy mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s from nh3-dev (same file, same mount) — that gap is this tunnel, not NFS and not the NAS. See 2026-08-23-smithy-mount-ana-ml2.

Access note

ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. sshpass is absent on nh3-dev; connect with paramiko via uv run --with paramiko. Password is vaulted at fortigate/ana-gw-infra-ops-password. diagnose vpn tunnel list prints live ESP session keys — never paste its output into althing, a booth, or a commit.

3.2 KiB Raw Blame History