esh-pfi-infrastructure/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md

[2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit

⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at the bottom before acting on anything here. The headline is wrong (the relevant ceiling is NH3's 1 Gbps uplink, not Anaheim's 2 Gbps), the aggregate number is wrong (692 Mbit/s at 8 streams, not ~550 — the original stopped measuring at 4), and the proposed remedy is impossible: UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The per-stream observation and the parallelise-your-transfers mitigation are the parts that survive.

The operator noticed site-to-site transfers were slow for a datacenter fiber handoff and asked whether WireGuard was the limit. It is not WireGuard, and the circuit is fine.

Measured

ana-docker -> internet, 8 parallel     2,153 Mbit/s   <- the 2 Gbps handoff, delivering
ANA <-> NH3 through the tunnel, 4 par.   460 Mbit/s
FortiGate's own recorded peak            554 Mbit/s
ANA <-> NH3, single stream               227 Mbit/s
ANA <-> ESH, single stream               249-265 Mbit/s
ESH <-> NH3 (never touches ana-gw)       545-557 Mbit/s  on a SINGLE stream

Method: stdlib TCP probe (no ssh, no crypto, no compression) between site endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling.

What it is not

• Not WireGuard. Both Anaheim tunnels are IPsec on ana-gw (pfi-ana-nh3 -> 70.230.226.88, ana-eshudm-dyn -> the ESH UDM). WireGuard on ana-wg is remote-access only and is not in this path. Traceroute confirms: both slow paths have hop 1 = 10.250.50.1 (the FortiGate); the fast ESH<->NH3 path rides a 192.168.x Site Magic overlay and never touches it.
• Not CPU or crypto exhaustion. FortiGate CPU was 100% idle across all 8 cores during the tests, and both live tunnels report npu_flag=03 with dec_npuid=1 enc_npuid=1 — encrypt and decrypt are hardware-offloaded.
• Not a 250 Mbit/s cap. That was the first number and it is misleading — single-stream TCP. Four parallel streams doubled it. Quote the aggregate.
• Not the interface. wan1: rxe=0 txe=0 rxd=0 txd=0, no collisions.

Most likely cause

Both tunnels negotiate aes256-sha1 in phase 1 and phase 2 (dhgrp 14, IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly faster on AES-GCM, which combines encryption and authentication in one pass. The datasheet IPsec headline for an 80F assumes GCM with large packets, not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x shortfall is consistent with that.

Not executed

Changing the proposal is a production-edge change requiring a matching change at the far end (NH3 UDM and the ESH UDM), and each tunnel drops while it renegotiates. Left for the operator. See the index entry for authorization state.

Immediate mitigation, no config change

Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a single stream through IPsec gets 227. Anything moving bulk data across the Anaheim link should parallelise — that alone roughly doubles throughput today.

Practical consequence already observed

/mnt/smithy mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s from nh3-dev (same file, same mount) — that gap is this tunnel, not NFS and not the NAS. See 2026-08-23-smithy-mount-ana-ml2.

Access note

ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. sshpass is absent on nh3-dev; connect with paramiko via uv run --with paramiko. Password is vaulted at fortigate/ana-gw-infra-ops-password. diagnose vpn tunnel list prints live ESP session keys — never paste its output into althing, a booth, or a commit.

---

CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist

The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done, and the measurements taken while trying show there is very little left to win.

AES-GCM is unavailable on the far end — not a naming problem

The NH3 edge is a UDM Pro SE terminating pfi-nh3-ana (networkconf _id 697d64414c85dd2b6669b00a, ifname vti64). Its UniFi API validates the crypto enum and rejected every GCM spelling tried — aes256gcm, aes256gcm128, aes256gcm16, aes-256-gcm, aes256-gcm, aes256gcm12, gcm, aes128gcm128 — all HTTP 400 api.err.InvalidPayload, nothing applied.

The control that makes this conclusive: the identical request body with ipsec_esp_encryption: "aes256" returns HTTP 200 rc:ok. So the 400s are the enum rejecting the value, not a malformed body. Corroborating: zero case-insensitive gcm matches across 7.3 MB of UniFi OS UI bundles.

Accepted enum (probed): aes128, aes192, aes256, 3des → 200; des, chacha20poly1305 → 400. There is no AEAD option. Both Anaheim tunnels land on UniFi far ends, so this blocks the ESH tunnel too.

The FortiGate side was widened and is GCM-capable: phase2 pfi-ana-nh3 now reads set proposal aes256-sha1 aes256gcm. Left in place deliberately — it is functionally identical while the peer only offers CBC, and reverting it would cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched; IKE protects the control channel only and has no bearing on data throughput.

The numbers that retire this as a problem

Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on the same UDM and the same uplink, with the same stdlib TCP probe:

streams	IPsec NH3→ANA	WireGuard NH3→ESH
1	245 Mbit/s	557 Mbit/s
4	471 Mbit/s	767 Mbit/s
8	692 Mbit/s	795 Mbit/s

NH3's WAN is a 1 Gbps link (uplink.speed = 1000, port capable of 10G) — that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel. So the tunnel does ~69% of the achievable uplink at 8 streams, and the IPsec-vs-WireGuard gap collapses from 2.3× at one stream to 15% at eight.

Re-architecting the transport (site-to-site WireGuard via ana-wg, since FortiOS has no WireGuard) would chase that last 15%. Not worth it.

What the constraint actually is

A per-stream limit (~245 Mbit/s), not an aggregate crypto ceiling. Both endpoints are idle at load — FortiGate CPU 100% idle with npu_flag=03 (offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from a higher floor (557 → 795 is only 1.43× scaling).

Actionable consequence

Anything moving bulk data across this link should parallelise — 245 → 692 Mbit/s, a 2.8× win with no config change. For single-stream workloads that cannot be parallelised at the application layer, NFS nconnect=N is the lever: it opens N TCP connections per mount, converting a single-stream workload into a parallel one. The /mnt/smithy mount on ana-ml2 reading at 24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live example — remounting with nconnect=8 is the obvious test.

Foot-gun recorded

Probing the enum by PUTting candidate values applies the accepted ones. A probe loop here timed out with 3des briefly live on the NH3 side, which the FortiGate would not accept — a short tunnel outage until aes256 was restored (~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi config enum this way, restore the known-good value after every 200, not at the end of the loop. Post-change verification: the UDM object was diffed field-by-field against its pre-change snapshot and is byte-identical.

---

FOLLOW-UP (2026-08-23): what the per-stream limit actually is

The correction above called the constraint "per-SA/per-flow serialisation". That was a hand-wave. Measured properly, it is a hard per-flow rate cap of ~230–245 Mbit/s with a very deep buffer in front of it — not a tuning problem, not loss, not window size.

The evidence: pin the send buffer and sweep it

Single stream NH3 → ana-docker, SO_SNDBUF pinned, ss -ti sampled in flight:

in-flight cap	throughput	RTT in flight	minRTT	retrans
256 KB	224 Mbit/s	7.8 ms	5.3 ms	0
416 KB	225 Mbit/s	11.8 ms	6.6 ms	0
416 KB	245–247 Mbit/s	12.0 ms	5.6 ms	0
~3.3 MB (autotuned)	245 Mbit/s	107 ms	5.5 ms	0

Throughput is flat across a 13× range of in-flight data while RTT scales with it. That is the signature of a fixed service rate with a standing queue: the window controls only how much queue you build, never how fast you go. Had this been window-limited, throughput would have risen with the buffer. Had it been congestion, there would be retransmits — there are essentially none (retrans:0, 0% ping loss).

So net.ipv4.tcp_* tuning, window scaling and congestion-control choice are all red herrings here. Do not go there.

Bufferbloat: one bulk stream wrecks latency for everything else

Measured on the same tunnel, ping to ana-docker:

• idle: 6.9 ms avg
• during a single bulk TCP stream: 102 ms avg, 136 ms max, 0% loss

15× latency inflation from one transfer. This is the operationally important finding — any interactive traffic sharing the Anaheim link (ssh, RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it takes only one stream to do it. Parallelising transfers makes throughput better and this worse. If it starts biting, the fix is an AQM/shaper on the tunnel (or rate-limiting bulk jobs), not more buffer.

Where the cap lives — strong inference, not proof

Three paths, and the FortiGate is the only variable:

path	single-stream
FortiGate ↔ NH3 UDM (IPsec)	245 Mbit/s
FortiGate ↔ ESH UDM (IPsec)	249–265 Mbit/s
NH3 UDM ↔ ESH UDM (WireGuard, no FortiGate)	557 Mbit/s

Present in both slow paths, absent from the fast one. Aggregate over the same SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole — many flows spread out fine, one flow does not.

The mechanism that fits is FortiGate NPU IPsec offload being per-session: each firewall session is bound to one crypto engine, so a single TCP flow is capped at one engine's rate while many sessions spread across engines. This is inference from the throughput shape, not something confirmed on the box — diagnose sys session list was not captured for a TCP flow (the filter caught only traceroute UDP probes). A single-stream control through ana-gw without IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is window-limited and does not discriminate. If this matters, the clean test is a non-IPsec single stream between two Anaheim VLANs at low RTT.

Relevant to the FortiGate cutover decision: if the per-flow cap is the FortiGate's IPsec path, replacing the box plausibly lifts single-stream throughput toward the WireGuard figure. That is a point in favour of the cutover, and it is cheap to verify afterwards by re-running the sweep.

---

FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec

Operator asked directly whether the 80F "can't handle the traffic". It can. Two new measurements settle the shape of this, and correct an overstatement in FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded — every slow path was both IPsec and FortiGate, so protocol and box could not be separated by that argument).

The 80F routes a single flow at line rate when IPsec is not involved

ana-ml2 → pfi-pve, inter-VLAN through ana-gw (traceroute hop 1 = 10.250.50.1), 0.36 ms RTT, no tunnel:

streams	throughput
1	940.2 Mbit/s
8	939.3 Mbit/s

Single stream saturates 1 GbE. So the box does not cap single sessions in general, and there is no per-session ceiling in its plain forwarding path. The ~250 Mbit/s per-flow cap is specific to the IPsec datapath.

Both IPsec tunnels converge on the same numbers despite different far ends

Measured today with the same probe:

tunnel	far-end gateway	RTT	1 stream	8 streams
NH3 ↔ ANA	UDM Pro SE	6.7 ms	245 Mbit/s	692 Mbit/s
ESH ↔ ANA	UDM Pro Max	3.9 ms	268 Mbit/s	715 Mbit/s

Different gateway hardware, different sites, different uplinks, and RTT differing by 1.7× — yet single-stream differs by only 9%. If this were window-limited the 3.9 ms path would be ~1.7× faster. It is not, which is independent confirmation of a rate cap rather than a BDP effect.

Capacity summary — the box has headroom it will not give one flow

• plain routing, 1 stream: 940 Mbit/s (line rate)
• plain routing to internet, 8 streams: 2,153 Mbit/s
• IPsec, 8 streams: 692–715 Mbit/s
• IPsec, 1 stream: 245–268 Mbit/s
• CPU 100% idle throughout; IPsec NPU-offloaded (npu_flag=03)

Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath distributes work by inner session — consistent with IPsec offload binding a session to one crypto engine.

What is still NOT separated

Whether the cap belongs to the 80F's IPsec offload or to UniFi's IPsec implementation. Both tunnels have a UDM at the far end, and both UDMs run the same UniFi firmware, so identical caps are explainable either way. The Pro Max being only 9% faster than the Pro SE argues against the UniFi side (a beefier CPU should show more), but that is suggestive, not conclusive.

The test that closes it: an IPsec tunnel whose endpoints do not include the 80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured single-stream. If it also caps ~250, the FortiGate is exonerated and replacing it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM WireGuard achieves, the 80F is the limiter. Bears directly on the pending FortiGate cutover — worth running before that decision, not after.

---

FOLLOW-UP 3 (2026-08-23): WireGuard over the same internet path does 767 Mbit/s on ONE stream

Operator asked for a WireGuard test from ana-wg to NH3 over the public internet. It is the test that separates the path from the crypto, and the answer is unambiguous. It also overturns FOLLOW-UP 1's "re-architecting the transport is not worth it" — that conclusion compared 8-stream numbers and was wrong for single-stream workloads.

Setup (fully torn down afterwards)

ana-wg (10.250.50.252, Debian 12 LXC, 4 cores) already has an internet-reachable WireGuard endpoint: wg0 on UDP 31337, published by FortiGate VIP wg-to-ana-wg (extip 38.120.12.42 → 10.250.50.252:31337, policy 46, service WireGuard-LEET). No FortiGate change was needed. A temporary wgt0 was created on nh3-dev (10.30.10.200/32) as a fourth peer on wg0, measured, then removed — ana-wg is back to its original 3 peers and the keys were shredded. wireguard-tools was installed on nh3-dev and left in place (benign, and wanted if this becomes permanent).

In this topology neither gateway does crypto: the FortiGate and the NH3 UDM only NAT/forward UDP, and Linux does WireGuard at both ends.

The full comparison

path	crypto performed by	1 stream	8 streams
IPsec NH3↔ANA	FortiGate + UDM	245 Mbit/s	692 Mbit/s
IPsec ESH↔ANA	FortiGate + UDM	268 Mbit/s	715 Mbit/s
WireGuard NH3→ana-wg (same internet path)	Linux + Linux	767 Mbit/s	763 Mbit/s
WireGuard NH3↔ESH (Site Magic)	UDM + UDM	557 Mbit/s	795 Mbit/s
plain routing through the 80F (inter-VLAN)	none	940 Mbit/s	939 Mbit/s

One stream equals eight streams over Linux WireGuard (767 ≈ 763). There is no per-flow penalty at all, and a single flow already saturates the path. So the ~245 Mbit/s per-flow cap is not the ISP, not the circuit, not the NH3 uplink and not the physical path — all of which sustain 767 on one flow.

Per-flow penalty ranks by implementation:

• Linux WireGuard — none (767 → 763, flat)
• UDM WireGuard — mild, ~1.4× (557 → 795)
• IPsec on this pair — severe, ~2.8× (245 → 692)

Latency under load — the same story

path	idle	during ONE bulk stream
IPsec NH3↔ANA	6.9 ms	102 ms avg, 136 ms max
WireGuard NH3→ana-wg	6.2 ms	12.7 ms avg, 23 ms max

WireGuard carries 3.1× the single-stream throughput with 8× less latency inflation on the same wire.

Attribution — still not fully separated, and it no longer matters much

Both IPsec measurements have a FortiGate and a UDM doing IPsec, so this still does not isolate which one imposes the 2.8× penalty. Closing that would need Linux↔Linux IPsec or UDM↔UDM IPsec on the same path. But the practical decision no longer depends on the answer, because the fix is the same either way and it is already demonstrated.

Recommendation (supersedes FOLLOW-UP 1)

A WireGuard site-to-site between NH3 and Anaheim, terminated on ana-wg, is worth real consideration: 3.1× single-stream, flat scaling, far better latency under load, and it reuses infrastructure that already exists and is already internet-reachable. It is also the architecture already proven for NH3↔ESH. Open questions before committing: routing/failover if ana-wg (an LXC) is down, whether it replaces or parallels the IPsec tunnel, and firewall policy for the new transit. ana-wg CPU was only ~40% busy across 4 cores at 767 Mbit/s, so it has headroom.

AND: nconnect=8 on /mnt/smithy remains worth doing regardless — it is the same lever (turn one flow into many) and brokkr-smithy-dev has given standing approval to apply it once the FortiGate work settles, with no need to ask again.

---

RESOLVED (2026-08-23): it is the UDM's software AES-CBC. The FortiGate is exonerated.

Operator's theory — the UDM does IPsec in software with no crypto offload, so the cost of the cipher itself is the limit — is correct, and it is now demonstrated rather than inferred. He also correctly pointed out that UDM↔UDM Site Magic is WireGuard, not IPsec, so that row never said anything about UniFi's IPsec performance. It didn't, and I had leaned on it.

The controlled experiment: vary cipher cost, hold everything else

AES-128 is 10 rounds, AES-256 is 14. If software crypto is the binding constraint, throughput must rise when the cipher gets cheaper. If the limit were the FortiGate's NPU, it would not move at all — hardware crypto is not cipher-cost-sensitive in that range. Run A/B/A, single stream, 25–60 s each:

condition	ESP cipher	single-stream	UDM CPU
A	aes256-cbc + sha1	232.3 Mbit/s	35.4%
B	aes128-cbc + sha1	281.8, 274.9 Mbit/s	35.5%
A again	aes256-cbc + sha1	244.9, 242.5 Mbit/s	—

~1.16–1.20× faster on the cheaper cipher at identical CPU. Same bytes of CPU work, more payload through it. That is the signature of CPU-bound software crypto, and it rules out the FortiGate's NPU as the limiter.

Correcting two of my own earlier claims

1. "UDM CPU is only ~7%, so it isn't CPU-bound" was WRONG — a sampling artifact. UniFi's system-stats.cpu refreshes on the device report interval; 4-second sample windows were reading stale values. Under a sustained 60 s single-stream load it reads 35.4%, with load1 rising 0.60 → 1.17. On a 4-core UDM Pro SE that is ≈1.4 cores — one core saturated on crypto plus overhead. Always drive load for ≥60 s before trusting a UniFi CPU figure.
2. The "FortiGate per-session NPU offload" hypothesis is REFUTED, not merely unproven. It predicts no change from a cipher swap; a 20% change was measured.

Why the numbers all line up now

• 1 stream = 1 core of UDM crypto → ~240 Mbit/s on AES-256-CBC.
• 8 streams = ~3 usable cores → ~692 Mbit/s, ≈2.9× the single-stream figure on a 4-core box. Aggregate is noisy (492–692 across repeats on a live link) and is not cipher-sensitive, consistent with it being bounded by the path/ uplink rather than crypto once several cores are engaged.
• AES-CBC is the specific villain: it is serial. Each block depends on the previous one, so the ARM AES instructions cannot pipeline across blocks. GCM (CTR-based) and ChaCha20-Poly1305 both parallelise freely. That is why the same UDM does 557 Mbit/s single-stream on WireGuard and only 240 on IPsec.
• This retroactively vindicates the GCM cutover as the right idea aimed at the right box — GCM would have removed the serial dependency on the constrained end. UniFi simply does not offer it, which is what made it impossible.

Options this opens

• AES-128 instead of AES-256: ~16–20% for free, no topology change, one API call per end. 128-bit is not the weak link here (SHA1 integrity is more dated, and unchanged either way). Operator's call — not adopted, restored to aes256.
• WireGuard site-to-site via ana-wg: 767 Mbit/s single-stream (3.1×), and it sidesteps the UDM's IPsec datapath entirely. Still the biggest win available.
• Replacing the FortiGate will not help this — it was never the constraint. Worth knowing before the cutover.

State left behind

UDM network object verified byte-identical to its pre-test snapshot (aes256/sha1). Tunnel up, selectors 1/1. FortiGate phase2 pfi-ana-nh3 is left as aes256-sha1 aes256gcm aes128-sha1 — a permissive superset; the peer offers only aes256 so the extra entries are inert, but narrowing it back to aes256-sha1 is one line if the looser list is unwanted.

---

FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through

Operator's point: FortiOS has no WireGuard, so a WireGuard site-to-site must terminate on a box behind the edge. Correct — and ana-wg (LXC, CT 113 on pfi-pve, 10.250.50.252) already is that box.

This closes a gap in FOLLOW-UP 3. That 767 Mbit/s figure was measured with traffic terminating on ana-wg. Real traffic must be forwarded onward to other Anaheim hosts, which was never measured. Now it is:

topology	1 stream	8 streams
IPsec, FortiGate ↔ UDM (today)	245 Mbit/s	692 Mbit/s
WG terminating on ana-wg	767 Mbit/s	763 Mbit/s
WG transit: nh3 → wg → ana-wg → forward → ana-docker	763.8 Mbit/s	790.4 Mbit/s

Forwarding through the LXC is free (763.8 vs 767). The downstream-VM architecture delivers the full 3.1× single-stream for real transit traffic, not just for traffic landing on the tunnel box.

ana-wg while forwarding 764 Mbit/s: ~22% busy across 4 cores (77.8% idle), so roughly 0.9 cores. Note /proc/loadavg inside this LXC reports the host's load, not the container's — do not read it as ana-wg's own. For contrast the UDM burns 35.4% of its 4 cores to move 240 Mbit/s, so ana-wg has ample headroom.

Design consequences of terminating downstream — the parts that need decisions

1. Anaheim hosts must route to ana-wg, not to the FortiGate. The 763.8 figure was obtained with an explicit 10.30.10.200/32 via 10.250.50.252 route on ana-docker. Without that, a host sends 10.100.0.0/16 to its default gateway (ana-gw), which routes it back out the same interface to ana-wg — a LAN hairpin crossing the FortiGate twice. The hairpin variant was NOT measured. Options: DHCP option 121 pushing the route fleet-wide, a dedicated transit VLAN for ana-wg, or accept the hairpin.
2. New single point of failure. Today site-to-site dies only when the edge dies, which is total anyway. A downstream terminator fails independently. Mitigation: keep the IPsec tunnel configured as a higher-metric fallback route so it takes over when ana-wg is down.
3. ana-wg is an LXC on pfi-pve, so its ~0.9 cores and NIC traffic land on the hypervisor shared with the rest of the Anaheim VMs.
4. The NH3 end needs a terminator too, and there are two shapes:
   • Linux VM at NH3 (nh3-dev or a dedicated VM on nh3-pve) — this is what was measured: 764 Mbit/s.
   • NH3 UDM's existing WireGuard server (PFI-NH3-WG, wireguard-server on UDP 31337) accepting ana-wg as a peer — plausible but untested, and UniFi's WireGuard shows a per-flow penalty (557 Mbit/s single-stream on Site Magic), so expect ~557 rather than 764. Still 2.3× today.

Standing recommendation

Worth doing, but it is a project, not a config tweak — routing, failover and policy all need deciding. The cheap wins remain available meanwhile and are independent: nconnect=8 on NFS mounts (approved by brokkr-smithy-dev, pending the FortiGate work settling) and AES-128 for ~20%.

---

LANDED (2026-08-23): AES-128 on both tunnels; FortiGate public admin closed

Operator directed: adopt AES-128 on both Anaheim tunnels, make-before-break, then close the FortiGate's WAN and SSH admin surfaces. All done and verified.

Context that retires the WireGuard-in-a-VM design work: the FortiGate is being replaced by OPNsense on a Dell R420, which gives WireGuard on the edge device itself. The downstream-terminator architecture (FOLLOW-UP 4) is therefore moot — do not scope it. This also un-parks the OPNsense migration, which auto-memory recorded as PARKED pending "hardware acquisition"; the R420 is that trigger.

What changed

Make-before-break on the FortiGate first, so neither tunnel dropped waiting on a far end:

phase2	proposal now
pfi-ana-nh3	aes256-sha1 aes256gcm aes128-sha1
ana-eshudm-dyn	aes256-sha1 aes128-sha1

Then each UDM flipped to ipsec_esp_encryption: aes128:

tunnel	UDM object	before	after
NH3 ↔ ANA	pfi-nh3-ana 697d64414c85dd2b6669b00a @ 10.100.0.1	245 Mbit/s	269.7
ESH ↔ ANA	esh-ana 697723b9b9d4266dddf2bcc7 @ 10.0.0.1	268 Mbit/s	304.3

Single-stream gain ~10–13% here, against 16–20% in the earlier controlled A/B — the difference is live-link variance, not a different result. Both UDM objects were diffed field-by-field against pre-change snapshots: the only field that moved on either is ipsec_esp_encryption.

The FortiGate proposal lists were deliberately left permissive (still accepting aes256). The peers offer only aes128 so the extra entries are inert, and keeping them means a UDM reverting does not strand the tunnel. Narrowing to aes128-sha1 alone is a one-liner if the looser list is unwanted.

Admin surfaces closed

wan1 allowaccess → ping (https + ssh removed) and infra-ops trusthost → 10.0.0.0/8 only (the 8 wide-open ranges unset). Verified 443 and 22 closed from both NH3 and ESH; management over the tunnel at 10.250.0.1 still works. Sequencing that matters: the close was executed over the TUNNEL path, not over WAN — removing ssh from allowaccess while connected over WAN kills the session mid-command.

Consequence to hold in mind: ana-gw now has no out-of-band management path. If both tunnels drop it is console-only until someone is on site.

Gotcha: the two UDM vault items have DIFFERENT shapes

• unifi/pfi-udmse-api-key → a bare 32-char key. secret get output is the key.
• unifi/esh-udmpm-api-key → a 19-line documentation note with the key on a key: line. secret get piped straight into a header yields a 1396-byte value and the UDM answers 400 Bad Request from nginx. Extract with grep '^key:' | awk '{print $2}'.

The ESH key's first-ever confirmed WRITE happened here (auto-memory recorded it as read-verified only): a control PUT of the unchanged object returned rc:ok, then the real change did too. That key has a full read+write admin role.

---

CORRECTION (2026-08-23): port 80 on the WAN IP is the FortiOS ACME listener

The claim in the previous section that .42:80 was an ISP transparent proxy was WRONG, and so was the earlier warning that ACME renewal would fail with port 80 absent from allowaccess. Operator pushed back asking where the port-80 map terminated. It terminates on the FortiGate itself.

What it is: the FortiOS ACME HTTP-01 challenge listener. config system acme has set interface "wan1", and FortiOS opens port 80 on that interface to answer Let's Encrypt challenges independently of allowaccess — wan1 allowaccess reads ping only and the port is still open. Every non-challenge request returns a fixed 403 whose body is literally:

<!DOCTYPE html><html><head><title>ACME Access Only</title></head><body>ACME Access Only</body></html>

Not a DNAT. The full VIP table has 14 entries; only two land on .42 — Kokoro-In (:8880 → 10.250.50.51) and wg-to-ana-wg (:31337 → 10.250.50.252). Worth noting separately: four VIPs are all-port static NAT — that claim was WRONG, see the correction below. All fourteen VIPs are scoped.

The methodology error that produced the wrong answer — worth not repeating

The sniffer filter used was dst host 38.120.12.42 and tcp port 80. dst host matches only inbound packets, so outbound SYN-ACKs were excluded by construction; concluding "the box sends no SYN-ACK" from that capture was unsound. Re-run with the bidirectional host 38.120.12.42 and tcp port 80 it immediately shows wan1 out 38.120.12.42.80 -> <scanner>: syn ack.

Rule: when testing whether a box answers, the sniffer filter must be bidirectional. dst host silently answers a different question.

Consequences

• ACME renewal will work with allowaccess ping. The earlier "add http back or the cert expires" warning is retracted — FortiOS opens the challenge port itself. Cert valid to 2026-10-27, renewal attempt ~2026-09-27.
• It is not an admin surface — static 403, no auth, no GUI.
• Its practical value is now low: WAN admin is closed, so the cert only serves the internal GUI at 10.250.0.1, where the name would not match anyway. Killing it (config system acme → unset interface) would close the last WAN listener at the cost of cert renewal. Operator's call; not done.

---

CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE

ACME disabled — the WAN IP now exposes nothing

config system acme / unset interface (the account object is left in place; with no interface bound there is no listener). Verified:

• External scan of 38.120.12.42 across 55 ports: no open TCP ports at all.
• Internal GUI at 10.250.0.1 still answers 200, SSH still works.
• admin-server-cert is still ana-fw.pfi — the existing cert is untouched and serves the internal GUI until 2026-10-27; it simply will not auto-renew.

Reverse with config system acme / set interface "wan1".

RETRACTION: the four VIPs are NOT all-port

A previous section claimed Rustdesk, https-to-tacticalrmm, web-to-webhost and web-to-sfcontainer were unrestricted all-port static NATs. They are not. A FortiOS VIP can be scoped two different ways and the parser used only checked one:

1. set portforward enable + set extport <n> — a single mapped port, or
2. set service "<svc>" on the VIP object — constrains the VIP to that service.

All four use form 2. The custom services are narrow: Rustdesk = TCP 21115–21119 + UDP 21116 (the standard RustDesk range), ssh-mapped-2223 = TCP 2223 only. Every one of the 14 VIPs is scoped; none is unrestricted.

Lesson: absence of portforward does NOT mean all-port on a FortiOS VIP — check service too. Better still, do what settled it here: scan from outside rather than reading config.

Ground-truth public exposure (external TCP scan, post-change)

IP	open	maps to
38.120.12.41	nothing	—
38.120.12.42	nothing	the FortiGate itself — fully closed
38.120.12.43	80, 443	sf-ana-container 10.250.150.100 (SureFire tenant)
38.120.12.44	22, 80, 443, 8025, 21115–21119	gitea (→222), traefik, mailrise, RustDesk
38.120.12.45	80, 443, 2223	pfi-ana-webhost 10.250.50.52 (2223→22)
38.120.12.46	443	pfi-tacticalrmm 10.250.50.57

Configured-but-closed: 8443 (mattermost-calls), 8444 (webdav-nas), 8880 (Kokoro-In) — VIPs exist, nothing listening behind them. Worth a tidy-up during the OPNsense translation but not exposure.