ops: the tunnel's per-flow limit is the UDM's software AES-CBC

Varying only the cipher settles what inference could not. AES-128 has
ten rounds against AES-256's fourteen, so a software-bound path must
speed up when the cipher gets cheaper while a hardware-offloaded one
will not move. Run as A/B/A on a single stream, AES-256 returns 232,
245 and 243 Mbit/s and AES-128 returns 282 and 275, a gain of about a
fifth at identical CPU. The FortiGate's offload engine is therefore
not the constraint, and the operator's reading of the UniFi side as
software crypto without acceleration is correct.

Two earlier claims of mine were wrong and are corrected in place. The
observation that the UDM sat at seven percent CPU was a sampling
artifact, since UniFi refreshes that statistic on the device report
interval and the windows used were four seconds; a sustained run reads
thirty-five percent, around one and a half of four cores. The
per-session offload hypothesis is refuted rather than merely unproven,
because it predicts no response to a cipher change.

The remaining figures follow from this. One stream is one core, eight
streams engage about three, and aggregate stops responding to cipher
choice once several cores are working because the path bounds it. CBC
chaining is what makes this expensive, as it serialises blocks and
prevents the AES instructions from pipelining, which is also why the
same gateway manages far more over WireGuard.

The cipher was restored and the network object verified unchanged
against its pre-test snapshot.

persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md

@@ -371,3 +371,76 @@ has headroom.
 **AND: `nconnect=8` on /mnt/smithy remains worth doing regardless** — it is the
 same lever (turn one flow into many) and brokkr-smithy-dev has given standing
 approval to apply it once the FortiGate work settles, with no need to ask again.
+
+---
+
+## RESOLVED (2026-08-23): it is the UDM's software AES-CBC. The FortiGate is exonerated.
+
+Operator's theory — the UDM does IPsec in software with no crypto offload, so
+the cost of the cipher itself is the limit — is **correct**, and it is now
+demonstrated rather than inferred. He also correctly pointed out that
+UDM↔UDM Site Magic is **WireGuard, not IPsec**, so that row never said anything
+about UniFi's IPsec performance. It didn't, and I had leaned on it.
+
+### The controlled experiment: vary cipher cost, hold everything else
+
+AES-128 is 10 rounds, AES-256 is 14. If software crypto is the binding
+constraint, throughput must rise when the cipher gets cheaper. If the limit
+were the FortiGate's NPU, it would not move at all — hardware crypto is not
+cipher-cost-sensitive in that range. Run A/B/A, single stream, 25–60 s each:
+
+| condition | ESP cipher | single-stream | UDM CPU |
+|---|---|---|---|
+| A | aes256-cbc + sha1 | 232.3 Mbit/s | 35.4% |
+| B | **aes128**-cbc + sha1 | **281.8**, 274.9 Mbit/s | 35.5% |
+| A again | aes256-cbc + sha1 | 244.9, 242.5 Mbit/s | — |
+
+**~1.16–1.20× faster on the cheaper cipher at identical CPU.** Same bytes of
+CPU work, more payload through it. That is the signature of CPU-bound software
+crypto, and it rules out the FortiGate's NPU as the limiter.
+
+### Correcting two of my own earlier claims
+
+1. **"UDM CPU is only ~7%, so it isn't CPU-bound" was WRONG — a sampling
+   artifact.** UniFi's `system-stats.cpu` refreshes on the device report
+   interval; 4-second sample windows were reading stale values. Under a
+   sustained 60 s single-stream load it reads **35.4%**, with load1 rising
+   0.60 → 1.17. On a 4-core UDM Pro SE that is ≈1.4 cores — one core saturated
+   on crypto plus overhead. **Always drive load for ≥60 s before trusting a
+   UniFi CPU figure.**
+2. **The "FortiGate per-session NPU offload" hypothesis is REFUTED**, not merely
+   unproven. It predicts no change from a cipher swap; a 20% change was measured.
+
+### Why the numbers all line up now
+
+- **1 stream = 1 core of UDM crypto** → ~240 Mbit/s on AES-256-CBC.
+- **8 streams = ~3 usable cores** → ~692 Mbit/s, ≈2.9× the single-stream figure
+  on a 4-core box. Aggregate is noisy (492–692 across repeats on a live link)
+  and is *not* cipher-sensitive, consistent with it being bounded by the path/
+  uplink rather than crypto once several cores are engaged.
+- **AES-CBC is the specific villain: it is serial.** Each block depends on the
+  previous one, so the ARM AES instructions cannot pipeline across blocks. GCM
+  (CTR-based) and ChaCha20-Poly1305 both parallelise freely. That is why the
+  same UDM does 557 Mbit/s single-stream on WireGuard and only 240 on IPsec.
+- **This retroactively vindicates the GCM cutover as the right idea aimed at the
+  right box** — GCM would have removed the serial dependency on the constrained
+  end. UniFi simply does not offer it, which is what made it impossible.
+
+### Options this opens
+
+- **AES-128 instead of AES-256: ~16–20% for free**, no topology change, one API
+  call per end. 128-bit is not the weak link here (SHA1 integrity is more
+  dated, and unchanged either way). Operator's call — **not adopted**, restored
+  to aes256.
+- **WireGuard site-to-site via ana-wg: 767 Mbit/s single-stream** (3.1×), and it
+  sidesteps the UDM's IPsec datapath entirely. Still the biggest win available.
+- Replacing the FortiGate **will not help this** — it was never the constraint.
+  Worth knowing before the cutover.
+
+### State left behind
+
+UDM network object verified **byte-identical** to its pre-test snapshot
+(aes256/sha1). Tunnel up, selectors 1/1. FortiGate phase2 `pfi-ana-nh3` is
+left as `aes256-sha1 aes256gcm aes128-sha1` — a permissive superset; the peer
+offers only aes256 so the extra entries are inert, but **narrowing it back to
+`aes256-sha1` is one line** if the looser list is unwanted.