ops(ana-gw): close out the Anaheim tunnel cipher question
The AES-GCM cutover was authorised and attempted, NH3 side first. It cannot be completed: UniFi's manual site-to-site IPsec implements no AEAD cipher. Eight GCM spellings were rejected with api.err.InvalidPayload while an otherwise identical body carrying aes256 returned rc:ok, which isolates the enum rather than the request as the cause. The accepted set is aes128/aes192/aes256/3des. Both Anaheim tunnels terminate on UniFi gateways, so this blocks the ESH tunnel on the same grounds. Measuring while testing also retires the premise. NH3's uplink is a 1 Gbps link, so Anaheim's 2 Gbps circuit was never the relevant ceiling, and the tunnel sustains 692 Mbit/s across 8 streams rather than the ~550 previously recorded from a 4-stream run. Compared with WireGuard over the same gateway and uplink, the deficit narrows from 2.3x at one stream to 15% at eight, so moving this link onto a different transport is not worth doing. The constraint is per-stream, around 245 Mbit/s, with both endpoints idle under load. Parallelising bulk transfers remains the mitigation and is worth 2.8x at no cost; NFS nconnect is the equivalent lever for single-stream mounts such as /mnt/smithy on ana-ml2. FortiGate phase2 for pfi-ana-nh3 keeps the widened proposal list, which is inert while the peer offers only CBC and avoids a further renegotiation. The UDM network object was diffed field-by-field against its pre-change snapshot and is unchanged.
This commit is contained in:
@@ -1,5 +1,14 @@
|
||||
# [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit
|
||||
|
||||
> **⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at
|
||||
> the bottom before acting on anything here.** The headline is wrong (the
|
||||
> relevant ceiling is NH3's **1 Gbps** uplink, not Anaheim's 2 Gbps), the
|
||||
> aggregate number is wrong (**692 Mbit/s** at 8 streams, not ~550 — the
|
||||
> original stopped measuring at 4), and the proposed remedy is **impossible**:
|
||||
> UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The
|
||||
> per-stream observation and the parallelise-your-transfers mitigation are the
|
||||
> parts that survive.
|
||||
|
||||
The operator noticed site-to-site transfers were slow for a datacenter fiber
|
||||
handoff and asked whether WireGuard was the limit. It is not WireGuard, and the
|
||||
circuit is fine.
|
||||
@@ -69,3 +78,80 @@ on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is
|
||||
vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list`
|
||||
prints live ESP session keys** — never paste its output into althing, a
|
||||
booth, or a commit.
|
||||
|
||||
---
|
||||
|
||||
## CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist
|
||||
|
||||
The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done,
|
||||
and the measurements taken while trying show there is very little left to win.
|
||||
|
||||
### AES-GCM is unavailable on the far end — not a naming problem
|
||||
|
||||
The NH3 edge is a **UDM Pro SE** terminating `pfi-nh3-ana` (networkconf
|
||||
`_id 697d64414c85dd2b6669b00a`, `ifname vti64`). Its UniFi API **validates** the
|
||||
crypto enum and rejected every GCM spelling tried — `aes256gcm`, `aes256gcm128`,
|
||||
`aes256gcm16`, `aes-256-gcm`, `aes256-gcm`, `aes256gcm12`, `gcm`, `aes128gcm128`
|
||||
— all `HTTP 400 api.err.InvalidPayload`, nothing applied.
|
||||
|
||||
**The control that makes this conclusive:** the *identical* request body with
|
||||
`ipsec_esp_encryption: "aes256"` returns `HTTP 200 rc:ok`. So the 400s are the
|
||||
enum rejecting the value, not a malformed body. Corroborating: **zero
|
||||
case-insensitive `gcm` matches across 7.3 MB of UniFi OS UI bundles.**
|
||||
|
||||
Accepted enum (probed): `aes128`, `aes192`, `aes256`, `3des` → 200; `des`,
|
||||
`chacha20poly1305` → 400. There is no AEAD option. Both Anaheim tunnels land on
|
||||
UniFi far ends, so this blocks the ESH tunnel too.
|
||||
|
||||
The FortiGate side **was** widened and is GCM-capable: phase2 `pfi-ana-nh3` now
|
||||
reads `set proposal aes256-sha1 aes256gcm`. Left in place deliberately — it is
|
||||
functionally identical while the peer only offers CBC, and reverting it would
|
||||
cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched;
|
||||
IKE protects the control channel only and has no bearing on data throughput.
|
||||
|
||||
### The numbers that retire this as a problem
|
||||
|
||||
Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on
|
||||
the same UDM and the same uplink, with the same stdlib TCP probe:
|
||||
|
||||
| streams | IPsec NH3→ANA | WireGuard NH3→ESH |
|
||||
|---|---|---|
|
||||
| 1 | 245 Mbit/s | 557 Mbit/s |
|
||||
| 4 | 471 Mbit/s | 767 Mbit/s |
|
||||
| 8 | **692 Mbit/s** | **795 Mbit/s** |
|
||||
|
||||
**NH3's WAN is a 1 Gbps link** (`uplink.speed = 1000`, port capable of 10G) —
|
||||
that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel.
|
||||
So the tunnel does **~69% of the achievable uplink** at 8 streams, and the
|
||||
IPsec-vs-WireGuard gap collapses from 2.3× at one stream to **15% at eight**.
|
||||
|
||||
Re-architecting the transport (site-to-site WireGuard via `ana-wg`, since
|
||||
FortiOS has no WireGuard) would chase that last 15%. Not worth it.
|
||||
|
||||
### What the constraint actually is
|
||||
|
||||
A **per-stream** limit (~245 Mbit/s), not an aggregate crypto ceiling. Both
|
||||
endpoints are idle at load — FortiGate CPU 100% idle with `npu_flag=03`
|
||||
(offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The
|
||||
shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from
|
||||
a higher floor (557 → 795 is only 1.43× scaling).
|
||||
|
||||
### Actionable consequence
|
||||
|
||||
Anything moving bulk data across this link should **parallelise** — 245 → 692
|
||||
Mbit/s, a 2.8× win with no config change. For single-stream workloads that
|
||||
cannot be parallelised at the application layer, **NFS `nconnect=N` is the
|
||||
lever**: it opens N TCP connections per mount, converting a single-stream
|
||||
workload into a parallel one. The `/mnt/smithy` mount on ana-ml2 reading at
|
||||
24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live
|
||||
example — remounting with `nconnect=8` is the obvious test.
|
||||
|
||||
### Foot-gun recorded
|
||||
|
||||
Probing the enum by PUTting candidate values **applies the accepted ones**. A
|
||||
probe loop here timed out with `3des` briefly live on the NH3 side, which the
|
||||
FortiGate would not accept — a short tunnel outage until `aes256` was restored
|
||||
(~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi
|
||||
config enum this way, restore the known-good value after **every** 200, not at
|
||||
the end of the loop. Post-change verification: the UDM object was diffed
|
||||
field-by-field against its pre-change snapshot and is **byte-identical**.
|
||||
|
||||
@@ -108,11 +108,11 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-08-23 — a long multi-party ops session; open with the operator are the Anaheim tunnel cipher and how long WAN admin on ana-gw stays open._
|
||||
_As of 2026-08-23 — a long multi-party ops session. The Anaheim tunnel cipher is now settled (closed: the remedy does not exist). The one thing still open with the operator is how long WAN admin on ana-gw stays open._
|
||||
|
||||
- **🔓 ana-gw WAN admin is OPEN again as of 2026-08-23 — deliberately, and it should be re-closed after the FortiGate cutover.** Operator asked for it as the cutover contingency ("so I don't have to drive down there"). `https://ana-fw.phasefinal.com/` (wan1's own IP 38.120.12.42) serves the admin GUI on a browser-trusted LE cert; **https only**, and only the `infra-ops` account is WAN-reachable (`admin` remains 10.0.0.0/8-locked). Verified end-to-end with a real login from two sites. This partially reverses the 2026-08-12 hardening on a box still running the **EOL, actively-exploited FortiOS 7.2 branch** — so the exposure is CVE-shaped, and time-boxing it is the mitigation. Re-close = `config system interface / edit wan1 / set allowaccess ping / next / end`. Config backups pre+post at `/var/tmp/ana-gw-config-20260823T20*.conf` on nh3-dev. **Two follow-ons:** (a) ACME renewal for that cert needs **port 80** on wan1 — currently absent, next attempt ≈2026-09-27, expiry 2026-10-27; (b) ~5 SYN/s from `179.51.184.0/21` now draws SYN-ACKs, CPU impact nil, unmitigated by choice. Detail + the FortiOS trusthost gotchas: auto-memory `reference_fortigate_ana_gw_access`.
|
||||
|
||||
- **⏳ THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit.** Circuit measured at **2,153 Mbit/s**; the tunnel ceilings ~550 Mbit/s aggregate, ~230 single-stream. FortiGate CPU **idle**, IPsec **NPU-offloaded**, interface error-free — so it is not crypto exhaustion. Both tunnels negotiate **`aes256-sha1`**; **AES-GCM is the proposed change** and the operator has signalled he will authorize it. Not executed: production edge, needs a matching change at NH3 + ESH, each tunnel drops during renegotiation. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
|
||||
- **✅ CLOSED 2026-08-23: the Anaheim tunnel "problem" is mostly a measurement artefact, and the AES-GCM cutover is impossible.** Operator authorised the cutover; it was attempted NH3-side-first and **cannot be done — UniFi's manual site-to-site IPsec implements no AES-GCM** (8 spellings rejected `api.err.InvalidPayload` against a passing `aes256` control; accepted enum is `aes128/aes192/aes256/3des` only). This blocks the ESH tunnel too, since both far ends are UDMs. The framing was also wrong twice over: **NH3's uplink is 1 Gbps** (not Anaheim's 2 Gbps — that is the real ceiling), and the tunnel does **692 Mbit/s at 8 streams** (the original stopped at 4 and reported ~550). Against WireGuard on the same UDM and uplink, the gap collapses from 2.3× at one stream to **15% at eight** — so re-architecting onto WireGuard is not worth it. Real constraint = **per-stream ~245 Mbit/s**, both endpoints idle. **Standing mitigation: parallelise bulk transfers (2.8× for free); for single-stream NFS use `nconnect=N`** — the `/mnt/smithy` mount on ana-ml2 at 24.7 MB/s is exactly this case and is the obvious test. FortiGate phase2 `pfi-ana-nh3` was left widened to `aes256-sha1 aes256gcm` (inert while the peer offers only CBC); UDM verified byte-identical to its pre-change snapshot. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
|
||||
|
||||
- **🟢 SEAT MAP (unchanged this session except selene).** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, **now 7 aliases** (see the collision note). **`char-rp`** = MeroMero-v2 dual-mode, GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC on DFlash2, GPU1 :8019. **`selene` RETIRED** — 17.2 GiB reclaimed on GPU1 (free now ~19.4 GiB).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user