vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: snapshot — run 7 training on gx10; run 6 TRANSFERRED after CSAM adjudication; erp-tune-v6-nvfp4a16 live as trial; ESH/YTVC/webhook repairs; ana-ml2 routes persisted; tank/zroot actions deferred to next session. Index 830→271 lines: 27 decisions + 8 abandoned archived, superseded in-flight blocks archived verbatim

Browse Source
This commit is contained in:
vh
2026-09-09 00:26:44 -07:00
parent 3e18a044bd
commit 5ad948bf31
23 changed files with 2951 additions and 2800 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md
-663
View File
@@ -1,663 +0,0 @@
# [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit
> **⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at
> the bottom before acting on anything here.** The headline is wrong (the
> relevant ceiling is NH3's **1 Gbps** uplink, not Anaheim's 2 Gbps), the
> aggregate number is wrong (**692 Mbit/s** at 8 streams, not ~550 — the
> original stopped measuring at 4), and the proposed remedy is **impossible**:
> UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The
> per-stream observation and the parallelise-your-transfers mitigation are the
> parts that survive.
The operator noticed site-to-site transfers were slow for a datacenter fiber
handoff and asked whether WireGuard was the limit. It is not WireGuard, and the
circuit is fine.
## Measured
```
ana-docker -> internet, 8 parallel 2,153 Mbit/s <- the 2 Gbps handoff, delivering
ANA <-> NH3 through the tunnel, 4 par. 460 Mbit/s
FortiGate's own recorded peak 554 Mbit/s
ANA <-> NH3, single stream 227 Mbit/s
ANA <-> ESH, single stream 249-265 Mbit/s
ESH <-> NH3 (never touches ana-gw) 545-557 Mbit/s on a SINGLE stream
```
Method: stdlib TCP probe (no ssh, no crypto, no compression) between site
endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner
Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling.
## What it is not
- **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw
(`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard
on ana-wg is remote-access only and is not in this path. Traceroute confirms:
both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast
ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it.
- **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all
8 cores** during the tests, and both live tunnels report `npu_flag=03` with
`dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded.
- **Not a 250 Mbit/s cap.** That was the first number and it is misleading —
single-stream TCP. Four parallel streams doubled it. Quote the aggregate.
- **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions.
## Most likely cause
Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14,
IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly
faster on **AES-GCM**, which combines encryption and authentication in one
pass. The datasheet IPsec headline for an 80F assumes GCM with large packets,
not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x
shortfall is consistent with that.
## Not executed
Changing the proposal is a **production-edge change requiring a matching
change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while
it renegotiates. Left for the operator. See the index entry for authorization
state.
## Immediate mitigation, no config change
Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a
single stream through IPsec gets 227. **Anything moving bulk data across the
Anaheim link should parallelise** — that alone roughly doubles throughput
today.
## Practical consequence already observed
`/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s
from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and
not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]].
## Access note
ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent
on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is
vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list`
prints live ESP session keys** — never paste its output into althing, a
booth, or a commit.
---
## CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist
The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done,
and the measurements taken while trying show there is very little left to win.
### AES-GCM is unavailable on the far end — not a naming problem
The NH3 edge is a **UDM Pro SE** terminating `pfi-nh3-ana` (networkconf
`_id 697d64414c85dd2b6669b00a`, `ifname vti64`). Its UniFi API **validates** the
crypto enum and rejected every GCM spelling tried — `aes256gcm`, `aes256gcm128`,
`aes256gcm16`, `aes-256-gcm`, `aes256-gcm`, `aes256gcm12`, `gcm`, `aes128gcm128`
— all `HTTP 400 api.err.InvalidPayload`, nothing applied.
**The control that makes this conclusive:** the *identical* request body with
`ipsec_esp_encryption: "aes256"` returns `HTTP 200 rc:ok`. So the 400s are the
enum rejecting the value, not a malformed body. Corroborating: **zero
case-insensitive `gcm` matches across 7.3 MB of UniFi OS UI bundles.**
Accepted enum (probed): `aes128`, `aes192`, `aes256`, `3des` → 200; `des`,
`chacha20poly1305` → 400. There is no AEAD option. Both Anaheim tunnels land on
UniFi far ends, so this blocks the ESH tunnel too.
The FortiGate side **was** widened and is GCM-capable: phase2 `pfi-ana-nh3` now
reads `set proposal aes256-sha1 aes256gcm`. Left in place deliberately — it is
functionally identical while the peer only offers CBC, and reverting it would
cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched;
IKE protects the control channel only and has no bearing on data throughput.
### The numbers that retire this as a problem
Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on
the same UDM and the same uplink, with the same stdlib TCP probe:
| streams | IPsec NH3→ANA | WireGuard NH3→ESH |
|---|---|---|
| 1 | 245 Mbit/s | 557 Mbit/s |
| 4 | 471 Mbit/s | 767 Mbit/s |
| 8 | **692 Mbit/s** | **795 Mbit/s** |
**NH3's WAN is a 1 Gbps link** (`uplink.speed = 1000`, port capable of 10G) —
that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel.
So the tunnel does **~69% of the achievable uplink** at 8 streams, and the
IPsec-vs-WireGuard gap collapses from 2.3× at one stream to **15% at eight**.
Re-architecting the transport (site-to-site WireGuard via `ana-wg`, since
FortiOS has no WireGuard) would chase that last 15%. Not worth it.
### What the constraint actually is
A **per-stream** limit (~245 Mbit/s), not an aggregate crypto ceiling. Both
endpoints are idle at load — FortiGate CPU 100% idle with `npu_flag=03`
(offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The
shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from
a higher floor (557 → 795 is only 1.43× scaling).
### Actionable consequence
Anything moving bulk data across this link should **parallelise** — 245 → 692
Mbit/s, a 2.8× win with no config change. For single-stream workloads that
cannot be parallelised at the application layer, **NFS `nconnect=N` is the
lever**: it opens N TCP connections per mount, converting a single-stream
workload into a parallel one. The `/mnt/smithy` mount on ana-ml2 reading at
24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live
example — remounting with `nconnect=8` is the obvious test.
### Foot-gun recorded
Probing the enum by PUTting candidate values **applies the accepted ones**. A
probe loop here timed out with `3des` briefly live on the NH3 side, which the
FortiGate would not accept — a short tunnel outage until `aes256` was restored
(~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi
config enum this way, restore the known-good value after **every** 200, not at
the end of the loop. Post-change verification: the UDM object was diffed
field-by-field against its pre-change snapshot and is **byte-identical**.
---
## FOLLOW-UP (2026-08-23): what the per-stream limit actually is
The correction above called the constraint "per-SA/per-flow serialisation".
That was a hand-wave. Measured properly, it is a **hard per-flow rate cap of
~230–245 Mbit/s with a very deep buffer in front of it** — not a tuning
problem, not loss, not window size.
### The evidence: pin the send buffer and sweep it
Single stream NH3 → ana-docker, `SO_SNDBUF` pinned, `ss -ti` sampled in flight:
| in-flight cap | throughput | RTT in flight | minRTT | retrans |
|---|---|---|---|---|
| 256 KB | 224 Mbit/s | 7.8 ms | 5.3 ms | 0 |
| 416 KB | 225 Mbit/s | 11.8 ms | 6.6 ms | 0 |
| 416 KB | 245–247 Mbit/s | 12.0 ms | 5.6 ms | 0 |
| ~3.3 MB (autotuned) | 245 Mbit/s | **107 ms** | 5.5 ms | 0 |
**Throughput is flat across a 13× range of in-flight data while RTT scales with
it.** That is the signature of a fixed service rate with a standing queue: the
window controls only how much queue you build, never how fast you go. Had this
been window-limited, throughput would have risen with the buffer. Had it been
congestion, there would be retransmits — there are essentially none
(`retrans:0`, 0% ping loss).
So `net.ipv4.tcp_*` tuning, window scaling and congestion-control choice are all
**red herrings here**. Do not go there.
### Bufferbloat: one bulk stream wrecks latency for everything else
Measured on the same tunnel, ping to ana-docker:
- idle: **6.9 ms** avg
- during a **single** bulk TCP stream: **102 ms** avg, 136 ms max, 0% loss
**15× latency inflation from one transfer.** This is the operationally
important finding — any interactive traffic sharing the Anaheim link (ssh,
RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it
takes only one stream to do it. Parallelising transfers makes throughput
better and this *worse*. If it starts biting, the fix is an AQM/shaper on the
tunnel (or rate-limiting bulk jobs), not more buffer.
### Where the cap lives — strong inference, not proof
Three paths, and the FortiGate is the only variable:
| path | single-stream |
|---|---|
| FortiGate ↔ NH3 UDM (IPsec) | 245 Mbit/s |
| FortiGate ↔ ESH UDM (IPsec) | 249–265 Mbit/s |
| NH3 UDM ↔ ESH UDM (WireGuard, **no FortiGate**) | 557 Mbit/s |
Present in both slow paths, absent from the fast one. Aggregate over the same
SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole —
many flows spread out fine, one flow does not.
The mechanism that fits is **FortiGate NPU IPsec offload being per-session**:
each firewall session is bound to one crypto engine, so a single TCP flow is
capped at one engine's rate while many sessions spread across engines. **This
is inference from the throughput shape, not something confirmed on the box** —
`diagnose sys session list` was not captured for a TCP flow (the filter caught
only traceroute UDP probes). A single-stream control through ana-gw *without*
IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is
window-limited and does not discriminate. **If this matters, the clean test is
a non-IPsec single stream between two Anaheim VLANs at low RTT.**
**Relevant to the FortiGate cutover decision:** if the per-flow cap is the
FortiGate's IPsec path, replacing the box plausibly lifts single-stream
throughput toward the WireGuard figure. That is a point in favour of the
cutover, and it is cheap to verify afterwards by re-running the sweep.
---
## FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec
Operator asked directly whether the 80F "can't handle the traffic". It can.
Two new measurements settle the shape of this, and correct an overstatement in
FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded —
every slow path was *both* IPsec *and* FortiGate, so protocol and box could not
be separated by that argument).
### The 80F routes a single flow at line rate when IPsec is not involved
`ana-ml2 → pfi-pve`, inter-VLAN **through** ana-gw (traceroute hop 1 =
`10.250.50.1`), 0.36 ms RTT, no tunnel:
| streams | throughput |
|---|---|
| 1 | **940.2 Mbit/s** |
| 8 | 939.3 Mbit/s |
Single stream saturates 1 GbE. So the box does **not** cap single sessions in
general, and there is no per-session ceiling in its plain forwarding path. The
~250 Mbit/s per-flow cap is **specific to the IPsec datapath**.
### Both IPsec tunnels converge on the same numbers despite different far ends
Measured today with the same probe:
| tunnel | far-end gateway | RTT | 1 stream | 8 streams |
|---|---|---|---|---|
| NH3 ↔ ANA | UDM Pro **SE** | 6.7 ms | 245 Mbit/s | 692 Mbit/s |
| ESH ↔ ANA | UDM Pro **Max** | 3.9 ms | **268 Mbit/s** | **715 Mbit/s** |
Different gateway hardware, different sites, different uplinks, and RTT
differing by 1.7× — yet single-stream differs by only 9%. **If this were
window-limited the 3.9 ms path would be ~1.7× faster.** It is not, which is
independent confirmation of a rate cap rather than a BDP effect.
### Capacity summary — the box has headroom it will not give one flow
- plain routing, 1 stream: **940 Mbit/s** (line rate)
- plain routing to internet, 8 streams: **2,153 Mbit/s**
- IPsec, 8 streams: **692–715 Mbit/s**
- IPsec, 1 stream: **245–268 Mbit/s**
- CPU **100% idle** throughout; IPsec NPU-offloaded (`npu_flag=03`)
Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath
distributes work **by inner session** — consistent with IPsec offload binding a
session to one crypto engine.
### What is still NOT separated
Whether the cap belongs to **the 80F's IPsec offload** or to **UniFi's IPsec
implementation**. Both tunnels have a UDM at the far end, and both UDMs run the
same UniFi firmware, so identical caps are explainable either way. The Pro Max
being only 9% faster than the Pro SE argues against the UniFi side (a beefier
CPU should show more), but that is suggestive, not conclusive.
**The test that closes it:** an IPsec tunnel whose endpoints do not include the
80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured
single-stream. If it also caps ~250, the FortiGate is exonerated and replacing
it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM
WireGuard achieves, the 80F is the limiter. **Bears directly on the pending
FortiGate cutover** — worth running before that decision, not after.
---
## FOLLOW-UP 3 (2026-08-23): WireGuard over the same internet path does 767 Mbit/s on ONE stream
Operator asked for a WireGuard test from `ana-wg` to NH3 over the public
internet. It is the test that separates the *path* from the *crypto*, and the
answer is unambiguous. **It also overturns FOLLOW-UP 1's "re-architecting the
transport is not worth it" — that conclusion compared 8-stream numbers and was
wrong for single-stream workloads.**
### Setup (fully torn down afterwards)
`ana-wg` (10.250.50.252, Debian 12 LXC, 4 cores) already has an
internet-reachable WireGuard endpoint: wg0 on **UDP 31337**, published by
FortiGate VIP `wg-to-ana-wg` (extip **38.120.12.42** → 10.250.50.252:31337,
policy 46, service `WireGuard-LEET`). **No FortiGate change was needed.** A
temporary `wgt0` was created on nh3-dev (10.30.10.200/32) as a fourth peer on
wg0, measured, then removed — ana-wg is back to its original 3 peers and the
keys were shredded. `wireguard-tools` was installed on nh3-dev and **left in
place** (benign, and wanted if this becomes permanent).
In this topology **neither gateway does crypto**: the FortiGate and the NH3 UDM
only NAT/forward UDP, and Linux does WireGuard at both ends.
### The full comparison
| path | crypto performed by | 1 stream | 8 streams |
|---|---|---|---|
| IPsec NH3↔ANA | FortiGate + UDM | 245 Mbit/s | 692 Mbit/s |
| IPsec ESH↔ANA | FortiGate + UDM | 268 Mbit/s | 715 Mbit/s |
| **WireGuard NH3→ana-wg** (same internet path) | **Linux + Linux** | **767 Mbit/s** | 763 Mbit/s |
| WireGuard NH3↔ESH (Site Magic) | UDM + UDM | 557 Mbit/s | 795 Mbit/s |
| plain routing through the 80F (inter-VLAN) | none | 940 Mbit/s | 939 Mbit/s |
**One stream equals eight streams over Linux WireGuard (767 ≈ 763).** There is
no per-flow penalty at all, and a single flow already saturates the path. So
the ~245 Mbit/s per-flow cap is **not** the ISP, not the circuit, not the NH3
uplink and not the physical path — all of which sustain 767 on one flow.
Per-flow penalty ranks by implementation:
- **Linux WireGuard — none** (767 → 763, flat)
- **UDM WireGuard — mild**, ~1.4× (557 → 795)
- **IPsec on this pair — severe**, ~2.8× (245 → 692)
### Latency under load — the same story
| path | idle | during ONE bulk stream |
|---|---|---|
| IPsec NH3↔ANA | 6.9 ms | **102 ms** avg, 136 ms max |
| WireGuard NH3→ana-wg | 6.2 ms | **12.7 ms** avg, 23 ms max |
WireGuard carries **3.1× the single-stream throughput with 8× less latency
inflation** on the same wire.
### Attribution — still not fully separated, and it no longer matters much
Both IPsec measurements have a FortiGate *and* a UDM doing IPsec, so this still
does not isolate which one imposes the 2.8× penalty. Closing that would need
Linux↔Linux IPsec or UDM↔UDM IPsec on the same path. **But the practical
decision no longer depends on the answer**, because the fix is the same either
way and it is already demonstrated.
### Recommendation (supersedes FOLLOW-UP 1)
A **WireGuard site-to-site between NH3 and Anaheim, terminated on `ana-wg`**, is
worth real consideration: 3.1× single-stream, flat scaling, far better latency
under load, and it reuses infrastructure that already exists and is already
internet-reachable. It is also the architecture already proven for NH3↔ESH.
Open questions before committing: routing/failover if ana-wg (an LXC) is down,
whether it replaces or parallels the IPsec tunnel, and firewall policy for the
new transit. ana-wg CPU was only ~40% busy across 4 cores at 767 Mbit/s, so it
has headroom.
**AND: `nconnect=8` on /mnt/smithy remains worth doing regardless** — it is the
same lever (turn one flow into many) and brokkr-smithy-dev has given standing
approval to apply it once the FortiGate work settles, with no need to ask again.
---
## RESOLVED (2026-08-23): it is the UDM's software AES-CBC. The FortiGate is exonerated.
Operator's theory — the UDM does IPsec in software with no crypto offload, so
the cost of the cipher itself is the limit — is **correct**, and it is now
demonstrated rather than inferred. He also correctly pointed out that
UDM↔UDM Site Magic is **WireGuard, not IPsec**, so that row never said anything
about UniFi's IPsec performance. It didn't, and I had leaned on it.
### The controlled experiment: vary cipher cost, hold everything else
AES-128 is 10 rounds, AES-256 is 14. If software crypto is the binding
constraint, throughput must rise when the cipher gets cheaper. If the limit
were the FortiGate's NPU, it would not move at all — hardware crypto is not
cipher-cost-sensitive in that range. Run A/B/A, single stream, 25–60 s each:
| condition | ESP cipher | single-stream | UDM CPU |
|---|---|---|---|
| A | aes256-cbc + sha1 | 232.3 Mbit/s | 35.4% |
| B | **aes128**-cbc + sha1 | **281.8**, 274.9 Mbit/s | 35.5% |
| A again | aes256-cbc + sha1 | 244.9, 242.5 Mbit/s | — |
**~1.16–1.20× faster on the cheaper cipher at identical CPU.** Same bytes of
CPU work, more payload through it. That is the signature of CPU-bound software
crypto, and it rules out the FortiGate's NPU as the limiter.
### Correcting two of my own earlier claims
1. **"UDM CPU is only ~7%, so it isn't CPU-bound" was WRONG — a sampling
artifact.** UniFi's `system-stats.cpu` refreshes on the device report
interval; 4-second sample windows were reading stale values. Under a
sustained 60 s single-stream load it reads **35.4%**, with load1 rising
0.60 → 1.17. On a 4-core UDM Pro SE that is ≈1.4 cores — one core saturated
on crypto plus overhead. **Always drive load for ≥60 s before trusting a
UniFi CPU figure.**
2. **The "FortiGate per-session NPU offload" hypothesis is REFUTED**, not merely
unproven. It predicts no change from a cipher swap; a 20% change was measured.
### Why the numbers all line up now
- **1 stream = 1 core of UDM crypto** → ~240 Mbit/s on AES-256-CBC.
- **8 streams = ~3 usable cores** → ~692 Mbit/s, ≈2.9× the single-stream figure
on a 4-core box. Aggregate is noisy (492–692 across repeats on a live link)
and is *not* cipher-sensitive, consistent with it being bounded by the path/
uplink rather than crypto once several cores are engaged.
- **AES-CBC is the specific villain: it is serial.** Each block depends on the
previous one, so the ARM AES instructions cannot pipeline across blocks. GCM
(CTR-based) and ChaCha20-Poly1305 both parallelise freely. That is why the
same UDM does 557 Mbit/s single-stream on WireGuard and only 240 on IPsec.
- **This retroactively vindicates the GCM cutover as the right idea aimed at the
right box** — GCM would have removed the serial dependency on the constrained
end. UniFi simply does not offer it, which is what made it impossible.
### Options this opens
- **AES-128 instead of AES-256: ~16–20% for free**, no topology change, one API
call per end. 128-bit is not the weak link here (SHA1 integrity is more
dated, and unchanged either way). Operator's call — **not adopted**, restored
to aes256.
- **WireGuard site-to-site via ana-wg: 767 Mbit/s single-stream** (3.1×), and it
sidesteps the UDM's IPsec datapath entirely. Still the biggest win available.
- Replacing the FortiGate **will not help this** — it was never the constraint.
Worth knowing before the cutover.
### State left behind
UDM network object verified **byte-identical** to its pre-test snapshot
(aes256/sha1). Tunnel up, selectors 1/1. FortiGate phase2 `pfi-ana-nh3` is
left as `aes256-sha1 aes256gcm aes128-sha1` — a permissive superset; the peer
offers only aes256 so the extra entries are inert, but **narrowing it back to
`aes256-sha1` is one line** if the looser list is unwanted.
---
## FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through
Operator's point: FortiOS has no WireGuard, so a WireGuard site-to-site must
terminate on a box *behind* the edge. Correct — and `ana-wg` (LXC, CT 113 on
pfi-pve, 10.250.50.252) already is that box.
**This closes a gap in FOLLOW-UP 3.** That 767 Mbit/s figure was measured with
traffic terminating *on* ana-wg. Real traffic must be forwarded onward to other
Anaheim hosts, which was never measured. Now it is:
| topology | 1 stream | 8 streams |
|---|---|---|
| IPsec, FortiGate ↔ UDM (today) | 245 Mbit/s | 692 Mbit/s |
| WG terminating **on** ana-wg | 767 Mbit/s | 763 Mbit/s |
| **WG transit: nh3 → wg → ana-wg → forward → ana-docker** | **763.8 Mbit/s** | **790.4 Mbit/s** |
**Forwarding through the LXC is free** (763.8 vs 767). The downstream-VM
architecture delivers the full 3.1× single-stream for real transit traffic, not
just for traffic landing on the tunnel box.
ana-wg while forwarding 764 Mbit/s: **~22% busy across 4 cores** (77.8% idle),
so roughly 0.9 cores. Note `/proc/loadavg` inside this LXC reports the *host's*
load, not the container's — do not read it as ana-wg's own. For contrast the
UDM burns 35.4% of its 4 cores to move 240 Mbit/s, so ana-wg has ample headroom.
### Design consequences of terminating downstream — the parts that need decisions
1. **Anaheim hosts must route to ana-wg, not to the FortiGate.** The 763.8
figure was obtained with an explicit `10.30.10.200/32 via 10.250.50.252`
route on ana-docker. Without that, a host sends 10.100.0.0/16 to its default
gateway (ana-gw), which routes it back out the *same* interface to ana-wg — a
LAN hairpin crossing the FortiGate twice. **The hairpin variant was NOT
measured.** Options: DHCP option 121 pushing the route fleet-wide, a dedicated
transit VLAN for ana-wg, or accept the hairpin.
2. **New single point of failure.** Today site-to-site dies only when the edge
dies, which is total anyway. A downstream terminator fails independently.
Mitigation: keep the IPsec tunnel configured as a higher-metric fallback
route so it takes over when ana-wg is down.
3. **ana-wg is an LXC on pfi-pve**, so its ~0.9 cores and NIC traffic land on the
hypervisor shared with the rest of the Anaheim VMs.
4. **The NH3 end needs a terminator too**, and there are two shapes:
- **Linux VM at NH3** (nh3-dev or a dedicated VM on nh3-pve) — this is what
was measured: **764 Mbit/s**.
- **NH3 UDM's existing WireGuard server** (`PFI-NH3-WG`, wireguard-server on
UDP 31337) accepting ana-wg as a peer — plausible but **untested**, and
UniFi's WireGuard shows a per-flow penalty (557 Mbit/s single-stream on
Site Magic), so expect ~557 rather than 764. Still 2.3× today.
### Standing recommendation
Worth doing, but it is **a project, not a config tweak** — routing, failover and
policy all need deciding. The cheap wins remain available meanwhile and are
independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending
the FortiGate work settling) and AES-128 for ~20%.
---
## LANDED (2026-08-23): AES-128 on both tunnels; FortiGate public admin closed
Operator directed: adopt AES-128 on **both** Anaheim tunnels, make-before-break,
then close the FortiGate's WAN and SSH admin surfaces. All done and verified.
**Context that retires the WireGuard-in-a-VM design work:** the FortiGate is
being **replaced by OPNsense on a Dell R420**, which gives **WireGuard on the
edge device itself**. The downstream-terminator architecture (FOLLOW-UP 4) is
therefore moot — do not scope it. This also **un-parks the OPNsense migration**,
which auto-memory recorded as PARKED pending "hardware acquisition"; the R420
is that trigger.
### What changed
Make-before-break on the FortiGate first, so neither tunnel dropped waiting on
a far end:
| phase2 | proposal now |
|---|---|
| `pfi-ana-nh3` | `aes256-sha1 aes256gcm aes128-sha1` |
| `ana-eshudm-dyn` | `aes256-sha1 aes128-sha1` |
Then each UDM flipped to `ipsec_esp_encryption: aes128`:
| tunnel | UDM object | before | after |
|---|---|---|---|
| NH3 ↔ ANA | `pfi-nh3-ana` `697d64414c85dd2b6669b00a` @ 10.100.0.1 | 245 Mbit/s | **269.7** |
| ESH ↔ ANA | `esh-ana` `697723b9b9d4266dddf2bcc7` @ 10.0.0.1 | 268 Mbit/s | **304.3** |
Single-stream gain ~10–13% here, against 16–20% in the earlier controlled A/B —
the difference is live-link variance, not a different result. Both UDM objects
were diffed field-by-field against pre-change snapshots: **the only field that
moved on either is `ipsec_esp_encryption`.**
The FortiGate proposal lists were deliberately **left permissive** (still
accepting aes256). The peers offer only aes128 so the extra entries are inert,
and keeping them means a UDM reverting does not strand the tunnel. Narrowing to
`aes128-sha1` alone is a one-liner if the looser list is unwanted.
### Admin surfaces closed
`wan1 allowaccess` → **`ping`** (https + ssh removed) and `infra-ops` trusthost
→ **10.0.0.0/8 only** (the 8 wide-open ranges unset). Verified 443 and 22 closed
from both NH3 and ESH; management over the tunnel at 10.250.0.1 still works.
**Sequencing that matters: the close was executed over the TUNNEL path, not over
WAN** — removing `ssh` from allowaccess while connected over WAN kills the
session mid-command.
**Consequence to hold in mind: ana-gw now has no out-of-band management path.**
If both tunnels drop it is console-only until someone is on site.
### Gotcha: the two UDM vault items have DIFFERENT shapes
- `unifi/pfi-udmse-api-key` → a **bare 32-char key**. `secret get` output is the key.
- `unifi/esh-udmpm-api-key` → a **19-line documentation note** with the key on a
`key:` line. `secret get` piped straight into a header yields a 1396-byte
value and the UDM answers **`400 Bad Request` from nginx**. Extract with
`grep '^key:' | awk '{print $2}'`.
**The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded
it as read-verified only): a control PUT of the unchanged object returned
`rc:ok`, then the real change did too. That key has a full read+write admin role.
---
## CORRECTION (2026-08-23): port 80 on the WAN IP is the FortiOS ACME listener
The claim in the previous section that `.42:80` was an **ISP transparent proxy**
was **WRONG**, and so was the earlier warning that ACME renewal would fail with
port 80 absent from `allowaccess`. Operator pushed back asking where the port-80
map terminated. It terminates **on the FortiGate itself**.
**What it is:** the FortiOS **ACME HTTP-01 challenge listener**. `config system
acme` has `set interface "wan1"`, and FortiOS opens port 80 on that interface to
answer Let's Encrypt challenges **independently of `allowaccess`** — `wan1
allowaccess` reads `ping` only and the port is still open. Every non-challenge
request returns a fixed 403 whose body is literally:
```
<!DOCTYPE html><html><head><title>ACME Access Only</title></head><body>ACME Access Only</body></html>
```
**Not a DNAT.** The full VIP table has 14 entries; only two land on `.42` —
`Kokoro-In` (:8880 → 10.250.50.51) and `wg-to-ana-wg` (:31337 → 10.250.50.252).
~~Worth noting separately: four VIPs are all-port static NAT~~ — **that claim was
WRONG, see the correction below.** All fourteen VIPs are scoped.
### The methodology error that produced the wrong answer — worth not repeating
The sniffer filter used was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
matches only inbound packets**, so outbound SYN-ACKs were excluded *by
construction*; concluding "the box sends no SYN-ACK" from that capture was
unsound. Re-run with the bidirectional `host 38.120.12.42 and tcp port 80` it
immediately shows `wan1 out 38.120.12.42.80 -> <scanner>: syn ack`.
**Rule: when testing whether a box *answers*, the sniffer filter must be
bidirectional. `dst host` silently answers a different question.**
### Consequences
- **ACME renewal will work** with `allowaccess ping`. The earlier "add `http`
back or the cert expires" warning is retracted — FortiOS opens the challenge
port itself. Cert valid to 2026-10-27, renewal attempt ~2026-09-27.
- **It is not an admin surface** — static 403, no auth, no GUI.
- Its practical value is now low: WAN admin is closed, so the cert only serves
the internal GUI at 10.250.0.1, where the name would not match anyway. Killing
it (`config system acme` → unset interface) would close the last WAN listener
at the cost of cert renewal. Operator's call; **not done**.
---
## CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE
### ACME disabled — the WAN IP now exposes nothing
`config system acme / unset interface` (the account object is left in place;
with no interface bound there is no listener). Verified:
- **External scan of 38.120.12.42 across 55 ports: no open TCP ports at all.**
- Internal GUI at 10.250.0.1 still answers **200**, SSH still works.
- `admin-server-cert` is still `ana-fw.pfi` — the existing cert is untouched and
serves the internal GUI until **2026-10-27**; it simply will not auto-renew.
Reverse with `config system acme / set interface "wan1"`.
### RETRACTION: the four VIPs are NOT all-port
A previous section claimed `Rustdesk`, `https-to-tacticalrmm`, `web-to-webhost`
and `web-to-sfcontainer` were unrestricted all-port static NATs. **They are not.**
A FortiOS VIP can be scoped **two different ways** and the parser used only
checked one:
1. `set portforward enable` + `set extport <n>` — a single mapped port, **or**
2. `set service "<svc>"` on the VIP object — constrains the VIP to that service.
All four use form 2. The custom services are narrow: `Rustdesk` = TCP
21115–21119 + UDP 21116 (the standard RustDesk range), `ssh-mapped-2223` = TCP
2223 only. **Every one of the 14 VIPs is scoped; none is unrestricted.**
**Lesson: absence of `portforward` does NOT mean all-port on a FortiOS VIP —
check `service` too.** Better still, do what settled it here: scan from outside
rather than reading config.
### Ground-truth public exposure (external TCP scan, post-change)
| IP | open | maps to |
|---|---|---|
| 38.120.12.41 | *nothing* | — |
| **38.120.12.42** | ***nothing*** | the FortiGate itself — fully closed |
| 38.120.12.43 | 80, 443 | sf-ana-container 10.250.150.100 (SureFire tenant) |
| 38.120.12.44 | 22, 80, 443, 8025, 21115–21119 | gitea (→222), traefik, mailrise, RustDesk |
| 38.120.12.45 | 80, 443, 2223 | pfi-ana-webhost 10.250.50.52 (2223→22) |
| 38.120.12.46 | 443 | pfi-tacticalrmm 10.250.50.57 |
Configured-but-closed: 8443 (mattermost-calls), 8444 (webdav-nas), 8880
(Kokoro-In) — VIPs exist, nothing listening behind them. Worth a tidy-up during
the OPNsense translation but not exposure.