299 lines
20 KiB
Markdown
299 lines
20 KiB
Markdown
# Headscale overlay mesh — plan (2026-09-06)
|
||
|
||
**Status:** planning. Operator direction 2026-09-06: replace Site Magic (NH3↔ESH) and the
|
||
FortiGate IPsec tunnels (colo↔NH3, colo↔ESH) with a self-hosted Tailscale-protocol overlay
|
||
(Headscale), one ultra-light LXC per PVE host, keeping the old tunnels as an emergency
|
||
backup. This document is the plan that a `.contract.md` will be cut from; nothing here is
|
||
provisioned yet.
|
||
|
||
## Why Headscale (decided 2026-09-06)
|
||
|
||
Tailscale clients are the best in class on the devices actually used for remote access
|
||
(iPad, Mac, Linux); the future OPNsense colo edge can be a Tailscale node natively
|
||
(`os-tailscale`); Headscale is one binary + one SQLite file to self-host, versus NetBird's
|
||
six-service stack with a mandatory IdP. NetBird's built-in UI and HA routing groups are
|
||
real but not decisive for three sites. UDP-blocked networks (in-flight Wi-Fi) work because
|
||
the client falls back to a relay over TCP 443; plain WireGuard (ana-wg) cannot.
|
||
|
||
## Site facts that drive placement
|
||
|
||
| site | edge | v4 | v6 | change risk |
|
||
|---|---|---|---|---|
|
||
| ANA colo | FortiGate 80F → **OPNsense on R420 within the month** | public static | pending | HIGH — edge in flux; incident history 2026 (breaker, PSU1, WAN admin closed) |
|
||
| NH3 | UDM SE, stable | dynamic-but-stable (DDNS) | single /64, **reserved for meshing** | low |
|
||
| ESH | UDM Pro Max | CGNAT now, **static soon**; 2G symmetric soon | /56 | low, improving |
|
||
|
||
**Control plane at NH3 now** (only stable edge; static v6 + DDNS v4 → one DNS name).
|
||
**Reassess ESH** once its static v4 lands — it becomes the relay site regardless (2G
|
||
symmetric). **Never the colo:** the mesh's brain must not live in the building it exists
|
||
to reach in an emergency. Migration later = copy one SQLite file + move the DNS name;
|
||
clients follow the URL and never re-enrol.
|
||
|
||
## Topology
|
||
|
||
```
|
||
headscale.phasefinal.com (A via DDNS, AAAA static)
|
||
│ HTTPS 443 (+ DERP later)
|
||
┌─────────────────────────────┼───────────────────────────────┐
|
||
│ NH3 nh3-pve │ │
|
||
│ CT nh3-headscale ──control plane (1 core / 512M / 8G) │
|
||
│ CT nh3-mesh-rtr ──subnet router, advertises 10.100.0.0/16│
|
||
├──────────────────────────────────────────────────────────────┤
|
||
│ ESH esh-pve │
|
||
│ CT esh-mesh-rtr ──subnet router, advertises 10.0.0.0/16 │ (relay/DERP here later)
|
||
├──────────────────────────────────────────────────────────────┤
|
||
│ ANA pfi-pve │
|
||
│ CT ana-mesh-rtr ──subnet router, advertises 10.250.0.0/16│ (OPNsense takes this over later)
|
||
└──────────────────────────────────────────────────────────────┘
|
||
Per-device clients: nh3-dev, laptops/iPad, corviduo-dev, ana-ml2 … (MagicDNS names)
|
||
```
|
||
|
||
Four unprivileged Debian 13 LXCs, each 1 vCPU / 512 MB / 8 GB, `onboot=1`, backed up by
|
||
the existing `pbs-ana` job. Routers need `/dev/net/tun` passed in and `ip_forward` (both
|
||
routine for unprivileged CTs; net sysctls are namespaced).
|
||
|
||
Proposed ids/names (DHCP with reservation like every existing CT; recorded in
|
||
`dns/internal.yaml` as `<name>.<site>.internal`):
|
||
|
||
| PVE | next id | CT | note |
|
||
|---|---|---|---|
|
||
| nh3-pve (`root@10.100.250.60`, PVE 8.4.1) | 106 | `nh3-headscale` | control plane |
|
||
| nh3-pve | 107 | `nh3-mesh-rtr` | subnet router |
|
||
| esh-pve (`root@10.0.250.35`, PVE 8.4.20) | 108 | `esh-mesh-rtr` | subnet router |
|
||
| pfi-pve (`root@10.250.250.31`, PVE 8.3.5) | 114 | `ana-mesh-rtr` | subnet router; ana-wg (113) stays as independent WG fallback |
|
||
|
||
`debian-13-standard_13.6-1_amd64.tar.zst` is in `pveam available` on all three (not yet
|
||
downloaded on any).
|
||
|
||
## Access & credentials — all in hand (verified 2026-09-06)
|
||
|
||
| need | have |
|
||
|---|---|
|
||
| provision LXCs on all three PVEs | `infra-ops@` with NOPASSWD sudo on all four PVE hosts since 2026-09-06 (elway `bootstrap-infra-ops-user.yaml`; PVE needed `sudo` installed first); `root@` also works |
|
||
| public DNS name + DDNS | Cloudflare all-zones DNS-edit token, vault `nh3-dev/.config/cloudflare/infra-ops-dns-token` |
|
||
| NH3 UDM port-forward 443 → nh3-headscale; static routes on both UDMs | UDM API keys, vault `unifi/pfi-udmse-api-key`, `unifi/esh-udmpm-api-key` (classic `/rest/*` read+write) |
|
||
| colo static route toward ana-mesh-rtr | FortiGate infra-ops SSH pw vaulted; reachable at 10.250.0.1 via the tunnel (`execute backup config` first). Moot once OPNsense lands |
|
||
| TLS for headscale | Let's Encrypt via Cloudflare DNS-01 (same token) — no inbound 80 needed |
|
||
| secrets (pre-auth keys, API key, DB) | vault under `nh3-headscale/…` |
|
||
|
||
Nothing outstanding on credentials. Inputs still needed from the operator: confirm names/ids
|
||
above; 443 direct on the UDM vs behind the existing Caddy on nh3-dev; ACL posture (flat
|
||
"everything can reach everything" first, tighten later, is the recommendation).
|
||
|
||
## Phases
|
||
|
||
0. **Pre-flight (no changes):** confirm NH3 v6 /64 address for the AAAA; confirm DHCP
|
||
reservation ranges on the three sites; `pveam download` Debian 13 on all three.
|
||
1. **Control plane:** CT `nh3-headscale`; headscale in a container or the .deb (prefer .deb —
|
||
fewer layers in a 512M CT); Caddy/own TLS via DNS-01; UDM forward 443; DDNS updater
|
||
(UDM → Cloudflare, or a `ddclient`/cron in the CT); DERP = Tailscale public map
|
||
initially; headplane UI optional. Backup: `/var/lib/headscale/db.sqlite` is the only
|
||
state — PBS covers the CT; add a nightly `sqlite3 .backup` to `backupStore` too.
|
||
2. **First nodes:** nh3-dev + the operator's laptop/iPad enrol → prove MagicDNS, prove
|
||
TCP-443 relay path from a UDP-blocked network.
|
||
3. **Subnet routers:** three router CTs, `--advertise-routes` per site, approve routes in
|
||
headscale, SNAT off (`--snat-subnet-routes=false`) so source IPs survive, static routes
|
||
on each site gateway pointing the other two /16s at the local router. Test
|
||
site-to-site from clientless hosts.
|
||
4. **Cut over:** move day-to-day traffic onto the mesh; **disable (not delete)** Site Magic
|
||
and the two IPsec tunnels. Running all three at once makes route precedence murky on
|
||
the UDM and produces asymmetric paths. ana-wg stays as the out-of-band WG fallback.
|
||
5. **Later:** relay (DERP) at ESH when the 2G circuit is in; OPNsense as the colo router
|
||
node; HA router pairs (Tailscale HA subnet routers) per site; tighten ACLs.
|
||
|
||
## Risks / open questions
|
||
|
||
- Control-plane outage does not drop existing tunnels but stalls new logins and key
|
||
rotation — hence the SQLite backup and the "not at the colo" rule.
|
||
- Tailscale public DERPs carry relayed (encrypted) traffic through third-party infra until
|
||
a self-hosted DERP exists. Acceptable at first; fix in phase 5.
|
||
- CGNAT at ESH: outbound-only is fine; nothing at ESH needs to be dialled inbound.
|
||
- Headscale lags Tailscale feature-wise (tailnet lock, some ACL syntax). Not relevant to
|
||
phases 0–4.
|
||
- VM 106 on pfi-pve was once named "PFI-Tailscale" (pre-2026 inventory). No config
|
||
survives; nothing to reuse.
|
||
|
||
## Status — 2026-09-06 05:35Z: phases 0–1 done, phase 3 half-done, NOT cut over
|
||
|
||
Operator inputs (05:00Z): 443 direct on the UDM; names `nh3-headscale`, `nh3-scale`,
|
||
`esh-scale`, `ana-scale`; ids my call → sequential. ACLs: flat (database policy mode,
|
||
empty policy) until told otherwise.
|
||
|
||
**Live:**
|
||
|
||
| what | where | detail |
|
||
|---|---|---|
|
||
| control plane | CT 106 `nh3-headscale` 10.100.50.45 (nh3-pve, Debian 12, 1c/512M/8G) | headscale **v0.29.3** (.deb), `https://headscale.phasefinal.com` on **443 direct**, Let's Encrypt via TLS-ALPN-01 (no Caddy, no DNS-01), SQLite at `/var/lib/headscale/db.sqlite`, DERP = Tailscale public map, MagicDNS base `mesh.phasefinal.com`, split DNS `internal` → the three AdGuards, `override_local_dns: false`. Public health check 200 with a valid cert. |
|
||
| public name | Cloudflare A `headscale.phasefinal.com` → NH3 WAN (70.230.226.88 at creation), unproxied, TTL 120 | **DDNS:** `~/.local/bin/headscale-ddns.sh` on nh3-dev, user timer `headscale-ddns.timer` every 10 min, token pulled from the vault at run time (nothing stored in the CT). No AAAA yet — NH3 LAN v6 is deliberately off. |
|
||
| UDM forward | NH3 UDM SE `headscale-443`: WAN tcp/443 → 10.100.50.45:443 | added via the classic REST API. Hairpin from inside NH3 works. |
|
||
| routers | CT 107 `nh3-scale` 10.100.50.46 (100.64.0.1) · CT 108 `esh-scale` 10.0.50.65 (100.64.0.2, Debian 13) · CT 114 `ana-scale` 10.250.50.45 (100.64.0.3) | tailscale 1.102.3, `/dev/net/tun` passed, forwarding on, each advertising its site /16, **routes approved and serving**, `--snat-subnet-routes=false`, `--accept-routes=false` (see lesson 1). |
|
||
| first client | nh3-dev (100.64.0.4) | `--accept-routes=false`, `--accept-dns=false`. Direct paths to all three routers (7–10 ms). |
|
||
| DNS | `*.internal` names for all four CTs synced to the three AdGuards | |
|
||
| backups | pfi-pve + nh3-pve jobs are `all`; esh-pve job vmid list extended with 108 | |
|
||
| vault | `headscale/preauth-router-48h-20260906`, `headscale/preauth-client-48h-20260906` (reusable, expire 2026-09-08) | mint fresh ones on the CT: `headscale preauthkeys create --user 1 --reusable --expiration 48h` |
|
||
|
||
**Lessons (do not repeat):**
|
||
|
||
1. ⚠ **`--accept-routes` on a client BEFORE the return path exists breaks that client's
|
||
LAN.** nh3-dev accepted all three /16s including its OWN site's; Linux table-52 routes
|
||
beat the main table, so every 10.100.x packet went into the tunnel to nh3-scale and
|
||
nothing came back (no SNAT → the far host has no route to 100.64/10). Took ssh to
|
||
nh3-pve, esh-pve and pfi-pve down for ~1 min until `tailscale set --accept-routes=false`.
|
||
Phase order is therefore: **return path first** (either SNAT on the routers, or static
|
||
routes for `100.64.0.0/10` → local router on every site gateway), *then* accept-routes on
|
||
clients, and exclude a node's own site route.
|
||
2. The routers currently find each other "direct" via LAN addresses (`10.0.50.65:41641`
|
||
etc.) — i.e. **the mesh is riding inside the old Site Magic / IPsec tunnels**. Public-
|
||
endpoint / relay path discovery is untested until those tunnels are disabled (phase 4).
|
||
Test that with one tunnel down before declaring cut-over ready.
|
||
3. pfi-pve (PVE 8.3.5) and nh3-pve (8.4.1) refuse the Debian 13 template
|
||
(`unsupported debian version '13.6'`); esh-pve (8.4.20) accepts it. Three CTs are
|
||
Debian 12, esh-scale is Debian 13. A failed `pct create` followed by an `>>` into the
|
||
conf leaves a stub conf that makes `pct` say "missing arch" — delete the stub. Follow-up:
|
||
bring pfi-pve/nh3-pve up to current 8.4.
|
||
4. headscale 0.29 removed `randomize_client_port` and renamed
|
||
`ephemeral_node_inactivity_timeout` → `node.ephemeral.inactivity_timeout`; with a bad
|
||
key `headscale serve` silently falls back to defaults (127.0.0.1:8080). Always
|
||
`headscale configtest` first.
|
||
|
||
**Next (in order):** (a) decide the return path — recommendation: static routes for
|
||
`100.64.0.0/10` on each site gateway → local router (keeps source IPs), SNAT as the fallback;
|
||
(b) enrol the operator's devices with the client key; (c) test a UDP-blocked path (relay over
|
||
443); (d) disable ONE old tunnel and re-test router paths; (e) cut over: static routes for the
|
||
other two /16s on each gateway, disable Site Magic + IPsec; (f) v6 AAAA once NH3 LAN v6 is
|
||
on; (g) self-hosted DERP at ESH; (h) PVE upgrades on pfi-pve/nh3-pve.
|
||
|
||
### 06:05Z — return path built and verified (operator approved 05:50Z)
|
||
|
||
Static route `100.64.0.0/10` → local router on all three gateways: NH3 UDM
|
||
(`mesh-100.64-via-nh3-scale` → 10.100.50.46, classic REST), ESH UDM
|
||
(`mesh-100.64-via-esh-scale` → 10.0.50.65; ⚠ that vault item is a 19-line note — extract with
|
||
`secret get unifi/esh-udmpm-api-key | grep '^key:' | awk '{print $2}'`, a raw pipe gives nginx
|
||
400 and the NH3 key gives 401), FortiGate (`config router static`, dst 100.64.0.0/255.192.0.0,
|
||
gw 10.250.50.45, device `servers`; config backed up to flash as `pre-mesh-route-20260906`
|
||
first). Routers re-set to `--accept-routes=true` — a router MUST accept the other sites' routes
|
||
or its replies to remote-LAN sources leave via its default gateway with a 100.64 source and die;
|
||
a node never accepts its own advertised route, so the nh3-dev failure mode does not apply to
|
||
routers.
|
||
|
||
Verified (LAN host → gateway → local router → mesh → far router, and back): nh3-docker→ana-scale
|
||
9 ms, esh-docker-vm→ana-scale and →nh3-scale 0% loss, ana-docker→nh3-scale and →esh-scale 0%
|
||
loss. All three sites can reach mesh addresses from plain LAN hosts with no client installed.
|
||
|
||
**LAN-resident clients (nh3-dev) keep `--accept-routes=false`:** they reach other sites via
|
||
their gateway like any LAN host. `--accept-routes` is for roaming devices (laptop/iPad off-site)
|
||
only. Consequence: a LAN-resident client is reachable by its mesh IP only from the mesh, not
|
||
from other sites' LANs (ana-docker→100.64.0.4 fails by design; →100.64.0.1 works).
|
||
|
||
**Enrolling GUI devices (Mac/iPad) without OIDC:** the apps do not take a pre-auth key. Point
|
||
the app at `https://headscale.phasefinal.com`, it opens a registration page showing a
|
||
`headscale nodes register --user pfi --key mkey:…` command; run that on nh3-headscale
|
||
(`pct exec 106 -- headscale nodes register --user 1 --key <mkey>`). CLI Linux/macOS builds can
|
||
use the vaulted client key instead.
|
||
|
||
### 06:20Z — first roaming client verified
|
||
|
||
Operator's MacBook enrolled via the GUI (Option-click → Debug → Custom Login Server; headscale
|
||
0.29 registration command is `headscale auth register --auth-id hskey-authreq-… --user pfi`),
|
||
node 5 `vhlk-mba26` (100.64.0.5), "Use Tailscale subnets" on. From ESH: `ping 100.64.0.3`
|
||
and `ssh infra-ops@ana-docker.ana.internal` both work → colo subnet route + split DNS for
|
||
`*.internal` proven from a client. (Path was still via the ESH LAN / old tunnels.)
|
||
|
||
### Note — the mesh is also the durable fix for CGNAT CrowdSec false-bans
|
||
|
||
2026-09-06: CrowdSec on ana-docker banned ESH's shared CGNAT egress (23.164.40.160,
|
||
`custom/gitea-aggressive-crawl`) and the fortigate-mirror bouncer pushed it to the colo
|
||
edge, blackholing Matrix/gitea/chat for the whole ESH site (see
|
||
`~/.claude/.../memory/incident_crowdsec_cgnat_false_ban.md`). Once ESH consumes colo
|
||
services over the mesh (100.64/10 via ana-scale) rather than the public FortiGate VIP,
|
||
that traffic never reaches CrowdSec — a concrete win beyond replacing the tunnels, worth
|
||
weighing when prioritising the cut-over.
|
||
|
||
### 06:35Z — ESH egress whitelisted in CrowdSec (operator-directed, temporary)
|
||
|
||
`/opt/docker/conf/crowdsec/postoverflows/s01-whitelist/pfi-esh-egress.yaml` (bind mount,
|
||
persists) whitelists 23.164.40.160; SIGHUP reload, `crowdsec -t` clean, parser loaded.
|
||
TEMPORARY — remove when ESH gets its static IP. crowdsec stack is NOT in stacks/ canonical
|
||
(only stacks-mirror), so this lives on the host; re-mirror with sync-stacks.sh.
|
||
|
||
## CUTOVER EXECUTED 2026-09-06 — IPsec dormant, mesh primary; Site Magic pending operator
|
||
|
||
Operator goal: replace Site Magic + IPsec with headscale, tunnels dormant as backup.
|
||
|
||
**Safety net first:** FortiGate WAN SSH enabled — `wan1 set allowaccess ping ssh`, admin
|
||
`infra-ops` trusthost2/3 = 70.230.226.88/32 (NH3 egress) + 23.164.40.160/32 (ESH egress);
|
||
trusthost1 10.0.0.0/8 kept. Config backed up to flash `pre-wan-ssh-cutover-20260906`.
|
||
Verified: `ssh infra-ops@38.120.12.42` works from nh3-dev, tunnel-independent. ⚠ TEMPORARY —
|
||
scoped to two source IPs, not 0.0.0.0; remove after the edge is retired. Drive the FortiGate
|
||
via WAN SSH (`scratchpad/fgw.py`) during/after cutover, not over the tunnel.
|
||
|
||
**Method (learned the hard way):** the tunnel and a mesh static route for the same /16 on the
|
||
same gateway FIGHT — forward/return split across transports, asymmetric traffic dropped
|
||
(broke colo from NH3 at 06:xx while both were up; rolled back). Correct order per pair:
|
||
disable the tunnel end(s) FIRST (route withdraws), THEN add the mesh /16 route (now
|
||
unambiguous). Never do it with the tunnel up.
|
||
|
||
**colo↔NH3 (IPsec):** NH3 UDM `pfi-nh3-ana` disabled (enabled=false, API); mesh routes
|
||
NH3 `10.250.0.0/16→10.100.50.46 (nh3-scale)`, FortiGate `10.100.0.0/16→10.250.50.45
|
||
(ana-scale)`. Verified: nh3-dev→ana-docker + pfi-pve OPEN; FortiGate reachable via mesh AND
|
||
WAN SSH; ESH→rest-server-ana (restic) OPEN. FortiGate phase1 left enabled (no peer → idle);
|
||
re-enable NH3 side to restore.
|
||
|
||
**colo↔ESH (IPsec):** ESH UDM `esh-ana` disabled; mesh routes ESH
|
||
`10.250.0.0/16→10.0.50.65 (esh-scale)`, FortiGate `10.0.0.0/16→10.250.50.45`. Verified
|
||
bidirectional: esh-docker-vm↔ana-docker/pfi-pve OPEN both ways. Proves ESH (CGNAT) mesh over
|
||
public/DERP.
|
||
|
||
**NH3↔ESH (Site Magic) — NOT done by API.** `sdwan-mesh-tunnel` objects (`ESH-UDMPM` on NH3,
|
||
`PFI-UDMSE` on ESH) return `api.err.NoEdit` — Site Magic is cloud-orchestrated, not togglable
|
||
via the classic REST API. Mesh routes PRE-STAGED (NH3 `10.0.0.0/16→nh3-scale`, ESH
|
||
`10.100.0.0/16→esh-scale`), currently SHADOWED by Site Magic (traffic still on the SD-WAN
|
||
tunnel, traceroute via 192.168.1.x transit). esh-scale↔nh3-scale hold a DERP(lax) 9ms path
|
||
independent of Site Magic → when the operator disables Site Magic in the UniFi UI / Site
|
||
Manager, the staged routes take over immediately. **Operator to toggle; mesh verified ready.**
|
||
|
||
**Dormancy = disabled-but-retained**, not auto-failover (overlapping /16s across two
|
||
transports reintroduce the asymmetry). Re-enable a pair = flip the UDM object back to
|
||
enabled=true (+ re-enable Site Magic in UI). No automatic mesh→tunnel failover is wired.
|
||
|
||
**Bonus (operator noted):** exit nodes at each site give a free multi-location egress proxy
|
||
(`tailscale up --advertise-exit-node` + approve); replaces the nh3-dev SOCKS5 residential
|
||
egress with a native feature. Not yet configured — parked follow-up.
|
||
|
||
**Follow-ups:** operator disables Site Magic; (optional) disable FortiGate phase1 ends for
|
||
tidiness; advertise exit nodes; self-hosted DERP at ESH on the 2G circuit; remove the
|
||
FortiGate WAN-SSH trusthosts when the edge is retired; PVE 8.4 upgrade on pfi-pve/nh3-pve.
|
||
|
||
## ✅ CUTOVER COMPLETE 2026-09-06 — all three site-pairs on the mesh
|
||
|
||
Operator disabled Site Magic in the UniFi UI. NH3↔ESH re-homed to the mesh immediately —
|
||
traceroute now esh-scale(10.0.50.65)→nh3-scale(100.64.0.1)→dest, no 192.168.1.x SD-WAN
|
||
transit; esh-scale↔nh3-scale hole-punched a DIRECT public path (70.230.226.88:41641, 8ms),
|
||
not even DERP. Full matrix verified, all six directions OPEN:
|
||
NH3↔colo, NH3↔ESH, colo↔ESH. FortiGate mgmt via WAN SSH OPEN; ana-wg WG fallback untouched.
|
||
|
||
State: **Site Magic disabled (UI); both IPsec tunnels dormant (UDM ends disabled, config
|
||
retained); headscale is the sole active site-to-site transport.** Tunnels re-enablable for
|
||
backup (Site Magic in UI; IPsec by flipping the UDM objects enabled=true). Goal met.
|
||
|
||
## 2026-09-06 — exit nodes advertised (multi-location egress proxy)
|
||
|
||
All three routers advertise + serve exit nodes; approved on headscale (each shows
|
||
`0.0.0.0/0` + `::/0` Serving). Clients see three: `nh3-scale` (100.64.0.1, NH3 residential
|
||
egress 70.230.226.88), `esh-scale` (100.64.0.2, ESH 23.164.40.160), `ana-scale`
|
||
(100.64.0.3, colo 38.120.12.42).
|
||
|
||
⚠ **Design reversal:** exit nodes REQUIRE SNAT, so `--snat-subnet-routes` was flipped back to
|
||
**true** (NoSNAT=false) on all three routers — this undoes the earlier source-preserving
|
||
choice. Consequence: cross-site subnet-route traffic now appears to come from the local
|
||
router's IP, not the original host. Site-to-site re-verified OK after the flip. If per-host
|
||
source visibility is later needed, the alternative is NoSNAT + a manual selective masquerade
|
||
rule (`-s 100.64.0.0/10 ! -d 10.0.0.0/8 -o eth0 -j MASQUERADE`) on each router, made
|
||
persistent — not done.
|
||
|
||
**Use one from a client:** `tailscale set --exit-node=nh3-scale` (or `esh-scale` /
|
||
`ana-scale`), `--exit-node=` to clear. A node that itself advertises an exit node cannot
|
||
consume one (so the routers can't be used to test egress; use the laptop/iPad).
|
||
Replaces the nh3-dev SOCKS5 residential-egress proxy — retire that once confirmed on real
|
||
clients.
|