From 38bb20cedadbe61214761bf489b71729334c3c34 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 22 Sep 2026 08:21:10 -0700 Subject: [PATCH] fix(irv-ml1): tailscaled could never add its IPv6 mesh address, and the README described a topology two cutovers old MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TAILSCALE IPv6. `tailscale status` had been reporting, continuously: 2 add route failures; first was: permission denied adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied with tailscale0 carrying only 100.64.0.6/32 while headscale had assigned it an IPv6 address it could not use. Not a capability problem -- tailscaled runs as root with the full bounding set. /etc/sysctl.conf:59 sets net.ipv6.conf.default.disable_ipv6=1, and `default` is inherited by NEWLY CREATED interfaces; tailscale0 is created at daemon start, inherits it, and the kernel returns EPERM for every attempt. Fixed with a scoped systemd drop-in rather than flipping the global default. That line carries no comment, but IPv6-off-by-default on a host with ~26 docker bridges reads as deliberate, and changing it would hand IPv6 to every future bridge as a side effect of fixing Tailscale. ⚠ It must be ExecStartPost, not /etc/sysctl.d. A sysctl.d entry for a per-interface key is applied at boot, BEFORE tailscale0 exists, and is silently ignored -- the setting would look present and do nothing. Also learned: setting the sysctl on the LIVE interface is not enough. tailscaled only attempts the address at startup or on a netmap change, so the verify failed for 60s until the daemon was restarted. Restart is part of the operation, not an afterthought. Verified: fd7a:115c:a1e0::6/128 present on tailscale0, health clean, mesh and services (arbo, ytvc) up. README. It documented the pre-headscale topology as current -- "Reachable IP: 10.100.79.3 (WireGuard tunnel endpoint)", "No direct LAN access", and a refresh caveat telling you to bring WG up. That sends anyone triaging this host to the wrong layer, which is the exact tax the file exists to prevent. Now: mesh primary at 100.64.0.6, LAN 10.6.110.50, and wg0 documented as STILL UP with a live peer -- tailscale uses that address as its direct endpoint, so it is load-bearing, not vestigial. Recorded with it, because these cost hours tonight and will cost them again: - Irvine is a TENANCY behind a Fortinet PFI does not control. Its TLS inspection breaks Tailscale's relay and control channels (41 cert warnings/week, 4 control-plane episodes in 14 days). Usually invisible because direct peer paths carry the data. No fix on our side. - Diagnose reachability with `tailscale ping`, NOT the status output: headscale said "online" and status said "active, 19.7 GB" while nothing on the host answered. Both are last-known state; only a round trip is liveness. - The ~26 docker bridges make tailscaled report captive portals. Two stale claims corrected: the hostname rename it called "pending" is done, and `ollama` is listed as running on :11434 when it is gone -- verified, no unit file, nothing listening, no process. It is banned fleet-wide. --- playbooks/irv-ml1-tailscale-ipv6.yaml | 48 +++++++++++++++++ servers/irv-ml1/README.md | 77 ++++++++++++++++++--------- services/irv-ml1/tailscaled-ipv6.conf | 10 ++++ 3 files changed, 111 insertions(+), 24 deletions(-) create mode 100644 playbooks/irv-ml1-tailscale-ipv6.yaml create mode 100644 services/irv-ml1/tailscaled-ipv6.conf diff --git a/playbooks/irv-ml1-tailscale-ipv6.yaml b/playbooks/irv-ml1-tailscale-ipv6.yaml new file mode 100644 index 0000000..7ed872f --- /dev/null +++ b/playbooks/irv-ml1-tailscale-ipv6.yaml @@ -0,0 +1,48 @@ +# irv-ml1: let tailscaled add its own IPv6 mesh address. +# +# THE FAULT. `tailscale status` has been reporting, continuously: +# 2 add route failures; first was: permission denied +# adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied +# and `tailscale0` carries only 100.64.0.6/32 — no IPv6 — while headscale has +# assigned it fd7a:115c:a1e0::6. +# +# It is NOT a capability problem: tailscaled runs as root with the full +# bounding set including cap_net_admin. It is the kernel returning EPERM +# because /etc/sysctl.conf:59 sets +# net.ipv6.conf.default.disable_ipv6=1 +# and `default` is inherited by NEWLY CREATED interfaces. tailscale0 is created +# at daemon start, inherits disable_ipv6=1, and every attempt to add the +# address is refused. +# +# WHY NOT JUST FLIP THE DEFAULT. That line carries no comment, but the shape of +# it — IPv6 off for new interfaces on a Docker host with many bridges — reads as +# deliberate. Changing it would hand IPv6 to every future docker bridge as a +# side effect of fixing Tailscale. Scope the exception instead. +# +# WHY A DROP-IN AND NOT /etc/sysctl.d. A sysctl.d entry for a per-interface key +# is applied at boot, BEFORE tailscale0 exists, and is silently ignored — the +# setting would look present and do nothing. ExecStartPost runs after the +# interface is created, which is the only moment the key can be set. +steps: + - name: Install the tailscaled drop-in that re-enables IPv6 on tailscale0 + sudo: true + upload: + src: services/irv-ml1/tailscaled-ipv6.conf + dest: /etc/systemd/system/tailscaled.service.d/10-tailscale0-ipv6.conf + mode: '0644' + - name: Reload systemd so the drop-in is live for the next start + sudo: true + shell: systemctl daemon-reload + - name: Apply it to the RUNNING interface too, so no restart is needed + sudo: true + shell: sysctl -qw net.ipv6.conf.tailscale0.disable_ipv6=0 + changed_when: 'true' +verify: + - name: tailscale0 has its IPv6 mesh address and the health error is gone + shell: >- + for i in $(seq 1 20); do + ip -6 addr show tailscale0 2>/dev/null | grep -q "fd7a:115c:a1e0" && break || sleep 3; + done; + ip -6 addr show tailscale0 | grep -q "fd7a:115c:a1e0" && + echo "IPv6 present:" && ip -brief addr show tailscale0 + changed_when: 'false' diff --git a/servers/irv-ml1/README.md b/servers/irv-ml1/README.md index fb56604..c1624cb 100644 --- a/servers/irv-ml1/README.md +++ b/servers/irv-ml1/README.md @@ -2,33 +2,59 @@ Secondary AI/ML inference host at the Irvine site. Formerly known as `ana-ml1` when colocated at Anaheim; moved to Irvine and slated for -hostname rename to `irv-ml1` (OS-side rename pending — see below). +hostname rename to `irv-ml1` — **done**; `hostnamectl` reports `irv-ml1` +(verified 2026-09-22). ## Network -- **Reachable IP:** `10.100.79.3` (WireGuard tunnel endpoint) -- **No direct LAN access** — this host is reachable **only** via - WireGuard. Tunnel terminates at the NH3 site (10.100.0.0/16 WG - subnet). If WG is down, `scripts/refresh-server-info.sh irv-ml1` - will fail with "No route to host" — that's a WG issue, not a host - issue. -- **SSH:** `ssh irv-ml1` (config alias → `lkraven@10.100.79.3`, - key auth). +⚠ **This section was wrong until 2026-09-22** — it described the +pre-headscale topology as current, which sends anyone triaging this host to +the wrong layer. Verified against the host: -## Pending hostname rename +- **Primary path: the headscale mesh** — `100.64.0.6` (plus + `fd7a:115c:a1e0::6`). This is how the fleet reaches it. +- **Irvine LAN:** `10.6.110.50/24` on `enp68s0f0`, routed via the mesh. +- **`wg0` lifeline:** `10.100.79.3/32` — **still up with a live peer**, not + dormant. Tailscale currently uses this address as its endpoint for the + direct path to nh3-scale, so it is load-bearing rather than vestigial. +- **SSH:** `ssh infra-ops@100.64.0.6` (NOPASSWD sudo). -OS hostname still reports `ana-ml1` (both in `hostnamectl` and in -`system-details.txt`). To finish the rename: +### Known network behaviour — it lives behind someone else's firewall -```bash -ssh -t irv-ml1 'sudo hostnamectl set-hostname irv-ml1; \ - sudo sed -i "s/ana-ml1/irv-ml1/g" /etc/hosts; \ - cat /etc/hosts; hostname' -``` +**Irvine is a tenancy: this host sits behind a Fortinet PFI does not +control** (operator, 2026-09-22). Two consequences to recognise rather than +re-diagnose: -Then refresh the inventory snapshot so it reflects the new identity. -Not blocking anything — services don't care about the kernel's idea -of hostname. +- **TLS interception breaks Tailscale's relay and control channels + intermittently.** tailscaled logs `server cert for "derpNN.tailscale.com" + looks like "Fortinet" equipment` — 41 in one week — and the headscale + control connection times out periodically (4 episodes in 14 days). + **Usually invisible**, because direct peer paths keep carrying data + (netcheck: UDP true, `MappingVariesByDestIP: false`, an easy NAT). On + 2026-09-22 03:13 it went all the way to unreachable for ~5 minutes and + paged — the only such page in 14 days. **There is no fix on our side of + the boundary**; do not go looking for one. +- **Captive-portal false positives on Docker bridges.** ~26 `br-*` + interfaces here, and tailscaled probes them all: + `DetectCaptivePortal(found=true, ifName=br-…)`. Noise, not a fault. + +⚠ **Diagnose reachability with `tailscale ping`, not the status output.** +On 2026-09-22 headscale reported the node `online` and `tailscale status` +showed the peer `active` with 19.7 GB transferred, while nothing on the host +answered at all. Both are last-known state; only a round trip is liveness. + +### IPv6 on tailscale0 + +`/etc/sysctl.conf` sets `net.ipv6.conf.default.disable_ipv6=1`, which newly +created interfaces inherit — so `tailscale0` came up IPv6-disabled and +tailscaled could never add `fd7a:115c:a1e0::6/128` (logged as *permission +denied*, though it runs as root; the EPERM is the kernel refusing a disabled +interface). Fixed 2026-09-22 by a scoped systemd drop-in, +`services/irv-ml1/tailscaled-ipv6.conf`, which sets the per-interface sysctl +in `ExecStartPost` — **after** the interface exists. A `/etc/sysctl.d` entry +would run at boot, before `tailscale0` is created, and be silently ignored. +The global default is deliberately untouched so Docker bridges keep their +current behaviour. ## Hardware @@ -55,7 +81,9 @@ generative-AI stack: - ComfyUI, SillyTavern, SDNext, fluxgym (image gen / SD) - alltalk, alltalkv2, bark, kokoro, Orpheus-FastAPI, stablediffusion (TTS + voice) - llama.cpp, llama-swap, koboldcpp, aphrodite (LLM inference) -- ollama (port 11434 listening on all interfaces) — native binary, not the container +- (⚠ `ollama` was listed here as running on `:11434`. It is **gone** — + verified 2026-09-22: no unit file, nothing listening, no process. Ollama + is banned fleet-wide; if it reappears, tear it down.) - ai-toolkit, chat-ui, h2ogpt, o-textgen, lollms, bitsandbytes (misc ML frameworks) - sillytavern-extras, simple-proxy-for-tavern (lkraven-owned) @@ -68,7 +96,7 @@ need it. | Stack | Port | GPU | Role | |-------|------|-----|------| | dockge | 5001 | — | Per-host Compose UI | -| beszel-agent-irv | 45876 | — | Metrics agent → Beszel hub on ana-docker (token mode through WG) | +| beszel-agent-irv | 45876 | — | Metrics agent → Beszel hub on ana-docker (token mode, over the mesh) | | dozzle-agent-irv | 7007 | — | Log agent → Dozzle hub on ana-docker | | comfyui | 8188 | all | ComfyUI (node-based SD/Flux) — runs independently of `/opt/ComfyUI` native install | | kokoro | 8193 | all | Kokoro 82M TTS — lowest-latency English TTS, OpenAI-compatible /v1/audio/speech | @@ -119,5 +147,6 @@ regenerable). scripts/refresh-server-info.sh irv-ml1 ``` -**Caveat:** requires the WG tunnel to be up. If the refresh shows -"No route to host", bring WG up before retrying. +**Caveat:** depends on the headscale mesh, not `wg0`. If the refresh fails, +check with `tailscale ping 100.64.0.6` from a mesh node before assuming the +host is down — see Network on why the status output can claim it is fine. diff --git a/services/irv-ml1/tailscaled-ipv6.conf b/services/irv-ml1/tailscaled-ipv6.conf new file mode 100644 index 0000000..884f16f --- /dev/null +++ b/services/irv-ml1/tailscaled-ipv6.conf @@ -0,0 +1,10 @@ +# tailscale0 inherits net.ipv6.conf.default.disable_ipv6=1 (/etc/sysctl.conf) +# at creation, so tailscaled cannot add its fd7a:115c:a1e0::6/128 mesh address +# and logs "permission denied" forever. This re-enables IPv6 on that ONE +# interface, after it exists — a /etc/sysctl.d entry runs at boot, before +# tailscale0 is created, and is silently ignored. +# +# Scoped deliberately: the global default stays as-is so Docker bridges on this +# host keep their current (no-IPv6) behaviour. +[Service] +ExecStartPost=/bin/sh -c 'for i in 1 2 3 4 5 6 7 8 9 10; do if [ -e /sys/class/net/tailscale0 ]; then sysctl -qw net.ipv6.conf.tailscale0.disable_ipv6=0; exit 0; fi; sleep 1; done; exit 0'