vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(irv-ml1): tailscaled could never add its IPv6 mesh address, and the README described a topology two cutovers old

Browse Source 
TAILSCALE IPv6. `tailscale status` had been reporting, continuously:
    2 add route failures; first was: permission denied
    adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
with tailscale0 carrying only 100.64.0.6/32 while headscale had assigned it
an IPv6 address it could not use.

Not a capability problem -- tailscaled runs as root with the full bounding
set. /etc/sysctl.conf:59 sets net.ipv6.conf.default.disable_ipv6=1, and
`default` is inherited by NEWLY CREATED interfaces; tailscale0 is created at
daemon start, inherits it, and the kernel returns EPERM for every attempt.

Fixed with a scoped systemd drop-in rather than flipping the global default.
That line carries no comment, but IPv6-off-by-default on a host with ~26
docker bridges reads as deliberate, and changing it would hand IPv6 to every
future bridge as a side effect of fixing Tailscale.

⚠ It must be ExecStartPost, not /etc/sysctl.d. A sysctl.d entry for a
per-interface key is applied at boot, BEFORE tailscale0 exists, and is
silently ignored -- the setting would look present and do nothing.

Also learned: setting the sysctl on the LIVE interface is not enough.
tailscaled only attempts the address at startup or on a netmap change, so
the verify failed for 60s until the daemon was restarted. Restart is part
of the operation, not an afterthought.

Verified: fd7a:115c:a1e0::6/128 present on tailscale0, health clean, mesh
and services (arbo, ytvc) up.

README. It documented the pre-headscale topology as current -- "Reachable
IP: 10.100.79.3 (WireGuard tunnel endpoint)", "No direct LAN access", and a
refresh caveat telling you to bring WG up. That sends anyone triaging this
host to the wrong layer, which is the exact tax the file exists to prevent.
Now: mesh primary at 100.64.0.6, LAN 10.6.110.50, and wg0 documented as
STILL UP with a live peer -- tailscale uses that address as its direct
endpoint, so it is load-bearing, not vestigial.

Recorded with it, because these cost hours tonight and will cost them again:
  - Irvine is a TENANCY behind a Fortinet PFI does not control. Its TLS
    inspection breaks Tailscale's relay and control channels (41 cert
    warnings/week, 4 control-plane episodes in 14 days). Usually invisible
    because direct peer paths carry the data. No fix on our side.
  - Diagnose reachability with `tailscale ping`, NOT the status output:
    headscale said "online" and status said "active, 19.7 GB" while nothing
    on the host answered. Both are last-known state; only a round trip is
    liveness.
  - The ~26 docker bridges make tailscaled report captive portals.

Two stale claims corrected: the hostname rename it called "pending" is done,
and `ollama` is listed as running on :11434 when it is gone -- verified, no
unit file, nothing listening, no process. It is banned fleet-wide.
This commit is contained in:
Vuong Hoang
2026-09-22 08:21:10 -07:00
parent 94899d6fa3
commit 38bb20ceda
3 changed files with 111 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
playbooks/irv-ml1-tailscale-ipv6.yaml
+48
View File
@@ -0,0 +1,48 @@
# irv-ml1: let tailscaled add its own IPv6 mesh address.
#
# THE FAULT. `tailscale status` has been reporting, continuously:
# 2 add route failures; first was: permission denied
# adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
# and `tailscale0` carries only 100.64.0.6/32 — no IPv6 — while headscale has
# assigned it fd7a:115c:a1e0::6.
#
# It is NOT a capability problem: tailscaled runs as root with the full
# bounding set including cap_net_admin. It is the kernel returning EPERM
# because /etc/sysctl.conf:59 sets
# net.ipv6.conf.default.disable_ipv6=1
# and `default` is inherited by NEWLY CREATED interfaces. tailscale0 is created
# at daemon start, inherits disable_ipv6=1, and every attempt to add the
# address is refused.
#
# WHY NOT JUST FLIP THE DEFAULT. That line carries no comment, but the shape of
# it — IPv6 off for new interfaces on a Docker host with many bridges — reads as
# deliberate. Changing it would hand IPv6 to every future docker bridge as a
# side effect of fixing Tailscale. Scope the exception instead.
#
# WHY A DROP-IN AND NOT /etc/sysctl.d. A sysctl.d entry for a per-interface key
# is applied at boot, BEFORE tailscale0 exists, and is silently ignored — the
# setting would look present and do nothing. ExecStartPost runs after the
# interface is created, which is the only moment the key can be set.
steps:
- name: Install the tailscaled drop-in that re-enables IPv6 on tailscale0
sudo: true
upload:
src: services/irv-ml1/tailscaled-ipv6.conf
dest: /etc/systemd/system/tailscaled.service.d/10-tailscale0-ipv6.conf
mode: '0644'
- name: Reload systemd so the drop-in is live for the next start
sudo: true
shell: systemctl daemon-reload
- name: Apply it to the RUNNING interface too, so no restart is needed
sudo: true
shell: sysctl -qw net.ipv6.conf.tailscale0.disable_ipv6=0
changed_when: 'true'
verify:
- name: tailscale0 has its IPv6 mesh address and the health error is gone
shell: >-
for i in $(seq 1 20); do
ip -6 addr show tailscale0 2>/dev/null | grep -q "fd7a:115c:a1e0" && break || sleep 3;
done;
ip -6 addr show tailscale0 | grep -q "fd7a:115c:a1e0" &&
echo "IPv6 present:" && ip -brief addr show tailscale0
changed_when: 'false'
servers/irv-ml1/README.md
+53 -24
View File
@@ -2,33 +2,59 @@
Secondary AI/ML inference host at the Irvine site. Formerly known as
`ana-ml1` when colocated at Anaheim; moved to Irvine and slated for
hostname rename to `irv-ml1` (OS-side rename pending — see below).
hostname rename to `irv-ml1` — **done**; `hostnamectl` reports `irv-ml1`
(verified 2026-09-22).
## Network
- **Reachable IP:** `10.100.79.3` (WireGuard tunnel endpoint)
- **No direct LAN access** — this host is reachable **only** via
WireGuard. Tunnel terminates at the NH3 site (10.100.0.0/16 WG
subnet). If WG is down, `scripts/refresh-server-info.sh irv-ml1`
will fail with "No route to host" — that's a WG issue, not a host
issue.
- **SSH:** `ssh irv-ml1` (config alias → `lkraven@10.100.79.3`,
key auth).
⚠ **This section was wrong until 2026-09-22** — it described the
pre-headscale topology as current, which sends anyone triaging this host to
the wrong layer. Verified against the host:
## Pending hostname rename
- **Primary path: the headscale mesh** — `100.64.0.6` (plus
`fd7a:115c:a1e0::6`). This is how the fleet reaches it.
- **Irvine LAN:** `10.6.110.50/24` on `enp68s0f0`, routed via the mesh.
- **`wg0` lifeline:** `10.100.79.3/32` — **still up with a live peer**, not
dormant. Tailscale currently uses this address as its endpoint for the
direct path to nh3-scale, so it is load-bearing rather than vestigial.
- **SSH:** `ssh infra-ops@100.64.0.6` (NOPASSWD sudo).
OS hostname still reports `ana-ml1` (both in `hostnamectl` and in
`system-details.txt`). To finish the rename:
### Known network behaviour — it lives behind someone else's firewall
```bash
ssh -t irv-ml1 'sudo hostnamectl set-hostname irv-ml1; \
sudo sed -i "s/ana-ml1/irv-ml1/g" /etc/hosts; \
cat /etc/hosts; hostname'
```
**Irvine is a tenancy: this host sits behind a Fortinet PFI does not
control** (operator, 2026-09-22). Two consequences to recognise rather than
re-diagnose:
Then refresh the inventory snapshot so it reflects the new identity.
Not blocking anything — services don't care about the kernel's idea
of hostname.
- **TLS interception breaks Tailscale's relay and control channels
intermittently.** tailscaled logs `server cert for "derpNN.tailscale.com"
looks like "Fortinet" equipment` — 41 in one week — and the headscale
control connection times out periodically (4 episodes in 14 days).
**Usually invisible**, because direct peer paths keep carrying data
(netcheck: UDP true, `MappingVariesByDestIP: false`, an easy NAT). On
2026-09-22 03:13 it went all the way to unreachable for ~5 minutes and
paged — the only such page in 14 days. **There is no fix on our side of
the boundary**; do not go looking for one.
- **Captive-portal false positives on Docker bridges.** ~26 `br-*`
interfaces here, and tailscaled probes them all:
`DetectCaptivePortal(found=true, ifName=br-…)`. Noise, not a fault.
⚠ **Diagnose reachability with `tailscale ping`, not the status output.**
On 2026-09-22 headscale reported the node `online` and `tailscale status`
showed the peer `active` with 19.7 GB transferred, while nothing on the host
answered at all. Both are last-known state; only a round trip is liveness.
### IPv6 on tailscale0
`/etc/sysctl.conf` sets `net.ipv6.conf.default.disable_ipv6=1`, which newly
created interfaces inherit — so `tailscale0` came up IPv6-disabled and
tailscaled could never add `fd7a:115c:a1e0::6/128` (logged as *permission
denied*, though it runs as root; the EPERM is the kernel refusing a disabled
interface). Fixed 2026-09-22 by a scoped systemd drop-in,
`services/irv-ml1/tailscaled-ipv6.conf`, which sets the per-interface sysctl
in `ExecStartPost` — **after** the interface exists. A `/etc/sysctl.d` entry
would run at boot, before `tailscale0` is created, and be silently ignored.
The global default is deliberately untouched so Docker bridges keep their
current behaviour.
## Hardware
@@ -55,7 +81,9 @@ generative-AI stack:
- ComfyUI, SillyTavern, SDNext, fluxgym (image gen / SD)
- alltalk, alltalkv2, bark, kokoro, Orpheus-FastAPI, stablediffusion (TTS + voice)
- llama.cpp, llama-swap, koboldcpp, aphrodite (LLM inference)
- ollama (port 11434 listening on all interfaces) — native binary, not the container
- (⚠ `ollama` was listed here as running on `:11434`. It is **gone** —
verified 2026-09-22: no unit file, nothing listening, no process. Ollama
is banned fleet-wide; if it reappears, tear it down.)
- ai-toolkit, chat-ui, h2ogpt, o-textgen, lollms, bitsandbytes (misc ML frameworks)
- sillytavern-extras, simple-proxy-for-tavern (lkraven-owned)
@@ -68,7 +96,7 @@ need it.
| Stack | Port | GPU | Role |
|-------|------|-----|------|
| dockge | 5001 | — | Per-host Compose UI |
| beszel-agent-irv | 45876 | — | Metrics agent → Beszel hub on ana-docker (token mode through WG) |
| beszel-agent-irv | 45876 | — | Metrics agent → Beszel hub on ana-docker (token mode, over the mesh) |
| dozzle-agent-irv | 7007 | — | Log agent → Dozzle hub on ana-docker |
| comfyui | 8188 | all | ComfyUI (node-based SD/Flux) — runs independently of `/opt/ComfyUI` native install |
| kokoro | 8193 | all | Kokoro 82M TTS — lowest-latency English TTS, OpenAI-compatible /v1/audio/speech |
@@ -119,5 +147,6 @@ regenerable).
scripts/refresh-server-info.sh irv-ml1
```
**Caveat:** requires the WG tunnel to be up. If the refresh shows
"No route to host", bring WG up before retrying.
**Caveat:** depends on the headscale mesh, not `wg0`. If the refresh fails,
check with `tailscale ping 100.64.0.6` from a mesh node before assuming the
host is down — see Network on why the status output can claim it is fine.
services/irv-ml1/tailscaled-ipv6.conf
+10
View File
@@ -0,0 +1,10 @@
# tailscale0 inherits net.ipv6.conf.default.disable_ipv6=1 (/etc/sysctl.conf)
# at creation, so tailscaled cannot add its fd7a:115c:a1e0::6/128 mesh address
# and logs "permission denied" forever. This re-enables IPv6 on that ONE
# interface, after it exists — a /etc/sysctl.d entry runs at boot, before
# tailscale0 is created, and is silently ignored.
#
# Scoped deliberately: the global default stays as-is so Docker bridges on this
# host keep their current (no-IPv6) behaviour.
[Service]
ExecStartPost=/bin/sh -c 'for i in 1 2 3 4 5 6 7 8 9 10; do if [ -e /sys/class/net/tailscale0 ]; then sysctl -qw net.ipv6.conf.tailscale0.disable_ipv6=0; exit 0; fi; sleep 1; done; exit 0'