Files
esh-pfi-infrastructure/playbooks/irv-ml1-tailscale-ipv6.yaml
T
vh 38bb20ceda fix(irv-ml1): tailscaled could never add its IPv6 mesh address, and the README described a topology two cutovers old
TAILSCALE IPv6. `tailscale status` had been reporting, continuously:
    2 add route failures; first was: permission denied
    adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
with tailscale0 carrying only 100.64.0.6/32 while headscale had assigned it
an IPv6 address it could not use.

Not a capability problem -- tailscaled runs as root with the full bounding
set. /etc/sysctl.conf:59 sets net.ipv6.conf.default.disable_ipv6=1, and
`default` is inherited by NEWLY CREATED interfaces; tailscale0 is created at
daemon start, inherits it, and the kernel returns EPERM for every attempt.

Fixed with a scoped systemd drop-in rather than flipping the global default.
That line carries no comment, but IPv6-off-by-default on a host with ~26
docker bridges reads as deliberate, and changing it would hand IPv6 to every
future bridge as a side effect of fixing Tailscale.

⚠ It must be ExecStartPost, not /etc/sysctl.d. A sysctl.d entry for a
per-interface key is applied at boot, BEFORE tailscale0 exists, and is
silently ignored -- the setting would look present and do nothing.

Also learned: setting the sysctl on the LIVE interface is not enough.
tailscaled only attempts the address at startup or on a netmap change, so
the verify failed for 60s until the daemon was restarted. Restart is part
of the operation, not an afterthought.

Verified: fd7a:115c:a1e0::6/128 present on tailscale0, health clean, mesh
and services (arbo, ytvc) up.

README. It documented the pre-headscale topology as current -- "Reachable
IP: 10.100.79.3 (WireGuard tunnel endpoint)", "No direct LAN access", and a
refresh caveat telling you to bring WG up. That sends anyone triaging this
host to the wrong layer, which is the exact tax the file exists to prevent.
Now: mesh primary at 100.64.0.6, LAN 10.6.110.50, and wg0 documented as
STILL UP with a live peer -- tailscale uses that address as its direct
endpoint, so it is load-bearing, not vestigial.

Recorded with it, because these cost hours tonight and will cost them again:
  - Irvine is a TENANCY behind a Fortinet PFI does not control. Its TLS
    inspection breaks Tailscale's relay and control channels (41 cert
    warnings/week, 4 control-plane episodes in 14 days). Usually invisible
    because direct peer paths carry the data. No fix on our side.
  - Diagnose reachability with `tailscale ping`, NOT the status output:
    headscale said "online" and status said "active, 19.7 GB" while nothing
    on the host answered. Both are last-known state; only a round trip is
    liveness.
  - The ~26 docker bridges make tailscaled report captive portals.

Two stale claims corrected: the hostname rename it called "pending" is done,
and `ollama` is listed as running on :11434 when it is gone -- verified, no
unit file, nothing listening, no process. It is banned fleet-wide.
2026-09-22 08:21:10 -07:00

49 lines
2.3 KiB
YAML

# irv-ml1: let tailscaled add its own IPv6 mesh address.
#
# THE FAULT. `tailscale status` has been reporting, continuously:
# 2 add route failures; first was: permission denied
# adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
# and `tailscale0` carries only 100.64.0.6/32 — no IPv6 — while headscale has
# assigned it fd7a:115c:a1e0::6.
#
# It is NOT a capability problem: tailscaled runs as root with the full
# bounding set including cap_net_admin. It is the kernel returning EPERM
# because /etc/sysctl.conf:59 sets
# net.ipv6.conf.default.disable_ipv6=1
# and `default` is inherited by NEWLY CREATED interfaces. tailscale0 is created
# at daemon start, inherits disable_ipv6=1, and every attempt to add the
# address is refused.
#
# WHY NOT JUST FLIP THE DEFAULT. That line carries no comment, but the shape of
# it — IPv6 off for new interfaces on a Docker host with many bridges — reads as
# deliberate. Changing it would hand IPv6 to every future docker bridge as a
# side effect of fixing Tailscale. Scope the exception instead.
#
# WHY A DROP-IN AND NOT /etc/sysctl.d. A sysctl.d entry for a per-interface key
# is applied at boot, BEFORE tailscale0 exists, and is silently ignored — the
# setting would look present and do nothing. ExecStartPost runs after the
# interface is created, which is the only moment the key can be set.
steps:
- name: Install the tailscaled drop-in that re-enables IPv6 on tailscale0
sudo: true
upload:
src: services/irv-ml1/tailscaled-ipv6.conf
dest: /etc/systemd/system/tailscaled.service.d/10-tailscale0-ipv6.conf
mode: '0644'
- name: Reload systemd so the drop-in is live for the next start
sudo: true
shell: systemctl daemon-reload
- name: Apply it to the RUNNING interface too, so no restart is needed
sudo: true
shell: sysctl -qw net.ipv6.conf.tailscale0.disable_ipv6=0
changed_when: 'true'
verify:
- name: tailscale0 has its IPv6 mesh address and the health error is gone
shell: >-
for i in $(seq 1 20); do
ip -6 addr show tailscale0 2>/dev/null | grep -q "fd7a:115c:a1e0" && break || sleep 3;
done;
ip -6 addr show tailscale0 | grep -q "fd7a:115c:a1e0" &&
echo "IPv6 present:" && ip -brief addr show tailscale0
changed_when: 'false'