fix(irv-ml1): tailscaled could never add its IPv6 mesh address, and the README described a topology two cutovers old

TAILSCALE IPv6. `tailscale status` had been reporting, continuously:
    2 add route failures; first was: permission denied
    adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
with tailscale0 carrying only 100.64.0.6/32 while headscale had assigned it
an IPv6 address it could not use.

Not a capability problem -- tailscaled runs as root with the full bounding
set. /etc/sysctl.conf:59 sets net.ipv6.conf.default.disable_ipv6=1, and
`default` is inherited by NEWLY CREATED interfaces; tailscale0 is created at
daemon start, inherits it, and the kernel returns EPERM for every attempt.

Fixed with a scoped systemd drop-in rather than flipping the global default.
That line carries no comment, but IPv6-off-by-default on a host with ~26
docker bridges reads as deliberate, and changing it would hand IPv6 to every
future bridge as a side effect of fixing Tailscale.

⚠ It must be ExecStartPost, not /etc/sysctl.d. A sysctl.d entry for a
per-interface key is applied at boot, BEFORE tailscale0 exists, and is
silently ignored -- the setting would look present and do nothing.

Also learned: setting the sysctl on the LIVE interface is not enough.
tailscaled only attempts the address at startup or on a netmap change, so
the verify failed for 60s until the daemon was restarted. Restart is part
of the operation, not an afterthought.

Verified: fd7a:115c:a1e0::6/128 present on tailscale0, health clean, mesh
and services (arbo, ytvc) up.

README. It documented the pre-headscale topology as current -- "Reachable
IP: 10.100.79.3 (WireGuard tunnel endpoint)", "No direct LAN access", and a
refresh caveat telling you to bring WG up. That sends anyone triaging this
host to the wrong layer, which is the exact tax the file exists to prevent.
Now: mesh primary at 100.64.0.6, LAN 10.6.110.50, and wg0 documented as
STILL UP with a live peer -- tailscale uses that address as its direct
endpoint, so it is load-bearing, not vestigial.

Recorded with it, because these cost hours tonight and will cost them again:
  - Irvine is a TENANCY behind a Fortinet PFI does not control. Its TLS
    inspection breaks Tailscale's relay and control channels (41 cert
    warnings/week, 4 control-plane episodes in 14 days). Usually invisible
    because direct peer paths carry the data. No fix on our side.
  - Diagnose reachability with `tailscale ping`, NOT the status output:
    headscale said "online" and status said "active, 19.7 GB" while nothing
    on the host answered. Both are last-known state; only a round trip is
    liveness.
  - The ~26 docker bridges make tailscaled report captive portals.

Two stale claims corrected: the hostname rename it called "pending" is done,
and `ollama` is listed as running on :11434 when it is gone -- verified, no
unit file, nothing listening, no process. It is banned fleet-wide.

playbooks/irv-ml1-tailscale-ipv6.yaml

@@ -0,0 +1,48 @@
+# irv-ml1: let tailscaled add its own IPv6 mesh address.
+#
+# THE FAULT. `tailscale status` has been reporting, continuously:
+#     2 add route failures; first was: permission denied
+#     adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
+# and `tailscale0` carries only 100.64.0.6/32 — no IPv6 — while headscale has
+# assigned it fd7a:115c:a1e0::6.
+#
+# It is NOT a capability problem: tailscaled runs as root with the full
+# bounding set including cap_net_admin. It is the kernel returning EPERM
+# because /etc/sysctl.conf:59 sets
+#     net.ipv6.conf.default.disable_ipv6=1
+# and `default` is inherited by NEWLY CREATED interfaces. tailscale0 is created
+# at daemon start, inherits disable_ipv6=1, and every attempt to add the
+# address is refused.
+#
+# WHY NOT JUST FLIP THE DEFAULT. That line carries no comment, but the shape of
+# it — IPv6 off for new interfaces on a Docker host with many bridges — reads as
+# deliberate. Changing it would hand IPv6 to every future docker bridge as a
+# side effect of fixing Tailscale. Scope the exception instead.
+#
+# WHY A DROP-IN AND NOT /etc/sysctl.d. A sysctl.d entry for a per-interface key
+# is applied at boot, BEFORE tailscale0 exists, and is silently ignored — the
+# setting would look present and do nothing. ExecStartPost runs after the
+# interface is created, which is the only moment the key can be set.
+steps:
+  - name: Install the tailscaled drop-in that re-enables IPv6 on tailscale0
+    sudo: true
+    upload:
+      src: services/irv-ml1/tailscaled-ipv6.conf
+      dest: /etc/systemd/system/tailscaled.service.d/10-tailscale0-ipv6.conf
+      mode: '0644'
+  - name: Reload systemd so the drop-in is live for the next start
+    sudo: true
+    shell: systemctl daemon-reload
+  - name: Apply it to the RUNNING interface too, so no restart is needed
+    sudo: true
+    shell: sysctl -qw net.ipv6.conf.tailscale0.disable_ipv6=0
+    changed_when: 'true'
+verify:
+  - name: tailscale0 has its IPv6 mesh address and the health error is gone
+    shell: >-
+      for i in $(seq 1 20); do
+        ip -6 addr show tailscale0 2>/dev/null | grep -q "fd7a:115c:a1e0" && break || sleep 3;
+      done;
+      ip -6 addr show tailscale0 | grep -q "fd7a:115c:a1e0" &&
+      echo "IPv6 present:" && ip -brief addr show tailscale0
+    changed_when: 'false'

servers/irv-ml1/README.md

@@ -2,33 +2,59 @@
 
 Secondary AI/ML inference host at the Irvine site. Formerly known as
 `ana-ml1` when colocated at Anaheim; moved to Irvine and slated for
-hostname rename to `irv-ml1` (OS-side rename pending — see below).
+hostname rename to `irv-ml1` — **done**; `hostnamectl` reports `irv-ml1`
+(verified 2026-09-22).
 
 ## Network
 
-- **Reachable IP:** `10.100.79.3` (WireGuard tunnel endpoint)
+⚠ **This section was wrong until 2026-09-22** — it described the
-- **No direct LAN access** — this host is reachable **only** via
+pre-headscale topology as current, which sends anyone triaging this host to
-  WireGuard. Tunnel terminates at the NH3 site (10.100.0.0/16 WG
+the wrong layer. Verified against the host:
-  subnet). If WG is down, `scripts/refresh-server-info.sh irv-ml1`
-  will fail with "No route to host" — that's a WG issue, not a host
-  issue.
-- **SSH:** `ssh irv-ml1` (config alias → `lkraven@10.100.79.3`,
-  key auth).
 
-## Pending hostname rename
+- **Primary path: the headscale mesh** — `100.64.0.6` (plus
+  `fd7a:115c:a1e0::6`). This is how the fleet reaches it.
+- **Irvine LAN:** `10.6.110.50/24` on `enp68s0f0`, routed via the mesh.
+- **`wg0` lifeline:** `10.100.79.3/32` — **still up with a live peer**, not
+  dormant. Tailscale currently uses this address as its endpoint for the
+  direct path to nh3-scale, so it is load-bearing rather than vestigial.
+- **SSH:** `ssh infra-ops@100.64.0.6` (NOPASSWD sudo).
 
-OS hostname still reports `ana-ml1` (both in `hostnamectl` and in
+### Known network behaviour — it lives behind someone else's firewall
-`system-details.txt`). To finish the rename:
 
-```bash
+**Irvine is a tenancy: this host sits behind a Fortinet PFI does not
-ssh -t irv-ml1 'sudo hostnamectl set-hostname irv-ml1; \
+control** (operator, 2026-09-22). Two consequences to recognise rather than
-                sudo sed -i "s/ana-ml1/irv-ml1/g" /etc/hosts; \
+re-diagnose:
-                cat /etc/hosts; hostname'
-```
 
-Then refresh the inventory snapshot so it reflects the new identity.
+- **TLS interception breaks Tailscale's relay and control channels
-Not blocking anything — services don't care about the kernel's idea
+  intermittently.** tailscaled logs `server cert for "derpNN.tailscale.com"
-of hostname.
+  looks like "Fortinet" equipment` — 41 in one week — and the headscale
+  control connection times out periodically (4 episodes in 14 days).
+  **Usually invisible**, because direct peer paths keep carrying data
+  (netcheck: UDP true, `MappingVariesByDestIP: false`, an easy NAT). On
+  2026-09-22 03:13 it went all the way to unreachable for ~5 minutes and
+  paged — the only such page in 14 days. **There is no fix on our side of
+  the boundary**; do not go looking for one.
+- **Captive-portal false positives on Docker bridges.** ~26 `br-*`
+  interfaces here, and tailscaled probes them all:
+  `DetectCaptivePortal(found=true, ifName=br-…)`. Noise, not a fault.
+
+⚠ **Diagnose reachability with `tailscale ping`, not the status output.**
+On 2026-09-22 headscale reported the node `online` and `tailscale status`
+showed the peer `active` with 19.7 GB transferred, while nothing on the host
+answered at all. Both are last-known state; only a round trip is liveness.
+
+### IPv6 on tailscale0
+
+`/etc/sysctl.conf` sets `net.ipv6.conf.default.disable_ipv6=1`, which newly
+created interfaces inherit — so `tailscale0` came up IPv6-disabled and
+tailscaled could never add `fd7a:115c:a1e0::6/128` (logged as *permission
+denied*, though it runs as root; the EPERM is the kernel refusing a disabled
+interface). Fixed 2026-09-22 by a scoped systemd drop-in,
+`services/irv-ml1/tailscaled-ipv6.conf`, which sets the per-interface sysctl
+in `ExecStartPost` — **after** the interface exists. A `/etc/sysctl.d` entry
+would run at boot, before `tailscale0` is created, and be silently ignored.
+The global default is deliberately untouched so Docker bridges keep their
+current behaviour.
 
 ## Hardware
 
@@ -55,7 +81,9 @@ generative-AI stack:
 - ComfyUI, SillyTavern, SDNext, fluxgym (image gen / SD)
 - alltalk, alltalkv2, bark, kokoro, Orpheus-FastAPI, stablediffusion (TTS + voice)
 - llama.cpp, llama-swap, koboldcpp, aphrodite (LLM inference)
-- ollama (port 11434 listening on all interfaces) — native binary, not the container
+- (⚠ `ollama` was listed here as running on `:11434`. It is **gone** —
+  verified 2026-09-22: no unit file, nothing listening, no process. Ollama
+  is banned fleet-wide; if it reappears, tear it down.)
 - ai-toolkit, chat-ui, h2ogpt, o-textgen, lollms, bitsandbytes (misc ML frameworks)
 - sillytavern-extras, simple-proxy-for-tavern (lkraven-owned)
 
@@ -68,7 +96,7 @@ need it.
 | Stack | Port | GPU | Role |
 |-------|------|-----|------|
 | dockge | 5001 | — | Per-host Compose UI |
-| beszel-agent-irv | 45876 | — | Metrics agent → Beszel hub on ana-docker (token mode through WG) |
+| beszel-agent-irv | 45876 | — | Metrics agent → Beszel hub on ana-docker (token mode, over the mesh) |
 | dozzle-agent-irv | 7007 | — | Log agent → Dozzle hub on ana-docker |
 | comfyui | 8188 | all | ComfyUI (node-based SD/Flux) — runs independently of `/opt/ComfyUI` native install |
 | kokoro | 8193 | all | Kokoro 82M TTS — lowest-latency English TTS, OpenAI-compatible /v1/audio/speech |
@@ -119,5 +147,6 @@ regenerable).
 scripts/refresh-server-info.sh irv-ml1
 ```
 
-**Caveat:** requires the WG tunnel to be up. If the refresh shows
+**Caveat:** depends on the headscale mesh, not `wg0`. If the refresh fails,
-"No route to host", bring WG up before retrying.
+check with `tailscale ping 100.64.0.6` from a mesh node before assuming the
+host is down — see Network on why the status output can claim it is fine.

services/irv-ml1/tailscaled-ipv6.conf

@@ -0,0 +1,10 @@
+# tailscale0 inherits net.ipv6.conf.default.disable_ipv6=1 (/etc/sysctl.conf)
+# at creation, so tailscaled cannot add its fd7a:115c:a1e0::6/128 mesh address
+# and logs "permission denied" forever. This re-enables IPv6 on that ONE
+# interface, after it exists — a /etc/sysctl.d entry runs at boot, before
+# tailscale0 is created, and is silently ignored.
+#
+# Scoped deliberately: the global default stays as-is so Docker bridges on this
+# host keep their current (no-IPv6) behaviour.
+[Service]
+ExecStartPost=/bin/sh -c 'for i in 1 2 3 4 5 6 7 8 9 10; do if [ -e /sys/class/net/tailscale0 ]; then sysctl -qw net.ipv6.conf.tailscale0.disable_ipv6=0; exit 0; fi; sleep 1; done; exit 0'