vh
38bb20ceda
TAILSCALE IPv6. `tailscale status` had been reporting, continuously:
2 add route failures; first was: permission denied
adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
with tailscale0 carrying only 100.64.0.6/32 while headscale had assigned it
an IPv6 address it could not use.
Not a capability problem -- tailscaled runs as root with the full bounding
set. /etc/sysctl.conf:59 sets net.ipv6.conf.default.disable_ipv6=1, and
`default` is inherited by NEWLY CREATED interfaces; tailscale0 is created at
daemon start, inherits it, and the kernel returns EPERM for every attempt.
Fixed with a scoped systemd drop-in rather than flipping the global default.
That line carries no comment, but IPv6-off-by-default on a host with ~26
docker bridges reads as deliberate, and changing it would hand IPv6 to every
future bridge as a side effect of fixing Tailscale.
⚠ It must be ExecStartPost, not /etc/sysctl.d. A sysctl.d entry for a
per-interface key is applied at boot, BEFORE tailscale0 exists, and is
silently ignored -- the setting would look present and do nothing.
Also learned: setting the sysctl on the LIVE interface is not enough.
tailscaled only attempts the address at startup or on a netmap change, so
the verify failed for 60s until the daemon was restarted. Restart is part
of the operation, not an afterthought.
Verified: fd7a:115c:a1e0::6/128 present on tailscale0, health clean, mesh
and services (arbo, ytvc) up.
README. It documented the pre-headscale topology as current -- "Reachable
IP: 10.100.79.3 (WireGuard tunnel endpoint)", "No direct LAN access", and a
refresh caveat telling you to bring WG up. That sends anyone triaging this
host to the wrong layer, which is the exact tax the file exists to prevent.
Now: mesh primary at 100.64.0.6, LAN 10.6.110.50, and wg0 documented as
STILL UP with a live peer -- tailscale uses that address as its direct
endpoint, so it is load-bearing, not vestigial.
Recorded with it, because these cost hours tonight and will cost them again:
- Irvine is a TENANCY behind a Fortinet PFI does not control. Its TLS
inspection breaks Tailscale's relay and control channels (41 cert
warnings/week, 4 control-plane episodes in 14 days). Usually invisible
because direct peer paths carry the data. No fix on our side.
- Diagnose reachability with `tailscale ping`, NOT the status output:
headscale said "online" and status said "active, 19.7 GB" while nothing
on the host answered. Both are last-known state; only a round trip is
liveness.
- The ~26 docker bridges make tailscaled report captive portals.
Two stale claims corrected: the hostname rename it called "pending" is done,
and `ollama` is listed as running on :11434 when it is gone -- verified, no
unit file, nothing listening, no process. It is banned fleet-wide.
153 lines
7.3 KiB
Markdown
153 lines
7.3 KiB
Markdown
# irv-ml1
|
||
|
||
Secondary AI/ML inference host at the Irvine site. Formerly known as
|
||
`ana-ml1` when colocated at Anaheim; moved to Irvine and slated for
|
||
hostname rename to `irv-ml1` — **done**; `hostnamectl` reports `irv-ml1`
|
||
(verified 2026-09-22).
|
||
|
||
## Network
|
||
|
||
⚠ **This section was wrong until 2026-09-22** — it described the
|
||
pre-headscale topology as current, which sends anyone triaging this host to
|
||
the wrong layer. Verified against the host:
|
||
|
||
- **Primary path: the headscale mesh** — `100.64.0.6` (plus
|
||
`fd7a:115c:a1e0::6`). This is how the fleet reaches it.
|
||
- **Irvine LAN:** `10.6.110.50/24` on `enp68s0f0`, routed via the mesh.
|
||
- **`wg0` lifeline:** `10.100.79.3/32` — **still up with a live peer**, not
|
||
dormant. Tailscale currently uses this address as its endpoint for the
|
||
direct path to nh3-scale, so it is load-bearing rather than vestigial.
|
||
- **SSH:** `ssh infra-ops@100.64.0.6` (NOPASSWD sudo).
|
||
|
||
### Known network behaviour — it lives behind someone else's firewall
|
||
|
||
**Irvine is a tenancy: this host sits behind a Fortinet PFI does not
|
||
control** (operator, 2026-09-22). Two consequences to recognise rather than
|
||
re-diagnose:
|
||
|
||
- **TLS interception breaks Tailscale's relay and control channels
|
||
intermittently.** tailscaled logs `server cert for "derpNN.tailscale.com"
|
||
looks like "Fortinet" equipment` — 41 in one week — and the headscale
|
||
control connection times out periodically (4 episodes in 14 days).
|
||
**Usually invisible**, because direct peer paths keep carrying data
|
||
(netcheck: UDP true, `MappingVariesByDestIP: false`, an easy NAT). On
|
||
2026-09-22 03:13 it went all the way to unreachable for ~5 minutes and
|
||
paged — the only such page in 14 days. **There is no fix on our side of
|
||
the boundary**; do not go looking for one.
|
||
- **Captive-portal false positives on Docker bridges.** ~26 `br-*`
|
||
interfaces here, and tailscaled probes them all:
|
||
`DetectCaptivePortal(found=true, ifName=br-…)`. Noise, not a fault.
|
||
|
||
⚠ **Diagnose reachability with `tailscale ping`, not the status output.**
|
||
On 2026-09-22 headscale reported the node `online` and `tailscale status`
|
||
showed the peer `active` with 19.7 GB transferred, while nothing on the host
|
||
answered at all. Both are last-known state; only a round trip is liveness.
|
||
|
||
### IPv6 on tailscale0
|
||
|
||
`/etc/sysctl.conf` sets `net.ipv6.conf.default.disable_ipv6=1`, which newly
|
||
created interfaces inherit — so `tailscale0` came up IPv6-disabled and
|
||
tailscaled could never add `fd7a:115c:a1e0::6/128` (logged as *permission
|
||
denied*, though it runs as root; the EPERM is the kernel refusing a disabled
|
||
interface). Fixed 2026-09-22 by a scoped systemd drop-in,
|
||
`services/irv-ml1/tailscaled-ipv6.conf`, which sets the per-interface sysctl
|
||
in `ExecStartPost` — **after** the interface exists. A `/etc/sysctl.d` entry
|
||
would run at boot, before `tailscale0` is created, and be silently ignored.
|
||
The global default is deliberately untouched so Docker bridges keep their
|
||
current behaviour.
|
||
|
||
## Hardware
|
||
|
||
- **Chassis:** (TBD — captured on next physical inspection)
|
||
- **CPU:** AMD Ryzen Threadripper 3970X (32 cores / 64 threads)
|
||
- **RAM:** 251.6 GB
|
||
- **GPUs:** 2× (unlike ana-ml2's matched pair):
|
||
- GPU 0: **NVIDIA GeForce RTX 3090** (24 GB VRAM)
|
||
- GPU 1: **NVIDIA RTX A6000** (48 GB VRAM)
|
||
- Total VRAM: 72 GB across both
|
||
- **OS:** Debian 12 (bookworm), kernel 6.1.0-37
|
||
- **Storage:**
|
||
- `/` on `/dev/nvme0n1p2` — 1.8 TB (78% used, ~393 GB free)
|
||
- `/worktank` — 462 GB (1% used — dedicated to Docker stacks' user state, e.g. ComfyUI models + workflows)
|
||
- `/storetank` — 1.8 TB (64% used, ~660 GB free)
|
||
|
||
## What it runs
|
||
|
||
### Native toolchain (`/opt`, owned by `llmuser`)
|
||
|
||
Predates the PFI docker convention; still the primary runtime for the
|
||
generative-AI stack:
|
||
|
||
- ComfyUI, SillyTavern, SDNext, fluxgym (image gen / SD)
|
||
- alltalk, alltalkv2, bark, kokoro, Orpheus-FastAPI, stablediffusion (TTS + voice)
|
||
- llama.cpp, llama-swap, koboldcpp, aphrodite (LLM inference)
|
||
- (⚠ `ollama` was listed here as running on `:11434`. It is **gone** —
|
||
verified 2026-09-22: no unit file, nothing listening, no process. Ollama
|
||
is banned fleet-wide; if it reappears, tear it down.)
|
||
- ai-toolkit, chat-ui, h2ogpt, o-textgen, lollms, bitsandbytes (misc ML frameworks)
|
||
- sillytavern-extras, simple-proxy-for-tavern (lkraven-owned)
|
||
|
||
### Docker stacks (`/opt/docker/compose/`, owned by `lkraven`)
|
||
|
||
Docker 29.4.1 with `nvidia` and `runc` runtimes. `lkraven` is in the
|
||
`docker` group. `traefik-net` external network exists for stacks that
|
||
need it.
|
||
|
||
| Stack | Port | GPU | Role |
|
||
|-------|------|-----|------|
|
||
| dockge | 5001 | — | Per-host Compose UI |
|
||
| beszel-agent-irv | 45876 | — | Metrics agent → Beszel hub on ana-docker (token mode, over the mesh) |
|
||
| dozzle-agent-irv | 7007 | — | Log agent → Dozzle hub on ana-docker |
|
||
| comfyui | 8188 | all | ComfyUI (node-based SD/Flux) — runs independently of `/opt/ComfyUI` native install |
|
||
| kokoro | 8193 | all | Kokoro 82M TTS — lowest-latency English TTS, OpenAI-compatible /v1/audio/speech |
|
||
| chatterbox | 8196 | 0 (3090) | Chatterbox voice-cloning TTS |
|
||
| index-tts | 8192 | all | IndexTTS multilingual TTS |
|
||
| qwen3-tts | 8191 | all | Qwen3-TTS — 7B multilingual TTS |
|
||
| cosyvoice | 8190 | all | CosyVoice multilingual TTS |
|
||
| fish-s2 | 8195 | 1 (A6000) | Fish-Speech S2 TTS |
|
||
| kyutai-tts | 8198 | 0 (3090) | Kyutai TTS |
|
||
| vibevoice | 8194 | 1 (A6000) | Microsoft VibeVoice TTS |
|
||
| voxtral | 8197 | 1 (A6000) | Mistral Voxtral ASR |
|
||
| parakeet | 8765 | all | NVIDIA Parakeet ASR (transcription) — bare `{"text": …}`, no `no_speech_prob` |
|
||
| speaches | 8204 | 1 (A6000) | OpenAI-compatible faster-whisper ASR — `verbose_json` w/ per-segment `no_speech_prob`; serves Eyra. VAD pinned OFF, image digest-pinned |
|
||
| stable-audio-open | 8211 | 1 (A6000) | Stable Audio Open 1.0 — diffusion SFX/ambience generator |
|
||
| ace-step | 8210 | 1 (A6000) | ACE-Step 1.5 — Apache-2.0 hybrid diffusion+LLM music generation |
|
||
|
||
GPU pinning comes from the compose `NVIDIA_VISIBLE_DEVICES` default;
|
||
`all` means the model picks at runtime (typically GPU 0). The
|
||
asset-engine orchestrator (`docs/asset-engine/services.yaml` → per-service
|
||
`lifecycle:` block) gates start operations on per-device VRAM headroom.
|
||
|
||
Exposed Docker socket on `*:2375` (for the homepage integration hub on
|
||
esh-docker-vm, which auto-discovers containers on this host).
|
||
|
||
## Storage watch
|
||
|
||
Nothing acute. `/storetank` dropped from 92% → 64% after a prune pass
|
||
on the native-toolchain side; keep an eye on it since model weights
|
||
and training outputs accumulate steadily (misbehavior starts around
|
||
~95% on either ext4 or ZFS).
|
||
|
||
## Backup coverage
|
||
|
||
Restic via `resticprofile` + systemd timer (01:00 daily) → `rest-server-nh3`
|
||
(local to the WG endpoint site; lower latency than crossing back to
|
||
ana-side). Profile tracked at `configs/restic/irv-ml1/profiles.yaml`.
|
||
Excludes HuggingFace caches and bulk model files on `/storetank`
|
||
(regenerable from HF Hub).
|
||
|
||
When the ComfyUI stack ships, `/worktank/comfyui/basedir/{user,custom_nodes,input}`
|
||
should be added to the source set (workflows + hand-installed nodes);
|
||
`/worktank/comfyui/basedir/{models,output}` stay excluded (bulk /
|
||
regenerable).
|
||
|
||
## Refresh state
|
||
|
||
```bash
|
||
scripts/refresh-server-info.sh irv-ml1
|
||
```
|
||
|
||
**Caveat:** depends on the headscale mesh, not `wg0`. If the refresh fails,
|
||
check with `tailscale ping 100.64.0.6` from a mesh node before assuming the
|
||
host is down — see Network on why the status output can claim it is fine.
|