irv-ml1

Secondary AI/ML inference host at the Irvine site. Formerly known as ana-ml1 when colocated at Anaheim; moved to Irvine and slated for hostname rename to irv-ml1 — done; hostnamectl reports irv-ml1 (verified 2026-09-22).

Network

⚠ This section was wrong until 2026-09-22 — it described the pre-headscale topology as current, which sends anyone triaging this host to the wrong layer. Verified against the host:

Primary path: the headscale mesh — 100.64.0.6 (plus fd7a:115c:a1e0::6). This is how the fleet reaches it.
Irvine LAN: 10.6.110.50/24 on enp68s0f0, routed via the mesh.
wg0 lifeline: 10.100.79.3/32 — still up with a live peer, not dormant. Tailscale currently uses this address as its endpoint for the direct path to nh3-scale, so it is load-bearing rather than vestigial.
SSH: ssh infra-ops@100.64.0.6 (NOPASSWD sudo).

Known network behaviour — it lives behind someone else's firewall

Irvine is a tenancy: this host sits behind a Fortinet PFI does not control (operator, 2026-09-22). Two consequences to recognise rather than re-diagnose:

TLS interception breaks Tailscale's relay and control channels intermittently. tailscaled logs server cert for "derpNN.tailscale.com" looks like "Fortinet" equipment — 41 in one week — and the headscale control connection times out periodically (4 episodes in 14 days). Usually invisible, because direct peer paths keep carrying data (netcheck: UDP true, MappingVariesByDestIP: false, an easy NAT). On 2026-09-22 03:13 it went all the way to unreachable for ~5 minutes and paged — the only such page in 14 days. There is no fix on our side of the boundary; do not go looking for one.
Captive-portal false positives on Docker bridges. ~26 br-* interfaces here, and tailscaled probes them all: DetectCaptivePortal(found=true, ifName=br-…). Noise, not a fault.

⚠ Diagnose reachability with tailscale ping, not the status output. On 2026-09-22 headscale reported the node online and tailscale status showed the peer active with 19.7 GB transferred, while nothing on the host answered at all. Both are last-known state; only a round trip is liveness.

IPv6 on tailscale0

/etc/sysctl.conf sets net.ipv6.conf.default.disable_ipv6=1, which newly created interfaces inherit — so tailscale0 came up IPv6-disabled and tailscaled could never add fd7a:115c:a1e0::6/128 (logged as permission denied, though it runs as root; the EPERM is the kernel refusing a disabled interface). Fixed 2026-09-22 by a scoped systemd drop-in, services/irv-ml1/tailscaled-ipv6.conf, which sets the per-interface sysctl in ExecStartPost — after the interface exists. A /etc/sysctl.d entry would run at boot, before tailscale0 is created, and be silently ignored. The global default is deliberately untouched so Docker bridges keep their current behaviour.

Hardware

Chassis: (TBD — captured on next physical inspection)
CPU: AMD Ryzen Threadripper 3970X (32 cores / 64 threads)
RAM: 251.6 GB
GPUs: 2× (unlike ana-ml2's matched pair):
- GPU 0: NVIDIA GeForce RTX 3090 (24 GB VRAM)
- GPU 1: NVIDIA RTX A6000 (48 GB VRAM)
- Total VRAM: 72 GB across both
OS: Debian 12 (bookworm), kernel 6.1.0-37
Storage (measured 2026-10-05 0115, after that night's cleanup):
- / on /dev/nvme0n1p2: Samsung 970 EVO Plus 2 TB, single disk, no mirror. 54% used, 802 GB free. Holds /var/lib/docker and /home.
- worktank: ZFS mirror of 2× Samsung PM981 512 GB NVMe, the fastest storage on the box. 86% used, ~52 GB free. ComfyUI run dirs, TTS seat caches, and arbo's text encoders (next bullet).
- storetank: ZFS mirror of 2× Crucial MX500 2 TB SATA SSD. 79% used, 377 GB free. arbo's models.
- ⚠ /storetank/arbo/models/text_encoders is NOT on storetank. It is ZFS dataset worktank/arbo-text-encoders (recordsize 1M, atime off) mounted at that path since 2026-10-05, so every path (host, the comfyui container's /basedir/models/text_encoders, arbo's catalog) stayed the same. It moved for NVMe load speed and to relieve storetank (87% → 79%). Consequences: zfs list shows it under worktank, and du /storetank does not count it. A container that bind-mounts /storetank/arbo/models sees it only if it was started AFTER the mount (default rprivate propagation); zfs-mount runs before local-fs.target, so at boot it is in place before docker. If worktank fails to import, the path shows an empty directory and ComfyUI reports no text encoders.

What it runs

Native toolchain (`/opt`, owned by `llmuser`)

Predates the PFI docker convention; still the primary runtime for the generative-AI stack:

ComfyUI, SillyTavern, SDNext, fluxgym (image gen / SD)
alltalk, alltalkv2, bark, kokoro, Orpheus-FastAPI, stablediffusion (TTS + voice)
llama.cpp, llama-swap, koboldcpp, aphrodite (LLM inference)
(⚠ ollama was listed here as running on :11434. It is gone — verified 2026-09-22: no unit file, nothing listening, no process. Ollama is banned fleet-wide; if it reappears, tear it down.)
ai-toolkit, chat-ui, h2ogpt, o-textgen, lollms, bitsandbytes (misc ML frameworks)
sillytavern-extras, simple-proxy-for-tavern (lkraven-owned)

Docker stacks (`/opt/docker/compose/`, owned by `lkraven`)

Docker 29.4.1 with nvidia and runc runtimes. lkraven is in the docker group. traefik-net external network exists for stacks that need it.

Stack	Port	GPU	Role
dockge	5001	—	Per-host Compose UI
beszel-agent-irv	45876	—	Metrics agent → Beszel hub on ana-docker (token mode, over the mesh)
dozzle-agent-irv	7007	—	Log agent → Dozzle hub on ana-docker
comfyui	8188	all	ComfyUI (node-based SD/Flux) — runs independently of `/opt/ComfyUI` native install
kokoro	8193	all	Kokoro 82M TTS — lowest-latency English TTS, OpenAI-compatible /v1/audio/speech
chatterbox	8196	0 (3090)	Chatterbox voice-cloning TTS
index-tts	8192	all	IndexTTS multilingual TTS
qwen3-tts	8191	all	Qwen3-TTS — 7B multilingual TTS
cosyvoice	8190	all	CosyVoice multilingual TTS
fish-s2	8195	1 (A6000)	Fish-Speech S2 TTS
kyutai-tts	8198	0 (3090)	Kyutai TTS
vibevoice	8194	1 (A6000)	Microsoft VibeVoice TTS
voxtral	8197	1 (A6000)	Mistral Voxtral ASR
parakeet	8765	all	NVIDIA Parakeet ASR (transcription) — bare `{"text": …}`, no `no_speech_prob`
speaches	8204	1 (A6000)	OpenAI-compatible faster-whisper ASR — `verbose_json` w/ per-segment `no_speech_prob`; serves Eyra. VAD pinned OFF, image digest-pinned
stable-audio-open	8211	1 (A6000)	Stable Audio Open 1.0 — diffusion SFX/ambience generator
ace-step	8210	1 (A6000)	ACE-Step 1.5 — Apache-2.0 hybrid diffusion+LLM music generation

GPU pinning comes from the compose NVIDIA_VISIBLE_DEVICES default; all means the model picks at runtime (typically GPU 0). The asset-engine orchestrator (docs/asset-engine/services.yaml → per-service lifecycle: block) gates start operations on per-device VRAM headroom.

Exposed Docker socket on *:2375 (for the homepage integration hub on esh-docker-vm, which auto-discovers containers on this host).

Storage watch

2026-10-05 cleanup (Prime): Docker went from 968 GB to ~527 GB. 318 old pullable tags were removed (arbo except v1.17.0 + v1.16.2, vllm, old comfyui cuda12.6/12.8 tags, CUDA base images, python:3.11, tts-generation-webui, llama.cpp), plus dangling images and build cache unused for more than 30 days: +441 GB. Home entries untouched for more than 2 months were deleted: +82 GB (aimodels, granite-4.1-8b-bf16, r16-vmoan-harness, r17smoke/separator, morpheus, neovim, panns_data, small scripts/logs, old .cache dirs, 13 old HF cache entries). Kept although old, because they are live: act-runner (act_runner.service), arbo-webhook.py, ytvc-webhook.py, arbo-deploy.sh, deploy-engine.sh, dotfiles (stow target for .zshrc and friends) and fzf-git.sh (sourced by .zshrc). ⚠ An old mtime does not make a file unused: daemons started in May never update atime. Check /proc/*/{cwd,cmdline,fd,maps}, units and symlinks before deleting. Not done: ~81 GB of locally built TTS seat images with no container (qwen3-tts, ace-step, vibevoice, zonos, parakeet, kyutai-tts, old dots-tts, stable-audio-open). They cannot be re-pulled, and the asset-engine still lists several of those seats, so the TTS owner decides.

Before that: nothing acute. /storetank dropped from 92% → 64% after a prune pass on the native-toolchain side; keep an eye on it since model weights and training outputs accumulate steadily (misbehavior starts around ~95% on either ext4 or ZFS).

Backup coverage

Restic via resticprofile + systemd timer (01:00 daily) → rest-server-nh3 (local to the WG endpoint site; lower latency than crossing back to ana-side). Profile tracked at configs/restic/irv-ml1/profiles.yaml. Excludes HuggingFace caches and bulk model files on /storetank (regenerable from HF Hub).

When the ComfyUI stack ships, /worktank/comfyui/basedir/{user,custom_nodes,input} should be added to the source set (workflows + hand-installed nodes); /worktank/comfyui/basedir/{models,output} stay excluded (bulk / regenerable).

Refresh state

scripts/refresh-server-info.sh irv-ml1

Caveat: depends on the headscale mesh, not wg0. If the refresh fails, check with tailscale ping 100.64.0.6 from a mesh node before assuming the host is down — see Network on why the status output can claim it is fine.

9.6 KiB Raw Blame History Unescape Escape