# irv-ml1

Secondary AI/ML inference host at the Irvine site. Formerly known as
`ana-ml1` when colocated at Anaheim; moved to Irvine and slated for
hostname rename to `irv-ml1` — **done**; `hostnamectl` reports `irv-ml1`
(verified 2026-09-22).

## Network

⚠ **This section was wrong until 2026-09-22** — it described the
pre-headscale topology as current, which sends anyone triaging this host to
the wrong layer. Verified against the host:

- **Primary path: the headscale mesh** — `100.64.0.6` (plus
  `fd7a:115c:a1e0::6`). This is how the fleet reaches it.
- **Irvine LAN:** `10.6.110.50/24` on `enp68s0f0`, routed via the mesh.
- **`wg0` lifeline:** `10.100.79.3/32` — **still up with a live peer**, not
  dormant. Tailscale currently uses this address as its endpoint for the
  direct path to nh3-scale, so it is load-bearing rather than vestigial.
- **SSH:** `ssh infra-ops@100.64.0.6` (NOPASSWD sudo).

### Known network behaviour — it lives behind someone else's firewall

**Irvine is a tenancy: this host sits behind a Fortinet PFI does not
control** (operator, 2026-09-22). Two consequences to recognise rather than
re-diagnose:

- **TLS interception breaks Tailscale's relay and control channels
  intermittently.** tailscaled logs `server cert for "derpNN.tailscale.com"
  looks like "Fortinet" equipment` — 41 in one week — and the headscale
  control connection times out periodically (4 episodes in 14 days).
  **Usually invisible**, because direct peer paths keep carrying data
  (netcheck: UDP true, `MappingVariesByDestIP: false`, an easy NAT). On
  2026-09-22 03:13 it went all the way to unreachable for ~5 minutes and
  paged — the only such page in 14 days. **There is no fix on our side of
  the boundary**; do not go looking for one.
- **Captive-portal false positives on Docker bridges.** ~26 `br-*`
  interfaces here, and tailscaled probes them all:
  `DetectCaptivePortal(found=true, ifName=br-…)`. Noise, not a fault.

⚠ **Diagnose reachability with `tailscale ping`, not the status output.**
On 2026-09-22 headscale reported the node `online` and `tailscale status`
showed the peer `active` with 19.7 GB transferred, while nothing on the host
answered at all. Both are last-known state; only a round trip is liveness.

### IPv6 on tailscale0

`/etc/sysctl.conf` sets `net.ipv6.conf.default.disable_ipv6=1`, which newly
created interfaces inherit — so `tailscale0` came up IPv6-disabled and
tailscaled could never add `fd7a:115c:a1e0::6/128` (logged as *permission
denied*, though it runs as root; the EPERM is the kernel refusing a disabled
interface). Fixed 2026-09-22 by a scoped systemd drop-in,
`services/irv-ml1/tailscaled-ipv6.conf`, which sets the per-interface sysctl
in `ExecStartPost` — **after** the interface exists. A `/etc/sysctl.d` entry
would run at boot, before `tailscale0` is created, and be silently ignored.
The global default is deliberately untouched so Docker bridges keep their
current behaviour.

## Hardware

- **Chassis:** (TBD — captured on next physical inspection)
- **CPU:** AMD Ryzen Threadripper 3970X (32 cores / 64 threads)
- **RAM:** 251.6 GB
- **GPUs:** 2× (unlike ana-ml2's matched pair):
  - GPU 0: **NVIDIA GeForce RTX 3090** (24 GB VRAM)
  - GPU 1: **NVIDIA RTX A6000** (48 GB VRAM)
  - Total VRAM: 72 GB across both
- **OS:** Debian 12 (bookworm), kernel 6.1.0-37
- **Storage** (measured 2026-10-05 0115, after that night's cleanup):
  - `/` on `/dev/nvme0n1p2`: Samsung 970 EVO Plus 2 TB, **single disk, no mirror**. 54% used, 802 GB free. Holds
    `/var/lib/docker` and `/home`.
  - `worktank`: ZFS mirror of 2× Samsung PM981 512 GB NVMe, the fastest storage on the box. 86% used, ~52 GB free.
    ComfyUI run dirs, TTS seat caches, and arbo's text encoders (next bullet).
  - `storetank`: ZFS mirror of 2× Crucial MX500 2 TB SATA SSD. 79% used, 377 GB free. arbo's models.
  - ⚠ **`/storetank/arbo/models/text_encoders` is NOT on storetank.** It is ZFS dataset `worktank/arbo-text-encoders`
    (recordsize 1M, atime off) mounted at that path since 2026-10-05, so every path (host, the comfyui container's
    `/basedir/models/text_encoders`, arbo's catalog) stayed the same. It moved for NVMe load speed and to relieve storetank
    (87% → 79%). Consequences: `zfs list` shows it under worktank, and `du /storetank` does not count it. A container that
    bind-mounts `/storetank/arbo/models` sees it only if it was started AFTER the mount (default rprivate propagation);
    zfs-mount runs before local-fs.target, so at boot it is in place before docker. If worktank fails to import, the
    path shows an empty directory and ComfyUI reports no text encoders.

## What it runs

### Native toolchain (`/opt`, owned by `llmuser`)

Predates the PFI docker convention; still the primary runtime for the
generative-AI stack:

- ComfyUI, SillyTavern, SDNext, fluxgym (image gen / SD)
- alltalk, alltalkv2, bark, kokoro, Orpheus-FastAPI, stablediffusion (TTS + voice)
- llama.cpp, llama-swap, koboldcpp, aphrodite (LLM inference)
- (⚠ `ollama` was listed here as running on `:11434`. It is **gone** —
  verified 2026-09-22: no unit file, nothing listening, no process. Ollama
  is banned fleet-wide; if it reappears, tear it down.)
- ai-toolkit, chat-ui, h2ogpt, o-textgen, lollms, bitsandbytes (misc ML frameworks)
- sillytavern-extras, simple-proxy-for-tavern (lkraven-owned)

### Docker stacks (`/opt/docker/compose/`, owned by `lkraven`)

Docker 29.4.1 with `nvidia` and `runc` runtimes. `lkraven` is in the
`docker` group. `traefik-net` external network exists for stacks that
need it.

| Stack | Port | GPU | Role |
|-------|------|-----|------|
| dockge | 5001 | — | Per-host Compose UI |
| beszel-agent-irv | 45876 | — | Metrics agent → Beszel hub on ana-docker (token mode, over the mesh) |
| dozzle-agent-irv | 7007 | — | Log agent → Dozzle hub on ana-docker |
| comfyui | 8188 | all | ComfyUI (node-based SD/Flux) — runs independently of `/opt/ComfyUI` native install |
| kokoro | 8193 | all | Kokoro 82M TTS — lowest-latency English TTS, OpenAI-compatible /v1/audio/speech |
| chatterbox | 8196 | 0 (3090) | Chatterbox voice-cloning TTS |
| index-tts | 8192 | all | IndexTTS multilingual TTS |
| qwen3-tts | 8191 | all | Qwen3-TTS — 7B multilingual TTS |
| cosyvoice | 8190 | all | CosyVoice multilingual TTS |
| fish-s2 | 8195 | 1 (A6000) | Fish-Speech S2 TTS |
| kyutai-tts | 8198 | 0 (3090) | Kyutai TTS |
| vibevoice | 8194 | 1 (A6000) | Microsoft VibeVoice TTS |
| voxtral | 8197 | 1 (A6000) | Mistral Voxtral ASR |
| parakeet | 8765 | all | NVIDIA Parakeet ASR (transcription) — bare `{"text": …}`, no `no_speech_prob` |
| speaches | 8204 | 1 (A6000) | OpenAI-compatible faster-whisper ASR — `verbose_json` w/ per-segment `no_speech_prob`; serves Eyra. VAD pinned OFF, image digest-pinned |
| stable-audio-open | 8211 | 1 (A6000) | Stable Audio Open 1.0 — diffusion SFX/ambience generator |
| ace-step | 8210 | 1 (A6000) | ACE-Step 1.5 — Apache-2.0 hybrid diffusion+LLM music generation |

GPU pinning comes from the compose `NVIDIA_VISIBLE_DEVICES` default;
`all` means the model picks at runtime (typically GPU 0). The
asset-engine orchestrator (`docs/asset-engine/services.yaml` → per-service
`lifecycle:` block) gates start operations on per-device VRAM headroom.

Exposed Docker socket on `*:2375` (for the homepage integration hub on
esh-docker-vm, which auto-discovers containers on this host).

## Storage watch

**2026-10-05 cleanup (Prime):** Docker went from 968 GB to ~527 GB. 318 old pullable tags were removed (arbo except
v1.17.0 + v1.16.2, vllm, old comfyui cuda12.6/12.8 tags, CUDA base images, python:3.11, tts-generation-webui,
llama.cpp), plus dangling images and build cache unused for more than 30 days: +441 GB. Home entries untouched for more
than 2 months were deleted: +82 GB (aimodels, granite-4.1-8b-bf16, r16-vmoan-harness, r17smoke/separator, morpheus,
neovim, panns_data, small scripts/logs, old `.cache` dirs, 13 old HF cache entries). Kept although old, because they
are live: `act-runner` (act_runner.service), `arbo-webhook.py`, `ytvc-webhook.py`, `arbo-deploy.sh`,
`deploy-engine.sh`, `dotfiles` (stow target for `.zshrc` and friends) and `fzf-git.sh` (sourced by `.zshrc`).
⚠ An old mtime does not make a file unused: daemons started in May never update atime. Check `/proc/*/{cwd,cmdline,fd,maps}`,
units and symlinks before deleting.
Not done: ~81 GB of locally built TTS seat images with no container (qwen3-tts, ace-step, vibevoice, zonos, parakeet,
kyutai-tts, old dots-tts, stable-audio-open). They cannot be re-pulled, and the asset-engine still lists several of
those seats, so the TTS owner decides.

Before that: nothing acute. `/storetank` dropped from 92% → 64% after a prune pass
on the native-toolchain side; keep an eye on it since model weights
and training outputs accumulate steadily (misbehavior starts around
~95% on either ext4 or ZFS).

## Backup coverage

Restic via `resticprofile` + systemd timer (01:00 daily) → `rest-server-nh3`
(local to the WG endpoint site; lower latency than crossing back to
ana-side). Profile tracked at `configs/restic/irv-ml1/profiles.yaml`.
Excludes HuggingFace caches and bulk model files on `/storetank`
(regenerable from HF Hub).

When the ComfyUI stack ships, `/worktank/comfyui/basedir/{user,custom_nodes,input}`
should be added to the source set (workflows + hand-installed nodes);
`/worktank/comfyui/basedir/{models,output}` stay excluded (bulk /
regenerable).

## Refresh state

```bash
scripts/refresh-server-info.sh irv-ml1
```

**Caveat:** depends on the headscale mesh, not `wg0`. If the refresh fails,
check with `tailscale ping 100.64.0.6` from a mesh node before assuming the
host is down — see Network on why the status output can claim it is fine.