# irv-ml1

Secondary AI/ML inference host at the Irvine site. Formerly known as
`ana-ml1` when colocated at Anaheim; moved to Irvine and slated for
hostname rename to `irv-ml1` (OS-side rename pending — see below).

## Network

- **Reachable IP:** `10.100.79.3` (WireGuard tunnel endpoint)
- **No direct LAN access** — this host is reachable **only** via
  WireGuard. Tunnel terminates at the NH3 site (10.100.0.0/16 WG
  subnet). If WG is down, `scripts/refresh-server-info.sh irv-ml1`
  will fail with "No route to host" — that's a WG issue, not a host
  issue.
- **SSH:** `ssh irv-ml1` (config alias → `lkraven@10.100.79.3`,
  key auth).

## Pending hostname rename

OS hostname still reports `ana-ml1` (both in `hostnamectl` and in
`system-details.txt`). To finish the rename:

```bash
ssh -t irv-ml1 'sudo hostnamectl set-hostname irv-ml1; \
                sudo sed -i "s/ana-ml1/irv-ml1/g" /etc/hosts; \
                cat /etc/hosts; hostname'
```

Then refresh the inventory snapshot so it reflects the new identity.
Not blocking anything — services don't care about the kernel's idea
of hostname.

## Hardware

- **Chassis:** (TBD — captured on next physical inspection)
- **CPU:** AMD Ryzen Threadripper 3970X (32 cores / 64 threads)
- **RAM:** 251.6 GB
- **GPUs:** 2× (unlike ana-ml2's matched pair):
  - GPU 0: **NVIDIA GeForce RTX 3090** (24 GB VRAM)
  - GPU 1: **NVIDIA RTX A6000** (48 GB VRAM)
  - Total VRAM: 72 GB across both
- **OS:** Debian 12 (bookworm), kernel 6.1.0-37
- **Storage:**
  - `/` on `/dev/nvme0n1p2` — 1.8 TB (78% used, ~393 GB free)
  - `/worktank` — 462 GB (1% used — dedicated to Docker stacks' user state, e.g. ComfyUI models + workflows)
  - `/storetank` — 1.8 TB (64% used, ~660 GB free)

## What it runs

### Native toolchain (`/opt`, owned by `llmuser`)

Predates the PFI docker convention; still the primary runtime for the
generative-AI stack:

- ComfyUI, SillyTavern, SDNext, fluxgym (image gen / SD)
- alltalk, alltalkv2, bark, kokoro, Orpheus-FastAPI, stablediffusion (TTS + voice)
- llama.cpp, llama-swap, koboldcpp, aphrodite (LLM inference)
- ollama (port 11434 listening on all interfaces) — native binary, not the container
- ai-toolkit, chat-ui, h2ogpt, o-textgen, lollms, bitsandbytes (misc ML frameworks)
- sillytavern-extras, simple-proxy-for-tavern (lkraven-owned)

### Docker stacks (`/opt/docker/compose/`, owned by `lkraven`)

Docker 29.4.1 with `nvidia` and `runc` runtimes. `lkraven` is in the
`docker` group. `traefik-net` external network exists for stacks that
need it.

| Stack | Port | GPU | Role |
|-------|------|-----|------|
| dockge | 5001 | — | Per-host Compose UI |
| beszel-agent-irv | 45876 | — | Metrics agent → Beszel hub on ana-docker (token mode through WG) |
| dozzle-agent-irv | 7007 | — | Log agent → Dozzle hub on ana-docker |
| comfyui | 8188 | all | ComfyUI (node-based SD/Flux) — runs independently of `/opt/ComfyUI` native install |
| kokoro | 8193 | all | Kokoro 82M TTS — lowest-latency English TTS, OpenAI-compatible /v1/audio/speech |
| chatterbox | 8196 | 0 (3090) | Chatterbox voice-cloning TTS |
| index-tts | 8192 | all | IndexTTS multilingual TTS |
| qwen3-tts | 8191 | all | Qwen3-TTS — 7B multilingual TTS |
| cosyvoice | 8190 | all | CosyVoice multilingual TTS |
| fish-s2 | 8195 | 1 (A6000) | Fish-Speech S2 TTS |
| kyutai-tts | 8198 | 0 (3090) | Kyutai TTS |
| vibevoice | 8194 | 1 (A6000) | Microsoft VibeVoice TTS |
| voxtral | 8197 | 1 (A6000) | Mistral Voxtral ASR |
| parakeet | 8765 | all | NVIDIA Parakeet ASR (transcription) — bare `{"text": …}`, no `no_speech_prob` |
| speaches | 8204 | 1 (A6000) | OpenAI-compatible faster-whisper ASR — `verbose_json` w/ per-segment `no_speech_prob`; serves Eyra. VAD pinned OFF, image digest-pinned |
| stable-audio-open | 8211 | 1 (A6000) | Stable Audio Open 1.0 — diffusion SFX/ambience generator |
| ace-step | 8210 | 1 (A6000) | ACE-Step 1.5 — Apache-2.0 hybrid diffusion+LLM music generation |

GPU pinning comes from the compose `NVIDIA_VISIBLE_DEVICES` default;
`all` means the model picks at runtime (typically GPU 0). The
asset-engine orchestrator (`docs/asset-engine/services.yaml` → per-service
`lifecycle:` block) gates start operations on per-device VRAM headroom.

Exposed Docker socket on `*:2375` (for the homepage integration hub on
esh-docker-vm, which auto-discovers containers on this host).

## Storage watch

Nothing acute. `/storetank` dropped from 92% → 64% after a prune pass
on the native-toolchain side; keep an eye on it since model weights
and training outputs accumulate steadily (misbehavior starts around
~95% on either ext4 or ZFS).

## Backup coverage

Restic via `resticprofile` + systemd timer (01:00 daily) → `rest-server-nh3`
(local to the WG endpoint site; lower latency than crossing back to
ana-side). Profile tracked at `configs/restic/irv-ml1/profiles.yaml`.
Excludes HuggingFace caches and bulk model files on `/storetank`
(regenerable from HF Hub).

When the ComfyUI stack ships, `/worktank/comfyui/basedir/{user,custom_nodes,input}`
should be added to the source set (workflows + hand-installed nodes);
`/worktank/comfyui/basedir/{models,output}` stay excluded (bulk /
regenerable).

## Refresh state

```bash
scripts/refresh-server-info.sh irv-ml1
```

**Caveat:** requires the WG tunnel to be up. If the refresh shows
"No route to host", bring WG up before retrying.