diff --git a/servers/irv-ml1/README.md b/servers/irv-ml1/README.md new file mode 100644 index 0000000..0089384 --- /dev/null +++ b/servers/irv-ml1/README.md @@ -0,0 +1,105 @@ +# irv-ml1 + +Secondary AI/ML inference host at the Irvine site. Formerly known as +`ana-ml1` when colocated at Anaheim; moved to Irvine and slated for +hostname rename to `irv-ml1` (OS-side rename pending — see below). + +## Network + +- **Reachable IP:** `10.100.79.3` (WireGuard tunnel endpoint) +- **No direct LAN access** — this host is reachable **only** via + WireGuard. Tunnel terminates at the NH3 site (10.100.0.0/16 WG + subnet). If WG is down, `scripts/refresh-server-info.sh irv-ml1` + will fail with "No route to host" — that's a WG issue, not a host + issue. +- **SSH:** `ssh irv-ml1` (config alias → `lkraven@10.100.79.3`, + key auth). + +## Pending hostname rename + +OS hostname still reports `ana-ml1` (both in `hostnamectl` and in +`system-details.txt`). To finish the rename: + +```bash +ssh -t irv-ml1 'sudo hostnamectl set-hostname irv-ml1; \ + sudo sed -i "s/ana-ml1/irv-ml1/g" /etc/hosts; \ + cat /etc/hosts; hostname' +``` + +Then refresh the inventory snapshot so it reflects the new identity. +Not blocking anything — services don't care about the kernel's idea +of hostname. + +## Hardware + +- **Chassis:** (TBD — captured on next physical inspection) +- **CPU:** AMD Ryzen Threadripper 3970X (32 cores / 64 threads) +- **RAM:** 251.6 GB +- **GPUs:** 2× (unlike ana-ml2's matched pair): + - GPU 0: **NVIDIA GeForce RTX 3090** (24 GB VRAM) + - GPU 1: **NVIDIA RTX A6000** (48 GB VRAM) + - Total VRAM: 72 GB across both +- **OS:** Debian 12 (bookworm), kernel 6.1.0-37 +- **Storage:** + - `/` on `/dev/nvme0n1p2` — 1.8 TB (78% used, ~393 GB free) + - `/worktank` — 462 GB (1% used — dedicated to Docker stacks' user state, e.g. ComfyUI models + workflows) + - `/storetank` — 1.8 TB (64% used, ~660 GB free) + +## What it runs + +### Native toolchain (`/opt`, owned by `llmuser`) + +Predates the PFI docker convention; still the primary runtime for the +generative-AI stack: + +- ComfyUI, SillyTavern, SDNext, fluxgym (image gen / SD) +- alltalk, alltalkv2, bark, kokoro, Orpheus-FastAPI, stablediffusion (TTS + voice) +- llama.cpp, llama-swap, koboldcpp, aphrodite (LLM inference) +- ollama (port 11434 listening on all interfaces) — native binary, not the container +- ai-toolkit, chat-ui, h2ogpt, o-textgen, lollms, bitsandbytes (misc ML frameworks) +- sillytavern-extras, simple-proxy-for-tavern (lkraven-owned) + +### Docker stacks (`/opt/docker/compose/`, owned by `lkraven`) + +Docker 29.4.1 with `nvidia` and `runc` runtimes. `lkraven` is in the +`docker` group. `traefik-net` external network exists for stacks that +need it. + +| Stack | Port | Role | +|-------|------|------| +| dockge | 5001 | Per-host Compose UI | +| beszel-agent-irv | 45876 | Metrics agent → Beszel hub on ana-docker (token mode through WG) | +| dozzle-agent-irv | 7007 | Log agent → Dozzle hub on ana-docker | +| comfyui | 8188 | ComfyUI (node-based SD/Flux) — runs independently of `/opt/ComfyUI` native install | + +Exposed Docker socket on `*:2375` (for the homepage integration hub on +esh-docker-vm, which auto-discovers containers on this host). + +## Storage watch + +Nothing acute. `/storetank` dropped from 92% → 64% after a prune pass +on the native-toolchain side; keep an eye on it since model weights +and training outputs accumulate steadily (misbehavior starts around +~95% on either ext4 or ZFS). + +## Backup coverage + +Restic via `resticprofile` + systemd timer (01:00 daily) → `rest-server-nh3` +(local to the WG endpoint site; lower latency than crossing back to +ana-side). Profile tracked at `configs/restic/irv-ml1/profiles.yaml`. +Excludes HuggingFace caches and bulk model files on `/storetank` +(regenerable from HF Hub). + +When the ComfyUI stack ships, `/worktank/comfyui/basedir/{user,custom_nodes,input}` +should be added to the source set (workflows + hand-installed nodes); +`/worktank/comfyui/basedir/{models,output}` stay excluded (bulk / +regenerable). + +## Refresh state + +```bash +scripts/refresh-server-info.sh irv-ml1 +``` + +**Caveat:** requires the WG tunnel to be up. If the refresh shows +"No route to host", bring WG up before retrying. diff --git a/servers/irv-ml1/system-details.txt b/servers/irv-ml1/system-details.txt new file mode 100644 index 0000000..5f1edfa --- /dev/null +++ b/servers/irv-ml1/system-details.txt @@ -0,0 +1,383 @@ + +===== HOST ===== + +Hostname: irv-ml1.phasefinal.com +Date: 2026-04-23T22:25:44-07:00 +Uptime: up 3 weeks, 1 day, 7 hours, 20 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.1.0-37-amd64 +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 64 +CPU model: AMD Ryzen Threadripper 3970X 32-Core Processor +MemTotal: 251.6 GB +MemAvailable: 246.6 GB + +===== GPUS ===== + +index, name, memory.total [MiB], memory.free [MiB], driver_version +0, NVIDIA GeForce RTX 3090, 24576 MiB, 24126 MiB, 570.124.06 +1, NVIDIA RTX A6000, 49140 MiB, 48539 MiB, 570.124.06 + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/nvme0n1p2 1.8T 1.4T 393G 78% / +/dev/nvme0n1p1 511M 5.9M 506M 2% /boot/efi +worktank 462G 128K 462G 1% /worktank +storetank 1.8T 1.2T 660G 64% /storetank + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +UUID=a71486c6-b6db-4484-957f-a62e0b9dd127 / ext4 errors=remount-ro 0 1 +UUID=6C3D-4DC5 /boot/efi vfat umask=0077 0 1 +UUID=0f7132b5-ca2b-442a-bc1f-74b8fae44ccb none swap sw 0 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 187G) + total 140 + drwxrwxrwx 34 root root 4096 2026-04-23 15:18 . + drwxr-xr-x 21 root root 4096 2025-05-31 22:46 .. + drwxr-xr-x 20 llmuser llmuser 4096 2025-03-04 15:08 ai-toolkit + drwxr-xr-x 14 llmuser llmuser 4096 2025-02-18 13:03 alltalk + drwxr-xr-x 12 llmuser llmuser 4096 2025-01-28 11:36 alltalkv2 + drwxr-xr-x 2 llmuser llmuser 4096 2025-02-10 14:05 aphrodite + drwxr-xr-x 7 llmuser llmuser 4096 2023-09-26 15:44 bark + drwxr-xr-x 12 llmuser llmuser 4096 2023-09-27 16:23 bitsandbytes + drwxr-xr-x 10 llmuser llmuser 4096 2024-03-21 23:01 chat-ui + drwxr-xr-x 31 llmuser llmuser 4096 2025-05-04 09:30 ComfyUI + drwx--x--x 4 root root 4096 2026-04-23 15:18 containerd + drwxr-xr-x 2 llmuser llmuser 4096 2025-02-10 15:00 cuda + drwxr-xr-x 4 lkraven lkraven 4096 2026-04-23 14:56 docker + drwxr-xr-x 8 llmuser llmuser 4096 2025-03-07 14:03 fluxgym + drwxr-xr-x 3 root root 4096 2024-03-21 16:22 google + drwxr-xr-x 33 llmuser llmuser 4096 2024-03-21 16:39 h2ogpt + drwxr-xr-x 2 llmuser llmuser 4096 2025-03-23 22:51 koboldcpp + drwxr-xr-x 15 llmuser llmuser 4096 2025-06-04 15:34 kokoro + drwxr-xr-x 6 llmuser llmuser 4096 2025-02-05 23:05 kokovoicelab + drwxr-xr-x 24 llmuser llmuser 4096 2025-03-23 21:27 llama.cpp + drwxr-xr-x 2 llmuser llmuser 4096 2025-03-24 15:44 llama-swap + drwxr-xr-x 4 llmuser llmuser 4096 2024-03-18 10:50 lollms + drwxr-xr-x 4 root root 4096 2023-07-15 16:38 nvidia + drwxr-xr-x 2 llmuser llmuser 4096 2025-04-13 19:24 ollama + drwxr-xr-x 10 llmuser llmuser 4096 2025-03-21 22:00 Orpheus-FastAPI + drwxr-xr-x 24 llmuser llmuser 4096 2024-03-19 16:44 o-textgen + drwxr-xr-x 21 llmuser llmuser 4096 2023-09-27 10:06 sdnext + drwxr-xr-x 19 llmuser llmuser 4096 2025-03-05 13:50 SillyTavern + drwxrwxrwx 6 lkraven lkraven 4096 2023-07-17 19:55 SillyTavern-extras + +/opt/docker (total: 48K) + total 16 + drwxr-xr-x 4 lkraven lkraven 4096 2026-04-23 14:56 . + drwxrwxrwx 34 root root 4096 2026-04-23 15:18 .. + drwxr-xr-x 5 lkraven lkraven 4096 2026-04-23 16:50 compose + drwxr-xr-x 2 lkraven lkraven 4096 2026-04-23 14:56 conf + +/opt/docker/compose (total: 40K) + total 20 + drwxr-xr-x 5 lkraven lkraven 4096 2026-04-23 16:50 . + drwxr-xr-x 4 lkraven lkraven 4096 2026-04-23 14:56 .. + drwxr-xr-x 2 lkraven lkraven 4096 2026-04-23 17:09 beszel-agent-irv + drwxr-xr-x 2 lkraven lkraven 4096 2026-04-23 15:04 dockge + drwxr-xr-x 2 lkraven lkraven 4096 2026-04-23 16:57 dozzle-agent-irv + +/opt/docker/conf (total: 4.0K) + total 8 + drwxr-xr-x 2 lkraven lkraven 4096 2026-04-23 14:56 . + drwxr-xr-x 4 lkraven lkraven 4096 2026-04-23 14:56 .. + +/var/lib/docker (total: 4.0K) + +/srv (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2023-07-15 13:51 . + drwxr-xr-x 21 root root 4096 2025-05-31 22:46 .. + + +===== DOCKER ===== + +Server: 29.4.1 Client: 29.4.1 + +----- docker info ----- +Containers: 4 (running 3, paused 0, stopped 1) +Images: 44 +Runtimes: map[io.containerd.runc.v2:{{runc [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.2.1","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true},"mountExtensions":{"idmap":{"enabled":true}}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.3.5-0-g488fc13e","org.opencontainers.runc.version":"1.3.5\n"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]} nvidia:{{nvidia-container-runtime [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.2.1","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true},"mountExtensions":{"idmap":{"enabled":true}}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.3.5-0-g488fc13e","org.opencontainers.runc.version":"1.3.5\n"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]} runc:{{runc [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.2.1","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true},"mountExtensions":{"idmap":{"enabled":true}}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.3.5-0-g488fc13e","org.opencontainers.runc.version":"1.3.5\n"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]}] +Default runtime: runc +Storage driver: overlay2 +Root dir: /var/lib/docker +Server version: 29.4.1 + +----- running containers ----- +NAMES IMAGE STATUS PORTS +beszel-agent henrygd/beszel-agent:latest Up 5 hours (healthy) +dozzle-agent amir20/dozzle:latest Up 5 hours 0.0.0.0:7007->7007/tcp, 8080/tcp +dockge louislam/dockge:latest Up 7 hours (healthy) 0.0.0.0:5001->5001/tcp, [::]:5001->5001/tcp + +----- all containers ----- +NAMES IMAGE STATUS +beszel-agent henrygd/beszel-agent:latest Up 5 hours (healthy) +dozzle-agent amir20/dozzle:latest Up 5 hours +dockge louislam/dockge:latest Up 7 hours (healthy) +boring_bartik nvidia/cuda:11.5.2-base-ubuntu20.04 Exited (0) 13 months ago + +----- networks ----- +NAME DRIVER SCOPE +bridge bridge local +docker_default bridge local +host host local +none null local +textgen_default bridge local +traefik-net bridge local + +----- networks (external, non-default — worth knowing for compose external: true) ----- +docker_default +textgen_default +traefik-net + +----- named volumes ----- +VOLUME NAME DRIVER +beszel-agent-irv_beszel_agent_data local +dockge_dockge_data local +dozzle-agent-irv_dozzle_agent_data local + +----- compose projects currently running ----- +beszel-agent-irv +dockge +dozzle-agent-irv + +===== COMPOSE FILES (/opt/docker/compose/) ===== + + +>>> /opt/docker/compose/beszel-agent-irv/compose.yaml +# Beszel — lightweight server/container monitoring. +# +# Hub: single web UI with the SQLite store. Agents: per-host metric collectors +# that the hub pulls from over SSH. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → hub only (ana-docker) +# COMPOSE_PROFILES=hub,agent → hub + local agent on the same host +# COMPOSE_PROFILES=agent → agent only (ana-ml2, irv-ml1, +# nh3-docker, esh-docker-vm, +# vm-esh-nas) +# +# The agent uses network_mode: host so it sees real host CPU/mem/net/disk +# counters rather than container-scoped ones — that's why it can't share +# the tnet network with the hub. +# +# All tunables live in .env — edit that, not this file. + +services: + beszel: + image: henrygd/beszel:${BESZEL_VERSION} + container_name: beszel + profiles: [hub] + restart: unless-stopped + ports: + - "${BESZEL_PORT}:8090" + volumes: + - beszel_data:/beszel_data + healthcheck: + test: ["CMD", "/beszel", "health", "--url", "http://localhost:8090"] + interval: 120s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=Monitoring + - homepage.name=Beszel + - homepage.icon=mdi-chart-line + - homepage.description=Server + container monitoring + - homepage.href=http://10.250.50.70:${BESZEL_PORT} + + beszel-agent: + image: henrygd/beszel-agent:${BESZEL_VERSION} + container_name: beszel-agent + profiles: [agent] + restart: unless-stopped + network_mode: host + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - beszel_agent_data:/var/lib/beszel-agent + environment: + # Agent auth has two modes (v0.13+ supports both side-by-side): + # - KEY-mode: agent listens, hub connects inbound over SSH using KEY. + # Requires BESZEL_HUB_KEY in .env. + # - Token-mode: agent initiates an outbound connection to HUB_URL + # using TOKEN. Easier through NAT. Requires HUB_URL + BESZEL_TOKEN. + # Leave unused ones empty ("") in .env; both can be set simultaneously. + # + # For irv-ml1 (WireGuard-only): TOKEN mode is strongly preferred. The + # agent initiates outbound to the hub, so NAT/WG-firewall rules stay + # simple. KEY mode would require the hub to reach back into the WG + # tunnel to the agent's listening port. + - PORT=${BESZEL_AGENT_PORT:-45876} + - KEY=${BESZEL_HUB_KEY:-} + - HUB_URL=${HUB_URL:-} + - TOKEN=${BESZEL_TOKEN:-} + - EXTRA_FILESYSTEMS=${BESZEL_EXTRA_FS:-} + healthcheck: + test: ["CMD", "/agent", "health"] + interval: 120s + timeout: 10s + retries: 3 + start_period: 15s + +volumes: + beszel_data: + beszel_agent_data: + +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/dockge/compose.yaml +# Dockge — per-host Docker Compose UI (https://dockge.kuma.pet/). +# +# One instance runs on every Docker host so the compose dir is manageable +# from a browser. Each host sets DOCKGE_HOST_LABEL + DOCKGE_HOST_IP in its +# .env so the homepage card points at the right place. +# +# All tunables live in .env — edit that, not this file. + +services: + dockge: + image: louislam/dockge:${DOCKGE_VERSION:-latest} + container_name: dockge + restart: unless-stopped + ports: + - "${DOCKGE_PORT:-5001}:5001" + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - dockge_data:/app/data + - /opt/docker:/opt/docker + environment: + - DOCKGE_STACKS_DIR=/opt/docker/compose + networks: + - tnet + labels: + - homepage.group=Service Networking + - homepage.name=Dockge (${DOCKGE_HOST_LABEL}) + - homepage.icon=sh-dockge.png + - homepage.description=Compose UI on ${DOCKGE_HOST_LABEL} + - homepage.href=http://${DOCKGE_HOST_IP}:${DOCKGE_PORT:-5001} + +volumes: + dockge_data: + +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/dozzle-agent-irv/compose.yaml +# Dozzle — container log viewer. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → runs the web UI (deploy on ana-docker) +# COMPOSE_PROFILES=agent → runs the remote agent (deploy on ana-ml2, +# irv-ml1, nh3-docker, esh-docker-vm, vm-esh-nas) +# +# Same compose.yaml on all servers; per-host `.env` picks the profile. +# +# irv-ml1 specific: listens on the WG tunnel IP only (10.100.79.3) by +# default — the hub on ana-docker reaches it through the tunnel. Set +# DOZZLE_AGENT_BIND=0.0.0.0 in .env if you ever need to accept from +# other interfaces. +# +# All tunables live in .env — edit that, not this file. + +services: + dozzle: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle + profiles: [hub] + restart: unless-stopped + ports: + - "${DOZZLE_PORT}:8080" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + - DOZZLE_REMOTE_AGENT=${DOZZLE_REMOTE_AGENT:-} + - DOZZLE_AUTH_PROVIDER=${DOZZLE_AUTH_PROVIDER:-none} + - DOZZLE_USERNAME=${DOZZLE_USERNAME:-} + - DOZZLE_PASSWORD=${DOZZLE_PASSWORD:-} + healthcheck: + test: ["CMD", "/dozzle", "healthcheck"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=Monitoring + - homepage.name=Dozzle + - homepage.icon=mdi-text-box-search + - homepage.description=Container logs (ana-docker + ana-ml2) + - homepage.href=http://10.250.50.70:${DOZZLE_PORT} + + dozzle-agent: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle-agent + profiles: [agent] + restart: unless-stopped + command: agent + ports: + - "${DOZZLE_AGENT_BIND:-0.0.0.0}:${DOZZLE_AGENT_PORT}:7007" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_agent_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + networks: + - tnet + +volumes: + dozzle_data: + dozzle_agent_data: + +networks: + tnet: + name: traefik-net + external: true + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf + +===== LISTENING PORTS ===== + +0.0.0.0:22 +0.0.0.0:5001 +0.0.0.0:7007 +*:11434 +[::]:22 +*:2375 +[::]:5001 + +===== MODEL / HUGGINGFACE CACHES ===== + +/home/lkraven/.cache/huggingface (1.8M) + hub entries: + models--bert-base-uncased + models--openai--clip-vit-large-patch14 + version.txt + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +containerd.service running +docker.service running + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/stacks/comfyui/.env.example b/stacks/comfyui/.env.example new file mode 100644 index 0000000..8ec41e6 --- /dev/null +++ b/stacks/comfyui/.env.example @@ -0,0 +1,51 @@ +# ComfyUI stack tunables. Copy to `.env` on irv-ml1 before deploying. +# +# cp .env.example .env +# # edit as needed +# docker compose up -d + +# Image tag. Format is `ubuntu_cuda-`; the floating +# equivalent is `ubuntu_cuda-latest`. +# +# irv-ml1 runtime constraints: +# - Host driver 570.x → max runtime CUDA 12.8 (per nvidia-smi). Tags +# above 12.8 will fail at runtime on driver/CUDA mismatch even +# though the manifest pulls fine. +# - Host cuda-toolkit is 12.8 to match native Python venvs (ComfyUI, +# SDNext, fluxgym, llama.cpp, etc.) — keep the container in sync. +# +# See https://hub.docker.com/r/mmartial/comfyui-nvidia-docker/tags. +COMFYUI_VERSION=ubuntu24_cuda12.8-20260312 + +# Host port for the ComfyUI web UI (container listens on 8188) +COMFYUI_PORT=8188 + +# Bind address. 0.0.0.0 exposes on all interfaces including the WG +# tunnel IP (10.100.79.3) so homepage / the hub side can reach it. +# Use 127.0.0.1 to restrict to local / SSH-forwarded access only. +COMFYUI_BIND=0.0.0.0 + +# UID/GID the container user runs as. Must match the owner of +# COMFYUI_BASEDIR so workflows, custom nodes, input, and output are +# read/write from both the container and external host tooling. +# On irv-ml1: lkraven = 1000:1000. +COMFYUI_UID=1000 +COMFYUI_GID=1000 + +# Host path for all user state (models, workflows, custom_nodes, +# input, output). Must exist and be chowned to COMFYUI_UID:COMFYUI_GID +# before the first `docker compose up`. +COMFYUI_BASEDIR=/worktank/comfyui/basedir + +# Host path for container runtime (ComfyUI source + Python venv + +# pip cache, ~5 GB after bootstrap). Disposable — safe to `rm -rf` +# on version bumps to force a fresh re-bootstrap. Must be owned by +# COMFYUI_UID:COMFYUI_GID. +COMFYUI_RUNDIR=/worktank/comfyui/run + +# ComfyUI-Manager security level: +# normal → default; installs custom nodes from vetted sources only. +# weak → permits arbitrary pip installs via Manager. Use only if +# you trust every workflow author you'll import. +# strict → blocks pip installs via Manager entirely. +COMFYUI_SECURITY_LEVEL=normal diff --git a/stacks/comfyui/README.md b/stacks/comfyui/README.md new file mode 100644 index 0000000..87ada47 --- /dev/null +++ b/stacks/comfyui/README.md @@ -0,0 +1,121 @@ +# ComfyUI + +Node-based Stable Diffusion / Flux inference UI. + +**Server:** irv-ml1 (Irvine, WireGuard-only) +**Port:** 8188 (configurable via `.env`) +**GPUs:** both (RTX 3090 + RTX A6000 both exposed; select per-workflow) +**Image:** `mmartial/comfyui-nvidia-docker` +**Native coexistence:** runs independently of `/opt/ComfyUI` (the existing native install). Migrate models by copy or move — no cross-binding. + +## Path layout + +All user state lives under a single host tree on `/worktank`, owned +by `lkraven:lkraven` (UID/GID 1000) so external tooling can read and +write workflow files directly. ComfyUI-Manager puts downloaded weights +into the same tree, so everything is visible on the host. + +| Host path | Container path | Purpose | Restic? | +|---|---|---|---| +| `/worktank/comfyui/basedir/models/` | `/basedir/models/` | Checkpoints, loras, vae, unet, controlnet, upscale, etc. | excluded (bulk, regenerable) | +| `/worktank/comfyui/basedir/user/default/workflows/` | `/basedir/user/default/workflows/` | Workflow JSON files — **external tools edit here** | included | +| `/worktank/comfyui/basedir/custom_nodes/` | `/basedir/custom_nodes/` | ComfyUI-Manager-installed nodes | included | +| `/worktank/comfyui/basedir/input/` | `/basedir/input/` | Uploaded source images | included | +| `/worktank/comfyui/basedir/output/` | `/basedir/output/` | Generated images | excluded (regenerable) | +| `/worktank/comfyui/run/` | `/comfy/mnt` | ComfyUI source + venv + pip cache (~5 GB) | excluded (disposable) | + +Standard ComfyUI model subdirs the image creates under `models/`: +`checkpoints`, `clip`, `clip_vision`, `configs`, `controlnet`, +`diffusers`, `embeddings`, `gligen`, `hypernetworks`, `loras`, +`photomaker`, `style_models`, `unet`, `upscale_models`, `vae`, +`vae_approx`. + +## First-time deploy on irv-ml1 + +```bash +# 1. Push compose + env template from workstation +scripts/deploy-stack.sh irv-ml1 comfyui + +# 2. Create basedir + run on the host (the image expects both to exist +# as the target UID/GID before first up; it refuses to chown +# mounted paths). One-time sudo because /worktank is root-owned. +ssh -t irv-ml1 'sudo mkdir -p /worktank/comfyui/{basedir,run} && \ + sudo chown -R lkraven:lkraven /worktank/comfyui' + +# 3. Create .env and bring up. First `up` takes several minutes — +# the image installs ~5 GB of Python packages on first boot. +ssh irv-ml1 ' + cd /opt/docker/compose/comfyui && \ + cp -n .env.example .env && \ + docker compose config >/dev/null && \ + docker compose up -d && \ + docker compose logs -f --tail=30 +' +``` + +Once the logs settle on "Starting server" / "To see the GUI go to …", +the UI is reachable at `http://10.100.79.3:8188` (from the WG-tunnel +side). The homepage hub on ana-docker auto-discovers it via the Dozzle +agent + the homepage Docker socket integration already wired on +irv-ml1. + +## Populate models + +Native install lives at `/opt/ComfyUI/models/` (owned `llmuser`). +Copy or move into the container's tree — paths match 1:1: + +```bash +# Example: copy one subdir at a time, preserving timestamps +ssh -t irv-ml1 'sudo -u lkraven rsync -av \ + /opt/ComfyUI/models/checkpoints/ \ + /worktank/comfyui/basedir/models/checkpoints/' +``` + +Repeat per subdir. `rsync -av --remove-source-files` if you want a +true move (then clean up empty source dirs). The native install will +continue to work with whatever remains under `/opt/ComfyUI/models/`. + +## Editing workflows from outside the container + +Workflows live as JSON under +`/worktank/comfyui/basedir/user/default/workflows/` on the host, +owned `lkraven:lkraven`. External tools running as `lkraven` can +read and write them without coordination — ComfyUI picks up file +changes on the next workflow load (no container restart needed). + +## Deploy updates + +```bash +# After editing compose.yaml or .env.example here +scripts/deploy-stack.sh irv-ml1 comfyui +ssh irv-ml1 'cd /opt/docker/compose/comfyui && docker compose up -d' +``` + +## Image version bump + +```bash +# Pick a new tag from +# https://hub.docker.com/r/mmartial/comfyui-nvidia-docker/tags +ssh -t irv-ml1 ' + cd /opt/docker/compose/comfyui && \ + sed -i "s/^COMFYUI_VERSION=.*/COMFYUI_VERSION=/" .env && \ + docker compose pull && \ + docker compose up -d +' +``` + +Runtime lives at `/worktank/comfyui/run/`. For a fresh bootstrap +(force venv + pip cache rebuild against the new image): + +```bash +ssh irv-ml1 ' + cd /opt/docker/compose/comfyui && \ + docker compose down && \ + rm -rf /worktank/comfyui/run/* && \ + docker compose pull && \ + docker compose up -d +' +``` + +User state under `/worktank/comfyui/basedir/` is untouched by either +pattern. diff --git a/stacks/comfyui/compose.yaml b/stacks/comfyui/compose.yaml new file mode 100644 index 0000000..e67e93a --- /dev/null +++ b/stacks/comfyui/compose.yaml @@ -0,0 +1,63 @@ +# ComfyUI — node-based Stable Diffusion / Flux inference UI. +# +# Runs on irv-ml1 (dual GPU: RTX 3090 + RTX A6000). Both GPUs are +# exposed via NVIDIA_VISIBLE_DEVICES=all so GPU selection happens +# per-workflow inside ComfyUI. +# +# All user state — models, workflows, custom_nodes, input, output — +# lives under a single BASE_DIRECTORY tree on /worktank (462 GB +# dedicated), owned by lkraven:lkraven (1000:1000) so external +# tooling can read and write workflow files directly on the host. +# +# Runtime state (ComfyUI source, venv, pip cache) lives in a bind +# mount at ${COMFYUI_RUNDIR} — disposable (can be wiped on version +# bumps to force re-bootstrap), but owned by the host user so no +# sudo dance is needed. (Named volumes would be created root-owned +# and the image refuses to chown a mounted path.) +# +# First-run prerequisite: both ${COMFYUI_BASEDIR} and ${COMFYUI_RUNDIR} +# must exist on the host with ownership matching COMFYUI_UID:COMFYUI_GID +# before `up`. See README for the bootstrap command. +# +# All tunables live in .env — edit that, not this file. + +services: + comfyui: + image: mmartial/comfyui-nvidia-docker:${COMFYUI_VERSION} + container_name: comfyui + restart: unless-stopped + runtime: nvidia + ports: + - "${COMFYUI_BIND:-0.0.0.0}:${COMFYUI_PORT}:8188" + environment: + - NVIDIA_VISIBLE_DEVICES=all + - WANTED_UID=${COMFYUI_UID} + - WANTED_GID=${COMFYUI_GID} + - BASE_DIRECTORY=/basedir + - SECURITY_LEVEL=${COMFYUI_SECURITY_LEVEL:-normal} + - USE_UV=true + volumes: + - ${COMFYUI_BASEDIR}:/basedir + - ${COMFYUI_RUNDIR}:/comfy/mnt + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8188/ >/dev/null || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + # First boot installs ~5 GB of Python packages; allow generous + # start_period so the container isn't marked unhealthy during + # bootstrap. Subsequent starts are fast. + start_period: 600s + networks: + - tnet + labels: + - homepage.group=AI Systems + - homepage.name=ComfyUI + - homepage.icon=mdi-image-auto-adjust + - homepage.description=Node-based SD/Flux inference (irv-ml1) + - homepage.href=http://10.100.79.3:${COMFYUI_PORT} + +networks: + tnet: + name: traefik-net + external: true