vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0f032fac2c65a2b55514cb0b8370dfdb7140c03d
esh-pfi-infrastructure/servers/nh3-ml1/system-details.txt
T
vh 5960526c3f feat(nh3-ml1): second TEI embed/rerank backend live on nh3-pve; parity-verified vs esh-ml1 
NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled.

- nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml.
- nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4)
  deployed with HOST_NAME/HOST_IP labels.
- Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min
  0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control
  0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed
  identical within rep spread.
- gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix #7006).
  With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl
  reopen; applied on nh3-pve (one package).
- pve-nvidia-host.yaml: document that the headers meta drags in the newest
  kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot).
- Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage
  nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal.
- nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels,
  AMT cabled but unreachable on the network.

Gateway routing to nh3-ml1 is not changed.
2026-09-25 15:55:33 -07:00

453 lines
24 KiB
Plaintext
Raw Blame History

===== HOST =====
Hostname: nh3-ml1
Date: 2026-09-25T22:51:53+00:00
Uptime: up 18 minutes
OS: Debian GNU/Linux 12 (bookworm)
Kernel: 6.8.12-43-pve
Arch: x86_64
===== HARDWARE =====
CPU cores: 6
CPU model: 13th Gen Intel(R) Core(TM) i9-13900H
MemTotal: 16.0 GB
MemAvailable: 13.1 GB
===== GPUS =====
index, name, memory.total [MiB], memory.free [MiB], driver_version
0, NVIDIA RTX 2000E Ada Generation, 16380 MiB, 13390 MiB, 580.178.04
===== FILESYSTEMS (df) =====
Filesystem Size Used Avail Use% Mounted on
rpool/data/subvol-109-disk-0 80G 12G 69G 15% /
===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
===== TARGETED DATA PATHS =====
/opt (total: 3.2G)
total 11
drwxr-xr-x 5 root root 5 2026-09-25 22:30 .
drwxr-xr-x 17 root root 21 2026-09-25 22:33 ..
drwxrwsr-x 4 root docker 4 2026-09-25 22:35 aimodels
drwx--x--x 4 root root 4 2026-09-25 22:30 containerd
drwxrwsr-x 4 root docker 4 2026-09-25 22:30 docker
/opt/docker (total: 131K)
total 2
drwxrwsr-x 4 root docker 4 2026-09-25 22:30 .
drwxr-xr-x 5 root root 5 2026-09-25 22:30 ..
drwxrwsr-x 5 root docker 5 2026-09-25 22:49 compose
drwxrwsr-x 2 root docker 2 2026-09-25 22:30 conf
/opt/docker/compose (total: 130K)
total 27
drwxrwsr-x 5 root docker 5 2026-09-25 22:49 .
drwxrwsr-x 4 root docker 4 2026-09-25 22:30 ..
drwxr-sr-x 4 infra-ops docker 8 2026-09-25 22:43 beszel
drwxr-sr-x 2 infra-ops docker 6 2026-09-25 22:49 dozzle-agent
drwxr-sr-x 2 infra-ops docker 6 2026-09-25 22:35 embed-rerank
/opt/docker/conf (total: 512)
total 1
drwxrwsr-x 2 root docker 2 2026-09-25 22:30 .
drwxrwsr-x 4 root docker 4 2026-09-25 22:30 ..
/var/lib/docker (total: 8.5K)
/srv (total: 512)
total 9
drwxr-xr-x 2 root root 2 2025-09-07 15:14 .
drwxr-xr-x 17 root root 21 2026-09-25 22:33 ..
===== DOCKER =====
Server: 29.8.1 Client: 29.8.1
----- docker info -----
Containers: 4 (running 4, paused 0, stopped 0)
Images: 4
Runtimes: map[io.containerd.runc.v2:{{runc [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.3.0","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_LOONGARCH64","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG","SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true,"schemata":true,"monitoring":true},"memoryPolicy":{"modes":["MPOL_BIND","MPOL_DEFAULT","MPOL_INTERLEAVE","MPOL_LOCAL","MPOL_PREFERRED","MPOL_PREFERRED_MANY","MPOL_WEIGHTED_INTERLEAVE"],"flags":["MPOL_F_NUMA_BALANCING","MPOL_F_RELATIVE_NODES","MPOL_F_STATIC_NODES"]},"mountExtensions":{"idmap":{"enabled":true}},"netDevices":{"enabled":true}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.5.1-0-g8f2685a4","org.opencontainers.runc.version":"1.5.1"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]} nvidia:{{nvidia-container-runtime [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.3.0","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_LOONGARCH64","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG","SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true,"schemata":true,"monitoring":true},"memoryPolicy":{"modes":["MPOL_BIND","MPOL_DEFAULT","MPOL_INTERLEAVE","MPOL_LOCAL","MPOL_PREFERRED","MPOL_PREFERRED_MANY","MPOL_WEIGHTED_INTERLEAVE"],"flags":["MPOL_F_NUMA_BALANCING","MPOL_F_RELATIVE_NODES","MPOL_F_STATIC_NODES"]},"mountExtensions":{"idmap":{"enabled":true}},"netDevices":{"enabled":true}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.5.1-0-g8f2685a4","org.opencontainers.runc.version":"1.5.1"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]} runc:{{runc [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.3.0","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_LOONGARCH64","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG","SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true,"schemata":true,"monitoring":true},"memoryPolicy":{"modes":["MPOL_BIND","MPOL_DEFAULT","MPOL_INTERLEAVE","MPOL_LOCAL","MPOL_PREFERRED","MPOL_PREFERRED_MANY","MPOL_WEIGHTED_INTERLEAVE"],"flags":["MPOL_F_NUMA_BALANCING","MPOL_F_RELATIVE_NODES","MPOL_F_STATIC_NODES"]},"mountExtensions":{"idmap":{"enabled":true}},"netDevices":{"enabled":true}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.5.1-0-g8f2685a4","org.opencontainers.runc.version":"1.5.1"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]}]
Default runtime: runc
Storage driver: overlayfs
Root dir: /var/lib/docker
Server version: 29.8.1
----- running containers -----
NAMES IMAGE STATUS PORTS
dozzle-agent amir20/dozzle:v10.4.1 Up About a minute 10.100.50.80:7007->7007/tcp, 8080/tcp
beszel-agent henrygd/beszel-agent-nvidia:0.18.7 Up 7 minutes (healthy)
tei-rerank ghcr.io/huggingface/text-embeddings-inference:89-1.9.4 Up 15 minutes (healthy) 0.0.0.0:8013->80/tcp, [::]:8013->80/tcp
tei-embed ghcr.io/huggingface/text-embeddings-inference:89-1.9.4 Up 15 minutes (healthy) 0.0.0.0:8001->80/tcp, [::]:8001->80/tcp
----- all containers -----
NAMES IMAGE STATUS
dozzle-agent amir20/dozzle:v10.4.1 Up About a minute
beszel-agent henrygd/beszel-agent-nvidia:0.18.7 Up 7 minutes (healthy)
tei-rerank ghcr.io/huggingface/text-embeddings-inference:89-1.9.4 Up 15 minutes (healthy)
tei-embed ghcr.io/huggingface/text-embeddings-inference:89-1.9.4 Up 15 minutes (healthy)
----- networks -----
NAME DRIVER SCOPE
bridge bridge local
embed-rerank_default bridge local
host host local
none null local
traefik-net bridge local
----- networks (external, non-default — worth knowing for compose external: true) -----
embed-rerank_default
traefik-net
----- named volumes -----
VOLUME NAME DRIVER
beszel_beszel_agent_data local
dozzle-agent_dozzle_agent_data local
----- compose projects currently running -----
beszel
dozzle-agent
embed-rerank
===== COMPOSE FILES (/opt/docker/compose/) =====
>>> /opt/docker/compose/beszel/compose.yaml
# Beszel — lightweight server/container monitoring.
#
# Hub: single web UI with the SQLite store. Agents: per-host metric collectors
# that the hub pulls from over SSH.
#
# Multi-host layout via compose profiles:
# COMPOSE_PROFILES=hub → hub only (ana-docker)
# COMPOSE_PROFILES=hub,agent → hub + local agent on the same host
# COMPOSE_PROFILES=agent → agent only (ana-ml2, nh3-docker,
# esh-docker-vm, vm-esh-nas)
#
# The agent uses network_mode: host so it sees real host CPU/mem/net/disk
# counters rather than container-scoped ones — that's why it can't share
# the tnet network with the hub.
#
# All tunables live in .env — edit that, not this file.
services:
beszel:
image: henrygd/beszel:${BESZEL_VERSION}
container_name: beszel
profiles: [hub]
restart: unless-stopped
ports:
- "${BESZEL_PORT}:8090"
volumes:
- beszel_data:/beszel_data
healthcheck:
# Hub image is distroless — no wget/curl. Use the bundled `/beszel`
# binary's built-in health subcommand (https://beszel.dev/guide/healthchecks).
test: ["CMD", "/beszel", "health", "--url", "http://localhost:8090"]
interval: 120s
timeout: 10s
retries: 3
start_period: 15s
networks:
- tnet
labels:
- homepage.group=Monitoring
- homepage.name=Beszel
- homepage.icon=mdi-chart-line
- homepage.description=Server + container monitoring
- homepage.href=http://10.250.50.70:${BESZEL_PORT}
- homepage.widget.type=beszel
- homepage.widget.url=http://10.250.50.70:${BESZEL_PORT}
- homepage.widget.version=2
- homepage.widget.username={{HOMEPAGE_VAR_BESZEL_USERNAME}}
- homepage.widget.password={{HOMEPAGE_VAR_BESZEL_PASSWORD}}
beszel-agent:
image: henrygd/beszel-agent:${BESZEL_VERSION}
container_name: beszel-agent
profiles: [agent]
restart: unless-stopped
network_mode: host
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- beszel_agent_data:/var/lib/beszel-agent
environment:
# Agent auth has two modes (v0.13+ supports both side-by-side):
# - KEY-mode: agent listens, hub connects inbound over SSH using KEY.
# Requires BESZEL_HUB_KEY in .env.
# - Token-mode: agent initiates an outbound connection to HUB_URL
# using TOKEN. Easier through NAT. Requires HUB_URL + BESZEL_TOKEN.
# Leave unused ones empty ("") in .env; both can be set simultaneously.
- PORT=${BESZEL_AGENT_PORT:-45876}
- KEY=${BESZEL_HUB_KEY:-}
- HUB_URL=${HUB_URL:-}
- TOKEN=${BESZEL_TOKEN:-}
- EXTRA_FILESYSTEMS=${BESZEL_EXTRA_FS:-}
healthcheck:
# Agent image ships the `/agent` binary with a `health` subcommand.
# Verifies the agent process is up — not that the hub can reach it.
test: ["CMD", "/agent", "health"]
interval: 120s
timeout: 10s
retries: 3
start_period: 15s
volumes:
beszel_data:
beszel_agent_data:
networks:
tnet:
name: traefik-net
external: true
>>> /opt/docker/compose/beszel/synology/compose.yaml
services:
beszel-agent:
image: henrygd/beszel-agent:0.18.7
container_name: beszel-agent
restart: unless-stopped
network_mode: host
volumes:
- ./agent-data:/var/lib/beszel-agent
- /var/run/docker.sock:/var/run/docker.sock:ro
- /usr/share/zoneinfo:/extra-filesystems/system:ro
- ./:/extra-filesystems/volume1:ro
environment:
PORT: '45876'
KEY: 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt'
FILESYSTEM: /extra-filesystems/system
EXTRA_FILESYSTEMS: /extra-filesystems/volume1
>>> /opt/docker/compose/dozzle-agent/compose.yaml
# Dozzle — container log viewer.
#
# Multi-host layout via compose profiles:
# COMPOSE_PROFILES=hub → runs the web UI (deploy on ana-docker)
# COMPOSE_PROFILES=agent → runs the remote agent (deploy on ana-ml2)
#
# Same compose.yaml on both servers; per-host `.env` picks the profile.
#
# All tunables live in .env — edit that, not this file.
services:
dozzle:
image: amir20/dozzle:${DOZZLE_VERSION}
container_name: dozzle
profiles: [hub]
restart: unless-stopped
ports:
- "${DOZZLE_PORT}:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- dozzle_data:/data
environment:
- DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME}
- DOZZLE_REMOTE_AGENT=${DOZZLE_REMOTE_AGENT:-}
- DOZZLE_AUTH_PROVIDER=${DOZZLE_AUTH_PROVIDER:-none}
- DOZZLE_USERNAME=${DOZZLE_USERNAME:-}
- DOZZLE_PASSWORD=${DOZZLE_PASSWORD:-}
healthcheck:
test: ["CMD", "/dozzle", "healthcheck"]
interval: 30s
timeout: 10s
retries: 3
start_period: 15s
networks:
- tnet
labels:
- homepage.group=Monitoring
- homepage.name=Dozzle
- homepage.icon=mdi-text-box-search
- homepage.description=Container logs (ana-docker + ana-ml2)
- homepage.href=http://10.250.50.70:${DOZZLE_PORT}
dozzle-agent:
image: amir20/dozzle:${DOZZLE_VERSION}
container_name: dozzle-agent
profiles: [agent]
restart: unless-stopped
command: agent
ports:
- "${DOZZLE_AGENT_BIND:-0.0.0.0}:${DOZZLE_AGENT_PORT}:7007"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- dozzle_agent_data:/data
environment:
- DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME}
networks:
- tnet
volumes:
dozzle_data:
dozzle_agent_data:
networks:
tnet:
name: traefik-net
external: true
>>> /opt/docker/compose/embed-rerank/compose.yaml
# embed-rerank — THE fleet's embedding + reranking service, on esh-ml1 (CT 110 on
# esh-pve, RTX 2000E Ada, 16 GB). Served by Hugging Face Text Embeddings
# Inference (TEI). A second instance runs on nh3-ml1 (CT 109 on nh3-pve, the same
# card) since 2026-09-25; the per-host bits are HOST_NAME / HOST_IP in .env, and
# their defaults are esh-ml1's, so esh-ml1's live .env needs no change.
#
# Prime, 2026-09-25: "TEI is embed/reranker server for esh-ml1 and the FLEET in
# general, in future." It replaced vLLM here the same day, after a side-by-side
# bake-off on this card (docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md). TEI
# gives the same vectors as the old vLLM seats (no re-embedding), but it is
# ~1.3x slower on bulk work on this card. It is much lighter (2.6 GB VRAM for
# both, 8 GB image, ~4 s restart). fv-ml1's vLLM embed/rerank seats were
# retired after this went live.
#
# tei-embed Qwen/Qwen3-Embedding-0.6B → /v1/embeddings (OpenAI), /embed :8001
# tei-rerank BAAI/bge-reranker-v2-m3 → /rerank (body: query + texts) :8013
#
# Ports kept from the vLLM era (and fv-ml1), so the embedding gateway entry did
# not change address. ⚠ The RERANK gateway entry must use LiteLLM's
# `huggingface/` provider: `hosted_vllm/` sends `documents` and TEI answers 422
# "missing field texts".
#
# ⚠ Embedding vectors are model-specific. Never change EMBED_MODEL without a
# re-embedding plan for every index built on it (Worldtree, nevermore, Open WebUI).
#
# FAIL-CLOSED truncation (--auto-truncate false). TEI's default silently
# embedded the first 16,384 tokens of a ~40k-token input and returned 200.
# Turning it off requires --max-batch-tokens >= the model's max input (32,768
# for Qwen3-Embedding), or TEI refuses to start.
#
# NO `tnet`/traefik-net: esh-ml1 runs no traefik; consumers reach the published
# ports, and in practice only the LiteLLM gateway does (verified from the seats'
# logs 2026-09-25: every request matched a gateway spend-log row).
#
# Host setup: playbooks/pve-nvidia-host.yaml, then playbooks/gpu-lxc.yaml.
# Tunables live in .env.
name: embed-rerank
services:
tei-embed:
image: ghcr.io/huggingface/text-embeddings-inference:${TEI_TAG}
container_name: tei-embed
restart: unless-stopped
ports:
- "${EMBED_PORT}:80"
volumes:
- /opt/aimodels/tei-cache:/data
environment:
- HF_TOKEN=${HF_TOKEN:-}
command:
- --model-id
- ${EMBED_MODEL}
- --served-model-name
- ${EMBED_MODEL}
# TEI on CUDA is float16-only; parity vs the bf16 vLLM vectors was measured.
- --dtype
- float16
# Default 32; vLLM had no cap and callers batch 64.
- --max-client-batch-size
- "${MAX_CLIENT_BATCH_SIZE}"
- --auto-truncate
- "false"
- --max-batch-tokens
- "32768"
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["0"]
capabilities: [gpu]
healthcheck:
test: ["CMD", "curl", "-fsS", "http://localhost:80/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 60s
labels:
- homepage.group=AI - Eval & Retrieval
- homepage.name=Embed — Qwen3 0.6B (TEI, ${HOST_NAME:-esh-ml1})
- homepage.icon=mdi-vector-arrange-below
- homepage.description=Fleet embeddings (qwen3-embedding) via TEI on ${HOST_NAME:-esh-ml1}
- homepage.href=http://${HOST_IP:-10.0.50.80}:${EMBED_PORT}/docs
tei-rerank:
image: ghcr.io/huggingface/text-embeddings-inference:${TEI_TAG}
container_name: tei-rerank
restart: unless-stopped
ports:
- "${RERANK_PORT}:80"
volumes:
- /opt/aimodels/tei-cache:/data
environment:
- HF_TOKEN=${HF_TOKEN:-}
command:
- --model-id
- ${RERANK_MODEL}
- --dtype
- float16
- --max-client-batch-size
- "${MAX_CLIENT_BATCH_SIZE}"
# Fail-closed; bge-reranker-v2-m3's max input (8,192) fits the default
# max-batch-tokens (16,384).
- --auto-truncate
- "false"
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["0"]
capabilities: [gpu]
healthcheck:
test: ["CMD", "curl", "-fsS", "http://localhost:80/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 60s
labels:
- homepage.group=AI - Eval & Retrieval
- homepage.name=Rerank — bge-v2-m3 (TEI, ${HOST_NAME:-esh-ml1})
- homepage.icon=mdi-sort-variant
- homepage.description=Fleet reranker (reranker) via TEI on ${HOST_NAME:-esh-ml1}
- homepage.href=http://${HOST_IP:-10.0.50.80}:${RERANK_PORT}/docs
===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
/opt/docker/conf
===== LISTENING PORTS =====
*:22
*:45876
0.0.0.0:8001
0.0.0.0:8013
10.100.50.80:2375
10.100.50.80:7007
127.0.0.1:25
[::1]:25
[::]:8001
[::]:8013
===== MODEL / HUGGINGFACE CACHES =====
===== DOCKER-ADJACENT SYSTEMD SERVICES =====
container-getty@1.service running
container-getty@2.service running
containerd.service running
docker.service running
===== DONE =====
Paste the above back into the chat, or pass a path as argv[1] to save.
Reference in New Issue View Git Blame Copy Permalink