vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0f032fac2c65a2b55514cb0b8370dfdb7140c03d
esh-pfi-infrastructure/servers/esh-ml1/system-details.txt
T
vh 5402568b76 feat(esh-ml1): RTX 2000E Ada on esh-pve serves embed + rerank as a LiteLLM failover 
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from
NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the
device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia`
retired. playbooks/esh-pve-nvidia-host.yaml.

esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN,
NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce +
nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the
vzdump job on purpose. DNS esh-ml1.esh.internal.

stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on
the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed
cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise
floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking.

litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2
behind fv-ml1 (order 1). Order fallback proven with throwaway groups:
refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500.

Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the
fv-ml1 relocation (still named 10.250.50.54); documented the third
unkillable homepage wedge on esh-docker-vm.
2026-09-24 22:38:55 -07:00

268 lines
18 KiB
Plaintext
Raw Blame History

===== HOST =====
Hostname: esh-ml1.esteban.net
Date: 2026-09-25T05:35:31+00:00
Uptime: up 20 minutes
OS: Debian GNU/Linux 12 (bookworm)
Kernel: 6.8.12-42-pve
Arch: x86_64
===== HARDWARE =====
CPU cores: 6
CPU model: 13th Gen Intel(R) Core(TM) i9-13900H
MemTotal: 16.0 GB
MemAvailable: 9.8 GB
===== GPUS =====
index, name, memory.total [MiB], memory.free [MiB], driver_version
0, NVIDIA RTX 2000E Ada Generation, 16380 MiB, 11072 MiB, 580.178.04
===== FILESYSTEMS (df) =====
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-vm--110--disk--0 79G 34G 41G 45% /
===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
===== TARGETED DATA PATHS =====
/opt (total: 3.3G)
total 20
drwxr-xr-x 5 root root 4096 2026-09-25 05:16 .
drwxr-xr-x 18 root root 4096 2026-09-25 05:14 ..
drwxrwsr-x 3 root docker 4096 2026-09-25 05:16 aimodels
drwx--x--x 4 root root 4096 2026-09-25 05:15 containerd
drwxrwsr-x 4 root docker 4096 2026-09-25 05:15 docker
/opt/docker (total: 32K)
total 16
drwxrwsr-x 4 root docker 4096 2026-09-25 05:15 .
drwxr-xr-x 5 root root 4096 2026-09-25 05:16 ..
drwxrwsr-x 3 root docker 4096 2026-09-25 05:16 compose
drwxrwsr-x 2 root docker 4096 2026-09-25 05:15 conf
/opt/docker/compose (total: 24K)
total 12
drwxrwsr-x 3 root docker 4096 2026-09-25 05:16 .
drwxrwsr-x 4 root docker 4096 2026-09-25 05:15 ..
drwxrwsr-x 2 root docker 4096 2026-09-25 05:17 embed-rerank
/opt/docker/conf (total: 4.0K)
total 8
drwxrwsr-x 2 root docker 4096 2026-09-25 05:15 .
drwxrwsr-x 4 root docker 4096 2026-09-25 05:15 ..
/var/lib/docker (total: 4.0K)
/srv (total: 4.0K)
total 8
drwxr-xr-x 2 root root 4096 2025-09-07 15:14 .
drwxr-xr-x 18 root root 4096 2026-09-25 05:14 ..
===== DOCKER =====
Server: 29.8.1 Client: 29.8.1
----- docker info -----
Containers: 2 (running 2, paused 0, stopped 0)
Images: 2
Runtimes: map[io.containerd.runc.v2:{{runc [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.3.0","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_LOONGARCH64","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG","SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true,"schemata":true,"monitoring":true},"memoryPolicy":{"modes":["MPOL_BIND","MPOL_DEFAULT","MPOL_INTERLEAVE","MPOL_LOCAL","MPOL_PREFERRED","MPOL_PREFERRED_MANY","MPOL_WEIGHTED_INTERLEAVE"],"flags":["MPOL_F_NUMA_BALANCING","MPOL_F_RELATIVE_NODES","MPOL_F_STATIC_NODES"]},"mountExtensions":{"idmap":{"enabled":true}},"netDevices":{"enabled":true}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.5.1-0-g8f2685a4","org.opencontainers.runc.version":"1.5.1"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]} nvidia:{{nvidia-container-runtime [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.3.0","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_LOONGARCH64","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG","SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true,"schemata":true,"monitoring":true},"memoryPolicy":{"modes":["MPOL_BIND","MPOL_DEFAULT","MPOL_INTERLEAVE","MPOL_LOCAL","MPOL_PREFERRED","MPOL_PREFERRED_MANY","MPOL_WEIGHTED_INTERLEAVE"],"flags":["MPOL_F_NUMA_BALANCING","MPOL_F_RELATIVE_NODES","MPOL_F_STATIC_NODES"]},"mountExtensions":{"idmap":{"enabled":true}},"netDevices":{"enabled":true}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.5.1-0-g8f2685a4","org.opencontainers.runc.version":"1.5.1"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]} runc:{{runc [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.3.0","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_LOONGARCH64","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG","SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true,"schemata":true,"monitoring":true},"memoryPolicy":{"modes":["MPOL_BIND","MPOL_DEFAULT","MPOL_INTERLEAVE","MPOL_LOCAL","MPOL_PREFERRED","MPOL_PREFERRED_MANY","MPOL_WEIGHTED_INTERLEAVE"],"flags":["MPOL_F_NUMA_BALANCING","MPOL_F_RELATIVE_NODES","MPOL_F_STATIC_NODES"]},"mountExtensions":{"idmap":{"enabled":true}},"netDevices":{"enabled":true}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.5.1-0-g8f2685a4","org.opencontainers.runc.version":"1.5.1"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]}]
Default runtime: runc
Storage driver: overlayfs
Root dir: /var/lib/docker
Server version: 29.8.1
----- running containers -----
NAMES IMAGE STATUS PORTS
vllm-embed vllm/vllm-openai:v0.24.0 Up 14 minutes (healthy) 0.0.0.0:8001->8000/tcp, [::]:8001->8000/tcp
vllm-rerank-bge vllm/vllm-openai:v0.24.0 Up 15 minutes (healthy) 0.0.0.0:8013->8000/tcp, [::]:8013->8000/tcp
----- all containers -----
NAMES IMAGE STATUS
vllm-embed vllm/vllm-openai:v0.24.0 Up 14 minutes (healthy)
vllm-rerank-bge vllm/vllm-openai:v0.24.0 Up 15 minutes (healthy)
----- networks -----
NAME DRIVER SCOPE
bridge bridge local
embed-rerank_default bridge local
host host local
none null local
----- networks (external, non-default — worth knowing for compose external: true) -----
embed-rerank_default
----- named volumes -----
VOLUME NAME DRIVER
----- compose projects currently running -----
embed-rerank
===== COMPOSE FILES (/opt/docker/compose/) =====
>>> /opt/docker/compose/embed-rerank/compose.yaml
# embed-rerank — the fleet's embedding + reranking models, served locally at ESH
# on esh-ml1 (CT 110 on esh-pve, RTX 2000E Ada, 16 GB).
#
# WHY: Prime's decision 2026-09-24 — the Ada card offloads embedding and
# reranking so ESH consumers (Open WebUI RAG, Paperless) are not tied to one
# seat on fv-ml1. It serves the SAME models as fv-ml1's `vllm` stack, with the
# SAME vLLM version and flags, because embedding vectors are model-specific:
# a different embedding model here would silently poison every index built
# against fv-ml1's. Mirror stacks/vllm/compose.yaml when that changes.
#
# vllm-embed Qwen/Qwen3-Embedding-0.6B → /v1/embeddings :8001
# vllm-rerank-bge BAAI/bge-reranker-v2-m3 → /rerank, /score :8013
#
# Ports match fv-ml1 on purpose, so a LiteLLM deployment for either site
# differs only in the host part of api_base.
#
# `vllm-rerank-bge`, not fv-ml1's `vllm-rerank-a3`: that name carries R43
# bake-off provenance for THAT container; and plain `vllm-rerank` was the
# retired Qwen3-Reranker that measured harmful. Name the model instead.
#
# NO `tnet`/traefik-net: esh-ml1 runs no traefik, and an external network
# that does not exist would stop the stack from starting. Consumers reach the
# published ports directly.
#
# Host setup (driver, LXC, docker, toolkit): playbooks/esh-pve-nvidia-host.yaml
# then playbooks/esh-ml1-lxc.yaml. Tunables live in .env — edit that, not this.
name: embed-rerank
services:
vllm-embed:
image: vllm/vllm-openai:${VLLM_VERSION}
container_name: vllm-embed
restart: unless-stopped
ipc: host
ports:
- "${EMBED_PORT}:8000"
volumes:
- /opt/aimodels/huggingface:/hfcache
environment:
- HF_HOME=/hfcache
- HF_HUB_CACHE=/hfcache/hub
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
- VLLM_API_KEY=${API_KEY:-}
command:
- ${EMBED_MODEL}
- --served-model-name
- ${EMBED_MODEL}
- --runner
- pooling
- --host
- 0.0.0.0
- --port
- "8000"
- --gpu-memory-utilization
- ${EMBED_GPU_MEM_UTIL}
- --max-model-len
- ${EMBED_MAX_MODEL_LEN}
- --dtype
- auto
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["0"]
capabilities: [gpu]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 180s
labels:
- homepage.group=AI - Eval & Retrieval
- homepage.name=vLLM Embed (esh-ml1)
- homepage.icon=mdi-vector-arrange-below
- homepage.description=Qwen3 Embedding 0.6B via vLLM (esh-ml1, RTX 2000E Ada)
- homepage.href=http://10.0.50.80:${EMBED_PORT}/docs
vllm-rerank-bge:
image: vllm/vllm-openai:${VLLM_VERSION}
container_name: vllm-rerank-bge
restart: unless-stopped
ipc: host
ports:
- "${RERANK_PORT}:8000"
volumes:
- /opt/aimodels/huggingface:/hfcache
environment:
- HF_HOME=/hfcache
- HF_HUB_CACHE=/hfcache/hub
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
- VLLM_API_KEY=${API_KEY:-}
command:
- ${RERANK_MODEL}
- --served-model-name
- ${RERANK_MODEL}
- --runner
- pooling
# bge-reranker-v2-m3 is natively a cross-encoder; no --hf-overrides.
- --host
- 0.0.0.0
- --port
- "8000"
- --gpu-memory-utilization
- ${RERANK_GPU_MEM_UTIL}
- --max-model-len
- ${RERANK_MAX_MODEL_LEN}
- --dtype
- auto
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["0"]
capabilities: [gpu]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 180s
labels:
- homepage.group=AI - Eval & Retrieval
- homepage.name=vLLM Rerank bge-v2-m3 (esh-ml1)
- homepage.icon=mdi-sort-variant
- homepage.description=bge-reranker-v2-m3 via vLLM (esh-ml1, RTX 2000E Ada)
- homepage.href=http://10.0.50.80:${RERANK_PORT}/docs
===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
/opt/docker/conf
===== LISTENING PORTS =====
*:22
0.0.0.0:8001
0.0.0.0:8013
127.0.0.1:25
[::1]:25
[::]:8001
[::]:8013
===== MODEL / HUGGINGFACE CACHES =====
===== DOCKER-ADJACENT SYSTEMD SERVICES =====
container-getty@1.service running
container-getty@2.service running
containerd.service running
docker.service running
===== DONE =====
Paste the above back into the chat, or pass a path as argv[1] to save.
Reference in New Issue View Git Blame Copy Permalink