- esh-pve-nvidia-host -> pve-nvidia-host: headers/dkms/build-essential step,
nouveau blacklist + guarded unload (refuses if nouveau bound a device)
- esh-ml1-lxc -> gpu-lxc: host vars have no defaults (elway aborts on undefined),
rootfs storage/startup order parameterized, CT kept out of all-guests vzdump jobs
- embed-rerank: Homepage labels take HOST_NAME/HOST_IP, defaults = esh-ml1
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system
registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples
verified (RTX 2000E Ada util, VRAM, power, temp).
- Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013)
services — the sole backends behind the gateway, so the one exception to
"seats are out of Kuma's lane". Reward seat excluded (no consumer).
- Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step);
esh-ml1-docker added to docker.yaml.
- Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still
named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed
to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing).
- Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an
automounted NAS share); revived by hand, fix still open.
- Audit: Skywork-Reward-V2-Llama-3.1-8B is still #1 of 188 on AllenAI's
RewardBench 2 per-sample results; no Skywork V3; the -40M sibling is
vendor-marked experimental. Our AWQ W4A16 quant: 0.847 vs published bf16
0.860 on a 150-prompt sample (within +/-2.9 pt SE), 96.2% pairwise
agreement. Double BOS from vLLM on pre-templated text costs a further
~2.7 pts; callers must send add_special_tokens=false.
- No working consumer: 0 requests since 2026-09-13; Worldtree Domari points
at a dead IP with a non-vLLM schema (reported to worldtree-dev).
- Move: sha256-identical model copy; vLLM v0.24.0 on esh-ml1 :8003 at 0.55
util (KV 1.30x of a 16k request). Parity vs fv-ml1: 149/150 verdicts,
99.8% pairwise signs, raw |delta| median 0.049.
- Gateway /scalar-judge passthrough -> 10.0.50.80:8003; fv-ml1 vllm-reward
removed (~10.2 GB freed on GPU 1). stacks/vllm now holds only vllm-coder.
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.
- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
8001/8013, fail-closed truncation (--auto-truncate false; embed
--max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
`texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
CLAUDE.md gains the TEI convention.
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from
NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the
device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia`
retired. playbooks/esh-pve-nvidia-host.yaml.
esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN,
NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce +
nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the
vzdump job on purpose. DNS esh-ml1.esh.internal.
stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on
the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed
cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise
floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking.
litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2
behind fv-ml1 (order 1). Order fallback proven with throwaway groups:
refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500.
Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the
fv-ml1 relocation (still named 10.250.50.54); documented the third
unkillable homepage wedge on esh-docker-vm.
vh