feat(esh-ml1): RTX 2000E Ada on esh-pve serves embed + rerank as a LiteLLM failover
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia` retired. playbooks/esh-pve-nvidia-host.yaml. esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN, NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce + nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the vzdump job on purpose. DNS esh-ml1.esh.internal. stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking. litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2 behind fv-ml1 (order 1). Order fallback proven with throwaway groups: refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500. Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the fv-ml1 relocation (still named 10.250.50.54); documented the third unkillable homepage wedge on esh-docker-vm.
This commit is contained in:
@@ -73,3 +73,32 @@ Like `nh3-docker`, this host runs **Dozzle** and **Beszel** agents that report b
|
||||
## Placement rule
|
||||
|
||||
Home-lab workloads for the ESH site go here. Not part of the PFI colo topology.
|
||||
|
||||
## ⚠ Recurring: `homepage` wedges unkillably (3× — 2026-06-03, 2026-09-18, 2026-09-24)
|
||||
|
||||
The Homepage container stops answering (Uptime Kuma: `timeout of 16000ms
|
||||
exceeded`; healthcheck `Connecting to 127.0.0.1:3000` times out) and cannot be
|
||||
stopped. Signature captured 2026-09-24 ~2224 PT, guest kernel `6.1.0-41-amd64`:
|
||||
|
||||
- one `node` thread in **D state in `vm_mmap_pgoff`** — waiting for its own
|
||||
process's `mmap_lock` for write, with **no visible holder** (every other
|
||||
thread sat in `futex_wait`; a scan of every task's kernel stack found no
|
||||
reader in a fault, NFS or `access_remote_vm` path except the `ps` calls
|
||||
queued behind it).
|
||||
- `docker restart` → *"tried to kill container, but did not receive an exit
|
||||
event"*; the process then sits in **`exit_mmap`** (uninterruptible) with
|
||||
PID 1 of the container in `zap_pid_ns_processes`. The June entry in
|
||||
archival-memory records the same `exit_mmap` end state.
|
||||
- The rest of the VM is fine: DNS (AdGuard) answered, 17 other containers up.
|
||||
|
||||
⚠ **Diagnosing it can hang your shell.** `ps`, `pgrep` and `docker top` read
|
||||
`/proc/<pid>/cmdline|environ`, which takes the same lock, so they block in D
|
||||
state too. Read `/proc/<pid>/task/*/stat` and `sudo cat .../stack` instead
|
||||
(`ssh infra-ops@10.0.50.45`, NOPASSWD).
|
||||
|
||||
**Only a VM reboot clears it**, and this VM is ESH's DNS resolver, so the
|
||||
reboot is a short ESH-wide DNS outage — schedule it. Root cause not
|
||||
established; a kernel-side mmap_lock problem is the leading guess, not a
|
||||
finding. Moving Homepage to ana-docker (as was done for Uptime Kuma on
|
||||
2026-09-21 for the same box's history) would take the dashboard out of this
|
||||
failure domain.
|
||||
|
||||
Reference in New Issue
Block a user