- qwen3-embedding and reranker: second deployment on nh3-ml1 (config);
reranker-a3-bge-v2-m3: second DB deployment via /model/new.
- router_settings.enable_weighted_failover: true. Only the three
multi-deployment groups are affected.
- Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd;
with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage;
nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
For brokkr's dataset foundry (operator-approved 2026-09-26, relayed).
- stacks/lfm-vl-seat: llama.cpp server-cuda b11176 (digest-pinned), Q5_K_M +
mmproj Q8_0, :8030; gateway alias lfm25-vl-3b (LiteLLM restarted, 36 s).
Positive control exact; null control shows it describes a missing image.
- stacks/vibevoice-asr-seat: audio.cpp v0.8.2-audio8-perf-hotfix (the GGUF's
own runtime, not vibevoice.cpp) on cuda 12.8 runtime + libgomp + libsoxr,
sha256-pinned; :8031 direct. LibriSpeech WER 3/69, RTF 0.07-0.14; ~31 s
cold first request.
Same model revision (df3ce67, blob sha matches fv-ml1), vLLM v0.24.0 and flags;
0.33 of the Ada (~5.4 GB, fv-ml1's absolute budget). Healthy, KV 79,808 tokens.
Parity (teacher-forced true code, 40 FIM prompts, 2,215 tokens, cache-salted):
each host is bit-exact with itself; cross-host |dlogprob| median 0.049, top-1
agreement 0.966 (Blackwell vs Ada + fp8 KV); against ground truth no quality
difference (mean logprob diff +0.008 +/- 0.019 SE). Speed on-box: ~5x slower
(64-tok FIM p50 ~1.0 s vs ~0.2 s; 63 vs 338 tok/s). Gateway coder-fast left on
fv-ml1 pending Prime's call.
- AMT re-IP'd over WS-Man to static 10.100.250.61/24 (gw/DNS .1): AMT keeps
its old DHCP lease across a VLAN move, so it was unreachable until changed
(reached via a temporary on-link /32 on vmbr0 + ssh tunnel, since removed).
- KVM redirection enabled, redirection listener on, IPS_OptInService
OptInRequired=0; all read back. Password vaulted as nh3-pve/amt-admin.
- scripts/amt-wsman.py: stdlib WS-Man get/put/invoke client (digest auth,
AMT 16 legacy-renegotiation TLS), password from $AMT_PW.
- PFI-UDMSE port 6 override: native nh3-mgmt, tagged VLANs blocked (was
forward all / native default). Reservation nh3-pve-amt -> 10.100.250.61.
- nh3-pve: arp_ignore=8 / arp_announce=2 on enp88s0 (now on vmbr0's untagged
L2) so the host never answers ARP for 10.100.250.60 with the AMT port's MAC.
- DNS nh3-pve-amt.nh3.internal -> 10.100.250.61.
- Homepage: NH3-PVE-AMT card under Infra - NH3 (no siteMonitor/ping: AMT drops
ICMP and its legacy-renegotiation TLS fails Homepage's fetch).
- AMT keeps its old 10.100.0.151 lease until rebind/expiry (~1920-2224 PT
2026-09-26); it does not re-DHCP on a VLAN change or link drop (measured).
AMT had no IP because Linux had enp88s0 admin-down (it left vmbr0 on 09-25 and
had no auto stanza); igc powers off the I226-LM PHY when down, so AMT had no
link. With the port up (no IP, unbridged) AMT took a lease within minutes.
- nh3-pve: `auto enp88s0` (manual, unbridged) and sysctl.d disable_ipv6 on it;
backup /root/interfaces.bak-20260925-amt; runtime already matched, no reload.
- UDM: reservation nh3-pve-amt -> 10.100.0.151 (untagged nh3-default; the
cable is on an nh3-sw1 trunk, not the UDM).
- DNS nh3-pve-amt.nh3.internal. AMT 16.1.25 answers TLS :16993 and :664.
nh3-ml1's first Temperature alert (2026-09-25 2105, 88.8 C) was nh3-pve's CPU
package during the nightly vzdump; the GPU sat at 47 C. hwmon is not namespaced,
so an LXC's agent reports its host's coretemp sensors under the LXC's name.
- hosts/{nh3,esh}-ml1.yaml: SENSORS=-coretemp_*,acpitz (blacklist); GPU and
NVMe temperatures verified still reported.
- Hub: Temperature >95 C / 5 m on nh3-pve and esh-pve for the CPUs (i9-13900H
TjMax 100 C; nh3-pve 2-hour averages reach 88 C on busy days).
- nh3-pve README: runs warmer than esh-pve; workload confounds it, check
airflow on the next visit.
NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled.
- nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml.
- nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4)
deployed with HOST_NAME/HOST_IP labels.
- Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min
0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control
0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed
identical within rep spread.
- gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix#7006).
With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl
reopen; applied on nh3-pve (one package).
- pve-nvidia-host.yaml: document that the headers meta drags in the newest
kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot).
- Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage
nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal.
- nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels,
AMT cabled but unreachable on the network.
Gateway routing to nh3-ml1 is not changed.
- esh-pve-nvidia-host -> pve-nvidia-host: headers/dkms/build-essential step,
nouveau blacklist + guarded unload (refuses if nouveau bound a device)
- esh-ml1-lxc -> gpu-lxc: host vars have no defaults (elway aborts on undefined),
rootfs storage/startup order parameterized, CT kept out of all-guests vzdump jobs
- embed-rerank: Homepage labels take HOST_NAME/HOST_IP, defaults = esh-ml1
nh3-pve and esh-pve are the same Minisforum MS-01 (BIOS AHWSA.1.17). With a
card in the x16 slot its root port takes bus 01 and every NIC moves down a
bus (measured on esh-pve), so predictable names change (enp2s0f0np0 ->
enp3s0f0np0 etc.) and vmbr0 would boot with no uplink. systemd .link files
now pin all NICs by MAC, baked into every initramfs and synced to the ESP;
udev confirms the files apply. The AMT-capable I226-LM (enp88s0) leaves
vmbr0's bridge-ports in the file (next boot), so cabling it for AMT cannot
loop the STP-less bridge.
Also: documented the NanoKVM (https://10.100.250.171) as nh3-pve's console
OOB and that AMT is not wired; nh3-dev's Beszel agent no longer binds NAS
shares (it died on the last NH3 cold start); post-boot checklist in
persistent-memory.
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system
registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples
verified (RTX 2000E Ada util, VRAM, power, temp).
- Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013)
services — the sole backends behind the gateway, so the one exception to
"seats are out of Kuma's lane". Reward seat excluded (no consumer).
- Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step);
esh-ml1-docker added to docker.yaml.
- Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still
named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed
to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing).
- Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an
automounted NAS share); revived by hand, fix still open.
- Audit: Skywork-Reward-V2-Llama-3.1-8B is still #1 of 188 on AllenAI's
RewardBench 2 per-sample results; no Skywork V3; the -40M sibling is
vendor-marked experimental. Our AWQ W4A16 quant: 0.847 vs published bf16
0.860 on a 150-prompt sample (within +/-2.9 pt SE), 96.2% pairwise
agreement. Double BOS from vLLM on pre-templated text costs a further
~2.7 pts; callers must send add_special_tokens=false.
- No working consumer: 0 requests since 2026-09-13; Worldtree Domari points
at a dead IP with a non-vLLM schema (reported to worldtree-dev).
- Move: sha256-identical model copy; vLLM v0.24.0 on esh-ml1 :8003 at 0.55
util (KV 1.30x of a 16k request). Parity vs fv-ml1: 149/150 verdicts,
99.8% pairwise signs, raw |delta| median 0.049.
- Gateway /scalar-judge passthrough -> 10.0.50.80:8003; fv-ml1 vllm-reward
removed (~10.2 GB freed on GPU 1). stacks/vllm now holds only vllm-coder.
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.
- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
8001/8013, fail-closed truncation (--auto-truncate false; embed
--max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
`texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
CLAUDE.md gains the TEI convention.
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from
NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the
device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia`
retired. playbooks/esh-pve-nvidia-host.yaml.
esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN,
NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce +
nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the
vzdump job on purpose. DNS esh-ml1.esh.internal.
stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on
the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed
cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise
floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking.
litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2
behind fv-ml1 (order 1). Order fallback proven with throwaway groups:
refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500.
Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the
fv-ml1 relocation (still named 10.250.50.54); documented the third
unkillable homepage wedge on esh-docker-vm.
After the 2026-09-24 NH3 outage pfi-gx10 stayed off. It has no BMC, and the
ASUS firmware defaults to staying off after AC loss. With Prime's approval,
bytes 3-4 of the UEFI Setup variable were changed from 0x1d,0x00 to
0x1c,0x01, a community recipe that matches factory power-on units. The
readback shows exactly those two bytes changed. The original 85-byte raw
variable (attr header included) is committed here and also kept on the box.
The README records that an OS shutdown is not a test of this setting. It
stays off by design, and two such "tests" were invalid. The only valid test
is pulling and restoring AC, which Prime will do on 2026-09-25. It also
records the revert procedure.
After the 2026-09-24 NH3 power outage every NFS line from nh3-nas failed at
boot: nh3-docker's /mnt/compose and /mnt/backup, and nh3-dev's /mnt/backup.
The NAS is the slowest box to serve NFS, and the plain fstab mounts stayed
down until remounted by hand. nh3-dev's /mnt/smithy already used
x-systemd.automount, and the same boot shows it self-healing on its next
access. playbooks/nh3-nfs-automount.yaml brings the other lines to that
shape (_netdev,nofail,x-systemd.automount,x-systemd.mount-timeout=30).
Hard-mount semantics are unchanged; only WHEN the mount happens moves.
Applied and read back on nh3-docker and nh3-dev: every nh3-nas line carries
automount, each automount unit is active, and each share is NFS-mounted after
access. A second run on nh3-docker is a no-op. The fstab check judges only
errors the rewrite ADDS, because nh3-dev's installer cdrom line has a
pre-existing verify error. The first run caught that and aborted without
writing.
servers/nh3-pve/README.md now lists each guest's expected state after a
power loss: nh3-laser is on-demand and stays off (Prime), pbs-nh3 now has
onboot=1 (it had none and stayed down), and pfi-gx10 is bare metal that
does not auto-power-on.
Operator directive 2026-09-24: clock times are written in 24-hour local
(US Pacific) format, four digits, no colon. Claude seats get it from the
global CLAUDE.md (dotfiles fe63a67). Codex (~/.codex/AGENTS.md) and Grok
(~/.grok/rules/00-fleettools.md) read this file through symlinks, so the
same rule goes under Non-negotiables here.
Operator ruling 2026-09-24. On ana-docker the stack is `docker compose
down`: the container is removed and port 7878 is closed. Kept for revival:
- the data dir /opt/docker/conf/task-board/data (tasks.db, last written
2026-09-11)
- the task-board:local image
- stacks/task-board/ and the host's compose + .env
The Uptime Kuma monitor (id 3) was deleted before the stop so it could
not page, and its row is removed from monitors.yaml. Homepage drops the
card on its own, since it reads the container's labels.
Hooks: the container log showed no hook POSTs in 30 days. The only
traffic was open browser tabs holding /events, and the plugin was already
uninstalled on nh3-dev. Removed the paragraph that told sessions to call
task_* tools (CLAUDE.md, and the fork-fleet.sh template that seeds new
repos), the vestigial TASK_BOARD_SESSION env in .claude/settings.json, and
the listings in README and FLEETTOOLS.
Global CLAUDE.md now defines how to reach Miranda: every message becomes a
summary on Prime's phone, URGENT means she phones him, and no noise. The
project section had prefixed every message URGENT and said 'when unsure,
send it', which contradicted that. It now points to the global section for
mechanics and keeps only what is infra-ops-specific: the standing order
covers matters already fixed, the consequence test, and the channel's
dependency on svos.service and hermes-gateway (a routine svos restart is
about 100s of deafness).
vh