vh
300ecc1276
Live on vllm-voices (fv-ml1 GPU0 :8027) beside voices-base, lv-yarros and lv-bronte.
Healthy 190 s after recreate, four models served, GPU0 96,092 -> 96,090 MiB. The adapter
was verified byte-identical to checkpoint-850 by sha256 across both transfer hops, and the
seat was verified by generating, not by reading its config: base emits 170 words of <think>
planning and never writes the passage, lv-hemingway writes the scene.
Gate design was pre-registered before any generation existed (0bb4938). Three arms, 60
held-out beats, 4 seeds, 240 generations per arm.
A. VOICE PASS 6.4x +0.413 delta_cb, pairwise floor 0.064 -- and it clears the OLD
all-arms floor (0.113) too, so this verdict does not lean on the
rule change. Closes 73.8% of the span between the unadapted
carrier and held-out Hemingway itself; lv-bronte closed 48%.
B. NOT COPIED see below
C. NO DAMAGE PASS ran-on +0.08, on-beat -0.14, both inside a 0.217 floor
AXIS B: THE NEGATIVE CONTROL WAS THE WRONG ONE, AND FIXING IT MADE THE RESULT WORSE, NOT
BETTER. memorization_check.py uses the base-unadapted arm as its control. Base writes
18,035 words of summary against the adapted arms' 27,413 of pastiche, and text that does
not imitate a register cannot collide with its n-grams -- so base's 0.00 measures "different
register", not "did not memorise". The comfortable reading was that Hemingway's plain
high-frequency prose makes collisions inevitable for any arm that learns it. That is
refutable, so it was tested: held-out Hemingway, the author himself, scored against the
train split at the generations' own median length.
HELD-OUT HEMINGWAY (never trained) 370 chunks 0.01 hit-rate mean-longest 0.1 max 10
base-unadapted 240 gens 0.00 0.0 0
ckpt850 (shipped) 240 gens 0.07 0.6 9
positive control (train vs train) 160
The hypothesis is false: the adapter reproduces train n-grams ~7x more often than the
author reproduces himself. That is real and is on the record. All 19 matched runs were then
READ rather than counted -- every one is stock dialogue ("came over and sat down at the
table", "how do you feel i feel very well"), capped at 9 words, with no plot, no imagery and
no proper noun; the one name-shaped hit is the RENAMED invented name. Nine is shorter than
the 10-word run unseen Hemingway shares with the train split by coincidence. Elevated rate,
zero protectable content. Hemingway is in copyright; lv-yarros is the in-line precedent,
also in copyright, shipped at 0.10 against a 0.07 control. Unload is 0.003 s.
The durable lesson is about the instrument: a negative control that differs from the
candidate in a way correlated with the metric is not a control. memorization_selfsim.py and
memorization_dump_matches.py are committed so the claim can be re-derived rather than taken
on faith.
SHIPPED ckpt850, NOT the loss minimum at step 1750. The two are indistinguishable on voice
-- 0.072 apart against a 0.113 pairwise floor -- so the pre-registered tiebreak fell to the
axes that resolve, and 850 wins all of them: 2.3x tighter seed spread (0.050 vs 0.113),
lower memorisation, less ran-on, half an epoch less overfit. ckpt1750's spread is one seed
(0.491, 0.449, 0.468, then 0.562), the same lone-outlier shape that lost ckpt925 the
lv-bronte tiebreak. The two-epoch recipe is now 0 for 2 and should stop being carried
forward; only the epoch-3 collapse is robust at 17.4x jitter.
servers/fv-ml1/ssh-target was a bare IP, so deploy-stack.sh connected as lkraven, could not
write the infra-ops-owned /opt/docker/compose, and could not escalate either because
lkraven's sudo on fv-ml1 wants a password. Now infra-ops@10.251.50.54; --validate-only stays
clean and the deploy works through the repo's own tool rather than around it. Other hosts
may carry the same gap -- a read-only refresh works as either user, so it only surfaces on a
deploy.
fv-ml1
Primary AI inference host for PFI.
Network
- LAN IP: 10.251.50.54 (in-band, OS-side)
- BMC (OOB): 10.251.250.50 — Supermicro IPMI web UI at https://10.251.250.50 (homepage card: PFI-ANA-ML2 BMC)
- SSH: standard port 22 on 10.251.50.54
Hardware
- Chassis: Supermicro mid-range inferencing server (bare metal, NOT Dell / not the same box as sf-r630 / sfsrv-ana)
- CPU: AMD EPYC 9254 24-core (96 threads)
- RAM: 566 GB
- GPUs: 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
- Storage: ZFS
zroot(434 GB root) +tankpool (raidz2, 8× NVMe, 8.6 TB at/tank) — drive inventory below - OS: Debian 13 (trixie), kernel 6.12.x
- Docker: 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
NVMe drive inventory (tank, raidz2-0) — read 2026-09-09 via nvme-cli
All eight are Dell Express Flash PM1725b 1.6 TB SFF (Samsung OEM), PCIe 3.0 x4 behind
a Broadcom PEX switch. Two provenance batches: the S5CU… six (fw 1.2.2) and the
S47V… pair (fw 1.2.0 / 1.2.1) with thousands of prior-life power cycles.
| dev | PCI | serial | fw | pwr-on h | pwr cycles | unsafe shut. | media err | used |
|---|---|---|---|---|---|---|---|---|
| nvme0 | 46:00.0 | S5CUNEUMB05672 | 1.2.2 | 33856 | 196 | 175 | 0 | 0% |
| nvme1 | — | S5CUNEUMB05671 | 1.2.2 | 33856 | 199 | 178 | 0 | 0% |
| nvme2 | — | S5CUNEUMB05694 | 1.2.2 | 15688 | 90 | 75 | 0 | 0% |
| nvme3 | — | S5CUNEUMB05667 | 1.2.2 | 33857 | 197 | 176 | 0 | 0% |
| nvme4 | — | S5CUNEUMB05674 | 1.2.2 | 33856 | 198 | 177 | 0 | 0% |
| nvme5 | c6:00.0 | S47VNY0K600270 | 1.2.1 | 18823 | 5357 | 5342 | 0 | 1% |
| nvme6 | — | S5CUNEUMB05697 | 1.2.2 | 15570 | 88 | 73 | 0 | 0% |
| nvme7 | 07:00.0 (slot 0-5) | S47VNY0K600221 | 1.2.0 | 19525 | 3093 | 3083 | 2084 | 2% |
⚠ nvme7 was ABSENT from every boot 2026-04-23 → 2026-09-05 (kernel enumerated 7
NVMes per boot; PCIe downstream port 02:04.0 had nothing on bus 07). It reappeared at
the 09-05 14:26 cold boot, the pool resilvered 638 GB, and 2 CKSUM errors landed on it
at import. While it was missing tank was DEGRADED, and Debian's zfsutils-linux cron
(/usr/lib/zfs-linux/{scrub,trim}) only touches pools whose health is ONLINE, so tank
got no scrub and no trim from 04-12 to 09-06. ZED's ZED_EMAIL_ADDR=root has no
MTA behind it, so the 4½-month degradation alerted nobody. media_errors=2084 on
nvme7 is a lifetime counter.
Settled by the 2026-09-09 scrub (00:29–02:02 PT, scrub repaired 0B in 01:32:44 with 0 errors, then zpool clear tank → CKSUM 2 → 0): media_errors read 2084
before and 2084 after a full 6.84 TiB verify, so the counter is prior-life
history, not an active fault, and the 2 CKSUM were the stale-block artefact of the
09-05 late resilver. nvme7 stays in service; watch the counter at every visit and
replace on growth (zpool replace tank nvme7n1 <new>; any PM1725b 1.6 TB or
larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on
visit — a bay that dropped a drive for 4½ months is the likelier fault than the
drive. Playbook: playbooks/fv-ml1-pool-health.yaml (idempotent; rerunning is a
no-op). ⚠ Nothing alerts on this — see the open follow-up in
persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md.
Key paths
| Path | Purpose |
|---|---|
/opt/docker/compose/<stack>/ |
Compose files |
/opt/docker/conf/<stack>/ |
Config bind mounts |
/tank/aimodels/huggingface/ |
HF cache (267 GB, pre-downloaded models) |
/tank/aimodels/llm/ |
Legacy GGUF models (790 GB, referenced by llama-swap as /models/) |
/var/lib/docker/ |
Docker data (on zroot) |
Running stacks
Live inventory as of 2026-07-22. Each model is its own compose stack now
(container vllm-<x> / llama-<x>); the vllm stack proper is just the
embed/rerank/reward trio. GPUs are pinned per container via
deploy.resources.reservations.devices[].device_ids.
GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):
| Container | Port | Served model | Quant | Ctx |
|---|---|---|---|---|
vllm-gen (project gen-seat) |
8015 | qwen3.8-27b-uncensored — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP) |
NVFP4 W4A16 (compressed-tensors) | 262k |
vllm-charrp-reasoning-nvfp4 |
8018 | char-rp-reasoning (R36 reasoning RP) |
NVFP4 (modelopt) | 256k |
GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):
| Container | Port | Served model | Quant | Ctx |
|---|---|---|---|---|
vllm-granite |
8004 | granite-4.1-8b — fleet summarizer/classifier |
FP8 (compressed-tensors) | 131k |
llama-charrp |
8016 | Magidonia-24B-v4.3 Q6_K — char-RP (llama.cpp) |
GGUF Q6_K | — |
vllm-selene |
RETIRED 2026-08-23 — lost a head-to-head against gen on its own judge task (see stacks/selene/README.md); seat downed to reclaim 17.2 GiB on GPU 1. selene-1-mini-8b now 404s by design; use chat-judge. |
— | — | |
vllm-reward |
8003 | Skywork-Reward-V2-Llama-3.1-8B-AWQ — reward classifier |
AWQ | 16k |
vllm-embed |
8001 | Qwen3-Embedding-0.6B |
— | 8k |
vllm-rerank |
8002 | Qwen3-Reranker-0.6B |
— | 8k |
Infra / non-GPU:
| Container | Port | Notes |
|---|---|---|
dockge |
5001 | Docker stack management UI |
dozzle-agent |
7007 | Log agent → Dozzle hub on ana-docker |
beszel-agent |
45876 | Metrics agent → Beszel hub on ana-docker |
Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing it on the card with room or evicting a dormant one first.
Dormant (compose present on disk, containers stopped) — rollback / audition
seats, safe to leave: mistral-medium-3.5, mistral-small-4(-heretic),
ms32-24b-angel, qwen3.5-122b, qwopus3.5-122b, qwen35-vl, qwen36-vl,
qwen36-27b-aeon, qwen-image-bench, vibevoice, comfyui, kokoro,
vllm-qwen3.
Also on GPU 0 (non-vLLM):
| Container | Port | Serves | Notes |
|---|---|---|---|
parakeet |
8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM ext-stt / whisper-1. Relocated from irv-ml1 2026-09-15. ~800 MiB. stacks/parakeet/. |
⚠ GPU 3 is deliberately kept EMPTY (2 MiB). It is the only card that can still
take a full-size seat — flash-next needs 93 GiB of 96 — and vLLM sizes its KV
cache against total VRAM rather than free VRAM, so even a sub-1 GB tenant there
eats into a future big seat's profiling margin. Small seats go on GPU 0, which has
the most uncommitted headroom (its seats commit util 0.88; GPU 1 is at 0.975 and
GPU 2 at 0.96).
Retired:
llama-swap(former GGUF multiplexer on :9292) — replaced by dedicated per-model seats (e.g.llama-charrp); no longer running.infinity— replaced by thevllmstack (originallyvllm-qwen3, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping atransformersbuild that knew Qwen3.LibreChat (+ rag_api, vectordb, mongodb, meilisearch),searxng— removed from this host (searxng now on ana-docker fleet-wide).
Refresh state
scripts/refresh-server-info.sh fv-ml1
Latest snapshot: system-details.txt (regenerate as needed).
GPU allocation policy
Every seat is explicitly pinned via device_ids (no unpinned containers), and
both cards run ~90% full:
- GPU 0: the two heavy NVFP4 seats —
vllm-gen(gen) andvllm-charrp-reasoning-nvfp4. The live serving path (near-100% util under load), ~42 + 45 GB. - GPU 1: everything else — reward, embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle between calls, ~91 GB resident.
Pin with deploy.resources.reservations.devices[].device_ids: ["<id>"] in
compose. Each service caps its share with --gpu-memory-utilization; with both
cards near-full, placing a new seat means freeing room (evict a dormant one) or
trimming a neighbour's utilization first.