3.5 KiB
[2026-09-16] voices-seat: LoRA over merge, measured — and GPU 0 is now full
vllm-voices live on fv-ml1 GPU 0 :8027, one Qwen3-4B-Instruct carrier serving
voices-base plus lv-<author> LoRA adapters. stacks/voices-seat/, commit d17bd3d.
⭐ LORA COSTS 24.3% OF DECODE THROUGHPUT AND IT IS WORTH PAYING. n=30 per arm, interleaved,
A-vs-A noise floor 0.1%: base 143.0 tok/s median vs adapter 108.2. The measurement is
unusually clean because --enable-lora serves BOTH the base name and the adapter name from ONE
process — the arm is a per-request field, so no restart, no second seat, no cold-vs-warm confound.
Arms were interleaved rather than blocked because the card's co-tenants take traffic this seat
does not control, and a block design would alias their load onto one arm.
Why pay it: 3 authors cost 8.4 GB as adapters against ~23 GB merged; 6 cost 9.2 vs ~46. On a card with 1.8 GB free afterwards that is the whole argument. If a voice ever lands on a latency path, merge THAT one and serve it separately.
⭐ ADAPTER HOT-SWAP IS REAL AND FAST — MEASURED, not read from docs.
POST /v1/load_lora_adapter 200 in 0.24 s, POST /v1/unload_lora_adapter 200 in 0.003 s,
VRAM unchanged, container stayed healthy. Proven by performing the babyyarros→lv-yarros
rename through it with no restart. ⚠ A runtime-loaded adapter is GONE on the next
compose up -d unless it is also in --lora-modules (which costs a recreate + ~3 min reload).
Runtime load is for TRYING a voice; the compose list is what persists. Switching between loaded
voices is just the model field — not a LiteLLM alias; LiteLLM is a thinner layer on top,
one alias entry per voice, no new deployment.
⚠⚠ --gpu-memory-utilization IS A REQUEST AGAINST TOTAL VRAM THAT THE CARD MUST ALREADY BE
ABLE TO HONOUR — not a share of what is free. First bring-up REFUSED: "Free memory on device
cuda:0 (11.16/94.97 GiB) is less than desired GPU memory utilization (0.12, 11.4 GiB)". Refusing
was the right outcome — it protected cyberprev, gen-small and the Parakeet STT seat rather
than squeezing them.
⭐ PINNING --kv-cache-memory IN BYTES MAKES THE FRACTION PREDICTIVE. Requested 0.11
(10,700 MiB), got 10,740 MiB resident — a 40 MiB miss on a box where the fraction has been
wrong by 8–10 GB in BOTH directions (cyberprev 0.40→47.1 GB, gen-small 0.48→36.9 GB). Second
seat to prove it after gen-small. Do not remove the pin.
⚠ fv-ml1 GPU 0 is now 96.0 of 97.9 GB. GPU 1 ~5.7 free, GPU 2 ~2.4, GPU 3 is a HELD RESERVE
for a future full-card seat (flash-next alone needs 93 of 96 GiB). There is no room for
another seat on fv-ml1 without a placement decision.
⚠ SUPPORT WAS CHECKED, NOT ASSUMED, per the training playbook's own lesson that LoRA support
is per-ARCHITECTURE not per-family: vllm/model_executor/models/qwen3.py:271 declares
Qwen3ForCausalLM with SupportsLoRA plus packed_modules_mapping and embedding_modules.
Do not transplant this compose onto an MoE carrier without re-running that grep — the
playbook records a LoRA refusal on a Qwen3 MoE arch.
Naming (operator, 2026-09-16): lv-<author> — lv for lang-voice, retiring baby*, which
read fine for one experiment and invites confusion across a family. The adapter NAME is the
request's model field, so it is the public API of a voice. Historical persistent-memory entries
still say BabyYarros/BabyHemingway and were deliberately left as dated records.