51 lines
3.5 KiB
Markdown
51 lines
3.5 KiB
Markdown
# `[2026-09-16]` voices-seat: LoRA over merge, measured — and GPU 0 is now full
|
||
|
||
**`vllm-voices` live on fv-ml1 GPU 0 :8027**, one Qwen3-4B-Instruct carrier serving
|
||
`voices-base` plus `lv-<author>` LoRA adapters. `stacks/voices-seat/`, commit `d17bd3d`.
|
||
|
||
⭐ **LORA COSTS 24.3% OF DECODE THROUGHPUT AND IT IS WORTH PAYING.** n=30 per arm, interleaved,
|
||
A-vs-A noise floor **0.1%**: base **143.0 tok/s** median vs adapter **108.2**. The measurement is
|
||
unusually clean because `--enable-lora` serves BOTH the base name and the adapter name from ONE
|
||
process — the arm is a per-request field, so no restart, no second seat, no cold-vs-warm confound.
|
||
Arms were **interleaved rather than blocked** because the card's co-tenants take traffic this seat
|
||
does not control, and a block design would alias their load onto one arm.
|
||
|
||
**Why pay it:** 3 authors cost 8.4 GB as adapters against ~23 GB merged; 6 cost 9.2 vs ~46. On a
|
||
card with 1.8 GB free afterwards that is the whole argument. If a voice ever lands on a latency
|
||
path, merge THAT one and serve it separately.
|
||
|
||
⭐ **ADAPTER HOT-SWAP IS REAL AND FAST — MEASURED, not read from docs.**
|
||
`POST /v1/load_lora_adapter` **200 in 0.24 s**, `POST /v1/unload_lora_adapter` **200 in 0.003 s**,
|
||
VRAM unchanged, container stayed healthy. Proven by performing the `babyyarros`→`lv-yarros`
|
||
rename through it with no restart. ⚠ **A runtime-loaded adapter is GONE on the next
|
||
`compose up -d`** unless it is also in `--lora-modules` (which costs a recreate + ~3 min reload).
|
||
Runtime load is for TRYING a voice; the compose list is what persists. Switching between loaded
|
||
voices is just the `model` field — **not** a LiteLLM alias; LiteLLM is a thinner layer on top,
|
||
one alias entry per voice, no new deployment.
|
||
|
||
⚠⚠ **`--gpu-memory-utilization` IS A REQUEST AGAINST *TOTAL* VRAM THAT THE CARD MUST ALREADY BE
|
||
ABLE TO HONOUR — not a share of what is free.** First bring-up REFUSED: *"Free memory on device
|
||
cuda:0 (11.16/94.97 GiB) is less than desired GPU memory utilization (0.12, 11.4 GiB)"*. Refusing
|
||
was the right outcome — it protected `cyberprev`, `gen-small` and the Parakeet STT seat rather
|
||
than squeezing them.
|
||
|
||
⭐ **PINNING `--kv-cache-memory` IN BYTES MAKES THE FRACTION PREDICTIVE.** Requested 0.11
|
||
(10,700 MiB), got **10,740 MiB** resident — a 40 MiB miss on a box where the fraction has been
|
||
wrong by **8–10 GB in BOTH directions** (cyberprev 0.40→47.1 GB, gen-small 0.48→36.9 GB). Second
|
||
seat to prove it after `gen-small`. Do not remove the pin.
|
||
|
||
⚠ **fv-ml1 GPU 0 is now 96.0 of 97.9 GB.** GPU 1 ~5.7 free, GPU 2 ~2.4, GPU 3 is a HELD RESERVE
|
||
for a future full-card seat (`flash-next` alone needs 93 of 96 GiB). **There is no room for
|
||
another seat on fv-ml1 without a placement decision.**
|
||
|
||
⚠ **SUPPORT WAS CHECKED, NOT ASSUMED**, per the training playbook's own lesson that LoRA support
|
||
is per-ARCHITECTURE not per-family: `vllm/model_executor/models/qwen3.py:271` declares
|
||
`Qwen3ForCausalLM` with `SupportsLoRA` plus `packed_modules_mapping` and `embedding_modules`.
|
||
**Do not transplant this compose onto an MoE carrier without re-running that grep** — the
|
||
playbook records a LoRA refusal on a Qwen3 MoE arch.
|
||
|
||
**Naming (operator, 2026-09-16):** `lv-<author>` — lv for **lang-voice**, retiring `baby*`, which
|
||
read fine for one experiment and invites confusion across a family. The adapter NAME is the
|
||
request's `model` field, so it is the public API of a voice. Historical persistent-memory entries
|
||
still say BabyYarros/BabyHemingway and were deliberately left as dated records.
|