voices-seat: one carrier, lv-<author> LoRA adapters, with the measured cost and placement limits

2026-09-16 13:51:51 -07:00
parent 6282833669
commit d17bd3df86
2 changed files with 210 additions and 0 deletions
@@ -0,0 +1,91 @@
+# voices-seat — author voice adapters on one carrier
+
+`fv-ml1` GPU 0, port **8027**. One Qwen3-4B-Instruct base; each author is a LoRA adapter
+named `lv-<author>` (**lv = lang-voice**). Switching voices is a request field, not a
+deployment.
+
+```bash
+curl http://10.251.50.54:8027/v1/chat/completions -H 'Content-Type: application/json' \
+  -d '{"model":"lv-yarros","messages":[{"role":"user","content":"Beat: ..."}]}'
+#         ^^^^^^^^^ the only thing that changes between voices
+```
+
+`voices-base` serves the unadapted carrier from the same process, which is what makes an
+adapter-on / adapter-off comparison harness-matched.
+
+## Adding a voice
+
+Two paths, and they are not interchangeable.
+
+**Try one now — no restart, ~0.24 s:**
+
+```bash
+curl -X POST http://10.251.50.54:8027/v1/load_lora_adapter -H 'Content-Type: application/json' \
+  -d '{"lora_name":"lv-hemingway","lora_path":"/adapters/lv-hemingway-4b-v1"}'
+curl -X POST http://10.251.50.54:8027/v1/unload_lora_adapter -H 'Content-Type: application/json' \
+  -d '{"lora_name":"lv-hemingway"}'
+```
+
+⚠ **A runtime-loaded adapter is GONE on the next `compose up -d`.** To make it survive, add
+it to `--lora-modules` in `compose.yaml` — which costs a container recreate and a full model
+reload (~3 min). Runtime load is for *trying* a voice; the compose list is what persists.
+
+Adapters live on the host at `/tank/aimodels/voice-adapters/<name>/`, mounted read-only at
+`/adapters`. A rename on the host is visible inside immediately.
+
+## Reaching it through LiteLLM
+
+Each voice is one alias entry pointing at this seat with its own `model` value — no new
+deployment, no new container, no VRAM:
+
+```yaml
+- model_name: lv-yarros
+  litellm_params:
+    model: openai/lv-yarros
+    api_base: http://10.251.50.54:8027/v1
+```
+
+⚠ **Retiring a voice orphans scoped keys.** Any LiteLLM key whose allowlist names a removed
+alias starts returning silent per-endpoint 403s. Audit `/key/list` + `/key/info` on every
+repoint — that failure has cost this fleet two and a half months before.
+
+## What it costs, measured
+
+| | |
+|---|---|
+| decode, base | **143.0 tok/s** (median, n=30) |
+| decode, LoRA | **108.2 tok/s** (median, n=30) |
+| **LoRA overhead** | **−24.3%**, against an A-vs-A noise floor of **0.1%** |
+| resident VRAM | **10,740 MiB** |
+| KV cache | 10,912 tokens → 1.33x concurrency at 8k context |
+
+Arms were interleaved rather than blocked, because the co-tenants on that card take traffic
+this seat does not control and a block design would alias their load onto one arm.
+
+**The 24.3% is accepted deliberately.** Three authors cost 8.4 GB as adapters and ~23 GB
+merged; six cost 9.2 GB versus ~46 GB. On a card with 1.8 GB free after this seat, that is
+the whole argument. If a voice ever lands on a latency path, merge that one and serve it
+separately.
+
+## Placement warnings
+
+⚠ **GPU 0 is now at 96.0 of 97.9 GB.** This seat's 10.7 GB went into the last real gap on
+fv-ml1: GPU 1 has ~5.7 GB, GPU 2 has ~2.4 GB, and GPU 3 is a held reserve for a future
+full-card seat (`flash-next` alone needs 93 of 96 GiB). **There is no room for another seat
+here without a placement decision.**
+
+⚠ **`--gpu-memory-utilization` is a request against TOTAL VRAM that the card must already be
+able to honour** — not a share of what is free. The first bring-up refused at 0.12 with
+11.16 GiB free, and refusing was correct: it protected `cyberprev`, `gen-small` and the
+Parakeet STT seat rather than squeezing them.
+
+⚠ **Pin `--kv-cache-memory` in bytes.** With it, the requested fraction predicted residency
+to within 40 MiB. Without it, this box has been wrong by 8–10 GB in *both* directions.
+
+## Support was checked, not assumed
+
+`vllm/model_executor/models/qwen3.py` declares `Qwen3ForCausalLM` with `SupportsLoRA` plus
+`packed_modules_mapping` and `embedding_modules`. The training playbook records a LoRA
+refusal on a Qwen3 **MoE** architecture, and its lesson is that support is per-architecture,
+not per-family. **Do not transplant this compose onto an MoE carrier without re-running that
+check.**
@@ -0,0 +1,119 @@
+# voices-seat — one base carrier, N author LoRA adapters, hot-swappable. fv-ml1 GPU 0, :8027.
+#
+# NAMING: adapters are `lv-<author>` — lv for lang-voice (operator, 2026-09-16, retiring the
+# `baby*` prefix: it read fine for one experiment and invites confusion across a family).
+# The adapter NAME is the request's `model` field, so this string is the public API of a voice.
+#
+# WHY LORA AND NOT MERGE (operator decision 2026-09-16, after measurement): the R49 line
+# produces a FAMILY of author voices — lv-bronte, lv-yarros (shipped), lv-hemingway (in
+# build) — and Skaldsong switches between them per request. Merging bakes each 264 MB adapter
+# into a fresh 7.6 GB model:
+#
+#     3 authors   LoRA  7.6 GB + 3x264 MB ~ 8.4 GB      merged ~23 GB
+#     6 authors   LoRA  7.6 GB + 6x264 MB ~ 9.2 GB      merged ~46 GB
+#
+# On a box where GPU 1 has 5.7 GB free, GPU 2 has 2.4 GB and GPU 3 is a held reserve, that
+# is the whole argument. A new author becomes a directory drop, not a VRAM negotiation.
+#
+# ⚠ SUPPORT WAS CHECKED, NOT ASSUMED. The training-throughput playbook records a LoRA refusal
+# on a Qwen3 *MoE* arch ("get_expert_mapping must be implemented") and its lesson is that
+# feature support is per-architecture, not per-family. Verified on the fleet's own engine
+# before committing: vllm/model_executor/models/qwen3.py:271 declares Qwen3ForCausalLM with
+# SupportsLoRA, plus packed_modules_mapping and embedding_modules. Dense Qwen3 is fine; do NOT
+# transplant this compose onto an MoE carrier without re-running that grep.
+#
+# ⚠ KV IS PINNED IN BYTES, deliberately, following gen-small-seat's note. On THIS box
+# `--gpu-memory-utilization` has been measured wrong in both directions — cyberprev at 0.40
+# holds 47.1 GB (8 GB over its fraction), gen-small at 0.48 holds 36.9 (10 GB under) — so a
+# fraction cannot be used to place a seat beside live co-tenants. GPU 0 already carries
+# cyberprev, gen-small and the Parakeet STT seat; an explicit KV budget makes this seat's
+# footprint deterministic instead of negotiated.
+#
+# ⚠ ADAPTERS ARE MOUNTED READ-ONLY and named explicitly. A request selects a voice by putting
+# the adapter name in the `model` field — switching voices is a field, not a deployment.
+#
+# MEASURED on this seat, 2026-09-16, rather than taken from docs:
+#   POST /v1/load_lora_adapter   {"lora_name","lora_path"}   -> 200 in 0.24 s
+#   POST /v1/unload_lora_adapter {"lora_name"}               -> 200 in 0.003 s
+#   VRAM unchanged across the swap; container stayed healthy; no restart, no reload.
+# ⚠ Anything added that way is GONE on `compose up -d` unless it is ALSO listed in
+# --lora-modules below. Runtime load is for trying a voice; this list is what survives.
+#
+# MEASURED LORA COST on this seat (n=30 per arm, interleaved, A-vs-A floor 0.1%):
+#   base            median 143.0 tok/s
+#   lv-yarros       median 108.2 tok/s   -> -24.3%, far outside the floor.
+# Accepted deliberately: the seat is for prose, not a latency path, and 24% buys every future
+# author for 264 MB instead of 7.6 GB. If a voice ever lands on a hot path, merge THAT one.
+#
+# MEASURED RESIDENCY: 10,740 MiB, against a requested 0.11 x 94.97 GiB = 10,700 MiB — a 40 MiB
+# miss on a box where the util fraction has been wrong by 8-10 GB in BOTH directions. Pinning
+# --kv-cache-memory in bytes is what makes the fraction predictive; do not remove it.
+name: voices-seat
+
+services:
+  vllm-voices:
+    image: ${VOICES_IMAGE:-vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013}
+    container_name: ${VOICES_CONTAINER_NAME:-vllm-voices}
+    restart: unless-stopped
+    ipc: host
+    ports:
+      - "${VOICES_PORT:-8027}:8000"
+    volumes:
+      - /tank/aimodels/huggingface:/hfcache
+      - ${VOICES_MODEL:-/tank/aimodels/Qwen3-4B-Instruct}:/model:ro
+      - /tank/aimodels/voice-adapters:/adapters:ro
+    environment:
+      - VLLM_API_KEY=${API_KEY:-}
+      - VLLM_ALLOW_RUNTIME_LORA_UPDATING=${VOICES_RUNTIME_LORA:-1}
+    command:
+      - /model
+      - --served-model-name
+      - ${VOICES_SERVED_NAME:-voices-base}
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8000"
+      - --gpu-memory-utilization
+      - "${VOICES_GPU_MEM_UTIL:-0.12}"
+      - --kv-cache-memory
+      - "${VOICES_KV_CACHE_MEMORY:-2147483648}"
+      - --max-model-len
+      - "${VOICES_MAX_MODEL_LEN:-8192}"
+      - --max-num-seqs
+      - "${VOICES_MAX_NUM_SEQS:-8}"
+      - --dtype
+      - auto
+      - --enable-prefix-caching
+      - --enable-lora
+      - --max-loras
+      - "${VOICES_MAX_LORAS:-4}"
+      - --max-lora-rank
+      - "${VOICES_MAX_LORA_RANK:-32}"
+      - --lora-modules
+      - lv-yarros=/adapters/lv-yarros-4b-v1
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids:
+                - "${VOICES_GPU_ID:-0}"
+              capabilities: [gpu]
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS http://localhost:8000/health || exit 1"]
+      interval: 30s
+      timeout: 5s
+      retries: 20
+      start_period: 300s
+    labels:
+      - homepage.group=AI - Inference
+      - homepage.name=Voices
+      - homepage.icon=mdi-account-voice
+      - homepage.description=Author voice adapters (LoRA) on Qwen3-4B
+      - homepage.href=http://10.251.50.54:8027/docs
+    networks: [tnet]
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true