Vuong Hoang
17db37e1cc
Shipped on the operator's standing authorisation, "ship it if the gate passes".
The gate design of record for this line -- the operator's own v2 rule, ratified
2026-09-15 -- passes on all three axes:
VOICE +0.177 delta_cb at 3.2x the pairwise floor, the BEST arm of five,
and +0.128 at 2.8x on the pre-registered punctuation-stripped
secondary read, so ~3/4 of the gain is words rather than marks.
Closed 60.8% of the reachable span on an axis deliberately made
harder than Bronte's or Hemingway's.
NOT COPIED 0.12 hit-rate against HELD-OUT McCARTHY'S OWN 0.12 -- 1.0x the
author -- with a longest verbatim run of 10 words against the 12
genuinely unseen McCarthy shares with the train split by
coincidence. All 31 matches READ: stock grammar, name-shaped hits
are the renamed inventions, nothing protectable. This matters:
McCarthy is in copyright with a living estate.
NO DAMAGE ran-on +0.12 against the 0.200 floor, 40% headroom, and the
measured minimum across the five-arm ladder.
ckpt300 over ckpt900, the eval-loss minimum: the two-epoch recipe is now 0 for 3,
and this time the curve was CONFIDENT and wrong -- ckpt900 is the minimum and
ckpt300 sits +15.8x the median neighbour jitter above it, yet ckpt300 wins every
axis that resolves (3.2x vs 1.2x voice margin, 2.7x tighter seed spread,
memorisation 1.0x the author vs 1.8x, ran-on 0.13 vs 0.28). On this schedule read
the axes, not the curve.
THE COST IS REAL AND IS WRITTEN INTO THE COMPOSE, THE ADAPTER README AND THE
MEMORY RECORD RATHER THAN GLOSSED. In-band is 0.65 against base's 0.89 and on-beat
0.42 against 0.71 -- 35% of generations miss the requested 90-140 word band against
base's 11%. Axis C is ran-on only and is structurally blind to this; that blindness
was identified and written down before these numbers existed. No checkpoint choice
fixes it: every adapted arm is 33-48% out-of-band and ran-on is non-monotonic in
epoch. If a caller depends on a word budget, expect one generation in three to miss
it, and the fix is a retrain targeting length rather than a different checkpoint.
Rollback is one line and 0.003 s: drop the --lora-modules entry, or POST
/v1/unload_lora_adapter. The other three voices are untouched.
voices-seat — author voice adapters on one carrier
fv-ml1 GPU 0, port 8027. One Qwen3-4B-Instruct base; each author is a LoRA adapter
named lv-<author> (lv = lang-voice). Switching voices is a request field, not a
deployment.
curl http://10.251.50.54:8027/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"lv-yarros","messages":[{"role":"user","content":"Beat: ..."}]}'
# ^^^^^^^^^ the only thing that changes between voices
voices-base serves the unadapted carrier from the same process, which is what makes an
adapter-on / adapter-off comparison harness-matched.
Adding a voice
Two paths, and they are not interchangeable.
Try one now — no restart, ~0.24 s:
curl -X POST http://10.251.50.54:8027/v1/load_lora_adapter -H 'Content-Type: application/json' \
-d '{"lora_name":"lv-hemingway","lora_path":"/adapters/lv-hemingway-4b-v1"}'
curl -X POST http://10.251.50.54:8027/v1/unload_lora_adapter -H 'Content-Type: application/json' \
-d '{"lora_name":"lv-hemingway"}'
⚠ A runtime-loaded adapter is GONE on the next compose up -d. To make it survive, add
it to --lora-modules in compose.yaml — which costs a container recreate and a full model
reload (~3 min). Runtime load is for trying a voice; the compose list is what persists.
Adapters live on the host at /tank/aimodels/voice-adapters/<name>/, mounted read-only at
/adapters. A rename on the host is visible inside immediately.
Reaching it through LiteLLM
Each voice is one alias entry pointing at this seat with its own model value — no new
deployment, no new container, no VRAM:
- model_name: lv-yarros
litellm_params:
model: openai/lv-yarros
api_base: http://10.251.50.54:8027/v1
⚠ Retiring a voice orphans scoped keys. Any LiteLLM key whose allowlist names a removed
alias starts returning silent per-endpoint 403s. Audit /key/list + /key/info on every
repoint — that failure has cost this fleet two and a half months before.
What it costs, measured
| decode, base | 143.0 tok/s (median, n=30) |
| decode, LoRA | 108.2 tok/s (median, n=30) |
| LoRA overhead | −24.3%, against an A-vs-A noise floor of 0.1% |
| resident VRAM | 10,740 MiB |
| KV cache | 10,912 tokens → 1.33x concurrency at 8k context |
Arms were interleaved rather than blocked, because the co-tenants on that card take traffic this seat does not control and a block design would alias their load onto one arm.
The 24.3% is accepted deliberately. Three authors cost 8.4 GB as adapters and ~23 GB merged; six cost 9.2 GB versus ~46 GB. On a card with 1.8 GB free after this seat, that is the whole argument. If a voice ever lands on a latency path, merge that one and serve it separately.
Placement warnings
⚠ GPU 0 is now at 96.0 of 97.9 GB. This seat's 10.7 GB went into the last real gap on
fv-ml1: GPU 1 has ~5.7 GB, GPU 2 has ~2.4 GB, and GPU 3 is a held reserve for a future
full-card seat (flash-next alone needs 93 of 96 GiB). There is no room for another seat
here without a placement decision.
⚠ --gpu-memory-utilization is a request against TOTAL VRAM that the card must already be
able to honour — not a share of what is free. The first bring-up refused at 0.12 with
11.16 GiB free, and refusing was correct: it protected cyberprev, gen-small and the
Parakeet STT seat rather than squeezing them.
⚠ Pin --kv-cache-memory in bytes. With it, the requested fraction predicted residency
to within 40 MiB. Without it, this box has been wrong by 8–10 GB in both directions.
Support was checked, not assumed
vllm/model_executor/models/qwen3.py declares Qwen3ForCausalLM with SupportsLoRA plus
packed_modules_mapping and embedding_modules. The training playbook records a LoRA
refusal on a Qwen3 MoE architecture, and its lesson is that support is per-architecture,
not per-family. Do not transplant this compose onto an MoE carrier without re-running that
check.