feat(fv-ml1): add gen-small A3B seat; sec/gen/char to native 262K; coder reclaim; gateway
Full seat rebalance across GPU0/GPU1 (flash on GPU2 and empty GPU3 untouched), operator-directed. Every target seat now serves native 262,144 context with concurrency in the requested 1.2-2.5x band, verified from live boot logs: cyberprev (sec) 262144 @ 1.37x depth-probed CLEAN to 259,722 tokens flash-next (gen) 262144 @ 1.31x (untouched, already in band) gen-small (NEW) 262144 @ 2.56x MTP k=3 measured 69.6% accept / 3.09 len char-rp 262144 @ 1.22x (was 1.02x; util 0.40->0.52) char-rp-fast 262144 @ 2.04x (util cap 0.30->0.24, pinned KV unchanged) - gen-small: NEW seat, Qwen3.6-35B-A3B (3B active MoE), llmfan46 Heretic (MPOA) NVFP4 experts-only, already on disk at qwen36-35b-a3b-heretic-nvfp4. There is no general Qwen3.8 A3B (3.8 MoEs are Flash-Next and the 2.4T), so this is the 3.6 fallback the operator specified. GPU0, :8026, MTP k=3, coherent and MTP-verified before wiring. gen-small / gen-small-reasoning gateway aliases. - coder: 8192 @ 13.75x -> 16384 @ 4.70x (util 0.06->0.055). Context doubled, waste cut. Not the exact 2-3x target: the 1.5B weight+overhead floor (~4.2 GiB) sits just under the util knob's resolution, so hitting <=3x reliably needs a --kv-cache-memory byte pin (compose change) rather than the util fraction. - cyberprev raised 163840 -> 262144: depth-probed with non-repeating prompts to 259,722 tokens, clean (no OOM, memory flat). Unlike mog-sec (same base arch, capped at 163840 for depth crashes), this checkpoint holds native depth. - Gateway (operator calls): summarizer + classifier -> gen-small; new classifier-large -> gen-large (flash) for the accuracy tier; summarizer-large stays on flash. All verified end-to-end. - GPU1 hit its ceiling raising char-rp; resolved by trimming char-rp-fast's reservation cap (its KV is pinned, so concurrency held at 2.04x) rather than moving a utility seat -- the shared GPU_ID on reward/embed/rerank made a single-seat move messier than the in-GPU rebalance. Seat inventory regenerated from the live containers.
This commit is contained in:
@@ -6,7 +6,7 @@
|
||||
Hand-edits are overwritten and, worse, drift silently while looking
|
||||
authoritative — which is exactly the failure this file replaced. -->
|
||||
|
||||
**Generated 2026-09-14 16:10 UTC** by `scripts/seat-inventory.py`, read from the running
|
||||
**Generated 2026-09-14 17:24 UTC** by `scripts/seat-inventory.py`, read from the running
|
||||
containers on `100.64.0.7` — `docker inspect`, `nvidia-smi`, each model's own
|
||||
`config.json`, and the `.PROVENANCE.txt` siblings on `/tank`.
|
||||
|
||||
@@ -17,11 +17,12 @@ containers on `100.64.0.7` — `docker inspect`, `nvidia-smi`, each model's own
|
||||
|
||||
| GPU | seat | VRAM | weights | KV tokens | ctx | concurrency | util |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0 | `vllm-cyberprev` | 38.0 GiB | 25.11 GiB | 166,264 | 163840 | **1.01×** | 0.40 |
|
||||
| 1 | `vllm-meromero-rp` | 37.9 GiB | 19.51 GiB | 266,334 | 262144 | **1.02×** | 0.40 |
|
||||
| 1 | `vllm-erp-seat` | 26.3 GiB | 15.9 GiB | 534,649 | 262144 | **2.04×** | 0.30 |
|
||||
| 0 | `vllm-cyberprev` | 46.0 GiB | 25.11 GiB | 359,372 | 262144 | **1.37×** | 0.40 |
|
||||
| 0 | `vllm-gen-small` | 35.3 GiB | 23.98 GiB | 670,142 | 262144 | **2.56×** | 0.48 |
|
||||
| 1 | `vllm-meromero-rp` | 41.3 GiB | 19.51 GiB | 320,774 | 262144 | **1.22×** | 0.52 |
|
||||
| 1 | `vllm-erp-seat` | 26.4 GiB | 15.9 GiB | 534,649 | 262144 | **2.04×** | 0.24 |
|
||||
| 1 | `vllm-reward` | 9.0 GiB | 4.41 GiB | 26,224 | 16384 | **1.60×** | 0.10 |
|
||||
| 1 | `vllm-coder` | 6.0 GiB | 2.98 GiB | 112,624 | 8192 | **13.75×** | 0.06 |
|
||||
| 1 | `vllm-coder` | 5.5 GiB | 2.98 GiB | 77,056 | 16384 | **4.70×** | 0.055 |
|
||||
| 1 | `vllm-embed` | 3.4 GiB | 1.12 GiB | 10,272 | 8192 | **1.25×** | 0.03 |
|
||||
| 1 | `vllm-rerank-a3` | 2.1 GiB | 1.06 GiB | — | 8192 | — | 0.03 |
|
||||
| 2 | `vllm-flash-next` | 93.1 GiB | 76.82 GiB | 344,155 | 262144 | **1.31×** | 0.96 |
|
||||
@@ -41,11 +42,20 @@ context.
|
||||
- **speculative decoding:** `{"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}`
|
||||
- **image:** `vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013`
|
||||
|
||||
### `vllm-gen-small` — GPU 0
|
||||
|
||||
- **serves:** `gen-small`, `gen-small-thinking`
|
||||
- **model:** `/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4`
|
||||
- **architecture:** Qwen3_5MoeForConditionalGeneration (qwen3_5_moe), 40 layers, 256 experts
|
||||
- **quantization:** modelopt / None — W4A4 (None)
|
||||
- **speculative decoding:** `{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}`
|
||||
- **image:** `vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013`
|
||||
|
||||
### `vllm-coder` — GPU 1
|
||||
|
||||
- **serves:** `qwen2.5-coder-1.5b`
|
||||
- **model:** `?`
|
||||
- **image:** `vllm/vllm-openai:latest` ⚠ **floating tag**
|
||||
- **image:** `vllm/vllm-openai:v0.24.0`
|
||||
|
||||
### `vllm-embed` — GPU 1
|
||||
|
||||
@@ -94,7 +104,7 @@ context.
|
||||
|
||||
## Gateway aliases resolving to this host
|
||||
|
||||
19 aliases. Ports with no listening seat are marked dead.
|
||||
22 aliases. Ports with no listening seat are marked dead.
|
||||
|
||||
| alias | port |
|
||||
|---|---|
|
||||
@@ -102,20 +112,23 @@ context.
|
||||
| `char-rp-fast` | 8021 |
|
||||
| `char-rp-reasoning` | 8016 |
|
||||
| `chat-judge` | 8022 |
|
||||
| `classifier` | 8022 |
|
||||
| `classifier` | 8026 |
|
||||
| `classifier-large` | 8022 |
|
||||
| `coder-fast` | 8020 |
|
||||
| `erp-tune-v2` | 8098 |
|
||||
| `gemma4-26b-a4b-it-base` | 8099 |
|
||||
| `gen` | 8022 |
|
||||
| `gen-large` | 8022 |
|
||||
| `gen-reasoning` | 8022 |
|
||||
| `gen-small` | 8026 |
|
||||
| `gen-small-reasoning` | 8026 |
|
||||
| `image-judge` | 8022 |
|
||||
| `qwen-image-bench` | 8022 |
|
||||
| `qwen3-embedding` | 8001 |
|
||||
| `reranker` | 8013 |
|
||||
| `sec` | 8025 |
|
||||
| `sec-reasoning` | 8025 |
|
||||
| `summarizer` | 8022 |
|
||||
| `summarizer` | 8026 |
|
||||
| `summarizer-large` | 8022 |
|
||||
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user