17db37e1cc4622b911740b0b89d5b703afead25d
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
4954cf6756 |
test(flash-next-seat): depth-probe orca to 258K clean; record a blind instrument
Bisected context depth on the orcarouter checkpoint with non-repeating prompts (unique random hex per probe, so prefix caching cannot short-circuit the prefill). Six depths from 31,978 to 258,517 tokens, all served. The load-bearing evidence is the engine allocator log: zero OOM, CUBLAS, or illegal-memory entries across the run. That is the same detector that caught the dealignai near-miss at 155K on the previous checkpoint, where it did fire. The probe also ran under real concurrent operator load, making it a stricter test than a solo run rather than a weaker one. Positive control passed: a mis-sized first attempt produced a ~265K-token prompt and got a clean 400 naming the limit instead of killing the engine, so the probe could detect the failure mode it was looking for. Calibration for re-runs: random hex words tokenize at 7.9 tokens/word here. vLLM #54919 (long prefill starving decode for 3-7 minutes) did not reproduce: 258K prefilled in 28.9 s, roughly 8,900 tok/s, scaling near-linearly from 32K. Records that the probe's memory-headroom half was BLIND and must not be reused. It reported an identical 95,460 MiB used / 2,427 MiB free on every row across an 8x range of depths, which is the tell. Two causes: --kv-cache-memory pins the pool and the engine logs "skipped memory profiling", so GPU usage is flat with respect to depth; and the actual risk is a transient activation spike during prefill, which before/after nvidia-smi bracketing structurally cannot observe. Peak-activation headroom therefore remains unmeasured; the pass/fail result rests on the allocator log alone. Also qualifies the earlier 167.5 tok/s decode figure as a possibly-contended lower bound, and records the operator's independent 140 tok/s average measured in real use while this probe was loading the same card. |
||
|
vh
|
1b5d6ba23a |
docs(flash-next-seat): dealignai weights deleted — record that no local rollback exists
Operator instruction: delete the displaced dealignai checkpoint. 125 GiB reclaimed from /tank (59% -> 57% used). Verified before removing: not mounted by any running or exited container, no symlinks, no inodes shared with the converted orcarouter directory. Every "rollback is two .env keys" statement across the stack README, the .env.example, persistent-memory and its detail file was true when written and is false now -- the .env backup still names paths that no longer exist. Corrected in place rather than left as false reassurance, since a stale rollback instruction is discovered precisely when it is needed. Reverting this seat now costs a 126 GiB re-download. The quality A/B against dealignai is likewise no longer runnable locally: its reference arm is gone. The pristine 170 GiB orcarouter download is retained deliberately -- it is what makes the PLE bf16->FP8 conversion reproducible without re-fetching -- and that is now recorded so a future session does not reclaim it as an obvious duplicate. Also notes that ~75 GiB of non-PLE shards are duplicated between the pristine and converted orca directories (the convert's hardlinks hit EXDEV across two container bind mounts); both now sit directly on /tank, so relinking would reclaim it if /tank ever tightens. |
||
|
vh
|
4390be947d |
feat(flash-next-seat): serve orcarouter weight-only NVFP4 on gen-large
Swaps gen-large from the dealignai ModelOpt W4A4 build to orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, which is weight-only on both axes (W8 float attn, W4 float experts, input_activations: null) and so avoids the 4-bit-activation long-context degradation mode. The checkpoint was previously recorded as unloadable on any mainline vLLM, requiring a from-source PLE-loader patch. That conclusion was wrong on cost. Qwen4ExpPLEEmbeddingMethod.from_quant_config checks ple_embedding_dtype as branch 1, before any quant-config type check, and its NotImplementedError for CompressedTensorsConfig is scoped to the PLE path only -- experts and dense load through the ordinary compressed-tensors paths. Verified by instantiating the real config and calling the selector both ways before doing any work. orcarouter ships a bf16 PLE, so the fix was to make the declaration true: convert the 51.2B-param table to FP8 and declare it. Its 128 PLE tensors sit in one shard file with nothing else in it. Global amax 0.0894, per-shard outlier ratio 1.66x, scale chosen exactly representable in bf16 so no scale-rounding error stacks on quantization; amax maps to 446.17/448, no clipping. Round-trip 2.655% RMS relative, 0.002% underflow, 0 saturation -- the same FP8-PLE treatment dealignai already shipped. MTP head (31 tensors) and vision tower carried through untouched. A second, independent blocker followed: orcarouter labels its 12 QSA layers qwen_sparse_attention, which vLLM rejects; it accepts full_attention and selects QSA via indexer_n_heads. Confirmed indexer_n_heads == 4 in both this and the dealignai checkpoint before renaming -- without that check the rename silently selects plain attention and serves a subtly wrong model that still passes a healthcheck. Measured on the live seat: healthy, coherent, KV 344,155 tokens @ 262,144 ctx, MTP k=3 at 60.4% acceptance / 2.81 mean acceptance length, warm decode median 167.5 tok/s at conc=1 (n=5, spread 12.2%). The reorg note's dealignai figure came from a different harness, so this is not claimed as a win over it; what it does establish is that weight-only experts did not cost decode speed. Still open: controlled quality A/B vs dealignai, and a deep-prefill probe at 262K. Rollback is two .env keys; dealignai remains on disk. Also corrects the README's MTP-is-off section, stale since k=3 was deployed, and adds a superseded-claims row to the quantization playbook. |
||
|
vh
|
2d83a895c1 |
docs(fv-ml1): reconcile GPU seat inventory and model lineage against running containers
Read from docker inspect, nvidia-smi and each model's own config.json rather than from compose files or the gateway config, so the document records what is deployed rather than what was intended. Findings worth naming: - char-rp drift. The LiteLLM config documents char-rp as G4-MeroMero-v2-31B on ana-ml2 GPU 0; :8016 actually serves gemma4-26b-a4b-it-nvfp4, a 26B 30-layer 128-expert stock Gemma-4, on fv-ml1 GPU 1. Host renaming explains part of it, but 31B MeroMero-v2 and 26B stock gemma4 are different models and the comment block carries sampler defaults keyed to the model it names. - Two gateway aliases are dead: erp-tune-v2 (:8098) and gemma4-26b-a4b-it-base (:8099). Nothing is listening on either; callers get a connection failure rather than a clear retirement error. - Three different speculative-decoding methods coexist -- dflash k=7 with a separate drafter on mog-sec, qwen3_5_mtp k=3 on gen, mtp k=3 on flash-next. The method string is architecture-specific; copying a --speculative-config between seats without changing it will not work. - Three seats run the floating :latest tag, so what they would pull on recreate is not what is running and was never recorded. - Seven aliases resolve to vllm-gen on :8015, which is why GPU 0 shows draw during ordinary OpenWebUI use -- its auto-titling fires summarizer per turn. Also commits the flash-next-seat MTP k=3 change made today: compose gains the speculative-config with its measured rationale, and .env.example documents the KV-cache reduction MTP requires (14 GiB OOMs, 10 GiB verified in production). |
||
|
vh
|
7e62a07341 |
flash-next-seat: full 262K context, KV pinned at a measured 14 GiB, gen-large on the gateway
Operator-directed: raise context to the model's native maximum and take as much KV as the card safely allows, and expose the seat through LiteLLM as `gen-large`. max_model_len 131,072 -> 262,144 KV cache 8.76 -> 14.00 GiB (332,721 -> 560,654 tokens) concurrency 2.54x@128K -> 2.14x@262K ⚠ 16.00 GiB WAS TRIED FIRST AND IS TOO AGGRESSIVE. A 155,497-token non-repeating prefill drove GPU 2 to 97,074 of 97,887 MiB and the caching allocator logged "OOM on device 0 while trying to allocate 488636416 bytes (free: 422117376)" -- 466 MiB wanted against 403 MiB free. The request completed, so nothing failed visibly; that is one step before the shape that crashed stacks/mog-sec twice on 2026-09-10 (~1.04 GiB wanted, ~600 MB free). Backed off to 14.00 GiB, which re-probes clean: zero allocator warnings, a 155,557-token prefill in 14.2 s, and 2,085 MiB still free at peak. The reason the first estimate was wrong is worth keeping, because it is not obvious and it inverts the usual advice: --kv-cache-memory makes vLLM SKIP MEMORY PROFILING ENTIRELY and ignore --gpu-memory-utilization. The profiler was the thing accounting for deep-prefill activation, so pinning bytes switched off the protection that the pin was supposed to formalise. vLLM's own "--kv-cache-memory=18745235968 (17.46 GiB) to fully utilize gpu memory" line is computed from a profile measured at max-num-batched-tokens depth and sits 3.5 GiB above what a 150K-token request survives; open #54764 compounds it, since PLE short-conv prefill pads every request in a batch to the batch-MAX query length. max-num-batched-tokens stays at 8192 -- it is what bounds the activation peak, and doubling max_model_len left the profiled peak unchanged at 1.65 GiB precisely because the peak tracks chunk size, not context length. Gateway: `gen-large` added to the LiteLLM model_list, pointing at fv-ml1:8022. One alias on purpose -- a single alias cannot trip the shared-config enable_thinking mutation footgun, which needs two over the same (model, api_base). Sampling is the checkpoint's own declared set (temp 1.0 / top_p 0.95 / top_k 20); presence_penalty, min_p and repetition_penalty are left unset because the checkpoint declares no canonical value for them. Verified registered for both the infra-ops admin key and the shared all-agents key, since a new model behind a scoped allowlist 403s silently. Also adds services/flash-next-mtp-bench/ -- the MTP measurement campaign and its rationale. MTP stays off, but on "not yet measured here" rather than on vLLM's 4xH100 recipe number, which is a cross-harness comparison and not evidence about a TP=1 Blackwell seat. |
||
|
vh
|
f964a47262 |
flash-next-seat: Qwen3.8-Flash-Next on fv-ml1 GPU 2 with the n-gram table in host RAM
First seat whose weights do not fit its card. The model is 176B total -- a 125B main model plus a 51B n-gram (PLE) lookup table -- at ~6B active per token. The table is a pure embedding lookup, so it lives in pinned host RAM and the GPU reads rows directly over CUDA UVA: ~78 GiB resident on a 95.6 GiB card, 47.7 GiB pinned of 566 GB. GPU 2 and GPU 3 were both idle, so this displaced nothing. Checkpoint dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99, pinned by revision: NVFP4 W4A4 routed experts, FP8 PLE table, everything else at source precision. Chosen over better-liked builds because its provenance states protocols and repeat counts -- AIME26 pass@1 98.75% over 30x8 repeats with a stated SEM, full-set GSM8K, and a byte-equality audit covering all 31 MTP tensors -- and because it declares text_config.ple_embedding_dtype, which is the field vLLM reads first when selecting the PLE weight format. Builds that ship an FP8 table without that declaration resolve to the unquantized path and fail on load; the README records the check. Requires vLLM #54371 (UVA PLE-offload, merged 2026-09-09T14:32Z), verified by ancestry: the pinned nightly is +150 commits / behind_by 0 from the merge commit. Not in v0.29.0, cut six hours earlier. The older worker-based offload (#53899) is paused upstream and is not the path here -- its deadlocks, ptrace gate and stale-output-under-graphs bugs all came from the separate worker process that UVA does not have. Five deliberate departures from the other seats on this box, each from a measurement rather than a preference, all annotated in place: - no MTP: the vLLM recipe measured it worse at every concurrency on 4xH100 (8-36% less throughput, 32-173% more latency, ~36% acceptance) - modelopt_fp4, not compressed-tensors: only the ModelOpt reader honours the ignore list keeping attention, shared experts, PLE and MTP out of W4A4 - KV left at auto: fp8 KV on this model's QSA path is an unmerged RFC (#54426) - mamba-cache-mode stated explicitly: the model raises on mode "all" - 128K context and 8192 batched tokens, not the native 262K: #54764 and #54919 make depth the risky axis, and sizing to the KV pool has never fixed a depth-driven crash on this hardware Nothing is wired into LiteLLM. Pointing an alias at this seat changes what existing callers receive and is a separate decision. |