diff --git a/stacks/flash-next-seat/.env.example b/stacks/flash-next-seat/.env.example new file mode 100644 index 0000000..e850df4 --- /dev/null +++ b/stacks/flash-next-seat/.env.example @@ -0,0 +1,87 @@ +# flash-next-seat — Qwen3.8-Flash-Next (abliterated) on fv-ml1 GPU 2, :8022. +# Copy to .env on the host at /opt/docker/compose/flash-next-seat/.env. +# +# This is the INITIAL configuration, stood up 2026-09-13. Values marked FIRST-BOOT +# are deliberately conservative and expected to be revised once the seat has +# reported its own memory budget and been bisected for depth. Do not treat them as +# measured — they are not yet. + +# ── Image ─────────────────────────────────────────────────────────────────── +# ⚠ MUST contain vLLM #54371 (UVA PLE-offload), merged 2026-09-09T14:32Z. +# Verified by ancestry rather than version string: this commit is +150 / behind_by=0 +# from merge commit 3116c5d06bfe76501b3dd6b5434bfc7f3274f5e7. v0.29.0 does NOT +# contain it (cut ~6h before the merge) and neither does any nightly- tag +# dated 2026-09-09 or earlier — the nightly build runs ~06:16 UTC. +FN_IMAGE=vllm/vllm-openai:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 +API_KEY=replace-me + +# ── Placement ─────────────────────────────────────────────────────────────── +# GPU 2 was completely idle (2 MiB) before this seat; GPU 3 still is. Every other +# compose GPU pin on fv-ml1 is 0 or 1, so this seat displaced nothing. +FN_GPU_ID=2 +FN_PORT=8022 +FN_CONTAINER_NAME=vllm-flash-next + +# ── Model ─────────────────────────────────────────────────────────────────── +# dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b990578ef3031eccf9f28e675a289a09ee9 +FN_MODEL=/tank/aimodels/qwen38-flash-next-abliterated-nvfp4 +FN_QUANT=modelopt_fp4 +FN_SERVED_NAME=qwen3.8-flash-next-uncensored +FN_SERVED_NAME_THINK=qwen3.8-flash-next-uncensored-thinking + +# ── The offload ───────────────────────────────────────────────────────────── +# The 51B n-gram table (47.7 GiB FP8) lives in pinned host RAM; the GPU reads rows +# over CUDA UVA. Without this the checkpoint needs ~126 GiB of VRAM and will not +# start on one 96 GiB card. fv-ml1 has 566 GB RAM / ~388 GB available, so the +# host side is not a constraint here — unlike every DGX-Spark report upstream, +# where host and device share one unified pool and "offload" frees nothing. +FN_ENGRAM_CONFIG={"cpu_offload": true} + +# ── Memory and context ────────────────────────────────────────────────────── +# FIRST-BOOT. Card is dedicated, so 0.90 (the vLLM recipe's own figure for offload +# configs) is not competing with a neighbour. Expected resident weights ~78 GiB of +# 95.6, leaving ~8-9 GiB for KV + activation inside the 0.90 budget. +# ⚠ NEXT STEP after first boot: read vLLM's budget line and replace this with an +# explicit --kv-cache-memory in BYTES. A ratio sizes the KV calculation but does +# not bound peak activation — that is what bit stacks/mog-sec twice on 2026-09-10. +FN_GPU_MEM_UTIL=0.90 + +# FIRST-BOOT. 128K, against a 262,144 native ceiling. Raising this is gated on a +# depth bisect with a NON-REPEATING prompt, because two open upstream issues make +# depth the risky axis: #54764 (PLE short-conv prefill pads every request in a +# batch to the batch-max query length) and #54919 (long prefill starving decode +# for 3-7 minutes). See the compose comment. +FN_MAX_MODEL_LEN=131072 +FN_MAX_NUM_SEQS=16 +FN_MAX_NUM_BATCHED_TOKENS=8192 +FN_MAMBA_CACHE_MODE=align + +# ── Prefix caching ────────────────────────────────────────────────────────── +# ⚠ THE ROLLBACK LEVER for open #54173 (CUBLAS_STATUS_INTERNAL_ERROR / illegal +# memory access in the GDN path, WITH prefix caching). Set to the empty string to +# disable. Leave FN_MAMBA_CACHE_MODE=align either way — Qwen4Exp raises on "all". +FN_PREFIX_CACHING=--enable-prefix-caching + +# ── Vision ────────────────────────────────────────────────────────────────── +# 4194304 px = 2048x2048 -> ~5,125 image tokens. The checkpoint's own preprocessor +# declares 16777216 (4096x4096) -> ~16,384 tokens, which is both wasteful and fatal +# on builds enforcing the image-token count check. Same trap as stacks/mog-sec. +FN_MM_PROCESSOR_KWARGS={"size": {"longest_edge": 4194304, "shortest_edge": 65536}} +FN_LIMIT_MM={"image": 4} + +# ── Misc ──────────────────────────────────────────────────────────────────── +FN_REASONING_PARSER=qwen3 +FN_REASONING_EFFORT=medium +FN_TOOL_CALL_PARSER=qwen3_xml +# Intentionally EMPTY. expandable_segments has corrupted retained tensors on this +# box before (quant playbook §3.10) and has never been tested against a pinned +# host allocation handed to UVA. +FN_ALLOC_CONF= + +# ── NOT SET, on purpose ───────────────────────────────────────────────────── +# --speculative-config : MTP is off. vLLM's own recipe measured it WORSE at every +# concurrency on 4xH100 (8-36% less throughput, 32-173% +# more latency, ~36% acceptance) and open #55357 reports +# episodic 0% acceptance with repetition collapse. +# --kv-cache-dtype fp8 : fp8_e4m3 KV on this model's QSA path is an unmerged RFC +# (#54426). Do not copy it over from gen/mog-sec. diff --git a/stacks/flash-next-seat/README.md b/stacks/flash-next-seat/README.md new file mode 100644 index 0000000..4dfadd9 --- /dev/null +++ b/stacks/flash-next-seat/README.md @@ -0,0 +1,139 @@ +# flash-next-seat — Qwen3.8-Flash-Next (abliterated), fv-ml1 GPU 2, `:8022` + +The first seat on the fleet whose weights do not fit its card and run anyway. + +`Qwen3.8-Flash-Next` is 176B total — a 125B main model plus a **51B n-gram (PLE) +lookup table** — activating ~6B parameters per token. The n-gram table is a pure +embedding lookup with almost no compute per token, so it lives in **pinned host +RAM** and the GPU reads the rows it needs directly over **CUDA UVA** on a dedicated +stream with async prefetch. + +| | | +|---|---| +| Checkpoint | `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4` @ `be794b99…` (126.0 GiB) | +| On the card | ~78 GiB of 95.6 GiB — routed experts NVFP4 W4A4, rest at source precision | +| In host RAM | 47.7 GiB pinned, FP8 E4M3, 10 `model-plefp8-*` shards + per-table scalar scale | +| Context | 131,072 to start (native ceiling 262,144) — see *Raising context* | +| Speculative decoding | **none** — see *Why MTP is off* | +| Gateway wiring | **none yet** — this seat is not in LiteLLM; `gen` is untouched | + +## Deploy + +```bash +scripts/deploy-stack.sh fv-ml1 flash-next-seat # diffs vs live, prompts y/N +# then on the host, first boot only: +ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/flash-next-seat && docker compose config >/dev/null && docker compose up -d' +``` + +The `.env` lives on the host and is never committed. Copy `.env.example`, set +`API_KEY`, and read the FIRST-BOOT annotations before changing anything else. + +## Architecture, briefly + +Four ideas, and three of them shape the serving config: + +- **GDN + QSA.** 36 of 48 layers use Gated DeltaNet (linear attention) to compress + history; every fourth layer uses Qwen Sparse Attention for long-range retrieval. + This is why KV is cheap at depth and why `--mamba-cache-dtype float32` matters. +- **N-gram embedding.** The 51B lookup table that this seat offloads. Qwen's own + framing: capacity with almost no per-token compute. +- **Gated residual / hyper-connections.** Four residual branches; excluded from + quantization in this checkpoint. +- **MTP head.** Present and preserved byte-identically. Deliberately unused. + +## Why this checkpoint, and the trap that disqualifies most others + +vLLM selects the PLE table's weight format from **`text_config.ple_embedding_dtype`**, +as the *first* branch of `Qwen4ExpPLEEmbeddingMethod.from_quant_config`. This +checkpoint declares `"float8_e4m3fn"`. + +A build that ships an FP8 PLE table **without** that declaration resolves instead +through ModelOpt's `*.ple.*` exclude to the *unquantized* method, never registers +the `weight_scale` parameter, and dies on load with `no module or parameter named +'ngram_embedding.weight_scale'`. `gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8` +is exactly this case. **Check that field before trying another build.** + +Chosen over better-liked alternatives because its provenance states protocols and +repeat counts rather than adjectives. From its own `qualification-notes.md` and +metrics files, kept in the model directory: + +- HarmBench, 240 genuinely-harmful behaviours, greedy: 100% compliance at reasoning + off / low / xhigh. +- MMLU 82.11% → 81.93% (−0.18 pp) on an identical harness, 2,280 questions. +- GSM8K 97.27% (1283/1319), full set, single-shot, temp 0.6 — inside the stated + BF16 reference band 97.12–97.50. +- AIME26 pass@1 98.75% (237/240, **SEM 0.61 pp, 30 problems × 8 repeats**), + majority@8 100%, `max_tokens` 130,000, 4.9M completion tokens, stop_rate 99.17%. +- Byte-equality audit of unchanged tensors: 1,562 tensors / 118.4 GB compared, all + passed, **including all 31 MTP tensors**. + +⚠ Two honest gaps in that evidence. The routed experts are **NVFP4 W4A4**, and +nobody — including the publisher — has measured this checkpoint at the full 262K +context; AIME26's 130K-token generations are the deepest evidence that exists. +Separately, `validate_checkpoint_report.json` in the repo describes the *earlier +BF16-PLE revision* (204 shards / 173.6 GiB), not the published FP8-PLE one. + +Rejected alternatives, for the record: +`orcarouter/…-Uncensored-NVFP4` is gated (access request pending nothing — not +requested); `nvidia/…-NVFP4` is the cleanest ModelOpt MIXED_PRECISION build but is +not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts. + +## Why MTP is off + +Against our house graft-MTP habit, and on purpose. + +vLLM's own recipe for this model measured MTP on 4×H100 as **worse at every +concurrency tested** — 8–36% lower request throughput, 32–173% higher per-token +latency, driven by ~36% acceptance — and says do not enable it by default. Open +issue **#55357** reports episodic 0% draft acceptance with repetition collapse +inside thinking blocks. Open **#55496** reports ModelOpt `MIXED_PRECISION` failing +to load FP8_BLOCK_SCALES MTP experts. + +Turning it on is two lines in `compose.yaml` (documented in place). If you do, +measure it **here**, with repeats, against this seat's own baseline — the numbers +above are someone else's hardware. + +## The upstream situation, as of 2026-09-13 + +- **#53896** — model support. **Merged 2026-08-31.** In v0.29.0. +- **#54371** — *UVA PLE-offload and Engram tensor parallelism*. **Merged + 2026-09-09T14:32Z.** This is the offload this seat uses. **Not in v0.29.0**, + which was cut ~6 h earlier; present in `v0.29.1rc0` and in any nightly from + 2026-09-10 onward. +- **#53899** — the *older, worker-based* PLE offload. **Open and explicitly paused** + in favour of #54371. Do not go back to it. Its whole bug family — the TP=1 + startup deadlock (#53960), the `pidfd_getfd` / `kernel.yama.ptrace_scope` gate, + the shared-CUDA-event race under async scheduling, and silently one-step-stale + PLE outputs under CUDA graphs — came from the separate worker process and the + CUDA-IPC row transfer that the UVA path does not have. + +Open issues worth knowing about on SM120, none of them blocking: + +| Issue | What it does | Our exposure | +|---|---|---| +| **#54173** | CUBLAS internal error / illegal memory access in the GDN path **with prefix caching** | We enable prefix caching. `FN_PREFIX_CACHING=` is the one-line rollback. | +| **#54764** | PLE short-conv batched prefill pads every request to the batch-max query length | Why `--max-num-batched-tokens` is 8192, not 16384 | +| **#54919** | Long prefill starves active decode for 3–7 minutes | Why context starts at 128K | +| **#54521** | Greedy decoding non-deterministic from `persistent_topk` in prefill | Affects any A/B on this seat — establish a noise floor before comparing | +| **#54426** | fp8_e4m3 KV on the QSA path is an unmerged RFC | Why `--kv-cache-dtype` is **not** set to fp8 here | + +## Raising context + +128K is a starting value, not a measured one. Before raising it, bisect with a +**non-repeating** prompt — a repeated one hashes to cached blocks and never +prefills deep, so it proves nothing. The `stacks/mog-sec` README records this the +hard way: three successive context cuts all sized the *KV pool* while the crashes +were governed by *processing depth*, which is a different number. + +The point of a ceiling is the refusal. Below it the seat serves; above it vLLM +returns a clean 400 naming the limit, instead of the engine dying and taking every +in-flight request with it. + +## Not done yet + +- **Pin `--kv-cache-memory` in bytes** from the first boot's budget line, replacing + the 0.90 ratio. Same discipline as `stacks/mog-sec` and `stacks/erp-seat`. +- **Gateway wiring is deliberately absent.** Pointing any LiteLLM alias at this + seat — in particular displacing `gen` / `summarizer` / `classifier`, which is the + long-term intent recorded in henge item 49 — changes what every existing caller + receives and is the operator's call, not a deploy-time default. diff --git a/stacks/flash-next-seat/compose.yaml b/stacks/flash-next-seat/compose.yaml new file mode 100644 index 0000000..babfe46 --- /dev/null +++ b/stacks/flash-next-seat/compose.yaml @@ -0,0 +1,212 @@ +# flash-next-seat — Qwen3.8-Flash-Next (abliterated) on fv-ml1 GPU 2 (:8022). +# +# THE POINT OF THIS SEAT: it is the first model on the fleet whose weights do NOT +# fit the card, and it runs anyway. Qwen3.8-Flash-Next is 176B total — a 125B main +# model plus a 51B n-gram (PLE) lookup table — at ~6B active per token. The n-gram +# table is a pure lookup with almost no compute, so it lives in PINNED HOST RAM and +# the GPU reads the rows it needs directly over CUDA UVA. Resident on the card: +# ~78 GiB of 95.6. Resident in host RAM: 47.7 GiB of 566 GB. See --engram-config. +# +# Serves dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4, revision +# be794b990578ef3031eccf9f28e675a289a09ee9, pulled to +# /tank/aimodels/qwen38-flash-next-abliterated-nvfp4 (126.0 GiB on disk). +# - routed experts (294,912 tensors, 48 layers x 512 experts) NVFP4 W4A4 +# - PLE n-gram table FP8 E4M3, 10 `model-plefp8-*` shards + per-table scalar scale, +# taken from Qwen/Qwen3.8-Flash-Next-FP8 rather than re-derived +# - everything else source precision: self_attn, linear_attn, mlp.gate, +# shared_expert, hyper_connection, ple projections, visual tower, embeddings, +# lm_head and all 31 MTP tensors are in the quant config's `ignore` list +# - abliteration is a direct weight edit, no fine-tune: HarmBench 240-real-harm +# compliance 100% at reasoning off/low/xhigh, MMLU 82.11 -> 81.93 (-0.18pp), +# GSM8K 97.27% (full 1319), AIME26 pass@1 98.75% (30 x 8 repeats, SEM 0.61pp). +# Publisher's own reports are kept in the model dir; they state protocols and +# repeat counts, which is why this build was chosen over better-liked ones. +# +# ⚠ THE CHECKPOINT-SELECTION TRAP, recorded because it silently disqualifies most +# community builds. vLLM picks the PLE table's format from +# `text_config.ple_embedding_dtype` as the FIRST branch of +# Qwen4ExpPLEEmbeddingMethod.from_quant_config. This checkpoint declares +# "float8_e4m3fn". A checkpoint with an FP8 PLE table and NO such declaration +# (e.g. gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8) resolves instead +# through ModelOpt's `*.ple.*` exclude to the UNQUANTIZED method and fails to load +# on the scale tensor it does ship. Check that field before trying another build. +# +# ⚠ IMAGE: must contain vLLM PR #54371 ("UVA PLE-offload and Engram tensor +# parallelism", merged 2026-09-09T14:32Z). Verified by ancestry, not version string: +# nightly-eed1f3d0... is +150 commits / behind_by=0 from merge commit 3116c5d0. +# NOT in v0.29.0 — that release was cut ~6h before the merge. The older worker-based +# offload (#53899) is PAUSED upstream in favour of this one; do not go back to it, +# its entire bug family (TP=1 deadlock, pidfd_getfd/yama ptrace gate, shared-event +# race, silently one-step-stale outputs under CUDA graphs) came from the separate +# worker process that the UVA path does not have. +# +# ⚠⚠ MTP IS DELIBERATELY OFF, against our house graft-MTP habit. The model ships a +# working MTP head and this checkpoint preserves it byte-identically (31 tensors, +# audit passed) — we still do not enable it. vLLM's own recipe measured MTP on +# 4xH100 as WORSE AT EVERY CONCURRENCY tested: 8-36% lower request throughput, +# 32-173% higher per-token latency, ~36% acceptance, and says do not default it on. +# Open #55357 reports episodic 0% draft acceptance with repetition collapse inside +# thinking blocks. If you want it, it is FN_SPEC_CONFIG — but measure it here, with +# repeats, before believing it. +name: flash-next-seat +services: + vllm-flash-next: + image: ${FN_IMAGE:-vllm/vllm-openai:latest} + container_name: ${FN_CONTAINER_NAME:-vllm-flash-next} + restart: unless-stopped + ipc: host + # The PLE table is 47.7 GiB of PINNED host memory. Pinned pages are not + # swappable and count against the process's locked-memory limit, so lift it. + ulimits: + memlock: -1 + ports: + - "${FN_PORT:-8022}:8000" + volumes: + - /tank/aimodels/huggingface:/hfcache + - ${FN_MODEL:-/tank/aimodels/qwen38-flash-next-abliterated-nvfp4}:/model:ro + environment: + - HF_HOME=/hfcache + - HF_HUB_CACHE=/hfcache/hub + - VLLM_API_KEY=${API_KEY:-} + # Intentionally UNSET by default. expandable_segments has corrupted retained + # tensors elsewhere on this box (quant playbook §3.10) and the offload path + # hands UVA a pinned host allocation — not a combination anyone has tested. + - PYTORCH_CUDA_ALLOC_CONF=${FN_ALLOC_CONF:-} + command: + - /model + - --served-model-name + - ${FN_SERVED_NAME:-qwen3.8-flash-next-uncensored} + - ${FN_SERVED_NAME_THINK:-qwen3.8-flash-next-uncensored-thinking} + - --host + - 0.0.0.0 + - --port + - "8000" + # ⚠ modelopt_fp4, NOT compressed-tensors as on the other seats here. This + # checkpoint's quantization_config carries compressed-tensors-shaped + # `config_groups` AND `quant_method: modelopt` / `quant_algo: NVFP4` + # (producer nvidia-modelopt 0.46.0). The ModelOpt reader is the one whose + # per-prefix `is_layer_excluded` honours the 13-entry ignore list that keeps + # attention, the shared experts, the PLE and MTP out of W4A4. + - --quantization + - ${FN_QUANT:-modelopt_fp4} + # ── THE WHOLE REASON THIS SEAT FITS ────────────────────────────────────── + # Keeps the 51B n-gram table in pinned host RAM, read by the GPU over UVA on + # a dedicated stream with async prefetch. Legacy VLLM_PLE_CPU_OFFLOAD=1 still + # works but is deprecated in favour of this flag. + # `embedding_across_dp` is irrelevant at TP=1/DP=1 — left at its default. + - --engram-config + - '${FN_ENGRAM_CONFIG:-{"cpu_offload": true}}' + # GPU 2 is DEDICATED to this seat — nothing else is resident on the card, so + # unlike gen/mog-sec on GPU 0/1 this ratio is not competing with a neighbour. + # 0.90 is the vLLM recipe's own figure for offload configurations. + # ⚠ PIN --kv-cache-memory IN BYTES once the first boot reports its budget + # line. Same lesson as stacks/mog-sec and stacks/erp-seat: a ratio sizes the + # KV calculation but does not bound peak activation, and it silently yields a + # different cache depending on what is resident at start time. + - --gpu-memory-utilization + - ${FN_GPU_MEM_UTIL:-0.90} + # ⚠ 128K TO START, NOT the native 262,144 the checkpoint advertises, and the + # gap is deliberate. Two open upstream issues make depth the risky axis here: + # #54764 (PLE short-conv batched prefill pads every request to the batch-max + # query length, so one deep request inflates the whole step's activation) and + # #54919 (long prefill starving active decode for 3-7 minutes). The mog-sec + # lesson applies directly: what the KV pool can HOLD and what the card can + # PROCESS at depth are different numbers, and sizing to the pool never fixed + # a depth-driven crash. RAISE THIS ONLY AFTER BISECTING with a NON-REPEATING + # prompt — a repeated one hashes to cached blocks and never prefills deep. + - --max-model-len + - ${FN_MAX_MODEL_LEN:-131072} + - --max-num-seqs + - ${FN_MAX_NUM_SEQS:-16} + # 8192 rather than the 16384 used elsewhere, for the #54764 padding reason + # above: a chunk's activation cost here scales with the batch's DEEPEST + # request, not its average. Costs prefill throughput, costs no context. + - --max-num-batched-tokens + - ${FN_MAX_NUM_BATCHED_TOKENS:-8192} + - --trust-remote-code + - --dtype + - auto + # Hybrid backbone: 36 of 48 layers are Gated DeltaNet (linear attention), the + # other 12 are Qwen Sparse Attention. The model's own config asks for + # mamba_ssm_dtype float32. + - --mamba-cache-dtype + - float32 + # ⚠ MANDATORY-ISH: Qwen4Exp RAISES NotImplementedError on mamba_cache_mode + # "all" and tells you to use "align". "align" is already the implicit default + # when prefix caching is on, but it is stated here so that turning prefix + # caching off cannot silently select a mode the model rejects. + - --mamba-cache-mode + - ${FN_MAMBA_CACHE_MODE:-align} + # ⚠ FIRST THING TO TURN OFF if you see CUBLAS_STATUS_INTERNAL_ERROR or an + # illegal memory access: open #54173 is exactly that, in the GDN path, WITH + # prefix caching. On by default because every vLLM recipe config for this + # model enables it. + - ${FN_PREFIX_CACHING:---enable-prefix-caching} + - --enable-chunked-prefill + # ⚠ KV CACHE LEFT AT AUTO — do NOT copy `--kv-cache-dtype fp8` from the gen or + # mog-sec seats. fp8_e4m3 KV on this model's QSA path is an unmerged RFC + # (#54426: working patch, one machine, author asking for corroboration). + # + # Recipe sets this in every configuration; autotune has no value here and + # costs startup time. + - --no-enable-flashinfer-autotune + - --limit-mm-per-prompt + - '${FN_LIMIT_MM:-{"image": 4}}' + # ⚠ MANDATORY. This checkpoint's preprocessor_config.json declares + # size.longest_edge = 16777216 px (4096x4096), which expands to ~16,384 image + # tokens — one image eating 12% of a 128K context, and enough to fail startup + # on builds that enforce the text-vs-ids token count check. Same trap already + # documented on stacks/mog-sec. 4194304 = 2048x2048 -> ~5,125 tokens. + - --mm-processor-kwargs + - '${FN_MM_PROCESSOR_KWARGS:-{"size": {"longest_edge": 4194304, "shortest_edge": 65536}}}' + - --reasoning-parser + - ${FN_REASONING_PARSER:-qwen3} + # The Qwen3.8 chat template defaults to `xhigh`, where reasoning length grows + # with conversation depth and has a long tail — measured on the 27B sibling at + # 4,529-5,532 chars spiking to 9,261. `medium` is the seat-level default; + # per-request override via chat_template_kwargs still works. + - --default-chat-template-kwargs + - '{"reasoning_effort": "${FN_REASONING_EFFORT:-medium}"}' + - --enable-auto-tool-choice + # qwen3_xml, per the vLLM recipe for THIS model — not the qwen3_coder parser + # the mog-sec seat uses. + - --tool-call-parser + - ${FN_TOOL_CALL_PARSER:-qwen3_xml} + # ── NO --speculative-config LINE, ON PURPOSE ───────────────────────────── + # Speculative decoding is absent rather than disabled-by-a-flag, because + # there is no honest "off" value for --speculative-config and a placeholder + # flag in this slot is the kind of thing that gets copied forward and + # misread. To turn MTP on, add these two lines here and nothing else: + # - --speculative-config + # - '{"method": "mtp", "num_speculative_tokens": 3}' + # Read the MTP warning in the header first; the measured default is off. + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: + - "${FN_GPU_ID:-2}" + capabilities: + - gpu + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + # ⚠ LONG. Cold boot reads 126 GiB from /tank and pins 47.7 GiB of it; the + # closest published single-card timing is ~8.5 min load on a 96 GB card. + # 1800s so a slow first boot is not mistaken for a failed one. + start_period: 1800s + retries: 3 + networks: + - tnet + labels: + - homepage.group=AI - Inference + - homepage.name=Qwen3.8-Flash-Next (uncensored) + - homepage.icon=mdi-lightning-bolt-outline + - homepage.description=176B/6B-active MoE, abliterated, NVFP4 + 48GB n-gram table offloaded to host RAM — fv-ml1 GPU 2 + - homepage.href=http://10.251.50.54:${FN_PORT:-8022}/docs +networks: + tnet: + name: traefik-net + external: true