vh
dc3e47b3a2
The char-rp-reasoning seat on ana-ml2 GPU0 — NEO-CODE Heretic2 27B at modelopt NVFP4 with a grafted BF16 MTP head, ~77 tok/s via qwen3_5_mtp spec-decode, replacing the retired GGUF seat. It had been running untracked. Includes conf/mtp-workaround/sitecustomize.py, which is not optional: vLLM 0.24.0 does not propagate modelopt exclude_modules to the spec-decode DRAFT model, so the BF16 MTP head gets quantized and the engine dies at load. The shim force-skips mtp.* in is_layer_skipped. Both the mount and PYTHONPATH are load-bearing. Adds the two files house convention expects and the directory lacked: a .env.example naming every knob (all values are the compose defaults; the live host overrides only the three VRAM ones) and a README that points at docs/runbooks/heretic2-nvfp4-mtp-seat.md rather than duplicating it. No secrets: API_KEY is empty by default and the real .env stays on the host.
33 lines
1.4 KiB
Bash
33 lines
1.4 KiB
Bash
# heretic2-charrp-reasoning on ana-ml2. Real .env lives on the host, never committed.
|
|
#
|
|
# Every value below is the compose default — this file exists to name the knobs,
|
|
# not to change them. The live host currently sets only the bottom three.
|
|
|
|
# --- image / container ---
|
|
REASONING_IMAGE=vllm/vllm-openai:v0.24.0
|
|
REASONING_CONTAINER=vllm-charrp-reasoning-nvfp4
|
|
REASONING_PORT=8018
|
|
|
|
# --- model ---
|
|
REASONING_MODEL=/tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp
|
|
# Speculative tokens for the qwen3_5_mtp draft head. This is the MTP seat's whole
|
|
# point (~77 tok/s, ~1.3x over the retired GGUF seat); dropping it to 0 disables
|
|
# spec-decode and the workaround mount stops mattering.
|
|
SPEC_TOKENS=3
|
|
|
|
# --- auth ---
|
|
# vLLM bearer token. Empty = unauthenticated on the LAN, which is how the other
|
|
# ana-ml2 seats run; set it if this seat is ever exposed beyond the fleet.
|
|
API_KEY=
|
|
|
|
# --- placement / VRAM (the three the host actually overrides) ---
|
|
# GPU0 is shared with vllm-aeon-gen (gen) and llama-charrp (char-rp). NVFP4 27B
|
|
# weights are ~26 GB + KV, so util is held low to fit the ~33 GB free alongside
|
|
# them — and that is why max-model-len is 32768 rather than the 256K the lighter
|
|
# Q5 GGUF seat managed. Raising either without rebalancing GPU0 will OOM the
|
|
# neighbours, not just this container.
|
|
REASONING_GPU_ID=0
|
|
REASONING_GPU_MEM_UTIL=0.30
|
|
REASONING_MAX_MODEL_LEN=32768
|
|
REASONING_MAX_NUM_SEQS=4
|