esh-pfi-infrastructure/stacks/mog-sec/compose.yaml

# mog-sec — the pen-test seat on ana-ml2 GPU 0 (:8019).
#
# MOVED GPU 1 -> GPU 0 on 2026-08-28 (operator-directed). GPU 1 carries the five
# resident fleet seats (gen 46 GB + embed + coder + rerank + reward = ~69.9 GB of
# 97.9), leaving ~28 GB — less than the ~51 GB this seat reserves at
# MOG_GPU_MEM_UTIL=0.52, so it could no longer start there. GPU 0 has been idle
# since run 3c was stopped.
#
# ⚠ POWER: bringing this up re-arms the two-GPU load condition that tripped the
# Anaheim rack breaker on 2026-08-26. One circuit feeds the whole rack including
# ana-gw and ana-wg, so a trip costs the site AND the remote path in. Idle draw is
# negligible (~6-13 W/card); the risk is sec and gen under concurrent load.
#
# Serves Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (stock-Qwen3.8-27B-based,
# vision-intact Qwen3_5ForConditionalGeneration, base-graft MTP head), quantized
# in-house to mixed NVFP4 W4A4 (bulk MLP) + FP8 W8A8 (attn / linear_attn / lm_head /
# top MLP layers), with the bf16 MTP head grafted back. ⚠ the grafted MTP requires
# `re:^mtp.*` in config.json quantization_config.ignore or vLLM loads it uninitialised
# (0% accept) — handled by post_quant.py, verified present at build time.
#
# CONTEXT: served at native 262K, NOT the card's 1M. The 1M needs YaRN rope_scaling
# (absent from the weights' config) plus the SGLang/DFlash2 attention path the repo
# ships a deployment kit for — neither is our vLLM serving surface. 262K is the honest
# native ceiling here; a real 1M seat would be a separate SGLang project.
#
# ⚠⚠ GPU0 HEADROOM — the 2026-09-10 crash pair, and why the live .env now reads
# MOG_GPU_MEM_UTIL=0.50 / MOG_MAX_MODEL_LEN=393216 rather than 0.52 / 420000.
#
# At util 0.52 this seat and `gen` (0.38) together reserve 0.90 of the card, leaving
# ~4.6 GiB. vLLM's utilization figure covers weights and the KV pool but NOT all
# transient activation memory, and a long-context prefill chunk with the 7-wide dflash
# drafter lives in what is left. Twice in ten minutes (20:20:30Z and 20:30:02Z) a request
# ~151,700 tokens deep scheduling a further ~15,700-token chunk asked for ~1.04 GiB with
# ~600 MB free, EngineCore took a fatal error, and `restart: unless-stopped` bounced the
# seat. Each bounce is a hard 500 to every in-flight caller.
#
# ⚠ 0.50 AND 420000 ARE MUTUALLY EXCLUSIVE — dropping util alone does NOT work and the
# seat will crash-loop at startup instead of at runtime. A single 420,000-token sequence
# needs 17.88 GiB of KV; at 0.50 the pool is 17.4-17.5 GiB, so vLLM refuses:
#   "To serve at least one request with the model's max seq len (420000), 17.88 GiB KV
#    cache is needed, which is larger than the available KV cache memory (17.41 GiB)."
# The context length and the crash were directly coupled: 420k was only reachable at the
# utilization that left no transient headroom.
#
# ⚠ DO NOT pin max_model_len to vLLM's suggested maximum. It estimated 406,352 on one
# boot and 409,840 on the next -- the available-KV figure drifts ~0.1 GiB boot to boot, so
# the edge value fails to start on an unlucky one. 393,216 (384k) sits 3% under the lower
# estimate and leaves ~0.7 GiB of the pool unspent, which IS the transient headroom this
# change exists to buy. Measured after: KV 405,612 tokens, concurrency 1.03x at 393,216.
#
# ⚠ NOT the lever: `num_speculative_tokens`. The crash window logged 17.6% draft
# acceptance with positions 5-7 at 1.5-4.9%, which reads as an obvious cut from 7 to 3.
# Across 180 samples of that counter the median acceptance LENGTH is 3.12 of 7 (range
# 1.83-6.75) and median draft acceptance is 30.4% (range 11.9-82.1%). The crash window sat
# near the minimum; cutting to 3 would cap the workloads accepting nearly the full draft.
#
# Two served-names (base + `-thinking`): LiteLLM keys deployments by (model, api_base),
# so mog-sec and mog-sec-reasoning use distinct names to avoid the shared-config
# enable_thinking clobber. Same pinned nightly as the gen seat (carries the #51113
# qwen3_5_mtp x GDN partial-accept fix that the MTP-on config depends on).

name: mog-sec

services:
  vllm-mog-sec:
    image: ${MOG_IMAGE:-vllm/vllm-openai:latest}
    container_name: ${MOG_CONTAINER_NAME:-vllm-mog-sec}
    restart: unless-stopped
    ipc: host
    ports:
      - "${MOG_PORT:-8019}:8000"
    volumes:
      - /tank/aimodels/huggingface:/hfcache
      - ${MOG_MODEL:-/tank/aimodels/mog-sec-27b-nvfp4-mixed}:/model:ro
      # DFlash2 speculative drafter. Mounted unconditionally — it is inert if
      # MOG_SPEC_CONFIG selects an MTP method that does not reference /drafter.
      - ${MOG_DRAFT_MODEL:-/tank/aimodels/qwen38-27b-dflash2-drafter}:/drafter:ro
    environment:
      - HF_HOME=/hfcache
      - HF_HUB_CACHE=/hfcache/hub
      - VLLM_API_KEY=${API_KEY:-}
      # ⚠ DEFAULTS TO UNSET, deliberately. This seat previously hardcoded
      # PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. The DFlash2 config
      # validated 2026-08-22 ran WITHOUT it, and the quant playbook §3.10
      # records expandable_segments corrupting retained tensors in another
      # context. Do not re-enable it casually — that would ship a variable the
      # tested configuration did not have.
      - PYTORCH_CUDA_ALLOC_CONF=${MOG_ALLOC_CONF:-}
    command:
      - /model
      - --served-model-name
      - ${MOG_SERVED_NAME:-mog-sec-27b}
      - ${MOG_SERVED_NAME_THINK:-mog-sec-27b-thinking}
      - --host
      - 0.0.0.0
      - --port
      - "8000"
      - --quantization
      - ${MOG_QUANT:-compressed-tensors}
      - --gpu-memory-utilization
      - ${MOG_GPU_MEM_UTIL:-0.44}
      # ⚠ KV PINNED IN BYTES, added 2026-09-10 after the crash pair above. The
      # utilization ratio does NOT bound actual usage -- it sizes the KV calculation,
      # but peak activation is measured at profiling time and real long-context work
      # exceeds the profile. vLLM's own budget line proved this seat was running 0.9 GiB
      # OVER its 47.48 GiB reservation at util 0.50 (26.44 consumed + 3.53 peak activation
      # + 0.89 CUDAGraph + 17.52 KV = 48.38), and `gen` was over by 0.33 on the same card.
      # That overage came out of the shared card's slack, which is what kept OOMing.
      #
      # 17,697,765,376 B = 16.48 GiB is vLLM's OWN recommended figure from that line
      # ("Replace gpu_memory_utilization config with --kv-cache-memory=17697765376 to fit
      # into requested memory"), not a value anyone here invented. Same discipline as
      # stacks/erp-seat: an explicit figure is reproducible, a ratio silently yields a
      # different cache depending on what else is resident at start time.
      - --kv-cache-memory
      - ${MOG_KV_CACHE_MEMORY:-17697765376}
      - --max-model-len
      # ⚠⚠ 160k, AND THE NUMBER IS MEASURED, NOT DERIVED FROM THE KV POOL. That distinction
      # is the whole lesson of 2026-09-10: what the KV pool can HOLD and what the card can
      # PROCESS at depth are different numbers, and every earlier cut (420k -> 384k -> 320k)
      # sized the first while the crashes were governed by the second. Sizing to the pool
      # never worked because the pool was never the constraint.
      #
      # Bisected against the real reproducer -- a non-repeating prompt, because prefix
      # caching would let a repeated one hash to cached blocks and never prefill deep:
      #     113,247 prompt tokens  -> SURVIVED (27 s)
      #     200,088 prompt tokens  -> SURVIVED (174 s)
      #     ~285,000 prompt tokens -> ENGINE DIED, HTTP 500, container restarted
      # So the sustainable ceiling sits between 200k and 285k with `gen` idle. 163,840
      # leaves ~20% margin under the proven-good depth, which is deliberate: `gen` shares
      # this card and its load is an UNCONTROLLED variable, so a value set just under the
      # measured edge would fail whenever the neighbour happened to be busy.
      #
      # ⚠ THE POINT OF THE CEILING IS THE REFUSAL. Below it the seat serves; above it vLLM
      # returns a clean 400 naming the limit instead of the engine dying and taking every
      # in-flight request with it. A seat that refuses what it cannot serve is strictly
      # better than one that dies trying, and that is what this number buys.
      #
      # ⚠ The header's "served at native 262K" was therefore never actually deliverable on
      # a shared card. It had simply not been exercised at depth until today.
      - ${MOG_MAX_MODEL_LEN:-262144}
      - --max-num-seqs
      - ${MOG_MAX_NUM_SEQS:-16}
      - --max-num-batched-tokens
      # ⚠ Raising this costs peak-activation VRAM straight out of the KV pool
      # (measured 2026-08-22: 16384 -> 32768 cost ~3 GiB of KV for no benefit).
      #
      # ⚠⚠ LOWERED 16384 -> 4096 on 2026-09-10, and this is the lever that actually
      # matches the failure. The OOMs were DEPTH-DEPENDENT -- operator: "loading up the
      # context killed sec again" -- and the dumps confirm it: the crashes hit at
      # num_computed_tokens 151,728 and then 266,832, each scheduling a 15,696-token
      # chunk. A chunked-prefill step of 16,384 tokens attending over a quarter-million
      # tokens of KV is the transient that blows the budget, and it scales with DEPTH,
      # not with the KV pool size. Cutting context 420k -> 384k -> 320k shrank the pool
      # and never touched that spike, which is why three cuts did not stop the crashes.
      # Reclaim is roughly the inverse of the 2026-08-22 measurement (~3 GiB per 16k of
      # batched tokens), so 16384 -> 4096 should return ~2.3 GiB of peak activation --
      # more than the 0.9 GiB the seat was over by. Costs prefill throughput on long
      # prompts (4x the chunks) and costs NO context.
      - ${MOG_MAX_NUM_BATCHED_TOKENS:-16384}
      - --trust-remote-code
      - --dtype
      - auto
      - --mamba-cache-dtype
      - float32
      - --kv-cache-dtype
      - ${MOG_KV_CACHE_DTYPE:-fp8}
      - --enable-prefix-caching
      - --enable-chunked-prefill
      - --limit-mm-per-prompt
      - '${MOG_LIMIT_MM:-{"image": 4}}'
      # ⚠ MANDATORY on a newer vLLM. The model's own preprocessor_config.json
      # declares size.longest_edge = 16777216 px (4096x4096), which expands to
      # 16384 image tokens — one image eating 6% of a 262K context, and enough
      # to kill startup on builds that enforce the text-vs-ids count check.
      # This caps the dummy profiling image AND real images. Cost scales as
      # (edge/patch)^2 / merge^2, so 2048x2048 -> ~5125 tokens.
      - --mm-processor-kwargs
      - '${MOG_MM_PROCESSOR_KWARGS:-{"size": {"longest_edge": 4194304, "shortest_edge": 65536}}}'
      - --reasoning-parser
      - ${MOG_REASONING_PARSER:-qwen3}
      - --default-chat-template-kwargs
      - '{"reasoning_effort": "${MOG_REASONING_EFFORT:-medium}"}'
      - --enable-auto-tool-choice
      - --tool-call-parser
      - qwen3_coder
      # ONE env var carrying the whole JSON, because the two speculative shapes
      # are not interchangeable: dflash needs a "model" pointing at the drafter,
      # MTP must NOT have one. A method+tokens template cannot express both.
      #   DFlash2 : {"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}
      #   MTP     : {"method": "qwen3_5_mtp", "num_speculative_tokens": 3}
      # ⚠ Comparing the two requires matching num_speculative_tokens — see the
      # quant playbook §5.1: MTP runs a single-module head autoregressively, so
      # deeper k improves acceptance and DESTROYS throughput.
      - --speculative-config
      - '${MOG_SPEC_CONFIG:-{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}}'
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids:
                - "${MOG_GPU_ID:-0}"
              capabilities:
                - gpu
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 900s
    networks:
      - tnet
    labels:
      - homepage.group=AI - Inference
      - homepage.name=M.O.G.-SEC 27B (pen-test)
      - homepage.icon=mdi-shield-lock
      - homepage.description=Uncensored security model, Qwen3.8-27B NVFP4+MTP, 262K — the `mog-sec` seat (ana-ml2 GPU 0)
      - homepage.href=http://10.250.50.54:${MOG_PORT:-8019}/docs

networks:
  tnet:
    name: traefik-net
    external: true