vh
e58360668e
Two operator-authorised changes on the same afternoon.
## althing v3 (U9b flag day, one-way, no rollback)
The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.
73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.
Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.
nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.
The v2 database is left inert on disk. There is no import path and none
was improvised.
## sec onto GPU0
GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.
Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
143 lines
6.3 KiB
YAML
143 lines
6.3 KiB
YAML
# mog-sec — the pen-test seat on ana-ml2 GPU 0 (:8019).
|
|
#
|
|
# MOVED GPU 1 -> GPU 0 on 2026-08-28 (operator-directed). GPU 1 carries the five
|
|
# resident fleet seats (gen 46 GB + embed + coder + rerank + reward = ~69.9 GB of
|
|
# 97.9), leaving ~28 GB — less than the ~51 GB this seat reserves at
|
|
# MOG_GPU_MEM_UTIL=0.52, so it could no longer start there. GPU 0 has been idle
|
|
# since run 3c was stopped.
|
|
#
|
|
# ⚠ POWER: bringing this up re-arms the two-GPU load condition that tripped the
|
|
# Anaheim rack breaker on 2026-08-26. One circuit feeds the whole rack including
|
|
# ana-gw and ana-wg, so a trip costs the site AND the remote path in. Idle draw is
|
|
# negligible (~6-13 W/card); the risk is sec and gen under concurrent load.
|
|
#
|
|
# Serves Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (stock-Qwen3.8-27B-based,
|
|
# vision-intact Qwen3_5ForConditionalGeneration, base-graft MTP head), quantized
|
|
# in-house to mixed NVFP4 W4A4 (bulk MLP) + FP8 W8A8 (attn / linear_attn / lm_head /
|
|
# top MLP layers), with the bf16 MTP head grafted back. ⚠ the grafted MTP requires
|
|
# `re:^mtp.*` in config.json quantization_config.ignore or vLLM loads it uninitialised
|
|
# (0% accept) — handled by post_quant.py, verified present at build time.
|
|
#
|
|
# CONTEXT: served at native 262K, NOT the card's 1M. The 1M needs YaRN rope_scaling
|
|
# (absent from the weights' config) plus the SGLang/DFlash2 attention path the repo
|
|
# ships a deployment kit for — neither is our vLLM serving surface. 262K is the honest
|
|
# native ceiling here; a real 1M seat would be a separate SGLang project.
|
|
#
|
|
# Two served-names (base + `-thinking`): LiteLLM keys deployments by (model, api_base),
|
|
# so mog-sec and mog-sec-reasoning use distinct names to avoid the shared-config
|
|
# enable_thinking clobber. Same pinned nightly as the gen seat (carries the #51113
|
|
# qwen3_5_mtp x GDN partial-accept fix that the MTP-on config depends on).
|
|
|
|
name: mog-sec
|
|
|
|
services:
|
|
vllm-mog-sec:
|
|
image: ${MOG_IMAGE:-vllm/vllm-openai:latest}
|
|
container_name: ${MOG_CONTAINER_NAME:-vllm-mog-sec}
|
|
restart: unless-stopped
|
|
ipc: host
|
|
ports:
|
|
- "${MOG_PORT:-8019}:8000"
|
|
volumes:
|
|
- /tank/aimodels/huggingface:/hfcache
|
|
- ${MOG_MODEL:-/tank/aimodels/mog-sec-27b-nvfp4-mixed}:/model:ro
|
|
# DFlash2 speculative drafter. Mounted unconditionally — it is inert if
|
|
# MOG_SPEC_CONFIG selects an MTP method that does not reference /drafter.
|
|
- ${MOG_DRAFT_MODEL:-/tank/aimodels/qwen38-27b-dflash2-drafter}:/drafter:ro
|
|
environment:
|
|
- HF_HOME=/hfcache
|
|
- HF_HUB_CACHE=/hfcache/hub
|
|
- VLLM_API_KEY=${API_KEY:-}
|
|
# ⚠ DEFAULTS TO UNSET, deliberately. This seat previously hardcoded
|
|
# PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. The DFlash2 config
|
|
# validated 2026-08-22 ran WITHOUT it, and the quant playbook §3.10
|
|
# records expandable_segments corrupting retained tensors in another
|
|
# context. Do not re-enable it casually — that would ship a variable the
|
|
# tested configuration did not have.
|
|
- PYTORCH_CUDA_ALLOC_CONF=${MOG_ALLOC_CONF:-}
|
|
command:
|
|
- /model
|
|
- --served-model-name
|
|
- ${MOG_SERVED_NAME:-mog-sec-27b}
|
|
- ${MOG_SERVED_NAME_THINK:-mog-sec-27b-thinking}
|
|
- --host
|
|
- 0.0.0.0
|
|
- --port
|
|
- "8000"
|
|
- --quantization
|
|
- ${MOG_QUANT:-compressed-tensors}
|
|
- --gpu-memory-utilization
|
|
- ${MOG_GPU_MEM_UTIL:-0.44}
|
|
- --max-model-len
|
|
- ${MOG_MAX_MODEL_LEN:-262144}
|
|
- --max-num-seqs
|
|
- ${MOG_MAX_NUM_SEQS:-16}
|
|
- --max-num-batched-tokens
|
|
# ⚠ Raising this costs peak-activation VRAM straight out of the KV pool
|
|
# (measured 2026-08-22: 16384 -> 32768 cost ~3 GiB of KV for no benefit).
|
|
- ${MOG_MAX_NUM_BATCHED_TOKENS:-16384}
|
|
- --trust-remote-code
|
|
- --dtype
|
|
- auto
|
|
- --mamba-cache-dtype
|
|
- float32
|
|
- --kv-cache-dtype
|
|
- ${MOG_KV_CACHE_DTYPE:-fp8}
|
|
- --enable-prefix-caching
|
|
- --enable-chunked-prefill
|
|
- --limit-mm-per-prompt
|
|
- '${MOG_LIMIT_MM:-{"image": 4}}'
|
|
# ⚠ MANDATORY on a newer vLLM. The model's own preprocessor_config.json
|
|
# declares size.longest_edge = 16777216 px (4096x4096), which expands to
|
|
# 16384 image tokens — one image eating 6% of a 262K context, and enough
|
|
# to kill startup on builds that enforce the text-vs-ids count check.
|
|
# This caps the dummy profiling image AND real images. Cost scales as
|
|
# (edge/patch)^2 / merge^2, so 2048x2048 -> ~5125 tokens.
|
|
- --mm-processor-kwargs
|
|
- '${MOG_MM_PROCESSOR_KWARGS:-{"size": {"longest_edge": 4194304, "shortest_edge": 65536}}}'
|
|
- --reasoning-parser
|
|
- ${MOG_REASONING_PARSER:-qwen3}
|
|
- --default-chat-template-kwargs
|
|
- '{"reasoning_effort": "${MOG_REASONING_EFFORT:-medium}"}'
|
|
- --enable-auto-tool-choice
|
|
- --tool-call-parser
|
|
- qwen3_coder
|
|
# ONE env var carrying the whole JSON, because the two speculative shapes
|
|
# are not interchangeable: dflash needs a "model" pointing at the drafter,
|
|
# MTP must NOT have one. A method+tokens template cannot express both.
|
|
# DFlash2 : {"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}
|
|
# MTP : {"method": "qwen3_5_mtp", "num_speculative_tokens": 3}
|
|
# ⚠ Comparing the two requires matching num_speculative_tokens — see the
|
|
# quant playbook §5.1: MTP runs a single-module head autoregressively, so
|
|
# deeper k improves acceptance and DESTROYS throughput.
|
|
- --speculative-config
|
|
- '${MOG_SPEC_CONFIG:-{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}}'
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
device_ids:
|
|
- "${MOG_GPU_ID:-0}"
|
|
capabilities:
|
|
- gpu
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
start_period: 900s
|
|
networks:
|
|
- tnet
|
|
labels:
|
|
- homepage.group=AI - Inference
|
|
- homepage.name=M.O.G.-SEC 27B (pen-test)
|
|
- homepage.icon=mdi-shield-lock
|
|
- homepage.description=Uncensored security model, Qwen3.8-27B NVFP4+MTP, 262K — the `mog-sec` seat (ana-ml2 GPU 0)
|
|
- homepage.href=http://10.250.50.54:${MOG_PORT:-8019}/docs
|
|
|
|
networks:
|
|
tnet:
|
|
name: traefik-net
|
|
external: true
|