vh
91bda3c480
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16, and serving inference again. This lands the repo half of that. Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq reservation so the address the runbook, DNS and LiteLLM all assume is the address it actually has. Its headscale node is renamed too. The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the reason this diff touches current-state files and not the record. Dated persistent-memory entries, archival-memory and incident notes still say ana-ml2 in 31 and 62 places respectively, because that is what the box was when those things happened. Rewriting them would make the history lie. LiteLLM was the load-bearing piece and needed more than the api_base sed the runbook describes. Twenty api_base entries repointed, but a grep-and-verify pass also caught a LIVE pass_through_endpoints target for the scalar-judge reward route still on the old address -- an api_base-only substitution would have left it dead. Four prose references describing current state were repointed as well; one historical note recording where a hand-test was run is deliberately left pointing at 10.250.50.54. Two facts in the server tables were wrong and are corrected here. The site is Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell Max-Q, not two -- verified by nvidia-smi -L and independently by PCI enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196, which changes what fits on it. DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all three resolvers. The BMC record carries a warning that its 802.1q VLAN tag must stay disabled -- it shipped tagging VLAN 250 into an untagged port, which made it invisible to every network-side diagnostic and is the reason it appeared dead through several cable changes. Verified end to end: summarizer and sec both answer through the Anaheim gateway across the mesh to FV seats on different ports.
161 lines
8.7 KiB
YAML
161 lines
8.7 KiB
YAML
# erp-seat — the RP seat on fv-ml1 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant
|
|
# (G4-MeroMero-26B-A4B-it-uncensored-heretic) behind the gateway alias `char-rp-fast`.
|
|
#
|
|
# ⚠ THE STACK NAME IS HISTORICAL. It served Pfish-6 (the run-6 ERP-tune LoRA merge) until
|
|
# 2026-09-10, when the operator swapped the occupant: "replace that a4b moe over pfish-6 --
|
|
# remove the pfish-6 alias and create an alias for char-rp-fast." The compose PROJECT name is
|
|
# deliberately NOT renamed: asset-engine derives seat liveness from it, so a rename reads as
|
|
# OFFLINE. Pfish-6 remains on disk at /tank/aimodels/erp-tune-v6-nvfp4a16 and the pre-swap host
|
|
# env is at /opt/docker/compose/erp-seat/.env.pfish6.bak-20260910 -- one cp plus `up -d` back.
|
|
#
|
|
# ⚠ THE FIRST A4B QUANT SERVED NaN AND LOOKED HEALTHY DOING IT. It was built with the DENSE
|
|
# recipe, whose ignore list carries no router regex, so all 30 MoE routers were quantized to
|
|
# 4 bits and expert selection was destroyed (playbook §3.15). The seat passed its healthcheck,
|
|
# returned finish_reason=length with the full token count, and every response decoded to the
|
|
# empty string; the give-away was NaN logprobs. Re-quantized with
|
|
# services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py, whose target guard refuses exactly
|
|
# that. Use the MoE recipe for anything in this family; the dense one is for the v2-31B.
|
|
#
|
|
# PFISH-6 PROVENANCE, kept because it is still the rollback target. Run 7 was retired by
|
|
# operator ruling 2026-09-09 ("we're gonna stay on 6 for now"): its gate failure turned out to
|
|
# be a DETECTOR BUG (the adjective "minor" in a HARD rule, fixed cc42d76 in brokkr-smithy), but
|
|
# run 7 was independently a poor run (primary FLAT +2, both diversity families reduced,
|
|
# long-context coherence 1.0 -> 0.875). Run 6 was the standing seat here until the 2026-09-10
|
|
# swap above. Routing aliases live in the gateway, not here.
|
|
#
|
|
# Serve recipe copied from stacks/gemma4-charrp (same architecture + quant format, proven on this
|
|
# box): gemma4 tool + reasoning parsers, enable_thinking pinned false, model's own stock template.
|
|
# It carries over to MeroMero A4B unchanged -- verified 2026-09-10 end to end: clean prose with
|
|
# no channel-prefix leak, a solid-colour image read correctly (vision towers intact), and an
|
|
# auto tool_choice call parsed.
|
|
# GPU1 is SHARED (scriberr, embed, rerank, coder, reward, charrp live there): the live .env runs
|
|
# ERP_GPU_MEM_UTIL=0.30, and the KV pool is pinned in bytes below regardless, so the ratio only
|
|
# has to clear admission.
|
|
# Quant pipeline: services/erp-seat-quant/ (MoE recipe -- NOT services/meromero-quant/, which is
|
|
# the dense one). Tunables in .env.
|
|
|
|
name: erp-seat
|
|
|
|
services:
|
|
vllm-erp-seat:
|
|
image: ${ERP_IMAGE:-vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0}
|
|
container_name: ${ERP_CONTAINER:-vllm-erp-seat}
|
|
restart: unless-stopped
|
|
ipc: host
|
|
ports:
|
|
- "${ERP_PORT:-8021}:8000"
|
|
volumes:
|
|
- /tank/aimodels:/tank/aimodels
|
|
environment:
|
|
- VLLM_API_KEY=${API_KEY:-}
|
|
command:
|
|
- ${ERP_MODEL:-/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16}
|
|
- --quantization
|
|
- compressed-tensors
|
|
- --served-model-name
|
|
- ${ERP_SERVED_NAME:-G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16}
|
|
- --tool-call-parser
|
|
- gemma4
|
|
- --enable-auto-tool-choice
|
|
# --reasoning-parser gemma4 + the enable_thinking pin are BOTH required (measured 2026-09-08):
|
|
# without them the post-tool-response turn leaks a literal "<|channel>thought\n<channel|>"
|
|
# prefix into content (3/3), exactly as stacks/gemma4-charrp/README.md warns. They do NOT
|
|
# fix forced (named) tool_choice — that failure is independent of the reasoning parser
|
|
# (measured 1/9 with it, 3/9 without): the model free-writes Gemma native call syntax
|
|
# instead of grammar-constrained JSON. See README for the forced-tool_choice status.
|
|
- --reasoning-parser
|
|
- gemma4
|
|
- --default-chat-template-kwargs
|
|
- '{"enable_thinking": false}'
|
|
# tool_choice:"none" trap (measured 2026-09-08): without this flag vLLM still renders the
|
|
# tools into the prompt, the model emits a tool call anyway, and because tool parsing is
|
|
# off for tool_choice=none the reply comes back with content=null AND tool_calls=null —
|
|
# an empty turn. The flag drops the tools from the prompt so the model answers in prose.
|
|
- --exclude-tools-when-tool-choice-none
|
|
- --chat-template
|
|
- ${ERP_CHAT_TEMPLATE:-/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16/chat_template.jinja}
|
|
- --max-model-len
|
|
- "${ERP_MAX_MODEL_LEN:-262144}"
|
|
# KV pool pinned in BYTES, not inferred from the utilization ratio. GPU1 is
|
|
# SHARED with seven other containers (charrp, scriberr, embed, rerank, coder,
|
|
# reward), so a ratio silently yields a different cache depending on what else
|
|
# is resident at start time; an explicit figure is reproducible and is what the
|
|
# concurrency ceiling is actually computed from.
|
|
#
|
|
# ⚠ KV-per-token is NOT a constant you may carry between context settings.
|
|
# Measured on THIS build (2026-09-09): 63.2 KiB/token at max-model-len 32768,
|
|
# 16.6 KiB/token at 262144 — a 3.8x difference, because Gemma-4 is hybrid
|
|
# attention (sliding_window 1024 on most layers) and the sliding layers stop
|
|
# scaling with the context while the full-attention layers keep going.
|
|
# Extrapolating the 32k figure to 256k under-counted the achievable cache by
|
|
# 3.8x and would have sized this seat wrong. Re-measure per configuration.
|
|
#
|
|
# 8.49 GiB -> 534,649 tokens -> 2.04x a full 262,144-token context, which is
|
|
# the operator's sizing rule (KV = 2x max context, 2026-09-09).
|
|
#
|
|
# This figure SURVIVED the 2026-09-10 Pfish-6 -> MeroMero-A4B swap unchanged, and that
|
|
# is not luck: the two are the same architecture field for field (30 layers, kv 8,
|
|
# head_dim 256, sliding_window 1024, 25 sliding / 5 full, 128 experts top-8), so the
|
|
# KV-per-token is the same number. Confirmed by reading 534,649 tokens / 2.04x back out
|
|
# of the new engine's log rather than assuming the pinning carried.
|
|
- --kv-cache-memory
|
|
- "${ERP_KV_CACHE_MEMORY:-9114000000}"
|
|
- --max-num-seqs
|
|
- "${ERP_MAX_NUM_SEQS:-32}"
|
|
- --gpu-memory-utilization
|
|
- "${ERP_GPU_MEM_UTIL:-0.35}"
|
|
- --kv-cache-dtype
|
|
- fp8
|
|
# SPEED (2026-09-09). ⚠ NVFP4A16 is WEIGHT-ONLY: activations stay 16-bit, so
|
|
# there is no FP4 GEMM to issue and the FlashInfer/CUTLASS NVFP4 MoE kernels
|
|
# do not apply — vLLM's "your GPU does not have native support for FP4"
|
|
# warning is about the SCHEME, not the card. fv-ml1 is sm_120 and DOES have
|
|
# native FP4; a W4A4 build would use it, at a fidelity cost the operator
|
|
# already settled against for this seat. Marlin is the correct kernel here.
|
|
# PROVEN, not assumed: forcing moe_backend=flashinfer_cutedsl fails at engine
|
|
# init — "does not support the deployment configuration since kernel does not
|
|
# support current device cuda". Marlin is the only viable NVFP4 MoE kernel
|
|
# here, so `auto` is already optimal and this knob exists to document that
|
|
# rather than to be turned.
|
|
#
|
|
# THE REAL LEVER WAS CONCURRENCY. max-num-seqs was 8, which capped the seat
|
|
# far below its KV pool (534,649 tokens). Measured on this build 2026-09-09,
|
|
# 3 rounds each, 300-token generations:
|
|
# n=1 219 tok/s (unchanged by the seq limit)
|
|
# n=4 676 tok/s aggregate
|
|
# n=8 1269 tok/s aggregate <- was NOT REACHABLE at max-num-seqs 8
|
|
# n=16 2170 tok/s aggregate <- 3.2x the old ceiling
|
|
# Single-stream latency is identical, so the raise costs nothing and lifts
|
|
# the ceiling ~3.2x. CUDA-graph capture sizes followed automatically to 64.
|
|
- --kernel-config
|
|
- '{"moe_backend": "${ERP_MOE_BACKEND:-auto}"}'
|
|
- --trust-remote-code
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
device_ids:
|
|
- "${ERP_GPU_ID:-1}"
|
|
capabilities:
|
|
- gpu
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
start_period: 600s
|
|
networks:
|
|
- tnet
|
|
labels:
|
|
- homepage.group=AI - Inference
|
|
- homepage.name=char-rp-fast (MeroMero 26B-A4B, NVFP4A16 MoE)
|
|
- homepage.icon=mdi-fire
|
|
- homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (fv-ml1 GPU1)
|
|
- homepage.href=http://10.251.50.54:${ERP_PORT:-8021}/docs
|
|
|
|
networks:
|
|
tnet:
|
|
name: traefik-net
|
|
external: true
|