4cee4b179d04a47afd53ddee5b24b5b180ce66e0
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
b8435ceb6f |
Set mog-sec's context to a measured ceiling so it refuses instead of dying
Operator: "loading up the context killed sec again." That reproducer is what finally made the failure legible, and it showed the previous four fixes had all been aimed at the wrong quantity. What the KV pool can hold and what the card can process at depth are different numbers. Cutting context 420k to 384k to 320k, pinning the KV in bytes, and dropping the prefill chunk from 16384 to 4096 all sized the pool. The crashes were governed by the transient needed to process a prefill chunk against a quarter million tokens of resident KV, which scales with depth and not with pool size. Each change helped and none fixed it. Bisected against the real reproducer, with a non-repeating prompt because prefix caching would let a repeated one hash to cached blocks and never prefill deep: 113,247 prompt tokens SURVIVED (27 s) 200,088 prompt tokens SURVIVED (174 s) ~285,000 prompt tokens ENGINE DIED, HTTP 500, container restarted The sustainable ceiling therefore sits between 200k and 285k with gen idle, and gen shares the card with its load uncontrolled, so 163,840 takes about 20% margin under the proven-good depth rather than sitting at the measured edge. The ceiling's purpose is the refusal. Verified after the change: a 149,073-token request serves in 41 s, and requests at both 200k and the ~285k depth that killed the engine now return a clean 400 naming the limit in under a second with the seat untouched. A seat that refuses what it cannot serve is strictly better than one that dies trying. Concurrency went 1.03x to 2.09x. The compose header's "served at native 262K" was never actually deliverable on a shared card; it had simply not been exercised at depth until today. The probe is committed rather than described, so the ceiling can be re-measured when the card's tenancy changes. |
||
|
vh
|
8842ffe1fe |
Pin mog-sec's KV in bytes at 16.48 GiB and take it to 320k context
Operator: "yes, pin the kv and take it to 320k." The real finding, which took three crashes and two failed attempts to reach: --gpu-memory-utilization does not bound actual usage. It sizes the KV calculation, but peak activation is measured at profiling time and real long-context work exceeds the profile. vLLM's own budget line showed mog-sec running 0.9 GiB over its 47.48 GiB reservation -- 26.44 consumed plus 3.53 peak activation plus 0.89 CUDAGraph plus 17.52 KV equals 48.38 -- and gen was over by 0.33 on the same card. That overage came out of the shared card's slack, which is what kept OOMing after the utilization drop. The fix is the one vLLM printed itself: --kv-cache-memory=17697765376, its own recommended figure to fit inside the requested budget. Same discipline erp-seat already uses, and for the same stated reason -- an explicit figure is reproducible where a ratio silently yields a different cache depending on what else is resident at start time. The KV pin and the context length are coupled. 16.48 GiB yields about 383,730 tokens, so a 393,216 max_model_len falls under the 1.0x floor and vLLM refuses to start rather than crashing later; pinning the KV while keeping 384k was never an available combination. 327,680 leaves 1.15x, up from 1.03x. Verified: the engine now logs "reserved 16.48 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling", KV 375,901 tokens, GPU0 down to 90,561 MiB from 91,313, RestartCount 0, and both sec and sec-reasoning return 200 through the gateway. Also records the BabyBronte eyeball A/B, whose result is the operator's own: the voice transferred and the sense did not. Curly quotes went 1 of 18 to 18 of 18 and worksheet collapse 3 of 18 to 0 of 18 between arms. That voice is separable from coherence at 0.6B is the premise the lightweight-adapter regime rests on, so this is the informative outcome rather than a disappointing one. A corpus-prep defect surfaced with it: the tuned output is hard-wrapped at about 70 characters because the Gutenberg source kept its line breaks and the adapter learned the typography too. Cost: 320k of context instead of 420k, on a seat whose crashes happened at 151k. |
||
|
vh
|
77224619ee |
Give GPU0 real headroom: mog-sec to util 0.50 and 384k context
sec/sec-reasoning crash-bounced twice in ten minutes, not once in thirteen days. My earlier read of "rare, not chronic" came off a RestartCount of 1 and was wrong; the operator pushed back and the second and third failures arrived while that recommendation was still on screen. The memory entry making that call is replaced rather than left standing. Cause is unchanged from the first diagnosis: mog-sec at 0.52 plus gen at 0.38 reserve 0.90 of the card, leaving about 4.6 GiB, and vLLM's utilization figure covers weights and the KV pool but not transient activation memory. A request about 151,700 tokens deep scheduling a further 15,700-token chunk asked for 1.04 GiB with roughly 600 MB free. Dropping utilization alone does not work, and fails in a worse way: a single 420,000-token sequence needs 17.88 GiB of KV, and at 0.50 the pool is 17.4 to 17.5 GiB, so vLLM refuses to start at all and the seat crash-loops during startup instead of during a request. The context length and the crash were directly coupled -- 420k was only reachable at the utilization that left no transient headroom. So both moved: 0.50 and 393,216. 384k rather than vLLM's suggested maximum, deliberately. It estimated 406,352 on one boot and 409,840 on the next, because the available-KV figure drifts about 0.1 GiB boot to boot; pinning the edge value fails to start on an unlucky boot. 393,216 sits 3% under the lower estimate and leaves roughly 0.7 GiB of the pool unspent, which is the transient headroom the change exists to buy. Verified after: KV 405,612 tokens, concurrency 1.03x at 393,216, and both sec and sec-reasoning return 200 through the gateway. num_speculative_tokens is documented as NOT the lever. The crash window logged 17.6% draft acceptance with positions 5 through 7 at 1.5 to 4.9 percent, which reads as an obvious cut from 7 to 3; across 180 samples the median acceptance length is 3.12 of 7 and median draft acceptance is 30.4%, so the crash window sat near the minimum and cutting would cap the workloads accepting nearly the full draft. Cost: 384k of context instead of 420k, an 8.5% reduction on a seat whose crashes were happening at 151k. |
||
|
vh
|
e58360668e |
feat: althing v3.0.0 cutover (U9b) and the sec seat onto GPU0
Two operator-authorised changes on the same afternoon.
## althing v3 (U9b flag day, one-way, no rollback)
The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.
73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.
Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.
nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.
The v2 database is left inert on disk. There is no import path and none
was improvised.
## sec onto GPU0
GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.
Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
|
||
|
vh
|
8389470898 |
feat(mog-sec): promote the DFlash2 configuration into the compose stack
Operator approved after real-use testing. The experimental standalone container is retired and stacks/mog-sec is canonical again, with restart: unless-stopped so the configuration survives a reboot. Cutover verified against the container it replaces: KV pool 526,617 tokens at 1.10x concurrency, identical; zero restarts; both gateway aliases serving; DFlash2 confirmed drafting at k=7 with 231 draft tokens over 33 drafts; vision working at 2048x2048. One variable was deliberately dropped rather than carried over. The previous stack hardcoded PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, the validated container never set it, and the quant playbook records expandable_segments corrupting retained tensors in another context. The compose now defaults it empty via MOG_ALLOC_CONF. Promoting the stack as it stood would have shipped a variable the tested configuration did not have. The speculative config moves into a single MOG_SPEC_CONFIG carrying the whole JSON, because the two shapes are not interchangeable: dflash requires a model pointing at the drafter and MTP must not have one, so a method-plus-tokens template cannot express both. Also parameterised: MOG_DRAFT_MODEL, MOG_MM_PROCESSOR_KWARGS, MOG_MAX_NUM_BATCHED_TOKENS. The mm-processor image cap is now mandatory rather than incidental. The model's own preprocessor declares 4096x4096, which expands to 16384 image tokens and kills startup on builds that enforce the image-token count check. Adds the .env.example this stack never had, carrying the measured rationale for each value and the one-line rollback. |
||
|
vh
|
36c173c6a1 |
feat(mog-sec): quant + serve M.O.G.-SEC pen-test seat; PPL on gen; retire fable
Autonomous overnight run under the operator's full-autonomy grant. End state:
fleet up, gen seat untouched, a new verified pen-test seat serving where fable was.
PPL on the orcarouter gen seat (fable downed to free GPU1 for a nospec probe,
probe torn down after): mean 7.07 / median 5.76, within noise of heresy 6.910 /
5.625 and identical to our recipe's usual 7.059. The gen-seat search is settled.
M.O.G.-SEC: chose Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (rev deede677)
over the pre-made ModelOpt NVFP4, which was disqualified on W4A4 4-bit activations
(the AEON degradation mode, catastrophic on a 1M-context model), zero MTP tensors,
and ModelOpt format. Pulled, format-screened (P(<think>) 1.11e-05, clean), quanted
in-house to mixed NVFP4+FP8 (23.4 GB, MTP + vision preserved), and served in the
retired fable slot.
stacks/mog-sec ana-ml2 GPU1 :8019, KV 418,218 tok / 1.60x @ 262K
aliases mog-sec (non-thinking), mog-sec-reasoning (thinking)
gates surface 6/6, MTP 55.3%, format 0/15 leak, vision 7/3/1,
capability 4/4 (delivers offensive-security content)
Served at native 262K, NOT the card's 1M -- the 1M needs YaRN (absent from the
weights' config) plus the SGLang/DFlash2 path the repo ships a deployment kit for,
neither of which is our vLLM surface. A real 1M seat is a separate SGLang project.
Retired char-rp-reasoning + char-rp-fable (zero traffic, pointed at the downed
fable :8019; now 404 cleanly, not repointed -- a security model is not an RP model).
char-rp (meromero) untouched. Vision preprocessor built from the model's own
image_processor block, same trick as the MeroMero seat.
GPU0 seats (gen, meromero) were untouched and healthy throughout. The quant ran in
GPU1 free space with no production seat stopped except fable, which was replaced.
|