Operator: keep the benchmark. It has a named second use (brokkr-smithy-dev
wants gen vs a trained reward model once their tournament converges) and a
demonstrated first one -- it caught a seat that had been coin-flip-grade for
five weeks with nobody measuring it.
Harness promoted from scratch to tools/judge-bench/:
- paths de-hardcoded; runs from its own directory
- proper CLI: --models (REQUIRED), --repeats, --limit, --gateway.
Required on purpose: a stale default would silently benchmark a retired
seat, and the original default (selene-1-mini-8b) now 400s.
- README states the limitation rather than burying it: 24 items of the
author's own design, a screen and not a verdict. This harness scored the
same pair 83 vs 96 while brokkr's corpus ranking task scored it 47 (chance)
vs 94. Both honest; absolute scoring on designed items is an easier task
than ranking real text.
- records brokkr's technique, which is better than anything here: a control
constructed so the correct answer is DEFINITIONAL rather than judged cannot
inherit the designer's error (item vs itself, response vs its own
truncation, text vs its own clauses permuted). Add those before adding more
judged items.
Gateway: comment-only warning at the head of model_list. SEVEN aliases now
resolve to the same weights (chat-judge, classifier, gen, image-judge,
qwen-image-bench, summarizer, summarizer-large -> qwen3.8-27b-uncensored).
That is intended under ADR-0012, but it has a sharp edge brokkr flagged:
cross-checking a result against another alias measures NOTHING when they are
the same model -- agreement is an echo, not corroboration. The note names the
other current collisions (glm-5.2 x4, TTS x4, reranker x2), gives the
/model/info one-liner to check, and records that probes should resolve alias
-> backing at run start AND end because the response `model` field returns the
alias, so a swap is otherwise invisible.
Verified: config still parses, diff is comment-only, canonical re-synced.
Quants are hard-fought and we keep re-paying for the same lessons. A survey
found quant knowledge scattered across 18 files in four trees, with three
documents having independently discovered and recorded overlapping
"landmines" sections — and one of them now actively misleading.
Adds docs/pfi/model-quantization-playbook.md as the single home for the
TRANSFERABLE lessons, with per-model artifacts demoted to worked examples
that link up to it. Contents:
- scheme decision table, incl. that a literal "W4A8" NVFP4 checkpoint is
unservable on vLLM (two legal activation settings, FP8 is not one)
- the reference mixed-precision recipe and the three parts of it that are
load-bearing and easy to drop
- the recurring landmines, ordered by cost: the loader-class trap
(rediscovered THREE times), the three separate ways to lose the MTP head,
toolchain deadlocks, vision configs, memory/device placement
- pipeline shape: prove targets before spending GPU time; mandatory
post-steps that verify rather than assume
- the acceptance gate, and the three ways measurement has lied to us —
prefix caching faking both speed metrics, prompt_logprobs going uniform
under speculative decoding, and a 0600 .env making compose silently no-op
- hardware/co-residency, including that a SMALLER model can starve its
neighbour because gpu-memory-utilization is a fraction of the whole card
- a superseded-claims table, and measured negatives not to re-chase
The superseded table earns its place immediately: the heretic2 runbook tells
readers to use modelopt because "compressed-tensors can't load the BF16 MTP
head, 0% acceptance". That symptom was real but the cause was not the format
-- it was the missing re:^mtp.* ignore entry. compressed-tensors gives
47.7-83.2% acceptance in production. A fresh session following that doc would
be sent down the modelopt path that current memory calls dependency hell, so
the runbook now carries a stale-warning header pointing here.
Wires discovery: an orientation.md "Where to look for what" row, pointers
from the gen-seat / heretic2 / mistral artifacts, and a CLAUDE.md maintenance
rule so the playbook gets fed instead of going stale -- model-agnostic
lessons land in the playbook, model-specific ones stay put, and a wrong
claim earns a dated superseded row rather than a silent edit.
Motivated by Qwen3.8 having just released: the next model swap will need a
requant, and this is what that session should read first.
The TTS path was hardwired to the parked zonos-gateway: it force-reverted the
endpoint field back to zonos :8890 on load, hardcoded model=ext-tts, and decoded
the response as Zonos-specific raw float32 PCM @ 44.1kHz. Result: quoted-text TTS
was dead once zonos was parked, and pointing the field elsewhere silently failed.
- Honor the interface: set endpoint/model/voice defaults only when a field is
empty; never rewrite a user-typed value (removed the zonos auto-revert regex).
- Add a TTS model field (ttsModel); send the UI's model instead of hardcoding.
- Playback: request standard OpenAI /v1/audio/speech mp3 and decode via
audioCtx.decodeAudioData (handles wav/mp3/ogg/flac from any endpoint).
- Defaults: endpoint = LiteLLM ext-tts alias (fleet TTS gateway), voice = nova.
Point the in-page TTS at the zonos-gateway wrapper on irv-ml1:8890 (direct,
so streaming isn't buffered by LiteLLM): OpenAI-shape body (model: ext-tts,
input, voice), Cora default voice, float32@44.1kHz PCM decode. Quotes are
joined into a single stream call (prosody — no per-sentence chunking).
Migration regex rewrites stale saved endpoints (:8299/:8210, /tts[/stream],
:4000) to the new one.
Per-sentence chunking generated each sentence cold, flattening intonation/prosody that
spans the whole quoted line. Chunk by QUOTED SECTION instead — each contiguous quote is
generated whole (max_tokens 2400) so its prosody stays intact; multiple quotes in a reply
still play serially on the shared clock. extractQuotes already returns exactly these spans;
dropped splitSentences.
Split the quoted dialogue into sentences and stream each as its own short /tts/stream
request (max_tokens 900), queued back-to-back on one shared AudioContext clock (speechHead)
so playback is gapless and in order. First sentence starts fast; each chunk is short so it
generates cleanly (no ramble/cap risk); the next sentence generates while the current plays.
A newer reply supersedes via the ttsGen counter; 🔊 replays.
Browsers suspend the Web Audio AudioContext until a user gesture; speakQuotes fires on
reply-complete (no active gesture), so a suspended context played silently. Prime/resume
the context on any click or keydown (capture phase) so it's running before playback.
Server side was fine throughout (/tts + /tts/stream both 200 with valid audio).
- max_tokens default 2400->3500 (~42s) in wrapper + gateway-chat client, with a _cap()
clamp so prompt+gen never exceeds MAX_CTX (4096) — a cloning ref block is ~1100 tokens,
so an unclamped 3500 would overflow context on the clone path.
- Staged clone voices: /voices dir of <name>.wav + <name>.txt, each encoded to its Orpheus
reference block at startup; voice="<name>" zero-shot clones it. Beatrice (a chatterbox
reference) staged as the first normal-voice clone. GET /voices lists baddy + clones.
- compose: mount voices dir + pass MORPHEUS_MAX_LEN to the wrapper (clamp must match engine).
vLLM concurrency (measured, --max-num-seqs 8, 250-tok reqs): near-linear batching — 8
concurrent finish in the same ~2.8s as 1 (707 tok/s, 8.1x single, flat per-req latency).
Chunked-sentence production can fan out for ~8x throughput; CPU SNAC decode is the scale
bottleneck, not generation.
Cut-offs were the max_tokens=1200 ceiling (~14.6s of audio), not memory (~1250 tokens
<< 4096 context). Diagnosis: the repetition penalty is load-bearing for clean stops —
rep 1.0 => the model never emits end-of-speech and rambles to the cap; rep 1.1 (the
wrapper default) => clean natural stop. So normal lines already complete; only genuinely
long dialogue (>~14.6s, ~25+ words) hit the cap. Raised default + client max_tokens to
2400 (~29s), still within the 4096 context (no memory cost). Verified: a 49-word line
now finishes at 16.73s (was clipped at 14.6s).
Wrapper gains POST /tts/stream: reads the vLLM token stream, decodes SNAC in WINDOWED
CHUNKS (every 6 frames, decode [2 ctx | 6 | 2 ctx] and emit only the middle 6 — context
both sides => seamless), and streams raw PCM16 (24kHz mono) as it generates. Windowed
(not per-frame) because per-frame CPU decode's per-call overhead x ~60 frames serialized
to ~7s (RTF 2.2); windowed keeps up (RTF ~0.97). Whole-clip /tts kept for non-browser use.
gateway-chat plays the stream via the Web Audio API (fetch reader -> int16->float32 ->
scheduled AudioBufferSourceNodes on a running clock; a new reply supersedes the prior
stream via a generation counter; 🔊 replays). Measured: TTFA 0.80s (was ~4.5s whole-clip),
RTF 0.97, full-duration match. CORS already covers the new route.
Deployed: tts rebuilt on irv-ml1, page pushed to ana-docker.
Gateway-chat now auto-plays quoted text from each assistant reply through the mOrpheus
TTS endpoint. Sidebar gains a 🔊 toggle + endpoint/voice fields (persist in localStorage,
prefilled to irv-ml1:8299 / baddy). On reply-complete, straight and typographic double
quotes are extracted, joined, POSTed to /tts, and the returned WAV plays (click 🔊 to
replay; a new reply interrupts the prior clip).
Requires CORS on the wrapper (page served from ana-docker:8091 fetches irv-ml1:8299
cross-origin) — added CORSMiddleware(allow_origins=[*]) to the mOrpheus tts app (internal-
only endpoint). Verified end-to-end: preflight + POST return ACAO=*, valid 24kHz WAV.
Deployed: tts container rebuilt/recreated on irv-ml1; page pushed to ana-docker conf
(bind-mounted, live on next request).
Model field now pulls /v1/models (the ↻ control; new gateway models just appear)
instead of a hardcoded stale list; 📎 attaches an image (base64 data: URL in
image_url content) so the multimodal models (Qwopus, image-judge) can be smoked.
Static-verified (JS syntax + element-id consistency); headless smoke was blocked
by a shared-browser version skew in /opt/ms-playwright, not a tool defect.
Quantize a HF-format Mistral Small 4 (Mistral3ForConditionalGeneration MoE) to
NVFP4 with the vision tower intact, then convert HF NVFP4 -> Mistral native so
vLLM can serve it (there is no HF Mistral4 serving path in any vLLM version).
Built + validated end-to-end on ana-ml2 for the abliterated character-model
successor (darkc0de/Mistral-Small-4-119B-2603-heretic): quant -> dry-run (clean
vs the official native NVFP4 reference) -> convert -> serve-test (loads on the
native loader, correct text, vision functional).
Converter scaffold came from worldtree-codex (bf16 bin maps + fused-expert
split); fixed here: NVFP4 layer regexes (keep the `model.` prefix) + non-mmap
shard reads (ZFS large-mmap ENOMEM). nvfp4_quant.py is local. README documents
the pipeline + every gotcha that cost a failed run. Homed here per operator
direction (not Worldtree).
Zero-dependency, zero-backend HTML chat UI for the LiteLLM gateway. The
browser talks straight to :4000 (gateway CORS is open), so it's just one
file you open — no container, no stack. System-prompt textarea, model
datalist, streaming SSE, renders reasoning_content for the -thinking/
-reasoning models, settings persist in localStorage.
Deliberately never sends a `tools` field, sidestepping the vLLM "tools
must not be an empty array" bug that breaks the LiteLLM admin UI
playground for vLLM-backed models (litellm #6228; the gateway's
strip_empty_tools hook can't reach the UI's in-process completion call).
Verified against the live gateway: streams + parses a real completion
with no tools sent.
vh