Adds model_name kimi-k3 → openai/kimi-k3 @ https://api.moonshot.ai/v1
(OpenAI-compatible), keyed by MOONSHOT_API_KEY (compose env + .env.example
placeholder; real key on server only). Verified live through the gateway.
Two Moonshot constraints captured in the config comment + pinned: K3 accepts
ONLY temperature=1 (else 400), and it is a reasoning model (CoT in
reasoning_content, answer in content — needs adequate max_tokens or content
returns empty). Model id confirmed via /v1/models.
Mirror comfy-dev's operator-run allocator A/B result off irv-ml1: drop
--disable-cuda-malloc (ComfyUI keeps CUDA's default async allocator) and
remove PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (native-allocator-only,
inert under cudaMallocAsync). The native+expandable_segments combo was
fragmenting/over-reserving (~45 GB allocated-but-unused) and OOMing the LTX-2.3
v1.5.0 LoRA stack at Gemma TE load; cudaMallocAsync packs tighter + returns
freed blocks so the job fits (stress test peaks ~82% VRAM, 0 OOM). The
shared-A6000 phantom-OOM that --disable-cuda-malloc guarded is gone since TTS
moved to the 3090 (2026-06-18).
Record the RTXVideoSuperResolution node clone + the nvidia-vfx pip install
(scoped --extra-index-url, uid 1000) in the stacks/comfyui runbook. Flag the
durability split: the node is persistent (basedir/custom_nodes) but nvidia-vfx
lives in the disposable venv (run/), so it must re-run after every fresh
bootstrap. Deliberately not a global PIP_EXTRA_INDEX_URL (torch-pin safety).
Closes the comfy-dev provisioning ticket.
Mirror the canonical EMOTION-DIALS-SPEC.md from vh/zonos-gateway (now carries
the provisional per-voice emotion presets baked as gateway 0.2.1) and capture
the axes-sweep → bake arc in persistent memory.
Created private vh/zonos-gateway on gitea, imported the previously-unversioned
~/zonos-gateway working tree (source + dials-first spec + voices). Updated the
stack README, spec §8, and the sister-repos table to point at the repo. Remaining
follow-up: CI + deploy key to wire the irv-ml1 deploy tree to the repo.
Canonical direction: emotion set by twisting raw dials per-utterance, not preset
selection. Presets demoted to optional examples. Spec covers the dial vocabulary
+ ranges, the emotion_cfg_scale 'deaf by 1.5' rule (NO cap — documented ceiling,
explicit over implicit), measured RTF cost, starting-point dial-sets, an LLM
client system-prompt snippet, and clone reference guidance (~15-24s, no
transcript). Follow-ups flagged: align dials.py cfg help/metadata, trim in-code
PRESETS pending usage check, stand up vh/zonos-gateway for version control.
Captures the production Zonos TTS path that the repo was blind to: stock ZONOS2
:1920 engine (stacks/zonos-engine) fronted by zonos-gateway:0.2.0 :8890, reached
via the LiteLLM ext-tts alias. Documents the emotion-preset system (neutral/warm/
excited/sad/intense/whisper, the simple preset: caller path), the API, and the
measured real-time cost (calibrated steering free at RTF~0.52, cfg1.5 ~0.625 —
still realtime). Flags the gateway source (~/zonos-gateway on irv-ml1) as not yet
in gitea. Marks stacks/zonos (v0.1 Gradio) dead/superseded. Records the emotion-
lever finding (text-priming flat -> native steering works) in persistent memory.
The Zonos TTS engine that zonos-gateway fronts (irv-ml1 3090, feeds asset-engine +
gateway-chat) ran as a bare native process with its real invocation existing ONLY
in the running process argv — the committed harness/zonos_server.sh on irv-ml1 was
STALE (said A6000/:1919, no perf flags; live is 3090/:1920 with cuda-graph/num-pages/
max-running-requests/memory-ratio). Captured the corrected canonical invocation +
tunables + the containerization plan here so the config survives a process death.
Engine = stock Zyphra/Zonos2 @ 194c0a3 (no custom PFI server code); torch 2.9.1+cu128;
15 GB HF weights. Next: containerize in-place on the 3090 (operator: keep off the
A6000, it OOMs under ComfyUI). Not yet built — this commit is the config capture only.
granite (fleet fan-out summarizer/classifier) had no explicit --max-num-seqs,
so vLLM V1 resolved it to 128 — which caps concurrency BELOW granite's own KV
bound (~192 concurrent @ 1K-token calls, more for shorter classify calls).
Pinned it very high (1024) so the KV pool is the only bound; VRAM-neutral
(the KV pool is util-bound, unchanged). Added the flag to the granite command
+ GRANITE_MAX_NUM_SEQS to the env template. Live applied + verified
(resolved max_num_seqs=1024, seat healthy).
Operator-directed 2026-07-16. Moved the char-rp prose seat (Magidonia-24B,
llama-charrp) from GPU0 to GPU1 (CHARRP_GPU_ID 0->1; recreate llama-charrp
only -- the var is shared with the retired GGUF reasoning service), then
re-optimized every context-relevant seat on both cards to native/max context
with acceptable headroom:
GPU0 (both seats now 256K native, ~14 GB reserve):
- char-rp-reasoning 150K -> 256K (heretic2 stack, util 0.38->0.46, 1.56x)
- gen 256K, max-num-seqs 16 -> 32 (qwen36-27b-aeon, util 0.30->0.42, 5.43x)
GPU1 (~6.7 GB headroom):
- granite 64K -> 128K full-chapter (vllm stack, util 0.18->0.27, 1.50x)
- char-rp 128K native (4 slots), selene/reward/embed/rerank unchanged
All seats gateway-verified healthy. Live .env changes on ana-ml2 with per-stack
backups (*-20260716). Templates updated to match; the qwen36-27b-aeon template
carries a NOTE that its served-name/model still lag the 2026-07-08 gen model swap
(35B-A3B-heretic) -- separate reconciliation. persistent-memory records the full
layout + the util-floor / per-model-KV-cost lessons.
Note: the heretic2-charrp-reasoning stack (char-rp-reasoning's live config) is
still untracked in git -- standing open-loop, its .env change lives server-side only.
Operator-directed 2026-07-16: free ~10GB on ana-ml2 GPU1 to relocate a GPU0
model onto GPU1. granite-4.1-8b (fleet summarizer) was over-provisioned at
util 0.34 / max-model-len 131072 with a flat 0.0% KV usage.
Set GRANITE_GPU_MEM_UTIL 0.34 -> 0.18 and GRANITE_MAX_MODEL_LEN 131072 -> 65536
on the live /opt/docker/compose/vllm/.env (backup .env.bak-pre-granite-rightsize-
20260716), recreated vllm-granite ONLY (shared stack). Result: GPU1 62,641 ->
51,897 MiB used (~10.5GB freed, ~45GB free now); KV 6.45 GiB / 84,528 tok /
1.29x concurrency @ 65536; summarizer verified healthy.
The util drop required the max-len drop: on this shared card the effective KV
slope is ~950 MiB per 0.01 util, and vLLM refuses to start unless the KV pool
holds >= 1x max-model-len -- util 0.15 undershot (est max-len 47184 < 65536,
crash-loop, ~2-3 min summarizer outage) before 0.18 landed. 65536 is granite's
precedented summarizer ctx; a summarizer doesn't need 131072.
.env.example updated to the new util (max-len was already 65536 in the template;
live had drifted to 131072). persistent-memory.md updated (parked item closed).
Operator-directed 2026-07-15. The dedicated Qwen-Image-Bench NVFP4 judge
backend on ana-ml2 GPU1 (:8014) was stopped to reclaim ~32GB after the
arbo -> gen hero-judge switch. Both LiteLLM gateway aliases that pointed at
it -- image-judge and qwen-image-bench -- now repoint to the gen backend
(:8015, qwen3.6-35b-a3b-heretic, vision-intact), held at deterministic
judge sampling (temp 0 / top_k 1 / rep_pen 1.05) with enable_thinking:false
(a reasoning preamble breaks json_object). Verified live: both answer with
:8014 down, so they are definitively on gen.
Incidental: backfilled the canonical char-rp-reasoning litellm block, which
had lagged live since the 2026-07-14 NVFP4+MTP seat repoint (model
deckard-pkd-27b -> char-rp-reasoning, top_k 40 -> 20, min_p dropped,
enable_thinking:true added). Required so pushing the canonical would not
clobber the correct live block.
Live changes applied out-of-band (config push + litellm restart + stack
stop on ana-ml2); live config backup at
config.yaml.bak-pre-imagejudge-20260715. Revert path documented in the
config comment. persistent-memory.md updated (parked item closed).
Move the ~22-service flat "AI Systems" group off the Main tab into a new
four-tab layout (Main / AI / Infrastructure / Toolchain). The AI tab sorts
the inference fleet by function into seven groups:
AI - Inference gen, char-rp, char-rp-reasoning, Granite summarizer
AI - Eval & Retrieval Selene, Skywork Reward, Qwen3 rerank/embed, image-bench
AI - Gateways & Chat LiteLLM, Asset Engine, Gateway Chat, Open WebUI, ...
AI - Speech (TTS) Chatterbox Fast, Kokoro, mOrpheus
AI - Audio Tools Parakeet ASR, YT Voice Clipper
AI - Image & Media ComfyUI, Arbo
AI - Dormant stopped rollback seats + retired auditions
Relabel each stack's homepage.group so canonical stacks/ matches the live
containers on ana-ml2, ana-docker, and irv-ml1. Dormant stacks were refreshed
with `docker compose up --no-start` so they carry the new label while staying
stopped (compose-start rollback preserved). settings.yaml drives tab/order/
columns; services.yaml and README updated to the new scheme.
R36 gate (2026-07-14) validated NEO-CODE ships on all axes: tool-calling 0.967
(attach_tool 1.00, 0 runaways — #355 eliminated), prose genre-artifact-fine
(less clichéd than gen), refusal uncensored-as-spec + CSAM-clean.
#355 root cause was MODEL-level, not the reasoning-budget-forcing bug: Deckard
emitted Qwen's native qwen3_coder XML tool format malformed -> llama.cpp leaked
the closing tags into the arg value -> Bifrost attach_tool schema error -> retry
-> reasoning runaway to max_tokens. NEO-CODE emits the same native format cleanly
on the same seat/parser -> no schema error -> no runaway. The fix was the model
swap; there was never a wrong parser (the XML is Qwen3.5/3.6-native).
- reasoning seat: Deckard-PKD (Qwen3.5) -> NEO-CODE=Heretic2-Thinking (Qwen3.6-27B) Q5
- samplers: card defaults (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0), DRY dropped
- ctx: 256K max; custom llama.cpp kept (qwen3_coder parse + PR#25544 belt-and-suspenders)
- persistent-memory ACTIVE 1 marked resolved
The char-rp-reasoning (Deckard) seat now runs llamacpp-charrp:custom-latest via a
new LLAMA_REASONING_IMAGE var (Magidonia char-rp stays on stock — no reasoning bug).
Fixes Worldtree #355 at the source (budget multi-terminator handles Qwen3.5's
<tool_call> reasoning end-tag). Live 2026-07-13: Deckard loads on Blackwell, serves
coherent, reasoning bounds at the 400 budget. Rollback via .env LLAMA_REASONING_IMAGE.
Durable record of the custom llama.cpp the char-rp-reasoning seat will run to
fix Worldtree #355 (reasoning-budget forcing broken in stock b8840 — single
end-tag </think> can't match Qwen3.5's <tool_call> reasoning terminator, so the
budget never force-closes and reasoning runs away to max_tokens). PR #25544
adds multiple terminating sequences; unmerged upstream, so we build it.
- build.sh: reproducible recipe (clone master@6eddde0 + merge PR #25544 +
resolve the 1 server-common.cpp conflict + CUDA build for Blackwell sm_120)
- README.md: why + acceptance test + rollback + REMOVE-WHEN-MERGED tracking
Image llamacpp-charrp:6eddde0-pr25544 BUILT + smoke-tested on ana-ml2; seat
swap pending. See also auto-memory reference_charrp_custom_llamacpp_pr25544.
Drop-in system prompt for an agent whose quoted output is voiced by mOrpheus: speak in
double quotes (only quoted text is voiced), phrase each quoted line as one coherent
utterance (per-quote prosody), and the sparse/boundary/no-stack tag rules (measured
stability on the early checkpoint). Lists the honored tag set.
Per-sentence chunking generated each sentence cold, flattening intonation/prosody that
spans the whole quoted line. Chunk by QUOTED SECTION instead — each contiguous quote is
generated whole (max_tokens 2400) so its prosody stays intact; multiple quotes in a reply
still play serially on the shared clock. extractQuotes already returns exactly these spans;
dropped splitSentences.
Split the quoted dialogue into sentences and stream each as its own short /tts/stream
request (max_tokens 900), queued back-to-back on one shared AudioContext clock (speechHead)
so playback is gapless and in order. First sentence starts fast; each chunk is short so it
generates cleanly (no ramble/cap risk); the next sentence generates while the current plays.
A newer reply supersedes via the ttsGen counter; 🔊 replays.
Browsers suspend the Web Audio AudioContext until a user gesture; speakQuotes fires on
reply-complete (no active gesture), so a suspended context played silently. Prime/resume
the context on any click or keydown (capture phase) so it's running before playback.
Server side was fine throughout (/tts + /tts/stream both 200 with valid audio).
- max_tokens default 2400->3500 (~42s) in wrapper + gateway-chat client, with a _cap()
clamp so prompt+gen never exceeds MAX_CTX (4096) — a cloning ref block is ~1100 tokens,
so an unclamped 3500 would overflow context on the clone path.
- Staged clone voices: /voices dir of <name>.wav + <name>.txt, each encoded to its Orpheus
reference block at startup; voice="<name>" zero-shot clones it. Beatrice (a chatterbox
reference) staged as the first normal-voice clone. GET /voices lists baddy + clones.
- compose: mount voices dir + pass MORPHEUS_MAX_LEN to the wrapper (clamp must match engine).
vLLM concurrency (measured, --max-num-seqs 8, 250-tok reqs): near-linear batching — 8
concurrent finish in the same ~2.8s as 1 (707 tok/s, 8.1x single, flat per-req latency).
Chunked-sentence production can fan out for ~8x throughput; CPU SNAC decode is the scale
bottleneck, not generation.
Cut-offs were the max_tokens=1200 ceiling (~14.6s of audio), not memory (~1250 tokens
<< 4096 context). Diagnosis: the repetition penalty is load-bearing for clean stops —
rep 1.0 => the model never emits end-of-speech and rambles to the cap; rep 1.1 (the
wrapper default) => clean natural stop. So normal lines already complete; only genuinely
long dialogue (>~14.6s, ~25+ words) hit the cap. Raised default + client max_tokens to
2400 (~29s), still within the 4096 context (no memory cost). Verified: a 49-word line
now finishes at 16.73s (was clipped at 14.6s).
Wrapper gains POST /tts/stream: reads the vLLM token stream, decodes SNAC in WINDOWED
CHUNKS (every 6 frames, decode [2 ctx | 6 | 2 ctx] and emit only the middle 6 — context
both sides => seamless), and streams raw PCM16 (24kHz mono) as it generates. Windowed
(not per-frame) because per-frame CPU decode's per-call overhead x ~60 frames serialized
to ~7s (RTF 2.2); windowed keeps up (RTF ~0.97). Whole-clip /tts kept for non-browser use.
gateway-chat plays the stream via the Web Audio API (fetch reader -> int16->float32 ->
scheduled AudioBufferSourceNodes on a running clock; a new reply supersedes the prior
stream via a generation counter; 🔊 replays). Measured: TTFA 0.80s (was ~4.5s whole-clip),
RTF 0.97, full-duration match. CORS already covers the new route.
Deployed: tts rebuilt on irv-ml1, page pushed to ana-docker.
Gateway-chat now auto-plays quoted text from each assistant reply through the mOrpheus
TTS endpoint. Sidebar gains a 🔊 toggle + endpoint/voice fields (persist in localStorage,
prefilled to irv-ml1:8299 / baddy). On reply-complete, straight and typographic double
quotes are extracted, joined, POSTed to /tts, and the returned WAV plays (click 🔊 to
replay; a new reply interrupts the prior clip).
Requires CORS on the wrapper (page served from ana-docker:8091 fetches irv-ml1:8299
cross-origin) — added CORSMiddleware(allow_origins=[*]) to the mOrpheus tts app (internal-
only endpoint). Verified end-to-end: preflight + POST return ACAO=*, valid 24kHz WAV.
Deployed: tts container rebuilt/recreated on irv-ml1; page pushed to ana-docker conf
(bind-mounted, live on next request).
The gen seat's vLLM served-name was still qwen3.6-27b-aeon, a stale skin
left over from the AEON-27B → 35B-A3B-heretic swap — it named neither the
right family (aeon) nor size (27b vs 35B-A3B). Renamed the served-name to
qwen3.6-35b-a3b-heretic (+ -thinking) on ana-ml2 :8015 via the stack .env,
and repointed litellm's gen / gen-reasoning / summarizer-large model refs +
comments to match, so /v1/models, the gateway config, and spend-logs all
reveal the actual model in the request path.
Verified end-to-end: gen -> 'PIPELINE OK', gen-reasoning -> content + reasoning
surfaced, all three aliases healthy. char-rp / char-rp-reasoning untouched.
dvalin confirmed the live A/B-proven set IS canonical for Deckard as a dark-RP reasoning seat:
temp 1.0/top_p 0.95/top_k 40/min_p 0.05, no presence/rep penalty, DRY 0.8 server-side. Endorsed
over the card's base-thinking (top_k 20/min_p 0/presence 1.5). No value change; comment + memory
record the confirmation + tuning ladder (flat->min_p 0.08, loops->DRY 0.9, over-damped->DRY 0.6/off).
Operator wanted a reasoning-RP model that tolerates DRY (RpR-v4 forbids rep/DRY -> a
1/30 loop tail). Ran the full A/B on brokkr's 30-prompt D1 suite (content-only, slop-scored):
- Deckard-PKD (Qwen3.5-27B, DavidAU creative tune) WON: 0/30 loops, 0/30 refusals, clean
managed reasoning (native Qwen3.5 <think>/enable_thinking), DRY-tolerant, ~57 tok/s,
runs on the base llama-swap b8840 image. -> now the char-rp-reasoning seat (:8018).
- RpR-v4: 0 refusals but 1/30 loop (no-DRY). Pantheon-27B: clean slop but 7/30 explicit
refusals + needs the newer ggml-org/llama.cpp image (Qwen3.6 won't load on b8840).
Snowdrop + Gembrain (Gemma-4): floored (llama.cpp can't manage their reasoning without
the vetoed template hacks). Losers kept on disk as alternates.
- char-rp (Magidonia) unchanged; gen unchanged. gateway char-rp-reasoning -> Deckard
sampler (temp 1.0/top_p 0.95/top_k 40/min_p 0.05; DRY server-side).
char-rp -> TheDrummer Magidonia-24B-v4.3 Q6_K (Magistral prose, ~65 tok/s,
zero refusal, tight POV) via llama.cpp (:8016).
char-rp-reasoning -> ArliAI QwQ-32B-RpR-v4 Q5_K_M (abliterated managed reasoning,
~52 tok/s, reasoning surfaces in reasoning_content) via llama.cpp (:8018).
- New canonical stack stacks/char-rp-gguf/ (llama-server x2, GPU0-pinned, ~86/97G
co-resident with gen). GGUF sidesteps the vLLM-NVFP4 + Mistral-tokenizer traps that
killed the Angel serve. Never Ollama.
- Best-of-breed per seat: no single dense 24-32B is both an elite non-thinking prose
seat AND a clean managed-reasoning seat on llama.cpp (Magidonia [THINK] boundary is
loose; Cydonia-R1 <think> runs away; QwQ is template-managed). Pantheon-Reasoning-27B
stays rejected (re-censors in <think>; RpR-v4 abliterated reasoning is the fix).
- Gateway rewired: char-rp->:8016, char-rp-reasoning->:8018, Mistral/QwQ samplers,
dropped the Qwen enable_thinking kwarg. One-model Magidonia fallback documented.
- Retired the ms32-24b-angel stack.
New stacks/qwen36-27b-aeon: two co-located vLLM serves on ana-ml2 GPU0 —
gen (:8015, MTP off) and an RP seat (:8016, native MTP) — dense Qwen3.6-27B
(qwen3_5 GDN-hybrid, uncensored/abliterated), ModelOpt-NVFP4, multimodal,
256K context, depends_on-sequenced util split (~0.50/0.45). Each serve
carries a base + `-thinking` served-name so the `-reasoning` gateway records
target distinct LiteLLM deployments — otherwise a thinking-off request mutates
the shared litellm_params and clobbers enable_thinking (the shared-config
footgun that silently disabled char-rp-reasoning).
Gateway (stacks/litellm/conf/config.yaml): gen / gen-reasoning /
summarizer-large -> AEON :8015; char-rp / char-rp-reasoning added -> RP seat
:8016 (Qwen-RP sampler recs); gen-reasoning -> `-thinking`, char-rp-reasoning
-> `-rp-thinking`. Retired qwen3.5-122-a10b[-reasoning] + qwen-large[-reasoning]
(qwopus displaced; those named a 122B that no longer serves gen).
Probed live vs z.ai 2026-07-05: glm-5.2 = 1,048,576-token (1M) input context,
131,072 (128K) max output; no gateway-side cap (pure z.ai passthrough). Comment-only,
no runtime effect.
Displaced qwopus-122B on ana-ml2 GPU0:8013 with robbatt/Qwen3.6-40B-Deckard-NVFP4
(stock vLLM 0.23.0, loaded clean: hybrid attn + multimodal + fp4_gemm all green).
Repointed the 5 role aliases (gen, gen-reasoning, qwen-large, qwen-large-reasoning,
summarizer-large); added the qwen3.6-40b-deckard true-name record; left the true
names qwen3.5-122-a10b[-reasoning] to 404 (no-false-alias). Operator-directed
trial-by-fleet-traffic; revert path in the config banner + live backup
config.yaml.bak-pre-deckard-20260701-001036.
dvalin evidence pass: IBM canonical is temp 0; greedy-loop risk is an
open-ended-generation phenomenon, not summ/classify; temp 0.1 reduces
classification reproducibility without fixing loops (use repetition/presence
penalty if loops appear). image-judge stays 0 (Qwen judge card + W&B judge
practice = temp 0 for reproducibility; NVFP4-needs-0.1 unsupported). Both
gateway temps now 0, vendor-canonical.
Operator call: avoid pure-greedy rigidity/loop-risk on granite summ/classify
while staying near-deterministic; matches the house nonzero-temp-floor lean.
image-judge held at temp 0 (scoring reproducibility) pending operator review.
- granite-4.1-8b (+ summarizer/classifier): temperature 0 (IBM vendor-canonical
"temp 0 for inferencing"; top_p/top_k no-ops at temp 0, omitted). Deterministic
baseline for summ/classify; creative callers override.
- GLM family (z.ai cloud): temperature + top_p 0.95 only (the ONLY params z.ai
chat API accepts per its OpenAPI schema; top_k/min_p/penalties absent -> not set).
temp 1.0 for glm-5.1/5.2/5-turbo/4.7 + gen-frontier; temp 0.6 for glm-4.5-air.
Matches z.ai API defaults -> explicit-over-implicit, future-proofs vs vendor drift.
Round-2 dvalin-researched (provenance-labeled), verified live, granite+glm smoked 200.
Embeddings/rerankers excluded (no sampling). Fleet-wide canonical-defaults sweep complete.
The abliterated/NVFP4 Qwopus 122B "gen" model (+ qwen-large / summarizer-large
aliases) had no repetition control in its sampling defaults, causing degenerate
repetition loops. Add presence_penalty: 1.0 (Qwen-documented anti-repetition
lever, range 0-2) to all 7 qwen3.5-122-a10b gateway records. Overrideable
default; bake into the vLLM serving def once the value is validated.
Native-allocator expandable segments to cut Qwen-Image-Edit fragmentation
OOMs on the A6000 (a ~2 GB alloc failing with 1.75 GB free while 45 GB sat
allocated + reserved-but-unallocated). Cache-preserving — packs better
without unloading the checkpoint, so no edit-latency hit. Paired with the
existing --disable-cuda-malloc (incompatible with cudaMallocAsync).
Deployed + recreated on irv-ml1; verified env present, PyTorch reads it,
container healthy. comfy-dev request 2026-06-25.
Langfuse's ClickHouse member spewed ~94 GB of unrotated logs and filled ana-docker's
root disk (took the fleet host to 100%, 28/48 containers unhealthy). Its trace UI was
redundant with LiteLLM's native logging — store_prompts_in_spend_logs:true already
captures full prompts/responses/tokens/cost/latency at :4000/ui — and nothing used its
unique trace-grouping/eval features (it only received flat gateway success_callbacks).
Removed the callbacks (gateway observability stays fully native) and tore down the
6-container langfuse stack + volumes on ana-docker. Re-add the callbacks if it returns.
Capability aliases for the PAID frontier tier, mirroring glm-5.2 / glm-5.2-
reasoning (thinking off / on) → openai/glm-5.2 @ z.ai. Worldtree binds these for
a frontier-grade generation/reasoning capability so the backing frontier model
can be swapped gateway-side (operator jump-started WT's request). PAID: only
all-proxy-models / explicitly-scoped keys reach them; the free all-agents-local
key stays fenced off z.ai spend. Verified both resolve + route to GLM 5.2.
llama-swap (ana-ml2:9292) is decommissioned (:9292 confirmed down), so the
catch-all wildcard routed every unmatched / typo'd / stale model name to a DEAD
backend, surfacing a misleading "Connection error" instead of a clean
"model not found". This is the exact footgun that silently swallowed Worldtree's
defunct model names (mistral-small-4 etc.) instead of erroring. Removed (operator
call) so unknown models now 404 loudly. Verified: gateway healthy post-restart,
a bogus model name now returns a clean not-found error, real aliases (gen) still
serve. Re-add explicit per-model entries if a swappable zoo ever returns.
Stand up the gateway-side capability aliases for the role→capability model
indirection (worldtree-dev's transparent-swap direction; operator: no wt-
prefix, reuse the existing summarizer/classifier/gen alias convention).
- chat-judge -> selene-1-mini-8b (mode chat) — WT selene-judgment role.
- reranker -> qwen3-reranker (mode rerank) — generic name for the cap.
- scalar-judge -> Skywork-Reward-V2 via a pass_through_endpoint to ana-ml2:8003
(LiteLLM has no reward/pooling MODE, so it's a passthrough, gateway-key-gated;
consumers hit /scalar-judge/<route> e.g. /score|/pooling|/classify).
Deliberately NO generic `embedding` alias: embedding vectors are model-specific
(not swap-transparent), so that capability stays `qwen3-embedding` — the model-
specific name is the guardrail against treating it as freely swappable. Verified
all three live (chat-judge 200, reranker present, scalar-judge passthrough 200
returning a Skywork reward). Deployed + gateway health-gated.
Empirical follow-up to the streaming /tts smoke test on the 3090. OmniVoice
is diffusion: a ~fixed per-call overhead (~1.5s at 32 steps, ~0.7s at 16)
dominates regardless of chunk length, so the upstream-claimed 40x RTF does
NOT hold here (measured ~2.8x/32-step, ~5.6x/16-step) and the chatterbox-
tuned scheduler over-chunks and starves.
- Streaming /tts defaults to num_step=16 (TTFA ~1.5s -> ~0.7s); batch
/v1/audio/speech stays num_step=32 for quality. Per-request override intact.
- Scheduler prior raised to rtf_prior=20 (env OMNIVOICE_STREAM_RTF_PRIOR,
wired through compose + .env.example) so it packs whole-text-minus-first-
sentence into a few chunks: validated ~3 chunks, no starvation, total wall
~= one-shot, less per-chunk silence padding.
- Docs corrected: the "sub-second / 40x" claims were wrong; streaming has a
diffusion TTFA floor (~0.7s) and wins mainly on long replies. chatterbox-
fast (autoregressive, ~0.5s TTFA) stays the lowest-latency front-end;
OmniVoice is the multilingual / voice-design complement.
Add a live-consumer streaming path and text sanitation to the OmniVoice
wrapper, so it can front speech-to-speech chat engines (not just the
asset-engine's batch WAV use).
- POST /tts: chunked 24 kHz mono s16le PCM (or open-ended WAV), driven by
the adaptive buffer-ratchet scheduler. Emits the first sentence
immediately, then ratchets chunk size up on OmniVoice's ~40x realtime
headroom -> sub-second time-to-first-audio. Wire-compatible with
chatterbox-fast /tts (both 24 kHz mono PCM). Batch /v1/audio/speech is
unchanged for asset/file callers.
- scheduler.py: VENDORED byte-faithful copy of chatterbox-fast's pure-
Python (torch-free) scheduler, pinned to commit 7631462 (v0.1.0/v0.1.1).
Vendor-copy over a shared package (operator call 2026-06-19): the module
has no GPU deps, so reuse it without dragging chatterbox-fast's torch
tree into this image. Promote to a shared package only on a 3rd consumer
or real drift.
- sanitize.py: language-safe TTS sanitizer run on both endpoints. Strips
markdown, <think> blocks, HTML, and model control tokens; deliberately
SKIPS the fork's English-only number/phone normalization that would
corrupt OmniVoice's 600-language input. Preserves [laughter]-style tags.
- Refactor: shared GenParams base for SpeechRequest + TTSStreamRequest;
single GEN_LOCK serializes generation (single-stream interactive).
- Dockerfile/playbook: copy + upload the two new modules; build-time
`import app` smoke; correct stale "Gradio demo / no FastAPI" comments.
vh