Operator call: avoid pure-greedy rigidity/loop-risk on granite summ/classify
while staying near-deterministic; matches the house nonzero-temp-floor lean.
image-judge held at temp 0 (scoring reproducibility) pending operator review.
- granite-4.1-8b (+ summarizer/classifier): temperature 0 (IBM vendor-canonical
"temp 0 for inferencing"; top_p/top_k no-ops at temp 0, omitted). Deterministic
baseline for summ/classify; creative callers override.
- GLM family (z.ai cloud): temperature + top_p 0.95 only (the ONLY params z.ai
chat API accepts per its OpenAPI schema; top_k/min_p/penalties absent -> not set).
temp 1.0 for glm-5.1/5.2/5-turbo/4.7 + gen-frontier; temp 0.6 for glm-4.5-air.
Matches z.ai API defaults -> explicit-over-implicit, future-proofs vs vendor drift.
Round-2 dvalin-researched (provenance-labeled), verified live, granite+glm smoked 200.
Embeddings/rerankers excluded (no sampling). Fleet-wide canonical-defaults sweep complete.
The abliterated/NVFP4 Qwopus 122B "gen" model (+ qwen-large / summarizer-large
aliases) had no repetition control in its sampling defaults, causing degenerate
repetition loops. Add presence_penalty: 1.0 (Qwen-documented anti-repetition
lever, range 0-2) to all 7 qwen3.5-122-a10b gateway records. Overrideable
default; bake into the vLLM serving def once the value is validated.
Native-allocator expandable segments to cut Qwen-Image-Edit fragmentation
OOMs on the A6000 (a ~2 GB alloc failing with 1.75 GB free while 45 GB sat
allocated + reserved-but-unallocated). Cache-preserving — packs better
without unloading the checkpoint, so no edit-latency hit. Paired with the
existing --disable-cuda-malloc (incompatible with cudaMallocAsync).
Deployed + recreated on irv-ml1; verified env present, PyTorch reads it,
container healthy. comfy-dev request 2026-06-25.
Langfuse's ClickHouse member spewed ~94 GB of unrotated logs and filled ana-docker's
root disk (took the fleet host to 100%, 28/48 containers unhealthy). Its trace UI was
redundant with LiteLLM's native logging — store_prompts_in_spend_logs:true already
captures full prompts/responses/tokens/cost/latency at :4000/ui — and nothing used its
unique trace-grouping/eval features (it only received flat gateway success_callbacks).
Removed the callbacks (gateway observability stays fully native) and tore down the
6-container langfuse stack + volumes on ana-docker. Re-add the callbacks if it returns.
Capability aliases for the PAID frontier tier, mirroring glm-5.2 / glm-5.2-
reasoning (thinking off / on) → openai/glm-5.2 @ z.ai. Worldtree binds these for
a frontier-grade generation/reasoning capability so the backing frontier model
can be swapped gateway-side (operator jump-started WT's request). PAID: only
all-proxy-models / explicitly-scoped keys reach them; the free all-agents-local
key stays fenced off z.ai spend. Verified both resolve + route to GLM 5.2.
llama-swap (ana-ml2:9292) is decommissioned (:9292 confirmed down), so the
catch-all wildcard routed every unmatched / typo'd / stale model name to a DEAD
backend, surfacing a misleading "Connection error" instead of a clean
"model not found". This is the exact footgun that silently swallowed Worldtree's
defunct model names (mistral-small-4 etc.) instead of erroring. Removed (operator
call) so unknown models now 404 loudly. Verified: gateway healthy post-restart,
a bogus model name now returns a clean not-found error, real aliases (gen) still
serve. Re-add explicit per-model entries if a swappable zoo ever returns.
Stand up the gateway-side capability aliases for the role→capability model
indirection (worldtree-dev's transparent-swap direction; operator: no wt-
prefix, reuse the existing summarizer/classifier/gen alias convention).
- chat-judge -> selene-1-mini-8b (mode chat) — WT selene-judgment role.
- reranker -> qwen3-reranker (mode rerank) — generic name for the cap.
- scalar-judge -> Skywork-Reward-V2 via a pass_through_endpoint to ana-ml2:8003
(LiteLLM has no reward/pooling MODE, so it's a passthrough, gateway-key-gated;
consumers hit /scalar-judge/<route> e.g. /score|/pooling|/classify).
Deliberately NO generic `embedding` alias: embedding vectors are model-specific
(not swap-transparent), so that capability stays `qwen3-embedding` — the model-
specific name is the guardrail against treating it as freely swappable. Verified
all three live (chat-judge 200, reranker present, scalar-judge passthrough 200
returning a Skywork reward). Deployed + gateway health-gated.
Empirical follow-up to the streaming /tts smoke test on the 3090. OmniVoice
is diffusion: a ~fixed per-call overhead (~1.5s at 32 steps, ~0.7s at 16)
dominates regardless of chunk length, so the upstream-claimed 40x RTF does
NOT hold here (measured ~2.8x/32-step, ~5.6x/16-step) and the chatterbox-
tuned scheduler over-chunks and starves.
- Streaming /tts defaults to num_step=16 (TTFA ~1.5s -> ~0.7s); batch
/v1/audio/speech stays num_step=32 for quality. Per-request override intact.
- Scheduler prior raised to rtf_prior=20 (env OMNIVOICE_STREAM_RTF_PRIOR,
wired through compose + .env.example) so it packs whole-text-minus-first-
sentence into a few chunks: validated ~3 chunks, no starvation, total wall
~= one-shot, less per-chunk silence padding.
- Docs corrected: the "sub-second / 40x" claims were wrong; streaming has a
diffusion TTFA floor (~0.7s) and wins mainly on long replies. chatterbox-
fast (autoregressive, ~0.5s TTFA) stays the lowest-latency front-end;
OmniVoice is the multilingual / voice-design complement.
Add a live-consumer streaming path and text sanitation to the OmniVoice
wrapper, so it can front speech-to-speech chat engines (not just the
asset-engine's batch WAV use).
- POST /tts: chunked 24 kHz mono s16le PCM (or open-ended WAV), driven by
the adaptive buffer-ratchet scheduler. Emits the first sentence
immediately, then ratchets chunk size up on OmniVoice's ~40x realtime
headroom -> sub-second time-to-first-audio. Wire-compatible with
chatterbox-fast /tts (both 24 kHz mono PCM). Batch /v1/audio/speech is
unchanged for asset/file callers.
- scheduler.py: VENDORED byte-faithful copy of chatterbox-fast's pure-
Python (torch-free) scheduler, pinned to commit 7631462 (v0.1.0/v0.1.1).
Vendor-copy over a shared package (operator call 2026-06-19): the module
has no GPU deps, so reuse it without dragging chatterbox-fast's torch
tree into this image. Promote to a shared package only on a 3rd consumer
or real drift.
- sanitize.py: language-safe TTS sanitizer run on both endpoints. Strips
markdown, <think> blocks, HTML, and model control tokens; deliberately
SKIPS the fork's English-only number/phone normalization that would
corrupt OmniVoice's 600-language input. Preserves [laughter]-style tags.
- Refactor: shared GenParams base for SpeechRequest + TTSStreamRequest;
single GEN_LOCK serializes generation (single-stream interactive).
- Dockerfile/playbook: copy + upload the two new modules; build-time
`import app` smoke; correct stale "Gradio demo / no FastAPI" comments.
9 Recent decisions + 6 Tried-and-abandoned (dates [2026-06-14]/[2026-06-15]) moved
non-destructively to archival-memory.md, each stamped _Archived 2026-06-19._. Kept
the active [2026-06-14] 'migrate ALL infra access to Claude-specific credentials'
standing directive. Back-ref counts: Recent decisions 79->88, Tried-and-abandoned
70->76. persistent-memory.md 397->363 lines.
Stands up tools/gateway-chat.html as a permanent URL on ana-docker (http://10.250.50.70:8091)
via a tiny nginx:alpine static container (no GPU, no DB). conf/index.html is a deployed
mirror of tools/gateway-chat.html (re-sync one-liner in README). Homepage tile + tnet per
convention. The enhanced tool (auto-discovers /v1/models, system prompts, streaming +
reasoning, image upload for vision) is now always-on for smoking new gateway models.
Duplicate-entry aliases. classifier -> granite-4.1-8b (:8004, same backend as the
existing summarizer alias). summarizer-large -> gen/qwen3.5-122-a10b (:8013, thinking
off) for heavier summarization on the 122B Qwopus. summarizer -> granite already
existed (no-op). Config-staged + deployed without bouncing the gateway; like any
config-add these activate on the next restart (no live-add performed).
Model field now pulls /v1/models (the ↻ control; new gateway models just appear)
instead of a hardcoded stale list; 📎 attaches an image (base64 data: URL in
image_url content) so the multimodal models (Qwopus, image-judge) can be smoked.
Static-verified (JS syntax + element-id consistency); headless smoke was blocked
by a shared-browser version skew in /opt/ms-playwright, not a tool defect.
Source + deployed config cleaned without bouncing the gateway. NOTE: these were
config-loaded models, which the /model/delete API can't remove (DB-only -> 'not
found in db'), so the LIVE gateway still serves them until its next restart, at
which point the cleaned config drops them. No bounce performed.
Same Qwopus gen model as gen / gen-reasoning (served-name qwen3.5-122-a10b @
:8013, thinking off/on respectively), but each bakes a dummy 'noop' function tool
+ tool_choice:none into litellm_params so a NON-EMPTY tools array always reaches
vLLM — for consumers where the global strip_empty_tools hook isn't the right fix
(they need a valid tools structure present, not stripped). tool_choice:none means
the noop is never called. api_base = the real LAN endpoint http://10.250.50.54:8013
(the requested http://vllm:8000 template wouldn't resolve from the ana-docker
litellm container). Verified: gen-nt + gen-reasoning-nt both survive a client
tools:[] send; noop never invoked; reasoning split intact.
Replaces the bjk110 text-only qwen3.5-122b as the `gen` model on ana-ml2 GPU 0.
OpenYourMind/Qwopus3.5-122B-A10B-Kimi-K2.6-destilled-abliterated-NVFP4 — Kimi-
distilled, abliterated, NVFP4, and crucially VISION-INTACT (serves as plain
multimodal, no text-only patch). Served as qwen3.5-122-a10b so the litellm
gen / gen-reasoning / qwen-large records route here unchanged.
Tuned for full native context on the 96GB Blackwell:
- stable vLLM image + fp8 KV → 11GB pool = 870,014 tokens = 3.32x concurrency
at the full 262144 (256K) window. Nightly+turboquant-4bit was unnecessary.
- CUDA graphs ON (no --enforce-eager) → 92.7 tok/s warm single-stream.
- util 0.95 + PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — 0.96 OOM'd by
0.1GB on the 3.09GB FusedMoE transient workspace (the hard floor; defrag
reclaims the 4.2GB fragmentation, 0.95 adds margin).
- max-num-seqs 16 (short reqs fan out ~16x32k; 256K reqs pool-limit to 3.32x).
- text + image + video all enabled; tool-calling via qwen3_coder (XML), verified.
gen/gen-reasoning tool-calling 400'd (operator + brokkr's capability battery both caught it):
the bjk110 serve command shipped --reasoning-parser qwen3 but no tool flags. Qwen3.5 emits XML
tool calls (<tool_call><function=NAME><parameter=K>V</parameter></function></tool_call>), NOT
Hermes JSON — so `hermes` mis-parsed to raw text; `qwen3_xml` is the correct parser. Reasoning +
tools coexist (gen-reasoning keeps its thinking split). Verified live: a get_weather request
returns tool_calls=[get_weather {"city":"Paris"}].
bjk110/Qwen3.5-122B-A10B-abliterated-NVFP4 on ana-ml2 GPU 0 (heretic downed):
- stacks/qwen3.5-122b/ — vLLM serve via the repo's text-only patch (Qwen3.5 MoE is a
multimodal arch but this checkpoint is text-only weights), --reasoning-parser qwen3,
GPU 0 pin, :8013; entrypoint+patch mounted from the model dir.
- serve-qwen3.5-122b.yaml — displace heretic + serve + verify.
- litellm: REMOVED dead mistral-small-4 / -reasoning; added qwen3.5-122-a10b[-reasoning]
+ aliases qwen-large[-reasoning] + repointed gen[-reasoning] -> qwen (thinking split via
chat_template_kwargs.enable_thinking + --reasoning-parser qwen3).
Verified live: qwen healthy on :8013; gen / qwen-large / qwen3.5-122-a10b route, and
gen-reasoning returns reasoning_content; mistral-small-4 removed.
NOTE: Worldtree character backend (was bound to mistral-small-4) is dark until repointed
(operator-acknowledged).
Duplicate-entry aliases (NOT router_settings.model_group_alias — that's hidden from
/v1/models and can be silently ignored in config per litellm #15020/#5524):
- summarizer -> granite-4.1-8b
- gen -> mistral-small-4
- gen-reasoning -> mistral-small-4-reasoning (reasoning_effort:high preserved)
Each alias is a real model_name co-located with its target (keep api_base in sync).
Verified live: all 3 in /v1/models + route end-to-end; gen-reasoning returns
reasoning_content.
Wrapper /v1/audio/speech now accepts OmniVoice's whole surface:
- voice (clone, now OPTIONAL) and/or instruct (voice DESIGN). instruct is a CONTROLLED
vocabulary (gender/age/pitch/accent/whisper tags, comma-separated), not free prose —
discoverable at the new /v1/audio/instruct-items endpoint (23 items).
- language (Auto + 647, new /v1/audio/languages endpoint), speed, duration.
- diffusion controls: num_step, guidance_scale, denoise, preprocess_prompt,
postprocess_output; plus a generation_overrides JSON passthrough for expert
GenerationConfig knobs (t_shift, layer_penalty_factor, position/class temperature,
audio_chunk_*).
- at least one of voice/instruct required (else 400).
Catalog (services.yaml): omnivoice v1 -> v2, 13 schema-valid fields; instruct as a
controlled-vocab text field sourced from the items endpoint.
Verified live on irv-ml1: clone, voice-design (instruct-only), and tuned-param synths
all -> 24 kHz PCM_16 WAV; 647 languages; 23 instruct items.
- app.py: thin FastAPI wrapper exposing OpenAI /v1/audio/speech (+ /v1/audio/voices,
/healthz) around OmniVoice's Python API; precomputes a voice-clone prompt per voice
at startup (loaded Whisper auto-transcribes each reference). Replaces the Gradio demo.
- Dockerfile/compose: run the uvicorn wrapper, /healthz healthcheck, project name pinned
to "omnivoice" so the asset-engine liveness probe matches.
- deploy-omnivoice.yaml: stage chatterbox /refs/*.wav as clone voices (skip _* artifacts)
+ verify the API surface.
- services.yaml: catalog entry (id omnivoice, :8199/v1/audio/speech, voice list sourced
live from /v1/audio/voices) + reproducibility_audit row.
Verified live on irv-ml1: /healthz ok, 33 voices loaded, test synth -> 24kHz PCM_16 WAV.
Zero-shot, massively-multilingual (600+ language) voice-cloning + voice-design
TTS (diffusion-LM, Apache-2.0). No official image, so a thin CUDA container
around the pip package running upstream's own Gradio demo (no FastAPI wrapper).
Pinned to GPU 0 (3090) — the A6000 is ComfyUI-exclusive — port 8199. Built +
verified live on irv-ml1 (Gradio 200, container healthy). Surface is the Gradio
UI + Gradio API, NOT OpenAI-compat /v1/audio/speech (wrap later if asset-engine
should consume it). deploy-omnivoice.yaml builds local + verifies.
ComfyUI 0.24.1 added native attention selection; the node-based
BlehGlobalSageAttention errors "does not support the new ComfyUI attention
changes". Add --use-sage-attention to COMFY_CMDLINE_EXTRA so the in-image
sageattention v2.2.0 sm_86 build (rebuilt vs pinned torch 2.12.1) binds via
the native path. OOM flags preserved. Deployed to irv-ml1 + recreated; log
confirms "Using sage attention", container healthy, serving 200.
(comfy-dev request, thread 01KVE89T2DKC)
vh