Operator-directed 2026-07-16. Moved the char-rp prose seat (Magidonia-24B,
llama-charrp) from GPU0 to GPU1 (CHARRP_GPU_ID 0->1; recreate llama-charrp
only -- the var is shared with the retired GGUF reasoning service), then
re-optimized every context-relevant seat on both cards to native/max context
with acceptable headroom:
GPU0 (both seats now 256K native, ~14 GB reserve):
- char-rp-reasoning 150K -> 256K (heretic2 stack, util 0.38->0.46, 1.56x)
- gen 256K, max-num-seqs 16 -> 32 (qwen36-27b-aeon, util 0.30->0.42, 5.43x)
GPU1 (~6.7 GB headroom):
- granite 64K -> 128K full-chapter (vllm stack, util 0.18->0.27, 1.50x)
- char-rp 128K native (4 slots), selene/reward/embed/rerank unchanged
All seats gateway-verified healthy. Live .env changes on ana-ml2 with per-stack
backups (*-20260716). Templates updated to match; the qwen36-27b-aeon template
carries a NOTE that its served-name/model still lag the 2026-07-08 gen model swap
(35B-A3B-heretic) -- separate reconciliation. persistent-memory records the full
layout + the util-floor / per-model-KV-cost lessons.
Note: the heretic2-charrp-reasoning stack (char-rp-reasoning's live config) is
still untracked in git -- standing open-loop, its .env change lives server-side only.
Operator-directed 2026-07-16: free ~10GB on ana-ml2 GPU1 to relocate a GPU0
model onto GPU1. granite-4.1-8b (fleet summarizer) was over-provisioned at
util 0.34 / max-model-len 131072 with a flat 0.0% KV usage.
Set GRANITE_GPU_MEM_UTIL 0.34 -> 0.18 and GRANITE_MAX_MODEL_LEN 131072 -> 65536
on the live /opt/docker/compose/vllm/.env (backup .env.bak-pre-granite-rightsize-
20260716), recreated vllm-granite ONLY (shared stack). Result: GPU1 62,641 ->
51,897 MiB used (~10.5GB freed, ~45GB free now); KV 6.45 GiB / 84,528 tok /
1.29x concurrency @ 65536; summarizer verified healthy.
The util drop required the max-len drop: on this shared card the effective KV
slope is ~950 MiB per 0.01 util, and vLLM refuses to start unless the KV pool
holds >= 1x max-model-len -- util 0.15 undershot (est max-len 47184 < 65536,
crash-loop, ~2-3 min summarizer outage) before 0.18 landed. 65536 is granite's
precedented summarizer ctx; a summarizer doesn't need 131072.
.env.example updated to the new util (max-len was already 65536 in the template;
live had drifted to 131072). persistent-memory.md updated (parked item closed).
Operator-directed 2026-07-15. The dedicated Qwen-Image-Bench NVFP4 judge
backend on ana-ml2 GPU1 (:8014) was stopped to reclaim ~32GB after the
arbo -> gen hero-judge switch. Both LiteLLM gateway aliases that pointed at
it -- image-judge and qwen-image-bench -- now repoint to the gen backend
(:8015, qwen3.6-35b-a3b-heretic, vision-intact), held at deterministic
judge sampling (temp 0 / top_k 1 / rep_pen 1.05) with enable_thinking:false
(a reasoning preamble breaks json_object). Verified live: both answer with
:8014 down, so they are definitively on gen.
Incidental: backfilled the canonical char-rp-reasoning litellm block, which
had lagged live since the 2026-07-14 NVFP4+MTP seat repoint (model
deckard-pkd-27b -> char-rp-reasoning, top_k 40 -> 20, min_p dropped,
enable_thinking:true added). Required so pushing the canonical would not
clobber the correct live block.
Live changes applied out-of-band (config push + litellm restart + stack
stop on ana-ml2); live config backup at
config.yaml.bak-pre-imagejudge-20260715. Revert path documented in the
config comment. persistent-memory.md updated (parked item closed).
Split the 53 over-threshold dated log entries (Recent decisions, Tried
and abandoned) into per-entry persistent-memory.d/<slug>.md detail files,
leaving one-line pointers in the index; the 10 short entries stay inline.
Startup index drops 60,527 -> 25,256 bytes (492 -> 270 lines); entry
bodies move verbatim to on-demand detail files, so a fresh session loads
~25 KB instead of ~60 KB and pulls a detail file only when its pointer is
relevant. Top matter (Repo purpose, Tools & conventions, Current state)
is unchanged; both archival back-references preserved.
CLAUDE.md persistent-memory section now documents the index<->detail read
discipline (read the index, pull details on demand, never bulk-read the
dir, commit both together).
Auto-archival still held every dated entry back (all <30 days old); the
July burst begins aging past the 30-day guard ~2026-07-31.
The prior fix (_netdev,nofail + docker.service After=remote-fs.target) looked
correct but silently failed — paperless still Exited(255) on the 2026-07-14
reboot. Root cause: `nofail` drops a mount out of remote-fs.target's blocking
set, so ordering docker After=remote-fs.target does NOT wait for the nofail
NFS mounts. Fix: add x-systemd.before=docker.service,x-systemd.mount-timeout=30
to the 4 NFS fstab lines (direct mount->docker ordering, nofail-safe). Applied
+ verified live (systemctl show docker -p After now lists all 4 mnt-*.mount).
Playbook + verify updated to canonicalize.
Move the ~22-service flat "AI Systems" group off the Main tab into a new
four-tab layout (Main / AI / Infrastructure / Toolchain). The AI tab sorts
the inference fleet by function into seven groups:
AI - Inference gen, char-rp, char-rp-reasoning, Granite summarizer
AI - Eval & Retrieval Selene, Skywork Reward, Qwen3 rerank/embed, image-bench
AI - Gateways & Chat LiteLLM, Asset Engine, Gateway Chat, Open WebUI, ...
AI - Speech (TTS) Chatterbox Fast, Kokoro, mOrpheus
AI - Audio Tools Parakeet ASR, YT Voice Clipper
AI - Image & Media ComfyUI, Arbo
AI - Dormant stopped rollback seats + retired auditions
Relabel each stack's homepage.group so canonical stacks/ matches the live
containers on ana-ml2, ana-docker, and irv-ml1. Dormant stacks were refreshed
with `docker compose up --no-start` so they carry the new label while staying
stopped (compose-start rollback preserved). settings.yaml drives tab/order/
columns; services.yaml and README updated to the new scheme.
The fast char-rp-reasoning seat works: ~77 tok/s (vs GGUF ~59.5, base NVFP4 ~53),
MTP draft-acceptance 32-40%, mean acceptance length 2.19. Same Heretic2/NEO-CODE
model, NVFP4 + native qwen3_5_mtp spec-decode.
Full end-to-end recipe + the four landmines in docs/runbooks/heretic2-nvfp4-mtp-seat.md:
(1) load as AutoModelForImageTextToText not AutoModelForCausalLM (namespace/gibberish);
(2) modelopt format not compressed-tensors (compressed-tensors MTP = 0% accept);
(3) modelopt 0.45 <-> transformers 5.12.1 FusedMoE crash (guarded in quant_modelopt.py);
(4) vLLM 0.24.0 does NOT propagate modelopt exclude_modules to the spec-decode draft
model -> BF16 mtp head gets quantized -> shape crash; no checkpoint config fixes it
(is_layer_skipped is exact-membership not glob) -> fix is a mounted sitecustomize that
force-skips mtp.* in is_layer_skipped (upstream vLLM bug to report).
Scripts: quant_modelopt.py (FusedMoE guard + single-shard export + multimodal load),
finalize_modelopt_mtp.py (splice bf16 mtp), serve_modelopt_mtp.sh, run_quant_modelopt.sh,
sitecustomize-mtp-workaround.py.
Root-caused the NVFP4 gibberish to a quant-namespace bug: quant_nvfp4.py loaded
via AutoModelForCausalLM -> text-only Qwen3_5ForCausalLM -> flat model.layers.* keys,
but vLLM 0.24 serves only Qwen3_5ForConditionalGeneration (whose weight mapper needs
model.language_model.*). Fixed by loading as AutoModelForImageTextToText; NVFP4 now
serves coherent (validated greedy on ana-ml2 GPU0).
Base NVFP4 (compressed-tensors) measured ~53 tok/s (~= GGUF at batch-1, no single-stream
win) and its MTP is 0% acceptance (vLLM's Qwen3_5MTP drafter loads the bf16 mtp head only
off a modelopt main-model checkpoint). Added quant_modelopt.py (nvidia-modelopt PTQ,
matches AEON's NVFP4 W4A4 g16 + lm_head/linear_attn/visual exclusions) as the path to
working native MTP; graft + splice + serve otherwise unchanged.
Captures the full pipeline recipe (graft->quant->splice->config->serve) with every
gotcha found this session, the 3 gibberish suspects, and the diagnostic ladder
(validate native-config no-MTP coherence FIRST) for a fresh session to finish the
chase. Also stages the NVFP4 scripts + 512-row calib. Recent decisions: NEO-CODE
seat swap (R36), webhook ALLOWED_HOST_LIST fix. Lessons: validate-tracer-bullet-first,
mtp-graft-dropped-at-load, gitea-204-red-herring.
The ufw fix (prior commit) was necessary but insufficient. The DECISIVE blocker
was gitea webhook.ALLOWED_HOST_LIST = 'external, 10.100.0.0/16' (NH3 only) —
corviduo-dev is 10.250.50.152 (Anaheim), so gitea refused to deliver ('deny
10.250.50.152') and never opened the TCP connection. Fixed to 'external,
10.0.0.0/8' (whole fleet, matches the ufw choice) + gitea restart.
Listener now logs every delivery (source-IP/hmac_ok/ref/action) — the old
log_message=pass silence hid the whole failure. Proven end-to-end: real gitea
delivery -> hmac_ok=True, ref=main, 202 deploying -> green deploy.
render-verify caught it: Dvalin's calib tool_calls carry OpenAI wire-form JSON
string arguments, but the Qwen3.6 chat template does .items() on arguments (needs
a dict) → jinja TypeError. Parse string->dict in render_verify + the quant's
load_calib_chat. Confirmed: renders the exact qwen3_coder XML the seat emits
(prefixed bifrost.soong-lab.*, v0.3.13 generate_portrait, <think>, <tool_response>).
Extracted from the deployed backend (bifrost/tools.py) via the venv with a
capturing mock register_tool — the LIVE schema, not a stale copy. OpenAI-function
form for brokkr/Dvalin's ~128-row tool-call-XML calib slice (R36 #355 anchor).
The auto-deploy silently never worked: corviduo-dev's ufw is default-deny and
port 9010 was never allowed, so gitea's webhook deliveries timed out (DROP).
v0.3.6 was a manual deploy; v0.3.7-v0.3.13 never auto-deployed. The setup-time
'test-delivery 204' was gitea queuing, not the listener receiving. Fixed by
'ufw allow from 10.0.0.0/8' (operator-directed). Confirmed end-to-end.
R36 gate (2026-07-14) validated NEO-CODE ships on all axes: tool-calling 0.967
(attach_tool 1.00, 0 runaways — #355 eliminated), prose genre-artifact-fine
(less clichéd than gen), refusal uncensored-as-spec + CSAM-clean.
#355 root cause was MODEL-level, not the reasoning-budget-forcing bug: Deckard
emitted Qwen's native qwen3_coder XML tool format malformed -> llama.cpp leaked
the closing tags into the arg value -> Bifrost attach_tool schema error -> retry
-> reasoning runaway to max_tokens. NEO-CODE emits the same native format cleanly
on the same seat/parser -> no schema error -> no runaway. The fix was the model
swap; there was never a wrong parser (the XML is Qwen3.5/3.6-native).
- reasoning seat: Deckard-PKD (Qwen3.5) -> NEO-CODE=Heretic2-Thinking (Qwen3.6-27B) Q5
- samplers: card defaults (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0), DRY dropped
- ctx: 256K max; custom llama.cpp kept (qwen3_coder parse + PR#25544 belt-and-suspenders)
- persistent-memory ACTIVE 1 marked resolved
soong-dev found the studio serving a stale web/ (52015 vs 55025 bytes — missing the
01-Role section, favicon, thinking-status): the deploy rsynced backend/ but never web/,
so SOONG_LAB_WEB_DIR stayed pinned to the initial manual copy while the backend updated.
Deploy now rsyncs BOTH backend/->studio AND web/->SOONG_LAB_WEB_DIR (read from the env)
on every green run. Verified: served frontend now 55025 bytes, current.
Per operator call (no gitea write token on the Worldtree-team VM): a 2-min systemd
--user timer on nh3-dev polls corviduo's last-deploy.json and pings soong-dev via
althing on a NEW red deploy (green stays silent). Delivers soong-dev's red-run
visibility without a credential on corviduo. Tested (red detect+format DRY, green quiet).
The char-rp-reasoning (Deckard) seat now runs llamacpp-charrp:custom-latest via a
new LLAMA_REASONING_IMAGE var (Magidonia char-rp stays on stock — no reasoning bug).
Fixes Worldtree #355 at the source (budget multi-terminator handles Qwen3.5's
<tool_call> reasoning end-tag). Live 2026-07-13: Deckard loads on Blackwell, serves
coherent, reasoning bounds at the 400 budget. Rollback via .env LLAMA_REASONING_IMAGE.
Durable record of the custom llama.cpp the char-rp-reasoning seat will run to
fix Worldtree #355 (reasoning-budget forcing broken in stock b8840 — single
end-tag </think> can't match Qwen3.5's <tool_call> reasoning terminator, so the
budget never force-closes and reasoning runs away to max_tokens). PR #25544
adds multiple terminating sequences; unmerged upstream, so we build it.
- build.sh: reproducible recipe (clone master@6eddde0 + merge PR #25544 +
resolve the 1 server-common.cpp conflict + CUDA build for Blackwell sm_120)
- README.md: why + acceptance test + rollback + REMOVE-WHEN-MERGED tracking
Image llamacpp-charrp:6eddde0-pr25544 BUILT + smoke-tested on ana-ml2; seat
swap pending. See also auto-memory reference_charrp_custom_llamacpp_pr25544.
Adds the rsync --link-dest hourly snapshot job (nh3-dev:~/development ->
nh3-nas, 48-snapshot retention, secrets/build-dirs excluded) that closes the
no-off-box-backup gap exposed by the 2026-07-12 working-dir clobber. Script
mirrors the live ~/.config/dev-backup/dev-backup.sh; runbook covers restore.
Point the in-page TTS at the zonos-gateway wrapper on irv-ml1:8890 (direct,
so streaming isn't buffered by LiteLLM): OpenAI-shape body (model: ext-tts,
input, voice), Cora default voice, float32@44.1kHz PCM decode. Quotes are
joined into a single stream call (prosody — no per-sentence chunking).
Migration regex rewrites stale saved endpoints (:8299/:8210, /tts[/stream],
:4000) to the new one.
New TTS service entry + reproducibility_audit row for the zonos-gateway
wrapper (irv-ml1:8890) — the ext-tts-aliased OpenAI facade over Zonos.
23 fields across Text&voice / Expression / Prosody / Quality / Sampling /
Output section groups; live voice dropdown from /v1/voices; response
format pcm|wav (audition UI forces wav). Distinct from the older down
zonos :8203 entry. jsonschema-validated.
The live gateway config has served char-rp-reasoning as deckard-pkd-27b (:8018)
since the 2026-07-08 A/B; the standalone doc had frozen on QwQ-RpR-v4. Corrects
seat 4 (backend + samplers + server-side DRY/reasoning-budget notes).
Also snapshots session state in persistent-memory.md: phantom-qwen verified
already-clean, ana-docker docker log-cap (logrotate copytruncate, no bounce),
and the granite→gen memory_extractor bind live on demo+personal.
Drop-in system prompt for an agent whose quoted output is voiced by mOrpheus: speak in
double quotes (only quoted text is voiced), phrase each quoted line as one coherent
utterance (per-quote prosody), and the sparse/boundary/no-stack tag rules (measured
stability on the early checkpoint). Lists the honored tag set.
Per-sentence chunking generated each sentence cold, flattening intonation/prosody that
spans the whole quoted line. Chunk by QUOTED SECTION instead — each contiguous quote is
generated whole (max_tokens 2400) so its prosody stays intact; multiple quotes in a reply
still play serially on the shared clock. extractQuotes already returns exactly these spans;
dropped splitSentences.
Split the quoted dialogue into sentences and stream each as its own short /tts/stream
request (max_tokens 900), queued back-to-back on one shared AudioContext clock (speechHead)
so playback is gapless and in order. First sentence starts fast; each chunk is short so it
generates cleanly (no ramble/cap risk); the next sentence generates while the current plays.
A newer reply supersedes via the ttsGen counter; 🔊 replays.
Browsers suspend the Web Audio AudioContext until a user gesture; speakQuotes fires on
reply-complete (no active gesture), so a suspended context played silently. Prime/resume
the context on any click or keydown (capture phase) so it's running before playback.
Server side was fine throughout (/tts + /tts/stream both 200 with valid audio).
- max_tokens default 2400->3500 (~42s) in wrapper + gateway-chat client, with a _cap()
clamp so prompt+gen never exceeds MAX_CTX (4096) — a cloning ref block is ~1100 tokens,
so an unclamped 3500 would overflow context on the clone path.
- Staged clone voices: /voices dir of <name>.wav + <name>.txt, each encoded to its Orpheus
reference block at startup; voice="<name>" zero-shot clones it. Beatrice (a chatterbox
reference) staged as the first normal-voice clone. GET /voices lists baddy + clones.
- compose: mount voices dir + pass MORPHEUS_MAX_LEN to the wrapper (clamp must match engine).
vLLM concurrency (measured, --max-num-seqs 8, 250-tok reqs): near-linear batching — 8
concurrent finish in the same ~2.8s as 1 (707 tok/s, 8.1x single, flat per-req latency).
Chunked-sentence production can fan out for ~8x throughput; CPU SNAC decode is the scale
bottleneck, not generation.
Cut-offs were the max_tokens=1200 ceiling (~14.6s of audio), not memory (~1250 tokens
<< 4096 context). Diagnosis: the repetition penalty is load-bearing for clean stops —
rep 1.0 => the model never emits end-of-speech and rambles to the cap; rep 1.1 (the
wrapper default) => clean natural stop. So normal lines already complete; only genuinely
long dialogue (>~14.6s, ~25+ words) hit the cap. Raised default + client max_tokens to
2400 (~29s), still within the 4096 context (no memory cost). Verified: a 49-word line
now finishes at 16.73s (was clipped at 14.6s).
Wrapper gains POST /tts/stream: reads the vLLM token stream, decodes SNAC in WINDOWED
CHUNKS (every 6 frames, decode [2 ctx | 6 | 2 ctx] and emit only the middle 6 — context
both sides => seamless), and streams raw PCM16 (24kHz mono) as it generates. Windowed
(not per-frame) because per-frame CPU decode's per-call overhead x ~60 frames serialized
to ~7s (RTF 2.2); windowed keeps up (RTF ~0.97). Whole-clip /tts kept for non-browser use.
gateway-chat plays the stream via the Web Audio API (fetch reader -> int16->float32 ->
scheduled AudioBufferSourceNodes on a running clock; a new reply supersedes the prior
stream via a generation counter; 🔊 replays). Measured: TTFA 0.80s (was ~4.5s whole-clip),
RTF 0.97, full-duration match. CORS already covers the new route.
Deployed: tts rebuilt on irv-ml1, page pushed to ana-docker.
Gateway-chat now auto-plays quoted text from each assistant reply through the mOrpheus
TTS endpoint. Sidebar gains a 🔊 toggle + endpoint/voice fields (persist in localStorage,
prefilled to irv-ml1:8299 / baddy). On reply-complete, straight and typographic double
quotes are extracted, joined, POSTed to /tts, and the returned WAV plays (click 🔊 to
replay; a new reply interrupts the prior clip).
Requires CORS on the wrapper (page served from ana-docker:8091 fetches irv-ml1:8299
cross-origin) — added CORSMiddleware(allow_origins=[*]) to the mOrpheus tts app (internal-
only endpoint). Verified end-to-end: preflight + POST return ACAO=*, valid 24kHz WAV.
Deployed: tts container rebuilt/recreated on irv-ml1; page pushed to ana-docker conf
(bind-mounted, live on next request).
The gen seat's vLLM served-name was still qwen3.6-27b-aeon, a stale skin
left over from the AEON-27B → 35B-A3B-heretic swap — it named neither the
right family (aeon) nor size (27b vs 35B-A3B). Renamed the served-name to
qwen3.6-35b-a3b-heretic (+ -thinking) on ana-ml2 :8015 via the stack .env,
and repointed litellm's gen / gen-reasoning / summarizer-large model refs +
comments to match, so /v1/models, the gateway config, and spend-logs all
reveal the actual model in the request path.
Verified end-to-end: gen -> 'PIPELINE OK', gen-reasoning -> content + reasoning
surfaced, all three aliases healthy. char-rp / char-rp-reasoning untouched.
dvalin confirmed the live A/B-proven set IS canonical for Deckard as a dark-RP reasoning seat:
temp 1.0/top_p 0.95/top_k 40/min_p 0.05, no presence/rep penalty, DRY 0.8 server-side. Endorsed
over the card's base-thinking (top_k 20/min_p 0/presence 1.5). No value change; comment + memory
record the confirmation + tuning ladder (flat->min_p 0.08, loops->DRY 0.9, over-damped->DRY 0.6/off).
Operator wanted a reasoning-RP model that tolerates DRY (RpR-v4 forbids rep/DRY -> a
1/30 loop tail). Ran the full A/B on brokkr's 30-prompt D1 suite (content-only, slop-scored):
- Deckard-PKD (Qwen3.5-27B, DavidAU creative tune) WON: 0/30 loops, 0/30 refusals, clean
managed reasoning (native Qwen3.5 <think>/enable_thinking), DRY-tolerant, ~57 tok/s,
runs on the base llama-swap b8840 image. -> now the char-rp-reasoning seat (:8018).
- RpR-v4: 0 refusals but 1/30 loop (no-DRY). Pantheon-27B: clean slop but 7/30 explicit
refusals + needs the newer ggml-org/llama.cpp image (Qwen3.6 won't load on b8840).
Snowdrop + Gembrain (Gemma-4): floored (llama.cpp can't manage their reasoning without
the vetoed template hacks). Losers kept on disk as alternates.
- char-rp (Magidonia) unchanged; gen unchanged. gateway char-rp-reasoning -> Deckard
sampler (temp 1.0/top_p 0.95/top_k 40/min_p 0.05; DRY server-side).
dvalin-smithy offered a follow-up sampler pass for char-rp / char-rp-reasoning after
they accumulate real Worldtree/SillyTavern character-role traffic. Parked as a future
option (thread 01KX1DS6...) — nothing to tune until there's live-session data.
vh