A standing user-level web server (nh3-dev :8090) that renders drop-folders
under ~/booth-data as ephemeral media "booths" so Claude Code sessions can
surface A/B renders and smoke results to the operator, then let them self-wipe.
- Scan-and-serve model, no database, no upload API — a booth is just a folder.
A folder's own index.html is served verbatim; otherwise an auto-gallery of
images / webm+mp4 video / audio is rendered, with <file>.txt caption sidecars
folded in (labels A/B pairs).
- 24h TTL from newest mtime in the tree; background sweeper wipes stale booths.
- Path-traversal + symlink-escape guarded; delete via UI button or DELETE API.
- FastAPI + Jinja2, runs from the checkout under systemctl --user (booth.service),
alongside the other nh3-dev fleet sidecars. 15 tests, all green.
- Homepage tile added (Apps -> The Booth, siteMonitor /healthz).
- Harden the homepage rsync doc: exclude *.bak* and logs/ so --delete can't
wipe the host's dated services.yaml backups (footgun found deploying this).
Mirror comfy-dev's operator-run allocator A/B result off irv-ml1: drop
--disable-cuda-malloc (ComfyUI keeps CUDA's default async allocator) and
remove PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (native-allocator-only,
inert under cudaMallocAsync). The native+expandable_segments combo was
fragmenting/over-reserving (~45 GB allocated-but-unused) and OOMing the LTX-2.3
v1.5.0 LoRA stack at Gemma TE load; cudaMallocAsync packs tighter + returns
freed blocks so the job fits (stress test peaks ~82% VRAM, 0 OOM). The
shared-A6000 phantom-OOM that --disable-cuda-malloc guarded is gone since TTS
moved to the 3090 (2026-06-18).
Record the RTXVideoSuperResolution node clone + the nvidia-vfx pip install
(scoped --extra-index-url, uid 1000) in the stacks/comfyui runbook. Flag the
durability split: the node is persistent (basedir/custom_nodes) but nvidia-vfx
lives in the disposable venv (run/), so it must re-run after every fresh
bootstrap. Deliberately not a global PIP_EXTRA_INDEX_URL (torch-pin safety).
Closes the comfy-dev provisioning ticket.
soong-lab: WT-style CI-deploy step live (SSH deploy@corviduo-dev, compose
pull && up -d from /opt/soong-lab, /api/version health-gate). Dir relocated
+ old retired; dedicated soong deploy key; run #5 green (container recreated,
v0.3.25 live). worldtree-sdk 1.0.0 Python published to vh Gitea PyPI +
acceptance-verified.
Deferred-work capture: Vuong approved wiring soong-lab auto-redeploy; operator
deferred execution to a fresh context. Recorded in Recent decisions with the
mechanism (WT-style CI-deploy step), the blocker (a vh-owned deploy SSH-key
secret), and next-session steps; handoff updated to make it the primary task.
The "5 AM clock" report was a hallucination in another window — clock verified
correct (US/Pacific, NTP-synced), no change.
Mirror the canonical EMOTION-DIALS-SPEC.md from vh/zonos-gateway (now carries
the provisional per-voice emotion presets baked as gateway 0.2.1) and capture
the axes-sweep → bake arc in persistent memory.
Created private vh/zonos-gateway on gitea, imported the previously-unversioned
~/zonos-gateway working tree (source + dials-first spec + voices). Updated the
stack README, spec §8, and the sister-repos table to point at the repo. Remaining
follow-up: CI + deploy key to wire the irv-ml1 deploy tree to the repo.
Canonical direction: emotion set by twisting raw dials per-utterance, not preset
selection. Presets demoted to optional examples. Spec covers the dial vocabulary
+ ranges, the emotion_cfg_scale 'deaf by 1.5' rule (NO cap — documented ceiling,
explicit over implicit), measured RTF cost, starting-point dial-sets, an LLM
client system-prompt snippet, and clone reference guidance (~15-24s, no
transcript). Follow-ups flagged: align dials.py cfg help/metadata, trim in-code
PRESETS pending usage check, stand up vh/zonos-gateway for version control.
Captures the production Zonos TTS path that the repo was blind to: stock ZONOS2
:1920 engine (stacks/zonos-engine) fronted by zonos-gateway:0.2.0 :8890, reached
via the LiteLLM ext-tts alias. Documents the emotion-preset system (neutral/warm/
excited/sad/intense/whisper, the simple preset: caller path), the API, and the
measured real-time cost (calibrated steering free at RTF~0.52, cfg1.5 ~0.625 —
still realtime). Flags the gateway source (~/zonos-gateway on irv-ml1) as not yet
in gitea. Marks stacks/zonos (v0.1 Gradio) dead/superseded. Records the emotion-
lever finding (text-priming flat -> native steering works) in persistent memory.
worldtree-dev ruled the four other personal deltas code-default-optional (none
boot-required), cut staging/v1.0.0b125 (workflow 1859 green). Personal now on
b125 (877bb577ad36), healthy, booted off the staged bind-mount, live #365 blocks
byte-exact vs baked, clean boot. Demo + personal both confirmed. .bak-pre-365
rollback on both.
Demo auto-deployed to b125 and passed the boot-FATAL validator live against
the staged bind-mount; staged demo blocks byte-exact vs the baked image.
Personal staged (operator-directed) with the same verbatim blocks + backups,
all b125 checks pass, no restart. worldtree-dev holds staging/v1.0.0b125 until
they rule whether personal's other missing b107-b125 blocks (metrics/kb_audit/
reference_knowledge + affect-debug-observability-allow) are boot-required.
Staged the boot-blocking #365 delta on corviduo-dev DEMO /opt/worldtree/config
per worldtree-dev thread 01KXRE59 (verbatim from their 6a60fe3/b123 blocks):
internal + internal-restricted tiers, four internal-* rules, and the
internal_agent_comms:{enabled:false} gate. Validated via the demo container's
own PyYAML + the exact-set uniqueness guard; no restart (b123 activates off the
bind-mount). Backups .bak-pre-365. Flagged a pre-existing free==pro scope-set
dup to worldtree-dev as a b123 boot-block risk.
worldtree-dev flagged a boot-blocking config delta (commit 6a60fe3/b123, #365
internal-badge layer) needing a pre-sync of the demo policies.yaml + defaults.yaml
before its eventual deploy. Verbatim blocks received (althing 01KXRE59); the two
new scope-sets pre-verified unique vs the demo's 7 existing tiers. No urgency (push
held for heid gates) + FATAL-on-malformed, so deferred to a focused pass and captured
as an open loop (persistent-memory + /tmp handoff) rather than staged at the snapshot tail.
Current state rewritten to the active build task: containerize the production
Zonos2 :1920 TTS engine as a self-contained image (config captured in 14a0004)
and prototype the operator's prosody-priming hypothesis (prime→generate-one-
utterance→parakeet-clip→deliver, adapter-level, engine stays stock). New detail
file + Recent-decisions pointer capture the full plan + the crux (does AR prosody
carry the sentence boundary — A/B the join). /tmp handoff written for the fresh session.
The Zonos TTS engine that zonos-gateway fronts (irv-ml1 3090, feeds asset-engine +
gateway-chat) ran as a bare native process with its real invocation existing ONLY
in the running process argv — the committed harness/zonos_server.sh on irv-ml1 was
STALE (said A6000/:1919, no perf flags; live is 3090/:1920 with cuda-graph/num-pages/
max-running-requests/memory-ratio). Captured the corrected canonical invocation +
tunables + the containerization plan here so the config survives a process death.
Engine = stock Zyphra/Zonos2 @ 194c0a3 (no custom PFI server code); torch 2.9.1+cu128;
15 GB HF weights. Next: containerize in-place on the 3090 (operator: keep off the
A6000, it OOMs under ComfyUI). Not yet built — this commit is the config capture only.
granite (fleet fan-out summarizer/classifier) had no explicit --max-num-seqs,
so vLLM V1 resolved it to 128 — which caps concurrency BELOW granite's own KV
bound (~192 concurrent @ 1K-token calls, more for shorter classify calls).
Pinned it very high (1024) so the KV pool is the only bound; VRAM-neutral
(the KV pool is util-bound, unchanged). Added the flag to the granite command
+ GRANITE_MAX_NUM_SEQS to the env template. Live applied + verified
(resolved max_num_seqs=1024, seat healthy).
Operator-directed 2026-07-16. Moved the char-rp prose seat (Magidonia-24B,
llama-charrp) from GPU0 to GPU1 (CHARRP_GPU_ID 0->1; recreate llama-charrp
only -- the var is shared with the retired GGUF reasoning service), then
re-optimized every context-relevant seat on both cards to native/max context
with acceptable headroom:
GPU0 (both seats now 256K native, ~14 GB reserve):
- char-rp-reasoning 150K -> 256K (heretic2 stack, util 0.38->0.46, 1.56x)
- gen 256K, max-num-seqs 16 -> 32 (qwen36-27b-aeon, util 0.30->0.42, 5.43x)
GPU1 (~6.7 GB headroom):
- granite 64K -> 128K full-chapter (vllm stack, util 0.18->0.27, 1.50x)
- char-rp 128K native (4 slots), selene/reward/embed/rerank unchanged
All seats gateway-verified healthy. Live .env changes on ana-ml2 with per-stack
backups (*-20260716). Templates updated to match; the qwen36-27b-aeon template
carries a NOTE that its served-name/model still lag the 2026-07-08 gen model swap
(35B-A3B-heretic) -- separate reconciliation. persistent-memory records the full
layout + the util-floor / per-model-KV-cost lessons.
Note: the heretic2-charrp-reasoning stack (char-rp-reasoning's live config) is
still untracked in git -- standing open-loop, its .env change lives server-side only.
Operator-directed 2026-07-16: free ~10GB on ana-ml2 GPU1 to relocate a GPU0
model onto GPU1. granite-4.1-8b (fleet summarizer) was over-provisioned at
util 0.34 / max-model-len 131072 with a flat 0.0% KV usage.
Set GRANITE_GPU_MEM_UTIL 0.34 -> 0.18 and GRANITE_MAX_MODEL_LEN 131072 -> 65536
on the live /opt/docker/compose/vllm/.env (backup .env.bak-pre-granite-rightsize-
20260716), recreated vllm-granite ONLY (shared stack). Result: GPU1 62,641 ->
51,897 MiB used (~10.5GB freed, ~45GB free now); KV 6.45 GiB / 84,528 tok /
1.29x concurrency @ 65536; summarizer verified healthy.
The util drop required the max-len drop: on this shared card the effective KV
slope is ~950 MiB per 0.01 util, and vLLM refuses to start unless the KV pool
holds >= 1x max-model-len -- util 0.15 undershot (est max-len 47184 < 65536,
crash-loop, ~2-3 min summarizer outage) before 0.18 landed. 65536 is granite's
precedented summarizer ctx; a summarizer doesn't need 131072.
.env.example updated to the new util (max-len was already 65536 in the template;
live had drifted to 131072). persistent-memory.md updated (parked item closed).
Operator-directed 2026-07-15. The dedicated Qwen-Image-Bench NVFP4 judge
backend on ana-ml2 GPU1 (:8014) was stopped to reclaim ~32GB after the
arbo -> gen hero-judge switch. Both LiteLLM gateway aliases that pointed at
it -- image-judge and qwen-image-bench -- now repoint to the gen backend
(:8015, qwen3.6-35b-a3b-heretic, vision-intact), held at deterministic
judge sampling (temp 0 / top_k 1 / rep_pen 1.05) with enable_thinking:false
(a reasoning preamble breaks json_object). Verified live: both answer with
:8014 down, so they are definitively on gen.
Incidental: backfilled the canonical char-rp-reasoning litellm block, which
had lagged live since the 2026-07-14 NVFP4+MTP seat repoint (model
deckard-pkd-27b -> char-rp-reasoning, top_k 40 -> 20, min_p dropped,
enable_thinking:true added). Required so pushing the canonical would not
clobber the correct live block.
Live changes applied out-of-band (config push + litellm restart + stack
stop on ana-ml2); live config backup at
config.yaml.bak-pre-imagejudge-20260715. Revert path documented in the
config comment. persistent-memory.md updated (parked item closed).
Split the 53 over-threshold dated log entries (Recent decisions, Tried
and abandoned) into per-entry persistent-memory.d/<slug>.md detail files,
leaving one-line pointers in the index; the 10 short entries stay inline.
Startup index drops 60,527 -> 25,256 bytes (492 -> 270 lines); entry
bodies move verbatim to on-demand detail files, so a fresh session loads
~25 KB instead of ~60 KB and pulls a detail file only when its pointer is
relevant. Top matter (Repo purpose, Tools & conventions, Current state)
is unchanged; both archival back-references preserved.
CLAUDE.md persistent-memory section now documents the index<->detail read
discipline (read the index, pull details on demand, never bulk-read the
dir, commit both together).
Auto-archival still held every dated entry back (all <30 days old); the
July burst begins aging past the 30-day guard ~2026-07-31.
The prior fix (_netdev,nofail + docker.service After=remote-fs.target) looked
correct but silently failed — paperless still Exited(255) on the 2026-07-14
reboot. Root cause: `nofail` drops a mount out of remote-fs.target's blocking
set, so ordering docker After=remote-fs.target does NOT wait for the nofail
NFS mounts. Fix: add x-systemd.before=docker.service,x-systemd.mount-timeout=30
to the 4 NFS fstab lines (direct mount->docker ordering, nofail-safe). Applied
+ verified live (systemctl show docker -p After now lists all 4 mnt-*.mount).
Playbook + verify updated to canonicalize.
Move the ~22-service flat "AI Systems" group off the Main tab into a new
four-tab layout (Main / AI / Infrastructure / Toolchain). The AI tab sorts
the inference fleet by function into seven groups:
AI - Inference gen, char-rp, char-rp-reasoning, Granite summarizer
AI - Eval & Retrieval Selene, Skywork Reward, Qwen3 rerank/embed, image-bench
AI - Gateways & Chat LiteLLM, Asset Engine, Gateway Chat, Open WebUI, ...
AI - Speech (TTS) Chatterbox Fast, Kokoro, mOrpheus
AI - Audio Tools Parakeet ASR, YT Voice Clipper
AI - Image & Media ComfyUI, Arbo
AI - Dormant stopped rollback seats + retired auditions
Relabel each stack's homepage.group so canonical stacks/ matches the live
containers on ana-ml2, ana-docker, and irv-ml1. Dormant stacks were refreshed
with `docker compose up --no-start` so they carry the new label while staying
stopped (compose-start rollback preserved). settings.yaml drives tab/order/
columns; services.yaml and README updated to the new scheme.
The fast char-rp-reasoning seat works: ~77 tok/s (vs GGUF ~59.5, base NVFP4 ~53),
MTP draft-acceptance 32-40%, mean acceptance length 2.19. Same Heretic2/NEO-CODE
model, NVFP4 + native qwen3_5_mtp spec-decode.
Full end-to-end recipe + the four landmines in docs/runbooks/heretic2-nvfp4-mtp-seat.md:
(1) load as AutoModelForImageTextToText not AutoModelForCausalLM (namespace/gibberish);
(2) modelopt format not compressed-tensors (compressed-tensors MTP = 0% accept);
(3) modelopt 0.45 <-> transformers 5.12.1 FusedMoE crash (guarded in quant_modelopt.py);
(4) vLLM 0.24.0 does NOT propagate modelopt exclude_modules to the spec-decode draft
model -> BF16 mtp head gets quantized -> shape crash; no checkpoint config fixes it
(is_layer_skipped is exact-membership not glob) -> fix is a mounted sitecustomize that
force-skips mtp.* in is_layer_skipped (upstream vLLM bug to report).
Scripts: quant_modelopt.py (FusedMoE guard + single-shard export + multimodal load),
finalize_modelopt_mtp.py (splice bf16 mtp), serve_modelopt_mtp.sh, run_quant_modelopt.sh,
sitecustomize-mtp-workaround.py.
Root-caused the NVFP4 gibberish to a quant-namespace bug: quant_nvfp4.py loaded
via AutoModelForCausalLM -> text-only Qwen3_5ForCausalLM -> flat model.layers.* keys,
but vLLM 0.24 serves only Qwen3_5ForConditionalGeneration (whose weight mapper needs
model.language_model.*). Fixed by loading as AutoModelForImageTextToText; NVFP4 now
serves coherent (validated greedy on ana-ml2 GPU0).
Base NVFP4 (compressed-tensors) measured ~53 tok/s (~= GGUF at batch-1, no single-stream
win) and its MTP is 0% acceptance (vLLM's Qwen3_5MTP drafter loads the bf16 mtp head only
off a modelopt main-model checkpoint). Added quant_modelopt.py (nvidia-modelopt PTQ,
matches AEON's NVFP4 W4A4 g16 + lm_head/linear_attn/visual exclusions) as the path to
working native MTP; graft + splice + serve otherwise unchanged.
Captures the full pipeline recipe (graft->quant->splice->config->serve) with every
gotcha found this session, the 3 gibberish suspects, and the diagnostic ladder
(validate native-config no-MTP coherence FIRST) for a fresh session to finish the
chase. Also stages the NVFP4 scripts + 512-row calib. Recent decisions: NEO-CODE
seat swap (R36), webhook ALLOWED_HOST_LIST fix. Lessons: validate-tracer-bullet-first,
mtp-graft-dropped-at-load, gitea-204-red-herring.
The ufw fix (prior commit) was necessary but insufficient. The DECISIVE blocker
was gitea webhook.ALLOWED_HOST_LIST = 'external, 10.100.0.0/16' (NH3 only) —
corviduo-dev is 10.250.50.152 (Anaheim), so gitea refused to deliver ('deny
10.250.50.152') and never opened the TCP connection. Fixed to 'external,
10.0.0.0/8' (whole fleet, matches the ufw choice) + gitea restart.
Listener now logs every delivery (source-IP/hmac_ok/ref/action) — the old
log_message=pass silence hid the whole failure. Proven end-to-end: real gitea
delivery -> hmac_ok=True, ref=main, 202 deploying -> green deploy.
render-verify caught it: Dvalin's calib tool_calls carry OpenAI wire-form JSON
string arguments, but the Qwen3.6 chat template does .items() on arguments (needs
a dict) → jinja TypeError. Parse string->dict in render_verify + the quant's
load_calib_chat. Confirmed: renders the exact qwen3_coder XML the seat emits
(prefixed bifrost.soong-lab.*, v0.3.13 generate_portrait, <think>, <tool_response>).
Extracted from the deployed backend (bifrost/tools.py) via the venv with a
capturing mock register_tool — the LIVE schema, not a stale copy. OpenAI-function
form for brokkr/Dvalin's ~128-row tool-call-XML calib slice (R36 #355 anchor).
The auto-deploy silently never worked: corviduo-dev's ufw is default-deny and
port 9010 was never allowed, so gitea's webhook deliveries timed out (DROP).
v0.3.6 was a manual deploy; v0.3.7-v0.3.13 never auto-deployed. The setup-time
'test-delivery 204' was gitea queuing, not the listener receiving. Fixed by
'ufw allow from 10.0.0.0/8' (operator-directed). Confirmed end-to-end.
R36 gate (2026-07-14) validated NEO-CODE ships on all axes: tool-calling 0.967
(attach_tool 1.00, 0 runaways — #355 eliminated), prose genre-artifact-fine
(less clichéd than gen), refusal uncensored-as-spec + CSAM-clean.
#355 root cause was MODEL-level, not the reasoning-budget-forcing bug: Deckard
emitted Qwen's native qwen3_coder XML tool format malformed -> llama.cpp leaked
the closing tags into the arg value -> Bifrost attach_tool schema error -> retry
-> reasoning runaway to max_tokens. NEO-CODE emits the same native format cleanly
on the same seat/parser -> no schema error -> no runaway. The fix was the model
swap; there was never a wrong parser (the XML is Qwen3.5/3.6-native).
- reasoning seat: Deckard-PKD (Qwen3.5) -> NEO-CODE=Heretic2-Thinking (Qwen3.6-27B) Q5
- samplers: card defaults (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0), DRY dropped
- ctx: 256K max; custom llama.cpp kept (qwen3_coder parse + PR#25544 belt-and-suspenders)
- persistent-memory ACTIVE 1 marked resolved
soong-dev found the studio serving a stale web/ (52015 vs 55025 bytes — missing the
01-Role section, favicon, thinking-status): the deploy rsynced backend/ but never web/,
so SOONG_LAB_WEB_DIR stayed pinned to the initial manual copy while the backend updated.
Deploy now rsyncs BOTH backend/->studio AND web/->SOONG_LAB_WEB_DIR (read from the env)
on every green run. Verified: served frontend now 55025 bytes, current.
Per operator call (no gitea write token on the Worldtree-team VM): a 2-min systemd
--user timer on nh3-dev polls corviduo's last-deploy.json and pings soong-dev via
althing on a NEW red deploy (green stays silent). Delivers soong-dev's red-run
visibility without a credential on corviduo. Tested (red detect+format DRY, green quiet).
The char-rp-reasoning (Deckard) seat now runs llamacpp-charrp:custom-latest via a
new LLAMA_REASONING_IMAGE var (Magidonia char-rp stays on stock — no reasoning bug).
Fixes Worldtree #355 at the source (budget multi-terminator handles Qwen3.5's
<tool_call> reasoning end-tag). Live 2026-07-13: Deckard loads on Blackwell, serves
coherent, reasoning bounds at the 400 budget. Rollback via .env LLAMA_REASONING_IMAGE.
Durable record of the custom llama.cpp the char-rp-reasoning seat will run to
fix Worldtree #355 (reasoning-budget forcing broken in stock b8840 — single
end-tag </think> can't match Qwen3.5's <tool_call> reasoning terminator, so the
budget never force-closes and reasoning runs away to max_tokens). PR #25544
adds multiple terminating sequences; unmerged upstream, so we build it.
- build.sh: reproducible recipe (clone master@6eddde0 + merge PR #25544 +
resolve the 1 server-common.cpp conflict + CUDA build for Blackwell sm_120)
- README.md: why + acceptance test + rollback + REMOVE-WHEN-MERGED tracking
Image llamacpp-charrp:6eddde0-pr25544 BUILT + smoke-tested on ana-ml2; seat
swap pending. See also auto-memory reference_charrp_custom_llamacpp_pr25544.
Adds the rsync --link-dest hourly snapshot job (nh3-dev:~/development ->
nh3-nas, 48-snapshot retention, secrets/build-dirs excluded) that closes the
no-off-box-backup gap exposed by the 2026-07-12 working-dir clobber. Script
mirrors the live ~/.config/dev-backup/dev-backup.sh; runbook covers restore.
Point the in-page TTS at the zonos-gateway wrapper on irv-ml1:8890 (direct,
so streaming isn't buffered by LiteLLM): OpenAI-shape body (model: ext-tts,
input, voice), Cora default voice, float32@44.1kHz PCM decode. Quotes are
joined into a single stream call (prosody — no per-sentence chunking).
Migration regex rewrites stale saved endpoints (:8299/:8210, /tts[/stream],
:4000) to the new one.
New TTS service entry + reproducibility_audit row for the zonos-gateway
wrapper (irv-ml1:8890) — the ext-tts-aliased OpenAI facade over Zonos.
23 fields across Text&voice / Expression / Prosody / Quality / Sampling /
Output section groups; live voice dropdown from /v1/voices; response
format pcm|wav (audition UI forces wav). Distinct from the older down
zonos :8203 entry. jsonschema-validated.
vh