The first real arbo train 422'd: SDXL checkpoints live at
/storetank/arbo/models/checkpoints/ (the 2026-06-13 move to the 1.8TB
/storetank volume), which wasn't in ALLOWED_MODEL_ROOTS — the old roots
predated the move (/worktank/models is gone, /worktank/comfyui host path
is empty; ComfyUI mounts /storetank/arbo/models -> /basedir/models inside
its container). Allowlist /storetank/arbo/models (llmuser-readable,
world-readable tree), drop the two stale roots. Regression test added (15 green).
comfy-dev cross-check: Sindra v1/v2 used alpha=dim/2 (0.5 LoRA scaling),
which produced the validated likeness; the initial alpha=dim (1.0) was a
stronger, unvalidated default. Align the default to the proven value
(operator/per-request can still override). Tests updated (14 green).
Host service (runs as llmuser, owns /opt/fluxgym + GPU access) that runs
sd-scripts SDXL LoRA training on demand for arbo — the infra-ops half of the
in-arbo LoRA training Phase 1 ownership split (vh/arbo
docs/contracts/in-arbo-lora-training-phase1.contract.md §4.1/§2).
- Fixed-invocation only (INV-T7): bounded params -> one sd-scripts command
shape; every param range/allowlist/path-containment checked before spawn;
bad request = 422, never a silent downgrade. 14 unit tests green.
- Thin supervisor: never imports torch; subprocesses the fluxgym venv's
accelerate. 1-job-at-a-time (arbo lease is the serializer, 409 is backstop).
Durable job records + boot reconciliation (§4.3).
- API: POST /train, GET /train/{id}[/log], POST /train/{id}/cancel,
GET /gpu-status (per-device VRAM + tts_on_3090 co-OOM signal), GET /healthz.
- Wire-shape (§7 resolved with comfy-dev): shared /worktank/arbo/train handoff
(group arbotrain, setgid 2770); worker binds 0.0.0.0:8203, arbo reaches via
host.docker.internal:host-gateway (reachability proven on 172.20.0.1:8203);
device-aware TTS steering via /gpu-status.
Deployed to irv-ml1 via playbooks/deploy-lora-training-worker.yaml (elway,
idempotent); systemd unit active; /healthz + /gpu-status verified live.
- gen := AEON dual NVFP4 serves (vLLM 0.24 + LiteLLM v1.91.0); reasoning-trace bug
was the LiteLLM shared-config mutation, fixed durably via distinct -thinking served-names.
- Worldtree personal character/thoughtful-character repointed to char-rp/char-rp-reasoning.
- LitBench-RM torn down, comfyui restored on irv-ml1.
New stacks/qwen36-27b-aeon: two co-located vLLM serves on ana-ml2 GPU0 —
gen (:8015, MTP off) and an RP seat (:8016, native MTP) — dense Qwen3.6-27B
(qwen3_5 GDN-hybrid, uncensored/abliterated), ModelOpt-NVFP4, multimodal,
256K context, depends_on-sequenced util split (~0.50/0.45). Each serve
carries a base + `-thinking` served-name so the `-reasoning` gateway records
target distinct LiteLLM deployments — otherwise a thinking-off request mutates
the shared litellm_params and clobbers enable_thinking (the shared-config
footgun that silently disabled char-rp-reasoning).
Gateway (stacks/litellm/conf/config.yaml): gen / gen-reasoning /
summarizer-large -> AEON :8015; char-rp / char-rp-reasoning added -> RP seat
:8016 (Qwen-RP sampler recs); gen-reasoning -> `-thinking`, char-rp-reasoning
-> `-rp-thinking`. Retired qwen3.5-122-a10b[-reasoning] + qwen-large[-reasoning]
(qwopus displaced; those named a 122B that no longer serves gen).
Probed live vs z.ai 2026-07-05: glm-5.2 = 1,048,576-token (1M) input context,
131,072 (128K) max output; no gateway-side cap (pure z.ai passthrough). Comment-only,
no runtime effect.
R30 b15-b17 removed mood.decay_rate/mood.stale_hours from canonical (OCEAN
wall-clock OU replaced per-turn decay); deployed /opt/worldtree*/config
bind-mounts still carry them, harmless (CharacterSchema.mood is dict[str,Any]).
Tracked as an opportunistic edit-only/no-restart cleanup to restore byte-identity;
noted new optional mood.tau_base (unset->derived). Config-delta acked to worldtree-dev.
Records Vuong's 2026-07-02 call closing the granite-efficacy thread: no
intermediate real-efficacy granite spike (uninterpretable proxy — arch
gap + abliteration axis), efficacy validated on the real T1 run. Notes
the LitBench-less data/judge WIRING check as the correct pre-T1 de-risk
IF one is ever wanted, and that infra's remaining owed item is the queued
swappable-LoRA-on-NVFP4 load test (gated on the first T1 adapter).
The granite-8b harness spike proved the TRL SFT->DPO->eval seam (incl. the
in-loop HoldoutEvaluator base-vs-adapter leg) runs end-to-end, but used a
12-row/12-pair synthetic writing fixture — NOT the E-RP corpus — so the
~0 anti-slop delta (-0.002) is the expected null, not an efficacy signal.
Adapter reaped; nothing to A/B. Real behaviour-shift efficacy is a T1-run
question. Sharpen both the T1 in-flight bullet and the Recent-decisions
entry so 'green' no longer reads as efficacy-validated.
Displaced qwopus-122B on ana-ml2 GPU0:8013 with robbatt/Qwen3.6-40B-Deckard-NVFP4
(stock vLLM 0.23.0, loaded clean: hybrid attn + multimodal + fp4_gemm all green).
Repointed the 5 role aliases (gen, gen-reasoning, qwen-large, qwen-large-reasoning,
summarizer-large); added the qwen3.6-40b-deckard true-name record; left the true
names qwen3.5-122-a10b[-reasoning] to 404 (no-false-alias). Operator-directed
trial-by-fleet-traffic; revert path in the config banner + live backup
config.yaml.bak-pre-deckard-20260701-001036.
Add §9 "PFI LiteLLM Gateway — Deployed Sampling Defaults": the live fleet
sampling table (granite/qwen/judges/GLM) with provenance, overrideable-default
semantics, the GLM API-accepted-subset caveat, and the research-confirmed temp-0
rationale for granite + image-judge. Accepts the dvalin-smithy-dev recommendations
as deployed. §§1-8 vendor reference left intact.
dvalin evidence pass: IBM canonical is temp 0; greedy-loop risk is an
open-ended-generation phenomenon, not summ/classify; temp 0.1 reduces
classification reproducibility without fixing loops (use repetition/presence
penalty if loops appear). image-judge stays 0 (Qwen judge card + W&B judge
practice = temp 0 for reproducibility; NVFP4-needs-0.1 unsupported). Both
gateway temps now 0, vendor-canonical.
Operator call: avoid pure-greedy rigidity/loop-risk on granite summ/classify
while staying near-deterministic; matches the house nonzero-temp-floor lean.
image-judge held at temp 0 (scoring reproducibility) pending operator review.
- granite-4.1-8b (+ summarizer/classifier): temperature 0 (IBM vendor-canonical
"temp 0 for inferencing"; top_p/top_k no-ops at temp 0, omitted). Deterministic
baseline for summ/classify; creative callers override.
- GLM family (z.ai cloud): temperature + top_p 0.95 only (the ONLY params z.ai
chat API accepts per its OpenAPI schema; top_k/min_p/penalties absent -> not set).
temp 1.0 for glm-5.1/5.2/5-turbo/4.7 + gen-frontier; temp 0.6 for glm-4.5-air.
Matches z.ai API defaults -> explicit-over-implicit, future-proofs vs vendor drift.
Round-2 dvalin-researched (provenance-labeled), verified live, granite+glm smoked 200.
Embeddings/rerankers excluded (no sampling). Fleet-wide canonical-defaults sweep complete.
The abliterated/NVFP4 Qwopus 122B "gen" model (+ qwen-large / summarizer-large
aliases) had no repetition control in its sampling defaults, causing degenerate
repetition loops. Add presence_penalty: 1.0 (Qwen-documented anti-repetition
lever, range 0-2) to all 7 qwen3.5-122-a10b gateway records. Overrideable
default; bake into the vLLM serving def once the value is validated.
Native-allocator expandable segments to cut Qwen-Image-Edit fragmentation
OOMs on the A6000 (a ~2 GB alloc failing with 1.75 GB free while 45 GB sat
allocated + reserved-but-unallocated). Cache-preserving — packs better
without unloading the checkpoint, so no edit-latency hit. Paired with the
existing --disable-cuda-malloc (incompatible with cudaMallocAsync).
Deployed + recreated on irv-ml1; verified env present, PyTorch reads it,
container healthy. comfy-dev request 2026-06-25.
Langfuse's ClickHouse member spewed ~94 GB of unrotated logs and filled ana-docker's
root disk (took the fleet host to 100%, 28/48 containers unhealthy). Its trace UI was
redundant with LiteLLM's native logging — store_prompts_in_spend_logs:true already
captures full prompts/responses/tokens/cost/latency at :4000/ui — and nothing used its
unique trace-grouping/eval features (it only received flat gateway success_callbacks).
Removed the callbacks (gateway observability stays fully native) and tore down the
6-container langfuse stack + volumes on ana-docker. Re-add the callbacks if it returns.
Capability aliases for the PAID frontier tier, mirroring glm-5.2 / glm-5.2-
reasoning (thinking off / on) → openai/glm-5.2 @ z.ai. Worldtree binds these for
a frontier-grade generation/reasoning capability so the backing frontier model
can be swapped gateway-side (operator jump-started WT's request). PAID: only
all-proxy-models / explicitly-scoped keys reach them; the free all-agents-local
key stays fenced off z.ai spend. Verified both resolve + route to GLM 5.2.
llama-swap (ana-ml2:9292) is decommissioned (:9292 confirmed down), so the
catch-all wildcard routed every unmatched / typo'd / stale model name to a DEAD
backend, surfacing a misleading "Connection error" instead of a clean
"model not found". This is the exact footgun that silently swallowed Worldtree's
defunct model names (mistral-small-4 etc.) instead of erroring. Removed (operator
call) so unknown models now 404 loudly. Verified: gateway healthy post-restart,
a bogus model name now returns a clean not-found error, real aliases (gen) still
serve. Re-add explicit per-model entries if a swappable zoo ever returns.
Stand up the gateway-side capability aliases for the role→capability model
indirection (worldtree-dev's transparent-swap direction; operator: no wt-
prefix, reuse the existing summarizer/classifier/gen alias convention).
- chat-judge -> selene-1-mini-8b (mode chat) — WT selene-judgment role.
- reranker -> qwen3-reranker (mode rerank) — generic name for the cap.
- scalar-judge -> Skywork-Reward-V2 via a pass_through_endpoint to ana-ml2:8003
(LiteLLM has no reward/pooling MODE, so it's a passthrough, gateway-key-gated;
consumers hit /scalar-judge/<route> e.g. /score|/pooling|/classify).
Deliberately NO generic `embedding` alias: embedding vectors are model-specific
(not swap-transparent), so that capability stays `qwen3-embedding` — the model-
specific name is the guardrail against treating it as freely swappable. Verified
all three live (chat-judge 200, reranker present, scalar-judge passthrough 200
returning a Skywork reward). Deployed + gateway health-gated.
Empirical follow-up to the streaming /tts smoke test on the 3090. OmniVoice
is diffusion: a ~fixed per-call overhead (~1.5s at 32 steps, ~0.7s at 16)
dominates regardless of chunk length, so the upstream-claimed 40x RTF does
NOT hold here (measured ~2.8x/32-step, ~5.6x/16-step) and the chatterbox-
tuned scheduler over-chunks and starves.
- Streaming /tts defaults to num_step=16 (TTFA ~1.5s -> ~0.7s); batch
/v1/audio/speech stays num_step=32 for quality. Per-request override intact.
- Scheduler prior raised to rtf_prior=20 (env OMNIVOICE_STREAM_RTF_PRIOR,
wired through compose + .env.example) so it packs whole-text-minus-first-
sentence into a few chunks: validated ~3 chunks, no starvation, total wall
~= one-shot, less per-chunk silence padding.
- Docs corrected: the "sub-second / 40x" claims were wrong; streaming has a
diffusion TTFA floor (~0.7s) and wins mainly on long replies. chatterbox-
fast (autoregressive, ~0.5s TTFA) stays the lowest-latency front-end;
OmniVoice is the multilingual / voice-design complement.
Add a live-consumer streaming path and text sanitation to the OmniVoice
wrapper, so it can front speech-to-speech chat engines (not just the
asset-engine's batch WAV use).
- POST /tts: chunked 24 kHz mono s16le PCM (or open-ended WAV), driven by
the adaptive buffer-ratchet scheduler. Emits the first sentence
immediately, then ratchets chunk size up on OmniVoice's ~40x realtime
headroom -> sub-second time-to-first-audio. Wire-compatible with
chatterbox-fast /tts (both 24 kHz mono PCM). Batch /v1/audio/speech is
unchanged for asset/file callers.
- scheduler.py: VENDORED byte-faithful copy of chatterbox-fast's pure-
Python (torch-free) scheduler, pinned to commit 7631462 (v0.1.0/v0.1.1).
Vendor-copy over a shared package (operator call 2026-06-19): the module
has no GPU deps, so reuse it without dragging chatterbox-fast's torch
tree into this image. Promote to a shared package only on a 3rd consumer
or real drift.
- sanitize.py: language-safe TTS sanitizer run on both endpoints. Strips
markdown, <think> blocks, HTML, and model control tokens; deliberately
SKIPS the fork's English-only number/phone normalization that would
corrupt OmniVoice's 600-language input. Preserves [laughter]-style tags.
- Refactor: shared GenParams base for SpeechRequest + TTSStreamRequest;
single GEN_LOCK serializes generation (single-stream interactive).
- Dockerfile/playbook: copy + upload the two new modules; build-time
`import app` smoke; correct stale "Gradio demo / no FastAPI" comments.
9 Recent decisions + 6 Tried-and-abandoned (dates [2026-06-14]/[2026-06-15]) moved
non-destructively to archival-memory.md, each stamped _Archived 2026-06-19._. Kept
the active [2026-06-14] 'migrate ALL infra access to Claude-specific credentials'
standing directive. Back-ref counts: Recent decisions 79->88, Tried-and-abandoned
70->76. persistent-memory.md 397->363 lines.
Stands up tools/gateway-chat.html as a permanent URL on ana-docker (http://10.250.50.70:8091)
via a tiny nginx:alpine static container (no GPU, no DB). conf/index.html is a deployed
mirror of tools/gateway-chat.html (re-sync one-liner in README). Homepage tile + tnet per
convention. The enhanced tool (auto-discovers /v1/models, system prompts, streaming +
reasoning, image upload for vision) is now always-on for smoking new gateway models.
Duplicate-entry aliases. classifier -> granite-4.1-8b (:8004, same backend as the
existing summarizer alias). summarizer-large -> gen/qwen3.5-122-a10b (:8013, thinking
off) for heavier summarization on the 122B Qwopus. summarizer -> granite already
existed (no-op). Config-staged + deployed without bouncing the gateway; like any
config-add these activate on the next restart (no live-add performed).
Model field now pulls /v1/models (the ↻ control; new gateway models just appear)
instead of a hardcoded stale list; 📎 attaches an image (base64 data: URL in
image_url content) so the multimodal models (Qwopus, image-judge) can be smoked.
Static-verified (JS syntax + element-id consistency); headless smoke was blocked
by a shared-browser version skew in /opt/ms-playwright, not a tool defect.
Source + deployed config cleaned without bouncing the gateway. NOTE: these were
config-loaded models, which the /model/delete API can't remove (DB-only -> 'not
found in db'), so the LIVE gateway still serves them until its next restart, at
which point the cleaned config drops them. No bounce performed.
Same Qwopus gen model as gen / gen-reasoning (served-name qwen3.5-122-a10b @
:8013, thinking off/on respectively), but each bakes a dummy 'noop' function tool
+ tool_choice:none into litellm_params so a NON-EMPTY tools array always reaches
vLLM — for consumers where the global strip_empty_tools hook isn't the right fix
(they need a valid tools structure present, not stripped). tool_choice:none means
the noop is never called. api_base = the real LAN endpoint http://10.250.50.54:8013
(the requested http://vllm:8000 template wouldn't resolve from the ana-docker
litellm container). Verified: gen-nt + gen-reasoning-nt both survive a client
tools:[] send; noop never invoked; reasoning split intact.
vh