ComfyUI's ~325G model tree moved off the near-full worktank NVMe (97%->26%,
342G free) to /storetank/arbo (roomy SATA SSD on irv-ml1), overlay-mounted
back at /basedir/models so ComfyUI behaviour is unchanged. rsync byte-verified
(src==dst), one comfyui restart, worktank original removed. Inventory of the
set in docs/arbo-comfyui-model-catalog.md for the retain decision. The older
919G /storetank/image-models/comfy archive is untouched (separate reclaim).
Qwen3.5-9B vision-language served FP8 on ana-ml2 GPU1 (co-located with
granite + the embed/rerank/reward trio; GPU0 kept free for hot-loading
large models), :8007, fronted by LiteLLM as qwen3.5-9b-fp8.
Pinned to vllm/vllm-openai nightly@sha256:49211ab2 — :latest (v0.19.1)
quantizes the VL vision tower under fp8 and garbles vision; the nightly
correctly excludes it (LM stays FP8, vision tower BF16). util 0.40
(~38GB) on the shared card (vLLM needs free>=util*total here). Vision
verified end-to-end through the gateway.
SGLang 0.5.13 confirmed to support our formats on Blackwell sm_120
(compressed-tensors NVFP4 W4A4, fp8, modelopt_fp4, petit_nvfp4, fp4_e2m1 KV),
so the bench can be a real NVFP4 head-to-head. Parameterized compose (model/
quant/GPU via .env) + a common streaming load generator (bench.py: agg tok/s,
TTFT p50/p99, TPOT) so both engines are driven identically on an exclusive GPU.
Bench-oriented; promote to a real stack only if SGLang wins. Launch deferred
until the NVFP4 eval frees a GPU.
Ada->Blackwell swap doubled card VRAM, so the Ada-era fractions (0.07/0.07/
0.18) reserved ~2x the bytes for the same models. Empirically re-floored via
0.01-step climb until each service was stable under 20x parallel inference:
embed/rerank 0.05 (load-floor for the 0.6B models), reward 0.10 (the real
over-provision). Frees ~11 GB on GPU 1. Live .env on ana-ml2 already applied.
ana-ml2's Ada->Blackwell swap (2x96GB) frees GPU 0 entirely. Pin llama-swap
to GPU 0 via NVIDIA_VISIBLE_DEVICES so on-demand large-model hot-loads land
there, off GPU 1 where the always-on vLLM services (granite + embed/rerank/
reward) live. Closes the long-standing 'pin llama-swap to GPU 0' item.
Extends the catalog/graphs git-pull-mount pattern to the SPA frontend so
frontend changes reach prod via git pull + restart, no image rebuild.
Delivers the v0.11.2 auth-on catalog-load fix without a rebuild; baked
image frontend stays the fallback.
New stack mirroring the canonical convention for the Arbo (catalog) engine,
co-located beside comfyui on irv-ml1 per ADR-0001 D1/D3:
- engine<->ComfyUI over traefik-net container DNS (http://comfyui:8188),
SSH dependency eliminated; file ops bind the shared basedir input/output
- named local-disk volumes for the gallery SQLite (arbo_db, restic-backed)
and hero images (arbo_heroes); catalog as a ro git-checkout mount (D2)
- ENGINE_TOKEN + GRANITE_KEY via on-host .env; GRANITE via the LiteLLM gateway
- Q5 catalog-pull: manual day-1, ytvc-style webhook follow-on
Image build, /healthz, catalog in-container path, and non-root UID are
comfy-dev's to confirm (CONFIRM items in README).
glm-5.1 now disables GLM thinking by default via extra_body (LiteLLM strips
top-level thinking under drop_params but forwards extra_body verbatim to z.ai).
New glm-5.1-reasoning alias = same upstream with thinking enabled, so reasoning
is opt-in. Operator call 2026-06-11; primary driver is the pi coding harness.
Verified live: glm-5.1 reasoning_tokens=0, glm-5.1-reasoning reasoning_tokens>0.
Adds graphify-out/GRAPH_REPORT.md (knowledge-graph map of the repo) and a
.gitignore block that tracks only the lightweight map while ignoring the
regenerable graph.json/cache/html. Part of the fleet-wide Graphify rollout.
Local post-commit auto-rebuild hook retained.
- migration-plan.md: workstation-only migration of nh3-dev's dev env to
the M2 Ultra Mac Pro Rack (on-subnet, sidecars stay on Linux).
- nh3-dev README: drop volva from live sidecars + add retired note.
volva.service/heid.service removed by infra-ops at heid's request
(re-architected daemon->CC-orchestrator; volva.service was crash-looping).
LLM observability for the fleet — pretty trace UI over the gateway: prompts,
completions, reasoning, latency, token counts. The pretty layer LiteLLM's
spend_logs lacked.
- stacks/langfuse: v3 self-host stack (web/worker/postgres/clickhouse/redis/
minio) on ana-docker, adapted from upstream. UI on :3001 (gitea owns :3000).
Project + API keys auto-provisioned via LANGFUSE_INIT_*. HOSTNAME=0.0.0.0 on
langfuse-web so it's reachable via the published port while also on tnet.
- litellm: enabled success_callback/failure_callback: ["langfuse"] (the
passthrough env was already wired); keys + host go in the litellm .env.
Verified: stack healthy, project keys authenticate, and a real gateway call
landed a litellm-acompletion trace in Langfuse within ~6s. Secrets live only in
the server .env (never committed).
Granite 4.1 8B beat phi4-mini on precision in brokkr's R15 P03 eval, so it's
the new production summarizer/dreamer for nevermore.
- vllm-phi4 -> vllm-granite: official IBM FP8 (ibm-granite/granite-4.1-8b-fp8,
compressed-tensors), GPU 1, 50K ctx, FP8-KV, CUDA graphs. Same :8004 slot.
- GPU 1 retune: the embed/rerank/reward trio was over-provisioned (embed ran a
5.89x KV pool, reward 3.90x). Trimmed utils 0.20/0.20/0.30 -> 0.07/0.07/0.18,
freeing ~10 GB so granite runs with CUDA graphs (not --enforce-eager) and
keeps ~10 GB free as a hedge for future Granite text-LoRAs (--enable-lora).
- LiteLLM: phi4-mini model_list entry -> granite-4.1-8b (hosted_vllm @ :8004);
explicit entry shadows the '*' wildcard's llama-swap route.
- nevermore repointed (LLAMA_SWAP_MODEL=granite-4.1-8b via the gateway) live.
Verified end-to-end: vLLM :8004 generates, gateway routes (gateway-granite-ok),
KV 86,768 tokens/1.69x at 50K, 0 restarts, GPU 1 10.3 GB free.
ComfyUI's default cudaMallocAsync allocator phantom-OOMs ('allocation
would exceed allowed memory', 0 bytes allocated) when the A6000 is
shared with the pinned TTS services (~22 GB used). --disable-cuda-malloc
switches to PyTorch's native allocator. --fp8_e4m3fn-text-enc loads the
FLUX.2 Qwen3-8B text encoder as fp8 (~8.7 GB) instead of upcasting the
fp8 file to fp16 (~16 GB), matching the box's Ampere-fp8 posture.
Applied via COMFY_CMDLINE_EXTRA in the canonical compose; verified the
allocator flipped to 'native' and both flags are on the live cmdline.
IBM Granite 4.1 dense instruct (3B + 8B), unsloth Q4_K_M GGUF via -hf
syntax, 64K ctx with q8 KV. Auto-exposed through the LiteLLM gateway
wildcard (ana-docker:4000) and direct on llama-swap (:9292).
Replaces the retired irv-ml1 Ollama granite4.1 — Ollama is now banned
fleet-wide; serving consolidates onto the sanctioned llama-swap/vLLM
substrate.
The asset-engine catalog source of truth. Removes the chatterbox
exaggeration/cfg_weight sliders (proven Turbo no-ops) — reconciling the
canonical with the fix previously applied only to asset-engine's vendored
copy — and adds the csm-expressiva whisper TTS entry (irv-ml1:8198).
Reflects the shared install: browsers in /opt/ms-playwright (root-owned,
world-readable) + PLAYWRIGHT_BROWSERS_PATH wired globally via
/etc/environment + profile.d, so any user/project resolves the shared
binary without a per-home playwright install. Includes the add-a-browser
command for future bumps.
Adds the NH3 dev box (10.100.10.50) as a managed host: README, first
system-details snapshot, ssh-target. Adds it to the fleet bootstrap's
Tier 1 and the CLAUDE.md servers table. infra-ops identity bootstrapped
there (operator-run) — NOPASSWD sudo + key, so root-level infra work on
the box (it runs the egress proxy, ttyd seat, mead-hall, volva, and dev
checkouts) no longer needs a per-task password.
First use: installed Playwright headless-Chromium system deps + binary
for bloom_music's OSMD browser-geometry test harness; headless launch +
real SVG geometry verified.
Adds explicit gateway entries for the four z.ai GLM models (glm-5.1,
glm-5-turbo, glm-4.7, glm-4.5-air) routed to api.z.ai with Z_AI_API_KEY,
plus the compose env passthrough + .env.example doc. Explicit entries
win over the llama-swap wildcard (distinct IDs, no collision). Extends
the gateway's unified logging to cloud inference, not just local
vLLM/llama-swap.
Cost note: paid API — only gateway-keyed callers reach these, but calls
spend z.ai credits (documented in config + compose comments).
Adds a `model_name: "*"` entry routing any unmatched model to llama-swap
(ana-ml2:9292) so its whole swappable LLM zoo logs through the gateway
without per-model registration — add/swap models in llama-swap freely,
litellm logs them all. Exact entries (phi4-mini/qwen3-embedding/
qwen3-reranker → vLLM) still win; the wildcard only catches the rest.
litellm does no inference; llama-swap keeps loading + serving. Enables
routing worldtree-personal's generative chat through the gateway for
full req/resp logging while preserving llama-swap's on-demand swapping.
LiteLLM proxy fronting the vLLM services on ana-ml2 so every request +
response is captured and inspectable in a browser Logs UI — the
visibility vLLM itself lacks (Dozzle shows only connection metadata).
- compose: litellm (proxy + /ui Logs) + litellm-db (Postgres store)
- conf/config.yaml: routes phi4-mini (chat, :8004), qwen3-embedding
(:8001), qwen3-reranker (:8002); store_prompts_in_spend_logs persists
full prompt/completion text. reward classifier (:8003) stays direct
(no first-class LiteLLM route).
- Langfuse-ready: lean first cut intentionally skips Langfuse's heavy v3
stack; graduating is one env-var + callback step, no re-architecture.
- roadmap: mark the vLLM-observability item's first cut as shipped.
Lean first cut of docs/roadmap.md "Observability for the vLLM stack".
Operator chose option (ii): keep the OFFICIAL Phi-4 format globally rather than
impose Ollama's leaner scaffold on every phi4 consumer. Removes the
--chat-template override + the conf/phi4-chat-template.jinja file (90e08f0).
vLLM now uses the tokenizer's built-in template (system <|end|> present);
verified 7-token render via tokenize/detokenize. brokkr re-baselines its R15
canonical on the official scaffold so baseline == production.
vLLM's official Phi-4 tokenizer template emits <|end|> after the system turn;
Ollama's does not. That single boundary token regressed brokkr's R15 P02
admission eval (type macro-F1 -33pp) vs the Ollama-measured canonical, while
valid_format held at 1.0. Operator chose to make vLLM match Ollama's leaner
scaffold globally (baseline == production). Adds conf/phi4-chat-template.jinja
(drops the system <|end|>) + mounts it + --chat-template on vllm-phi4. Applied
prompt verified via tokenize/detokenize; brokkr re-smokes probe_vllm.yaml.
Default gcc is gcc-11 (CUDA/torch R14 pin) but the only gfortran is 12,
whose libgfortran-12-dev puts the unversioned libgfortran.so only in the
gcc-12 dir. R links shared objects with gcc-11 + -lgfortran, so the whole
Fortran dependency chain (quadprog/tseries/RcppArmadillo -> soundgen) failed
'cannot find -lgfortran'. Adds an idempotent step installing libgfortran-11-dev
(guarded on 'gcc -print-file-name=libgfortran.so' not resolving). Additive;
default toolchain untouched.
No-sudo phase: uv, R16 world venv + pyworld (setuptools<81 pin for
pyworld's pkg_resources import). Sudo phase (run as infra-ops): R +
soundgen system deps, and pin the Ollama service to the A6000 BY UUID
(CUDA FASTEST_FIRST ordering != nvidia-smi PCI order on this box, so an
index pin is ambiguous). R15 models share the A6000's free headroom —
yt-voice-clipper + R14 keep running, card not freed.
Adds a host-agnostic elway play + fleet driver that stand up an
`infra-ops` system user (dedicated ed25519 key, NOPASSWD sudo with
log_output audit, docker group) so the infra-ops agent completes
DevOps work end-to-end instead of handing sudo steps back to the
operator. Scoped to PFI-owned Linux boxes; tiered (compute/app/
sensitive-infra) with SureFire/corviduo/esh/Synology explicitly
excluded. Validated live on irv-ml1.
worldtree-dev sent infra-ops a manual deploy request for v0.30.6 to
personal:8081; operator confirmed all three corviduo-dev Worldtree
deployments go through the Worldtree team's CI/CD, never a manual
infra-ops deploy. The old 'personal — manually managed' line was wrong
and misleading. docker-as-root pattern is emergency-ops (key restore)
only.
Auto-deploy on push to main: gitea webhook → HMAC-validated listener on irv-ml1:9008
→ git fetch/reset + docker compose up -d --build. Documents the gitea-server
ALLOWED_HOST_LIST anti-SSRF relaxation (scoped to the WG net), the irv-ml1 components
(deploy key, git-clone deploy dir preserving the proxy override/secrets, listener +
user service), and verify/debug steps.
YouTube (and a growing set of services) hard-flag datacenter IPs, bot-gating
even public content regardless of cookies/PO-tokens. Origin case: yt-voice-clipper
on irv-ml1 (Irvine colo) — every yt-dlp fetch returned LOGIN_REQUIRED. Confirmed
pure IP reputation: the same public video fetches cleanly (no cookies) once routed
through nh3-dev's residential egress (70.230.226.88).
- scripts/setup-nh3-egress-proxy.sh: idempotent dante (SOCKS5) install + config.
Internal-only ACL (10.100.0.0/16), bound to the WG interface, systemd-managed.
- docs/runbooks/nh3-egress-proxy.md: purpose, usage, security model, caveats.
Reusable fleet egress, not yt-voice-clipper-specific.
chatterbox-fast is authored software with a test suite, not a config-mirror stack —
so it moves to its own MIT-licensed, versioned, CI'd repo (gitea vh/chatterbox-fast,
v0.1.0) following the sister-repo pattern. Replace stacks/chatterbox-fast/ with a
pointer README; the moved code (scheduler/app/bench/tests/Dockerfile/compose) now
lives in the new repo. The deployed :8197 service is unaffected (still runs the
legacy devnen-based image; self-contained-image migration is an optional follow-up).
The fleet catalog entry stays in docs/asset-engine/services.yaml.
- chatterbox-fast experimental -> ready: browser audition verified end-to-end
(operator confirmed progressive playback "excellent" 2026-06-02).
- vibevoice ready -> down: no container running on irv-ml1 (connection refused);
catalog status was stale.
- voxtral: NOT a stale typo — its stack genuinely claimed :8197, the port now held
by the live chatterbox-fast. voxtral is down, so moved IT to :8201 (catalog
endpoint + source_url, stacks/voxtral/.env.example + README, host .env) rather
than disturb the live service. No live clash existed (voxtral down) but it was a
latent deploy-time collision I introduced by placing chatterbox-fast on 8197.
No catalog_version bump (status changes + endpoint correction, additive). Validates
against the schema.
Land chatterbox-fast in the asset-engine catalog as an additive service, per
asset-engine-dev's shipped streaming-audition path (asset-engine v0.1.17-19):
- streamable:true -> UI routes Generate to an ephemeral progressive-<audio>
audition (no Job/Asset); re-run on `chatterbox` to keep output.
- New service-level `streamable` bool added to services.schema.json (additive,
default false; mirrors the Pydantic model asset-engine-dev regenerates).
- Fields: text, voice (select via /voices), temperature/top_p/top_k/
repetition_penalty/seed, format (pcm default; UI forces wav). exaggeration/
cfg_weight omitted — Turbo ignores them.
- status experimental until the first real browser audition verifies progressive
playback (the one thing asset-engine-dev couldn't machine-verify).
- reproducibility + audit entries added. No catalog_version bump (additive).
Validates against the updated schema.
TTSRequest gains `seed` (0=random); seeded once per request under the lock via
torch.manual_seed + cuda.manual_seed_all. One-shot output is then byte-reproducible
for a fixed seed+params (verified: seed=42 -> identical sha256 across runs).
Streaming stays non-reproducible by design — adaptive-chunk boundaries depend on
live-measured RTF. Needed for the asset-engine catalog reproducibility contract
(parity with the chatterbox sibling, which exposes seed).
stream=false + format=wav emitted the streaming 0xFFFFFFFF-length header, so a
buffered consumer reading a complete wav got bogus RIFF/data sizes. One-shot knows
the full length, so emit correct sizes; streaming keeps the open-ended header
(length genuinely unknown up front). Verified remote: one-shot wav data size ==
bytes-44, python wave.open() reads 2.20s cleanly; streaming still 0xFFFFFFFF.
Deployed on irv-ml1 beside live chatterbox (:8196): healthy on :8197, TTFB ~0.5s,
no starvation. Measured VRAM 5.34 GB (fp32) settles the placement: the 3090's
~3.8 GB free does NOT fit, A6000 (device 1) is the only viable card.
Revert the priming feature from d707439. Live A/B caught an audible artifact: the
context-priming discard-cut left part of the throwaway prefix in the output, so a
clause ("...without a trace of sarcasm,") was spoken an extra time.
Root cause is structural: generate() returns one finished waveform with no marker
for where the prefix ends, and the model renders the same prefix with different
timing when followed by content than when generated solo — so the duration-estimate
+ energy-minimum cut is a guess and can leave a sliver (or a whole clause) of prefix
in. A reliable cut would need token-level access (the abandoned native-streaming
arc) or a per-chunk ASR/alignment pass (heavy, still imperfect, eats the latency
budget). Fails the agreed bar: "keep only if it closes the gap without a seam."
Kept from d707439: the .gitignore (build artifacts). NOT re-applied: the bundled
margin_first fix — wiring it would shrink chunk 1 (more joins = worse coherence),
against the operator's priority, and margin=0.8 there is already starvation-safe.
Coherence loss at joins stays an accepted limitation; cold streaming was judged
"really good". Phase 1 + Phase 2 parity/perf untouched. Next: Phase 3 deploy.
Prime early joins by prepending the prior sentence as backward prosodic context,
generating context+content together, then discarding the context audio. The cut
snaps to the inter-sentence pause (energy-minimum search around the context's
solo duration) with a 5ms fade-in to kill any seam click (app: _cut_at_pause /
_fade_in / Engine.generate_primed). Opt-in via request `prime` (default off).
Scheduler: priming is AFFORDABILITY-GATED so it can never starve. A primed chunk
costs ~(2·context + content)/rtf (a 2nd context-solo pass); a chunk is only primed
when buffer ≥ prime_buffer_factor (1.5) × that cost, else it falls back to a cold
generate. Consequences proven in the GPU-free sim (17 tests):
- fires on early joins for any GPU at/above rtf_prior (3.4 = 3090; A6000 ~3.8-4.0)
- self-skips (degrades to cold) on a slower-than-fleet GPU rather than starving
- never primes chunk 0 (latency-critical)
Also fixed a latent Phase-1 bug: margin_first was applied at chunk 0 (budget always
0 there) so it never did anything — now applied at chunk 1 (the first transition).
Live A/B on irv-ml1 (A6000, GLaDOS): TTFB unaffected (445 vs 467ms), no starvation;
priming fired on chunk 2 (gen 1.6s for the doubled pass). On typical text exactly
ONE early join safely primes — priming chunk 2 flattens the buffer so later/larger
chunks no longer clear the safety gate. Samples: ~/chatterbox-ab/_p2_{cold,primed}.wav.
- /voices endpoint lists predefined voice stems (excludes `_`-prefixed bench/A-B
scratch wavs); shared _predefined_wavs() also feeds default-voice discovery.
- Perf levers: TF32 matmul/cudnn + flash/mem-efficient SDPA, default ON, env-gated
(CBF_TF32 / CBF_SDPA_FLASH). Startup logs model dtype.
Measured on irv-ml1 (turbo, A6000): the model loads FLOAT32 (not the fp16 older
notes assumed). TF32+SDPA do NOT move TTFA (489->514ms, noise) — first-sentence
latency is bound by the sequential AR token decode at batch-1, not matmul
throughput. bf16 (the lever that would help) is DEFERRED: from_pretrained() has no
dtype arg and turbo's fp32 conditioning path + dtype-sensitive vocoder make a
clean cast nontrivial; not worth the quality risk at ~0.5s TTFA. torch.compile
also deferred (batch-1 regression). Findings recorded in README.
Voice management parity (predefined dir + per-request clone refs) was already in
the Phase-1 resolve path; /voices completes the surface.
Build the streaming TTS server MVP per docs/design/chatterbox-fast-plan.md §4.
- scheduler.py: adaptive buffer-ratchet chunker (the meat) — GPU-free pure
logic. First sentence emitted alone for low TTFA, then chunks ratchet ~3x by
packing whole sentences to margin x buffered-audio; drives off measured RTF +
sec/char (EMA). relieve_leader() clause-splits a too-big mid-stream sentence
to avoid starvation (joins land on commas); a long comma-less sentence is the
one honored-but-flagged limitation.
- test_scheduler.py: GPU-free simulation, 13 tests — asserts no-starvation
(incl. overestimated RTF) and the ratchet.
- app.py: FastAPI model holder + POST /tts StreamingResponse (raw PCM s16le
default, wav optional, stream/oneshot) + GET /health.
- bench.py: client — ground-truth TTFB + real 1x-consumer starvation check.
Live test on irv-ml1 (turbo, A6000, GLaDOS voice): streaming TTFB 499ms vs
oneshot 5230ms (~10x), stayed ahead of a 1x player (no starvation), ratchet
1.64->4.08->8.60->8.60s audio, measured RTF self-corrected 3.38->4.01.
Kill the superseded docs/design/chatterbox-fast.md — its §5 windowed-token
streaming was the abandoned native-frame-streaming arc; the adaptive-chunk plan
supersedes it. Repoint persistent-memory + README at the canonical plan.
vh