The irv-ml1-arbo Gitea Actions runner (host-executor as lkraven under
systemd) gets the bare service PATH (/usr/local/bin:/usr/bin:/bin), which
omits ~/.local/bin — so the CI uv bootstrap failed with "uv: not found".
Symlink uv/uvx into /usr/local/bin (on the systemd PATH) to fix it and
retire the per-run curl|sh bootstrap. Idempotent (creates: guard);
re-applies cleanly after a runner rebuild. Authorized by comfy-dev
(engine owner) per althing thread 01KV94VTS27B.
Zero-dependency, zero-backend HTML chat UI for the LiteLLM gateway. The
browser talks straight to :4000 (gateway CORS is open), so it's just one
file you open — no container, no stack. System-prompt textarea, model
datalist, streaming SSE, renders reasoning_content for the -thinking/
-reasoning models, settings persist in localStorage.
Deliberately never sends a `tools` field, sidestepping the vLLM "tools
must not be an empty array" bug that breaks the LiteLLM admin UI
playground for vLLM-backed models (litellm #6228; the gateway's
strip_empty_tools hook can't reach the UI's in-process completion call).
Verified against the live gateway: streams + parses a real completion
with no tools sent.
vLLM's OpenAI server 400s on an empty tools array ("tools must not be an
empty array"), which broke every gateway call carrying tools:[] (clients
that send it to mean "no tools" -- OpenAI tolerates it, vLLM does not).
drop_params doesn't help: it drops unsupported PARAMS, not empty VALUES.
Add a CustomLogger async_pre_call_hook (conf/strip_empty_tools.py) that
pops an empty/None tools field (+ orphaned tool_choice) before forwarding,
registered globally via litellm_settings.callbacks so it covers every
vLLM-backed model, not just mistral-small-4. Mounted at
/app/strip_empty_tools.py beside config.yaml (LiteLLM resolves callbacks
relative to the config dir). Surgical: only fires when tools is present
and empty; real tools pass through untouched.
Verified on live gateway (1.87.0): mistral-small-4 and granite-4.1-8b
with tools:[] now 200 (were 400); no-tools baseline unchanged; a real
tool still passes through.
Append 10.100.10.50:8391 to BIFROST_CLIENT_ALLOWED_HOSTS on the personal
Worldtree (.env) so the consumer may bind the memory provider at session-create
(affect :8390 was already listed; the url-guard 422s un-allowlisted endpoints).
Idempotent elway playbook; surgical worldtree-api recreate that auto-derives the
image pin from the matrix sibling to avoid the stale-:latest crash-block footgun.
Repoint servers/corviduo-dev/ssh-target to infra-ops (operator granted durable
NOPASSWD admin on corviduo-dev 2026-06-15).
Put GPU1's idle ~11 GB buffer to work on the two KV-bound models that gained
live consumers from the worldtree migration (granite + the pooling models
under-use their util, so growing them is wasted):
- selene 0.13→0.17: KV 2.53→6.33 GiB, concurrency 1.27x→3.16x @32K (Domari judge)
- qwen36 0.32→0.34: KV 7.73→9.63 GiB, concurrency 2.92x→3.64x @131K (arbo judge +
worldtree actor/echo + gateway)
GPU1 free now ~5.6 GB (safe floor for single-service recreates).
selene: AtlaAI Selene-1-Mini-Llama-3.1-8B judge restored on vLLM after the
llama-swap teardown took its Q6_K GGUF offline. FP8 (dynamic --quantization
fp8; FP8 >= the validated Q6_K fidelity, and text-only Llama so no vision-
tower-noise risk; NVFP4's W4A4 too aggressive for a precision judge). GPU1
util 0.13 (8.51 GiB weights + 2.53 GiB KV, 32K ctx, 1.27x concurrency),
~11 GB GPU1 buffer left. Gateway selene-1-mini-8b → :8011 (shadows the *
wildcard that used to reach it via llama-swap). Judge smoke: scored an
unfaithful claim 1/5 correctly.
mistral-small-4: max-model-len 131072 → 262144 (full native 256K) for
novel-length consistency-checking. KV pool is util-bound (~862K tokens), so
256K costs no extra VRAM — max concurrency just drops to 3.29x at full length.
max-num-seqs 64 → 32 keeps the warmup transient flat (scales with seqs × len),
so it fits the tight GPU0 (free unchanged at 5.2 GB). Verified loaded + healthy.
Operator needs a verified-working vision tower as the abliteration/tuning
baseline. vLLM 0.23.0 crashes Mistral multimodal at startup (#44911
fetch_images regression, ~0.22.1+). Pinned the Mistral container to
v0.22.0 — the last pre-regression release — which loads the NVFP4
(compressed-tensors) AND serves vision: verified a half-blue/half-red
image read correctly ('left blue, right red'). Dropped --limit-mm
(vision re-enabled). qwen36 stays on 0.23.0 (separate container; needs it
for its ModelOpt NVFP4).
- gateway: add mistral-small-4-reasoning. Operator asked for effort=medium
but Mistral's reasoning_effort is BINARY (none/high only — medium 400s);
set to 'high' (sole reasoning-ON level). NOTE: reasoning fires but
reasoning_content-splitting is unreliable on v0.22.0 (lands in content);
clean split would need 0.23.0, which breaks vision — vision prioritized.
- mistral-small-4 (instant) + mistral-small-4-reasoning both gateway-live.
Mistral-Small-4-119B-2603-NVFP4 (119B/6.5B-active MoE, 65.3 GiB) on the
freed GPU 0 (dedicated 96 GB Blackwell), vLLM 0.23.0, :8010. NVFP4 is the
only variant that fits one card (FP8 ~119 GB / bf16 ~238 GB need 2 GPUs).
- TEXT-ONLY: vLLM 0.23.0's Mistral multimodal processor crashes at startup
(fetch_images bug); loaded with --limit-mm-per-prompt image/video=0.
Remove the flag to restore vision once vLLM patches it.
- MLA attn (TRITON_MLA), mistral tool-call + reasoning parsers, util 0.93,
max-len 131072 (capped from native 256K), image pinned by 0.23.0 digest.
- litellm: mistral-small-4 → :8010, shadows the * wildcard.
- GPU 0 reassigned from the (now-offline) llama-swap zoo per operator.
The qwen3.6-35b-a3b VL checkpoint is a single hybrid model with a per-
request enable_thinking switch (Qwen3-style), defaulting thinking ON.
Make the default non-thinking and add an opt-in reasoning variant,
mirroring the existing glm-5.1 / glm-5.1-reasoning gateway split.
- qwen36-vl compose: add --reasoning-parser qwen3 (model-matched) so the
single :8007 endpoint splits <think> into reasoning_content when on and
routes all output to content when off — serving both modes cleanly.
- litellm gateway: base qwen3.6-35b-a3b pins chat_template_kwargs
enable_thinking=false (non-thinking default); new qwen3.6-35b-a3b-thinking
pins enable_thinking=true (opt-in reasoning). Same upstream checkpoint,
no extra VRAM/container.
Deployed + verified on ana-ml2 (vLLM recreated, healthy) and ana-docker
(litellm reloaded): default returns a direct answer with no reasoning_content;
-thinking returns cleanly-separated reasoning_content, no raw tag leak.
Retire qwen35-vl (Qwen3.5-9B); add qwen36-vl serving the official FP8
Qwen3.6-35B-A3B vision MoE on :8007 under its TRUE name only — no alias.
qwen3.5-9b-fp8 is killed at vLLM AND the litellm gateway (404/400); a model is
never served under a prior model's name. Consumer (comfy-dev/arbo) notified +
migrated; arbo vkeys flipped to all-proxy-models; shared all-agents-local key
repointed to qwen3.6-35b-a3b.
GPU-1 rebalance for the heavier FP8 weights (~34 GB): granite 0.35->0.24 /
131K->64K, embed/rerank 0.05->0.03 (reclaimed util-reservation waste). Verified:
vision correct, 20-concurrent/endpoint load test = no OOM (~7.5 GB headroom).
Drop the llama-swap qwen3.5-9b GPU-0 pin (GPU 0 freed for the creative-writing
hot-swap card). NVFP4 was the lighter fit (~21 GB) but its vLLM ModelOpt-MoE
loader is broken (KeyError w2_input_scale / lm_head.input_scale, vllm #44081);
revisit when fixed.
Operator's call: keep the arbo stack in eshpfi and version its deploy machinery
alongside the compose (was host-only on irv-ml1 = recoverability foot-gun).
- arbo-webhook.py: :9009 HMAC listener (secret externalized to host file, not git)
- arbo-deploy.sh: internal-route fetch + catalog-only targeted restart
Document both in the README Q5 section + the internal-gitea-route gotcha.
Fleet/colo hosts must reach gitea over the internal route (ana-docker
container git-SSH at 10.250.50.70:222), not the public gitea.phasefinal.com:22
which fail2bans the host's egress IP and silently wedges webhook auto-deploys.
Bit irv-ml1's arbo deploy 2026-06-13.
storetank archive fully resolved (919 G -> 0): ~739 G killed (superseded/niche),
177 G migrated into arbo, rest dupes. Rewrite the archive doc as a decommission
record; refresh the arbo catalog to its post-migration 502 G state (+ SDXL/Pony
stack + 9 gen-agnostic utility categories).
Swept the orphaned llava_llama3 (HunyuanVideo text encoder, 23.5 G) after the
Hunyuan kill left it unreferenced. Update the curation table + remaining total.
Capture the 2026-06-13 archive curation pass (919->238 G, 681 G reclaimed:
Hunyuan + WAN2.1 + FLUX.1 + umt5 orphan, all superseded by arbo's current-gen
stack) and a detailed catalog of the remaining 238 G (SDXL/Pony stack, SD3.5/
Chroma, gen-agnostic utilities, shared encoders) for comfy-dev's migration
decisions into the active arbo set.
Operator decision 2026-06-13 (relayed by comfy-dev, confirmed in-session):
turn off the prod arbo engine's bearer auth and rely on the WireGuard
perimeter. Reverses ADR-0001's open-auth-hole-closed posture (comfy-dev owns
the ADR update on the vh/arbo side).
The app's protected-gate no-ops only when ENGINE_TOKEN is ABSENT — an empty
string still gates (verified: ENGINE_TOKEN="" -> /workflows 401). So both
inject paths are removed: the compose environment line is commented out and
the .env line deleted on the host. Result: tokenless GET /workflows 200 (was
401), matching the dev engine. Original token preserved in the host's
.env.pre-auth-off.bak for re-enable.
playbooks/arbo-disable-engine-token.yaml captures the reversible procedure.
The arbo-prompt-enhance vkey was extended to reach qwen3.5-9b-fp8 for the
hero auto-judge step (v0.11.3+), not granite-only. Confirmed via /v1/models
for the key. Docs-only; no version bump.
ana-ml2 was upgraded 2026-06 from dual RTX 6000 Ada (46GB, cc 8.9) to
dual RTX PRO 6000 Blackwell Max-Q (96GB, cc 12.0 / sm_120). Update the
stale hardware facts across the workspace:
- CLAUDE.md servers table row
- servers/ana-ml2/README.md hardware spec (+ refreshed system-details.txt)
- stacks/vllm compose + .env.example FP8/KV comments (Ada cc 8.9 -> Blackwell cc 12.0)
- stacks/llama-swap config VRAM-budget comment (48GB -> 96GB, GPU-0 pin)
Also corrects the adjacent stale 'Phi-4-mini' comment in the granite
service block (the service has been Granite 4.1 8B since 34a43a0).
Doc/comment-only; no runtime change.
Benched granite prefix caching at ~6.5x faster TTFT (45ms cached vs 292ms
uncached) on a shared ~4.5k-token summarizer template. granite already had it
on by vLLM-v1 default; pinned explicit so a version flip can't silently disable
it. qwen (nightly) defaulted it OFF -> flipped on (free for the text-chat path,
marginal for vision where each image is a distinct prefix). Soft/evictable KV,
zero memory change (GPU1 still ~3.7GB free), all 5 services healthy.
ComfyUI's ~325G model tree moved off the near-full worktank NVMe (97%->26%,
342G free) to /storetank/arbo (roomy SATA SSD on irv-ml1), overlay-mounted
back at /basedir/models so ComfyUI behaviour is unchanged. rsync byte-verified
(src==dst), one comfyui restart, worktank original removed. Inventory of the
set in docs/arbo-comfyui-model-catalog.md for the retain decision. The older
919G /storetank/image-models/comfy archive is untouched (separate reclaim).
Qwen3.5-9B vision-language served FP8 on ana-ml2 GPU1 (co-located with
granite + the embed/rerank/reward trio; GPU0 kept free for hot-loading
large models), :8007, fronted by LiteLLM as qwen3.5-9b-fp8.
Pinned to vllm/vllm-openai nightly@sha256:49211ab2 — :latest (v0.19.1)
quantizes the VL vision tower under fp8 and garbles vision; the nightly
correctly excludes it (LM stays FP8, vision tower BF16). util 0.40
(~38GB) on the shared card (vLLM needs free>=util*total here). Vision
verified end-to-end through the gateway.
SGLang 0.5.13 confirmed to support our formats on Blackwell sm_120
(compressed-tensors NVFP4 W4A4, fp8, modelopt_fp4, petit_nvfp4, fp4_e2m1 KV),
so the bench can be a real NVFP4 head-to-head. Parameterized compose (model/
quant/GPU via .env) + a common streaming load generator (bench.py: agg tok/s,
TTFT p50/p99, TPOT) so both engines are driven identically on an exclusive GPU.
Bench-oriented; promote to a real stack only if SGLang wins. Launch deferred
until the NVFP4 eval frees a GPU.
Ada->Blackwell swap doubled card VRAM, so the Ada-era fractions (0.07/0.07/
0.18) reserved ~2x the bytes for the same models. Empirically re-floored via
0.01-step climb until each service was stable under 20x parallel inference:
embed/rerank 0.05 (load-floor for the 0.6B models), reward 0.10 (the real
over-provision). Frees ~11 GB on GPU 1. Live .env on ana-ml2 already applied.
ana-ml2's Ada->Blackwell swap (2x96GB) frees GPU 0 entirely. Pin llama-swap
to GPU 0 via NVIDIA_VISIBLE_DEVICES so on-demand large-model hot-loads land
there, off GPU 1 where the always-on vLLM services (granite + embed/rerank/
reward) live. Closes the long-standing 'pin llama-swap to GPU 0' item.
Extends the catalog/graphs git-pull-mount pattern to the SPA frontend so
frontend changes reach prod via git pull + restart, no image rebuild.
Delivers the v0.11.2 auth-on catalog-load fix without a rebuild; baked
image frontend stays the fallback.
New stack mirroring the canonical convention for the Arbo (catalog) engine,
co-located beside comfyui on irv-ml1 per ADR-0001 D1/D3:
- engine<->ComfyUI over traefik-net container DNS (http://comfyui:8188),
SSH dependency eliminated; file ops bind the shared basedir input/output
- named local-disk volumes for the gallery SQLite (arbo_db, restic-backed)
and hero images (arbo_heroes); catalog as a ro git-checkout mount (D2)
- ENGINE_TOKEN + GRANITE_KEY via on-host .env; GRANITE via the LiteLLM gateway
- Q5 catalog-pull: manual day-1, ytvc-style webhook follow-on
Image build, /healthz, catalog in-container path, and non-root UID are
comfy-dev's to confirm (CONFIRM items in README).
glm-5.1 now disables GLM thinking by default via extra_body (LiteLLM strips
top-level thinking under drop_params but forwards extra_body verbatim to z.ai).
New glm-5.1-reasoning alias = same upstream with thinking enabled, so reasoning
is opt-in. Operator call 2026-06-11; primary driver is the pi coding harness.
Verified live: glm-5.1 reasoning_tokens=0, glm-5.1-reasoning reasoning_tokens>0.
Adds graphify-out/GRAPH_REPORT.md (knowledge-graph map of the repo) and a
.gitignore block that tracks only the lightweight map while ignoring the
regenerable graph.json/cache/html. Part of the fleet-wide Graphify rollout.
Local post-commit auto-rebuild hook retained.
- migration-plan.md: workstation-only migration of nh3-dev's dev env to
the M2 Ultra Mac Pro Rack (on-subnet, sidecars stay on Linux).
- nh3-dev README: drop volva from live sidecars + add retired note.
volva.service/heid.service removed by infra-ops at heid's request
(re-architected daemon->CC-orchestrator; volva.service was crash-looping).
LLM observability for the fleet — pretty trace UI over the gateway: prompts,
completions, reasoning, latency, token counts. The pretty layer LiteLLM's
spend_logs lacked.
- stacks/langfuse: v3 self-host stack (web/worker/postgres/clickhouse/redis/
minio) on ana-docker, adapted from upstream. UI on :3001 (gitea owns :3000).
Project + API keys auto-provisioned via LANGFUSE_INIT_*. HOSTNAME=0.0.0.0 on
langfuse-web so it's reachable via the published port while also on tnet.
- litellm: enabled success_callback/failure_callback: ["langfuse"] (the
passthrough env was already wired); keys + host go in the litellm .env.
Verified: stack healthy, project keys authenticate, and a real gateway call
landed a litellm-acompletion trace in Langfuse within ~6s. Secrets live only in
the server .env (never committed).
Granite 4.1 8B beat phi4-mini on precision in brokkr's R15 P03 eval, so it's
the new production summarizer/dreamer for nevermore.
- vllm-phi4 -> vllm-granite: official IBM FP8 (ibm-granite/granite-4.1-8b-fp8,
compressed-tensors), GPU 1, 50K ctx, FP8-KV, CUDA graphs. Same :8004 slot.
- GPU 1 retune: the embed/rerank/reward trio was over-provisioned (embed ran a
5.89x KV pool, reward 3.90x). Trimmed utils 0.20/0.20/0.30 -> 0.07/0.07/0.18,
freeing ~10 GB so granite runs with CUDA graphs (not --enforce-eager) and
keeps ~10 GB free as a hedge for future Granite text-LoRAs (--enable-lora).
- LiteLLM: phi4-mini model_list entry -> granite-4.1-8b (hosted_vllm @ :8004);
explicit entry shadows the '*' wildcard's llama-swap route.
- nevermore repointed (LLAMA_SWAP_MODEL=granite-4.1-8b via the gateway) live.
Verified end-to-end: vLLM :8004 generates, gateway routes (gateway-granite-ok),
KV 86,768 tokens/1.69x at 50K, 0 restarts, GPU 1 10.3 GB free.
ComfyUI's default cudaMallocAsync allocator phantom-OOMs ('allocation
would exceed allowed memory', 0 bytes allocated) when the A6000 is
shared with the pinned TTS services (~22 GB used). --disable-cuda-malloc
switches to PyTorch's native allocator. --fp8_e4m3fn-text-enc loads the
FLUX.2 Qwen3-8B text encoder as fp8 (~8.7 GB) instead of upcasting the
fp8 file to fp16 (~16 GB), matching the box's Ampere-fp8 posture.
Applied via COMFY_CMDLINE_EXTRA in the canonical compose; verified the
allocator flipped to 'native' and both flags are on the live cmdline.
IBM Granite 4.1 dense instruct (3B + 8B), unsloth Q4_K_M GGUF via -hf
syntax, 64K ctx with q8 KV. Auto-exposed through the LiteLLM gateway
wildcard (ana-docker:4000) and direct on llama-swap (:9292).
Replaces the retired irv-ml1 Ollama granite4.1 — Ollama is now banned
fleet-wide; serving consolidates onto the sanctioned llama-swap/vLLM
substrate.
The asset-engine catalog source of truth. Removes the chatterbox
exaggeration/cfg_weight sliders (proven Turbo no-ops) — reconciling the
canonical with the fix previously applied only to asset-engine's vendored
copy — and adds the csm-expressiva whisper TTS entry (irv-ml1:8198).
Reflects the shared install: browsers in /opt/ms-playwright (root-owned,
world-readable) + PLAYWRIGHT_BROWSERS_PATH wired globally via
/etc/environment + profile.d, so any user/project resolves the shared
binary without a per-home playwright install. Includes the add-a-browser
command for future bumps.
Adds the NH3 dev box (10.100.10.50) as a managed host: README, first
system-details snapshot, ssh-target. Adds it to the fleet bootstrap's
Tier 1 and the CLAUDE.md servers table. infra-ops identity bootstrapped
there (operator-run) — NOPASSWD sudo + key, so root-level infra work on
the box (it runs the egress proxy, ttyd seat, mead-hall, volva, and dev
checkouts) no longer needs a per-task password.
First use: installed Playwright headless-Chromium system deps + binary
for bloom_music's OSMD browser-geometry test harness; headless launch +
real SVG geometry verified.
vh