This website requires JavaScript.
76b317ce3e
feat(backups): freshness check + daily alert timer; record rest-server-ana recovery, fstab hardening, esh-pve-nas gap, worldtree admin-key provisioning
vh
2026-06-20 16:13:13 -07:00
a7b4a82dec
docs(backups): add backup architecture + freshness runbook; record rest-server-ana recovery + correct ana-docker sudo path
vh
2026-06-20 15:56:41 -07:00
8f15f6bb0d
memory: capture 2026-06-20 session — Worldtree v0.37.7 demo fix, gitea notifier recovery, backup diagnosis
vh
2026-06-20 15:46:14 -07:00
58ec80d58a
memory: snapshot — 2026-06-20
vh
2026-06-20 13:36:54 -07:00
f8eda1c333
chore(litellm): retire Langfuse — drop success/failure callbacks (redundant + crash-prone)
vh
2026-06-20 13:21:31 -07:00
7819f96003
feat(litellm): add gen-frontier / gen-frontier-reasoning aliases (→ GLM 5.2)
vh
2026-06-20 10:14:28 -07:00
d0eb09cac1
fix(litellm): remove the * → llama-swap wildcard (decommissioned backend)
vh
2026-06-20 10:08:20 -07:00
d3721034c1
feat(litellm): Worldtree capability aliases (chat-judge, reranker, scalar-judge)
vh
2026-06-20 09:53:09 -07:00
cd92b85157
feat(omnivoice): tune streaming defaults (16-step + aggressive packing)
vh
2026-06-19 22:58:55 -07:00
288d085236
feat(omnivoice): streaming /tts + language-safe sanitizer
vh
2026-06-19 22:47:15 -07:00
826c2a6a64
memory: archive 15 pre-2026-06-16 entries to archival-memory.md
vh
2026-06-19 21:54:45 -07:00
dfda60fac7
memory: snapshot — 2026-06-19 (pt2) litellm task-aliases (classifier->granite, summarizer-large->gen; gen-nt/gen-reasoning-nt added-then-removed as redundant with strip_empty_tools) + gateway-chat model-smoking web chat enhanced (auto-discover /v1/models + image upload) and stood up as a PERSISTENT nginx container on ana-docker :8091 + pi on nh3-dev wired to gen (vision, ~/.pi models.json + gen launcher, local box config) + foot-guns: litellm config-loaded models can't be hot-removed (/model/delete is DB-only; /model/new live-adds work no-bounce but dup on restart) and the * wildcard routes stale/typo'd names to decommissioned llama-swap -> misleading 'Connection error' not 'model not found' (bit a brokkr call to the renamed-away qwen-image-judge).
vh
2026-06-19 18:51:03 -07:00
740bcae45d
feat(gateway-chat): persistent static-serve stack for the model-smoking web chat
vh
2026-06-19 12:32:51 -07:00
ef45f6d826
feat(litellm): add classifier -> granite + summarizer-large -> gen aliases (operator)
vh
2026-06-19 12:08:26 -07:00
4c40b9fac6
feat(tools): gateway-chat.html — auto-discover gateway models + image upload for vision smoke
vh
2026-06-19 11:56:20 -07:00
75bd4c3679
remove gen-nt / gen-reasoning-nt litellm records (operator)
vh
2026-06-19 11:56:20 -07:00
2e5ab72e2c
feat(litellm): add gen-nt / gen-reasoning-nt (noop-tool + tool_choice:none compat variants)
vh
2026-06-19 11:35:48 -07:00
378261763c
memory: snapshot — 2026-06-19 gen model = Qwopus3.5-122B vision-intact NVFP4 LIVE on ana-ml2 GPU 0 (full 256K @ fp8 KV + CUDA graphs, util 0.95 + expandable_segments, 92.7 tok/s warm, 3.32x concurrency, text+image+video, tool-calling qwen3_coder; nightly+turboquant-4bit-KV proven UNNECESSARY — stable fp8 reaches 256K) replacing the bjk110 text-only qwen3.5-122b (which displaced mistral-small-4 → Worldtree character backend DARK until repointed, operator-acknowledged) + qwen-image-bench T2I judge replaced qwen3.6-35b-a3b on GPU 1 (alias image-judge) + TP=2 across both Blackwells REJECTED (PCIe-only PIX, no NVLink → all-reduce-bound, one-model-per-card is optimal; PP=2 only if a >96GB model is ever wanted) + foot-guns: MoE FusedMoE workspace is the ~3.1GB un-budgeted floor (can't fill to 0), discard cold tok/s reads (24.8 cold vs 92.7 warm).
vh
2026-06-19 11:15:32 -07:00
5b06514020
docs(litellm): gen records now describe Qwopus3.5-122B (vision-intact), not bjk110 text-only
vh
2026-06-19 10:25:45 -07:00
20e796cf6b
feat(qwopus3.5-122b): gen model → Qwopus3.5-122B vision-intact NVFP4, full 256K @ fp8
vh
2026-06-19 10:24:34 -07:00
a5b626b3d5
fix(qwen3.5-122b): enable tool-calling (--enable-auto-tool-choice --tool-call-parser qwen3_xml)
vh
2026-06-19 01:56:56 -07:00
5dfce049f4
rename(litellm): qwen-image-judge alias -> image-judge
vh
2026-06-19 01:43:25 -07:00
bfae924048
feat(qwen-image-bench): replace qwen3.6-35b-a3b on GPU1 with the T2I judge (NVFP4)
vh
2026-06-19 01:40:59 -07:00
3ba0e544db
tune(qwen3.5-122b): gpu-mem-util 0.90->0.95, max-num-seqs 4->8 (KV 260K->446K tokens, 3.4x concurrency @131K, no OOM)
vh
2026-06-19 01:21:28 -07:00
89c83c4271
feat(qwen3.5-122b): replace mistral-small-4 as gen (abliterated NVFP4, text-only)
vh
2026-06-19 00:49:04 -07:00
67102b5b94
feat(litellm): add model aliases summarizer / gen / gen-reasoning
vh
2026-06-18 23:56:05 -07:00
91688a234b
revert(litellm): remove mistral-medium-3.5 entry (GPU0 reverted to small-4 heretic)
vh
2026-06-18 23:50:34 -07:00
981ae4e6a1
feat(omnivoice): expose full generation surface (voice-design, language, diffusion params)
vh
2026-06-18 23:25:39 -07:00
71f5784016
feat(litellm): add mistral-medium-3.5 (RecViking NVFP4 :8012, temporary GPU0 tenant)
vh
2026-06-18 23:12:31 -07:00
06eb487a26
feat(omnivoice): wire to asset-engine via FastAPI wrapper + reuse chatterbox voices
vh
2026-06-18 23:03:20 -07:00
984b72757f
feat(omnivoice): new TTS stack — k2-fsa/OmniVoice on irv-ml1 3090
vh
2026-06-18 22:25:54 -07:00
715a68bee7
feat(comfyui): native --use-sage-attention (node path dead on 0.24.1)
vh
2026-06-18 14:16:56 -07:00
632124c8fb
memory: nh3-extdev pi-on-GLM-5.2 wired (mark in-flight item done + residual gates)
vh
2026-06-18 14:05:02 -07:00
527a844714
feat(nh3-extdev): install pi (earendil-works) + wire /opt/externs client agents to GLM 5.2
vh
2026-06-18 14:04:29 -07:00
a67d4950d0
memory: snapshot — 2026-06-18 heretic abliterated Mistral Small 4 NVFP4 BUILT + LIVE as mistral-small-4 (in-house quant device_map=cpu → native-format convert → drop-in stack under same served-name, A/B'd vs official, operator "heretic stays"; byte-equivalent to official NVFP4) + irv-ml1 VRAM consolidation (ComfyUI pinned to A6000 exclusive/48GB, audio zoo→3090, downed dia/ace-step/csm, comfy-dev torch-pin DISABLE_UPGRADES@2.12 .1 + SageAttention rebuilt) + ComfyUI 9-node accel set installed for comfy-dev + ana-ml2 durable vm.overcommit_memory=1 + GLM5.2 wired + nh3-extdev sudo-less manager box + /opt/externs pi-on-GLM client workspaces.
vh
2026-06-18 13:48:12 -07:00
a8550ad4bc
feat(irv-ml1): pin comfyui to A6000 + torch-pin; parakeet -> 3090 (VRAM consolidation)
vh
2026-06-18 11:07:43 -07:00
f566f61b24
feat(stacks): mistral-small-4-heretic drop-in (abliterated NVFP4 backend swap)
vh
2026-06-17 22:33:29 -07:00
dd3a5c93fd
feat(tools): Mistral Small 4 NVFP4 build pipeline (quant + HF->native converter)
vh
2026-06-17 22:19:58 -07:00
fc88eff06e
feat(ana-ml2): durable vm.overcommit_memory=1 sysctl playbook
vh
2026-06-17 22:19:58 -07:00
a841eab3ff
servers: register nh3-extdev (sudo-less infra-ops manager box)
vh
2026-06-17 14:57:06 -07:00
fe77a3596a
litellm: wire GLM 5.2 (glm-5.2 + glm-5.2-reasoning) via z.ai passthrough
vh
2026-06-17 08:56:16 -07:00
8cca365b78
memory: correct gitea action-log API note (per-job endpoint works)
vh
2026-06-16 14:56:12 -07:00
03358dccd1
arbo: mount repo pyproject.toml ro into the engine (catalog_version observability)
vh
2026-06-16 14:42:04 -07:00
3a7236d51f
playbook: put uv/uvx on the irv-ml1-arbo runner PATH
vh
2026-06-16 14:24:08 -07:00
47c30e85f9
memory: snapshot — 2026-06-16 litellm strip_empty_tools hook (d1bea13984ca3d#295 recall root-cause FLIPPED (score_breakdown-shape DISPROVEN → cold-recall agent_self scope axis vs ratatoskr conjunctive INV-005; Worldtree #297 ); lessons: litellm-500-Router.acompletion-missing-messages = a request missing Content-Type (NOT a gateway outage — cost 4 needless restarts), litellm-admin-UI-playground-cant-test-vLLM (#6228 empty-tools, proxy-hook-cant-reach-in-process-call), gitea-run-looks-like-never-fired-but-fired-then-skipped/failed-fast (check run list not runner). Archived the 06-09→06-13 cluster (24 entries: 17 decisions + 7 foot-guns).
vh
2026-06-16 14:14:36 -07:00
984ca3d383
feat(tools): single-file gateway chat playground
vh
2026-06-16 01:40:30 -07:00
d1bea13994
fix(litellm): strip empty tools:[] before forwarding to vLLM
vh
2026-06-16 00:43:57 -07:00
f9277f5440
memory: snapshot — 2026-06-16 ratatoskr Tier-3 MEMORY plane wired (allowlist :8391, key reused, persist+dispatch GREEN; recall-injection root-caused to the score_breakdown shape seam → worldtree-dev #295 ) + infra-ops durable admin on corviduo (ssh alias + ssh-target) + demo/personal character model qwen→mistral (first-bind-is-default reorder, pin-safe recreate); lessons: bifrost-allowlist-is-per-port, promotion-gate=consumer-agent-memory-block-not-agent_self_enabled, WORLDTREE_IMAGE-pin-from-matrix-sibling
vh
2026-06-16 00:13:14 -07:00
c99aa49cad
feat(corviduo): wire ratatoskr memory plane :8391 into personal Worldtree bifrost allowlist
vh
2026-06-15 23:22:06 -07:00
aeea377749
memory: snapshot — 2026-06-16 ana-ml2 dual-NVFP4 reshape (GPU0 Mistral Small 4 256K/v0.22.0-vision + GPU1 qwen36 FP8→NVFP4 + Selene FP8 judge + GPU1 grows) + NVFP4-MoE-loads-on-0.23.0 (supersedes blocked) + claude-bot service account (corviduo-org tabled) + arbo→comfy-dev ownership + gitea runner on irv-ml1 + Worldtree demo/personal capability-profile migration (pre-sync-first); lessons: vLLM-0.23-breaks-Mistral-vision (#44911 ), Mistral-TTFT=Triton-JIT-spikes, vh-user-not-org blocks scoped package-write, old-baseline-instances-need-full-config-set; archived the 2026-06-05/08 cluster (11 entries)
vh
2026-06-15 22:26:11 -07:00
e124a2f233
tune(gpu1): grow selene 0.13→0.17 + qwen36 0.32→0.34 into the buffer
vh
2026-06-15 20:36:04 -07:00
c985ede07b
feat(selene+mistral): restore Selene judge (FP8, GPU1) + push Mistral to 256K
vh
2026-06-15 18:52:26 -07:00
9a49963d07
feat(mistral-small-4): pin v0.22.0 for working VISION baseline + reasoning entry
vh
2026-06-15 17:56:37 -07:00
c77a9aa4d8
feat(mistral-small-4): deploy NVFP4 119B MoE on GPU 0 (text-only) + gateway
vh
2026-06-15 17:28:15 -07:00
c6d76051a4
feat(qwen36-vl): swap FP8→NVFP4 + GPU1 rebalance (granite restored)
vh
2026-06-15 17:28:15 -07:00
6de0844323
feat(qwen36-vl): split thinking — non-thinking default + qwen3.6-35b-a3b-thinking variant
vh
2026-06-15 13:55:11 -07:00
0943d145fb
memory: snapshot — 2026-06-15 (cont.) arbo v0.11.22 engine rebuild + catalog v0.11.23 (curated /workflows footer live) + althing-core v0.14.1 box-wide refresh (monitor lock fix) + comfyui VAE-decode SEGFAULT diagnosis (aimdo 0.4.8 cuda-hooks vs torch cu129/cu130 mismatch, NOT OOM); lessons: comfyui-segfault-not-OOM diagnostic, never blanket-kill peer light-monitors
vh
2026-06-15 13:35:50 -07:00
12bcd06442
memory: snapshot — 2026-06-15 ratatoskr affect smoke GREEN (Heimdall key mint+inject, allowlist, handshake+emit) + infra-ops bootstrapped on corviduo + dense Qwen3-VL-32B-NVFP4 judge A/B (lost, torn down) + MastMed cloudflared public + R18 clip+caption staged (stub smoke passed; real-voice gate pending) + LiteLLM infra-ops key; lessons: corviduo stale-:latest recreate crash, .claude.json ENOSPC repair, pkill self-match
vh
2026-06-15 00:17:44 -07:00
10f346b39e
memory: snapshot — 2026-06-14 FP8 vision cutover (qwen35-vl→qwen36-vl, truthful naming, GPU-1 rebalance) + llama-swap pin drop + R16 yield probe executed + standing credential-migration directive; NVFP4-on-vLLM-blocked + sampler-warmup/profiling-race/embed-rerank-waste lessons
vh
2026-06-14 14:47:01 -07:00
a0fed13801
feat(ana-ml2): replace Qwen3.5-9B vision with Qwen3.6-35B-A3B FP8 on GPU 1
vh
2026-06-14 14:41:49 -07:00
b45d0cd86d
memory: snapshot — 2026-06-14 arbo auth-off + deploy-pipeline fix (v0.11.6, internal gitea route, catalog-only restart, scripts tracked) + storetank archive decommission (919G -> arbo 502G) + R16 inline arc closed (v1 final); archive 11 (2026-06-04 cluster)
vh
2026-06-14 08:56:12 -07:00
6d66bc2f30
feat(arbo): track webhook deploy scripts (arbo-deploy.sh + arbo-webhook.py)
vh
2026-06-13 17:17:49 -07:00
6e58e57362
docs(orientation): gitea internal-route gotcha (fleet hosts -> 10.250.50.70:222)
vh
2026-06-13 17:04:36 -07:00
5007ec1236
docs(catalog): archive decommissioned + arbo refreshed (502 G post-migration)
vh
2026-06-13 15:40:56 -07:00
308ca6f5d2
docs(catalog): record llava_llama3 sweep (919->214 G, 705 G reclaimed)
vh
2026-06-13 15:21:11 -07:00
1902425682
docs(catalog): record storetank image-models curation + remaining inventory
vh
2026-06-13 15:18:21 -07:00
db97899037
feat(arbo): disable ENGINE_TOKEN bearer auth on prod (WireGuard = boundary)
vh
2026-06-13 14:05:07 -07:00
f32c6ddaab
docs(arbo): GRANITE_KEY scope now granite + qwen-vision (extended)
vh
2026-06-13 13:45:32 -07:00
355a2407a2
docs(ana-ml2): correct GPU spec Ada -> RTX PRO 6000 Blackwell (96GB, cc 12.0)
vh
2026-06-13 13:36:14 -07:00
0fc9083d16
memory: snapshot — 2026-06-13 ana-ml2 Ada→Blackwell + NVFP4-infeasible + Qwen3.5-VL FP8 + comfyui→arbo + GPU-1 rebalance + prefix-caching; archive 8 (2026-06-03 cluster)
vh
2026-06-13 13:30:19 -07:00
a9a2be7060
tune(vllm): pin --enable-prefix-caching on granite + qwen
vh
2026-06-13 12:30:20 -07:00
1e2a3a13b5
tune(vllm): GPU-1 rebalance — granite 131k ctx, qwen 65k ctx, ~3.5GB free
vh
2026-06-13 08:46:46 -07:00
38186be1a7
feat(comfyui): migrate 325G model tree worktank -> /storetank/arbo
vh
2026-06-13 03:31:56 -07:00
2e3dcc2d3d
feat(qwen35-vl): Qwen3.5-9B VL FP8 stack on ana-ml2 GPU1 + LiteLLM entry
vh
2026-06-13 02:39:33 -07:00
5f049cb4ad
feat(sglang): stage vLLM-vs-SGLang bench stack on ana-ml2
vh
2026-06-12 22:40:26 -07:00
19a07b96ab
tune(vllm): re-floor trio GPU util for Blackwell (96GB), 20x-parallel-stable
vh
2026-06-12 17:41:13 -07:00
edf0f912f8
feat(llama-swap): pin to GPU 0, reserving it for large-model hot-loads
vh
2026-06-12 15:17:46 -07:00
922e8ad3d5
feat(arbo): ro-mount frontend from checkout (v0.11.2 delivery, ADR-0001 D2)
vh
2026-06-12 11:06:17 -07:00
bdb3312298
fix(arbo): python-based healthcheck (slim image ships no curl/wget)
vh
2026-06-12 10:49:49 -07:00
ee57e69ce8
feat(arbo): add irv-ml1 co-located engine stack (ADR-0001)
vh
2026-06-12 10:25:04 -07:00
005effd664
memory: snapshot — 2026-06-11 Mac Pro migration framing + GLM thinking-off + R17 v2 corpus; archive TTS-streaming arc
vh
2026-06-11 08:27:06 -07:00
95b2701c00
feat(litellm): default glm-5.1 to thinking-off; add glm-5.1-reasoning opt-in
vh
2026-06-10 21:23:34 -07:00
01bb7f24ce
merge: graphify-tooling — Granite-labeled codebase map
vh
2026-06-10 06:39:10 -07:00
28620ba78b
chore(graphify): add Granite-labeled codebase map
vh
2026-06-10 06:28:28 -07:00
6e2f80e18e
docs(nh3-dev): add Mac Pro migration plan; retire vestigial volva/heid daemons
vh
2026-06-07 21:45:17 -07:00
5349da567c
memory: snapshot — 2026-06-05 granite-FP8 cutover + Langfuse observability + worldtree #254 deferral
vh
2026-06-05 14:14:44 -07:00
9171e6a20f
feat(langfuse): stand up Langfuse v3 + wire the LiteLLM trace callback
vh
2026-06-05 11:35:01 -07:00
34a43a0bc5
feat(vllm): replace phi4-mini with Granite 4.1 8B summarizer + retune GPU 1
vh
2026-06-05 09:34:22 -07:00
89a020a3cc
fix(comfyui): native allocator + fp8 text-enc to stop FLUX.2 phantom-OOM
vh
2026-06-05 01:43:28 -07:00
03d73a212e
feat(llama-swap): serve Granite 4.1 3B + 8B instruct for brokkr R15 P03
vh
2026-06-05 00:31:39 -07:00
cf35adaf6d
catalog(canonical): remove chatterbox dead levers + add csm-expressiva
vh
2026-06-04 18:02:13 -07:00
df340c808f
docs(nh3-dev): Playwright/Chromium is box-wide, not lkraven-only
vh
2026-06-04 14:09:33 -07:00
4f0c5f1f66
feat(inventory): bring nh3-dev under infra-ops management
vh
2026-06-04 14:03:52 -07:00
c7ee8884c6
feat(litellm): front z.ai GLM cloud models through the gateway
vh
2026-06-04 11:13:32 -07:00
349b3614db
feat(litellm): front llama-swap through the gateway via wildcard passthrough
vh
2026-06-04 11:02:35 -07:00
83b2ec1a8a
feat(litellm): add vLLM request/response logging gateway on ana-docker
vh
2026-06-04 01:17:50 -07:00
1dba05d8c8
memory: snapshot — 2026-06-04 infra session (phi4/vLLM, infra-ops identity, R15/R16, brokkr svc)
vh
2026-06-04 01:08:50 -07:00
27eb53735a
revert(vllm-phi4): back to canonical/official Phi-4 chat template
vh
2026-06-04 00:39:36 -07:00
90e08f0502
fix(vllm-phi4): Ollama-matching chat template to recover R15 baseline
vh
2026-06-04 00:29:05 -07:00
40a374b809
feat(vllm): add phi4-mini FP8 summarizer/dreamer on ana-ml2; retire granite from llama-swap
vh
2026-06-03 23:21:45 -07:00
vh