fix(litellm): retire image-bench backend, repoint image-judge + qwen-image-bench aliases to gen
Operator-directed 2026-07-15. The dedicated Qwen-Image-Bench NVFP4 judge backend on ana-ml2 GPU1 (:8014) was stopped to reclaim ~32GB after the arbo -> gen hero-judge switch. Both LiteLLM gateway aliases that pointed at it -- image-judge and qwen-image-bench -- now repoint to the gen backend (:8015, qwen3.6-35b-a3b-heretic, vision-intact), held at deterministic judge sampling (temp 0 / top_k 1 / rep_pen 1.05) with enable_thinking:false (a reasoning preamble breaks json_object). Verified live: both answer with :8014 down, so they are definitively on gen. Incidental: backfilled the canonical char-rp-reasoning litellm block, which had lagged live since the 2026-07-14 NVFP4+MTP seat repoint (model deckard-pkd-27b -> char-rp-reasoning, top_k 40 -> 20, min_p dropped, enable_thinking:true added). Required so pushing the canonical would not clobber the correct live block. Live changes applied out-of-band (config push + litellm restart + stack stop on ana-ml2); live config backup at config.yaml.bak-pre-imagejudge-20260715. Revert path documented in the config comment. persistent-memory.md updated (parked item closed).
This commit is contained in:
@@ -118,7 +118,7 @@ _As of 2026-07-15 — a long fleet-ops + infra session. Everything below either
|
||||
- **Statusline overhaul** (`~/.claude/statusline-command.sh`, NOT this repo) — git state (branch/dirty/unpushed), 🔔/🔕 althing-monitor-armed, project tag, absolute tokens, **per-session cost** (`.cost.total_cost_usd`), threshold-colored ctx% + rate %s (green<60 / yellow60-90 / red>90).
|
||||
|
||||
**PARKED (operator-directed holds):**
|
||||
- **image-bench (qwen-image-bench, GPU1) eviction — PENDING post-bake.** Operator directed a full arbo -> gen switch (comfy-dev executed it, LIVE in prod: `DEFAULT_LLM_MODEL=gen` + `ARBO_VISION_MODEL=gen`); image-judge now has ZERO arbo tenancy. Sequencing = **short prod bake on gen (~few days) THEN evict** -> reclaim ~30 GB GPU1. Keep `vllm-qwen-image-bench` WARM as the instant-revert backstop; comfy-dev watches NSFW-anatomy scoring + tight-grounding (flip `ARBO_VISION_MODEL` back on regression). On a clean bake: evict + ping comfy-dev. **Fully tracked in auto-memory `project_arbo_gen_switch_imagebench_evict`.**
|
||||
- **image-bench (qwen-image-bench, GPU1) eviction — DONE 2026-07-15 (operator-directed, same-day as the switch).** Stopped `vllm-qwen-image-bench` on ana-ml2 (Exited(0), revert via `docker compose start`) → GPU1 freed ~32 GB (95→63 GB used). LiteLLM `image-judge` + `qwen-image-bench` aliases repointed to gen (:8015, qwen3.6-35b-a3b-heretic), deterministic judge sampling (temp0/top_k1/rep_pen1.05) + `enable_thinking:false` kept; verified live with :8014 down. comfy-dev pinged as regression watcher (NSFW-anatomy + tight-grounding; thread 01KXMRDJ). Backstop = the stopped stack (revert ~90 s) + live config backup `config.yaml.bak-pre-imagejudge-20260715`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`.
|
||||
- **granite right-size — un-started GPU1 win.** granite (GPU1) is ~29 GB (util 0.34 -> ~16.5 GB oversized KV for an 8B summarizer). Right-sizing util 0.34 -> ~0.13 frees ~16 GB WHILE keeping the summarizer (fleet's dedicated summarizer/classifier; aliases granite-4.1-8b/summarizer/classifier). Independent of image-bench; operator was freeing GPU1 for a TBD "something else".
|
||||
|
||||
**Open loops for the fresh session:**
|
||||
@@ -128,6 +128,8 @@ _As of 2026-07-15 — a long fleet-ops + infra session. Everything below either
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-07-15]` **image-bench eviction DONE (parked item closed).** Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM `image-judge`+`qwen-image-bench` → gen :8015 (judge samplers + thinking-off), verified with :8014 down; comfy-dev pinged; also backfilled the canonical char-rp-reasoning litellm block (was lagging live). Revert ~90 s. auto-memory `project_arbo_gen_switch_imagebench_evict`.
|
||||
|
||||
- `[2026-07-15]` arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake → `persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md`
|
||||
|
||||
- `[2026-07-15]` esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service` → `persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md`
|
||||
|
||||
@@ -72,35 +72,40 @@ model_list:
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
# --- Qwen-Image-Bench (NVFP4) — text-to-image quality JUDGE, vision-intact. vLLM
|
||||
# on ana-ml2 GPU 1, :8014. REPLACED qwen3.6-35b-a3b (operator 2026-06-19; the
|
||||
# qwen36 VL judge was downed + its FP8 weights freed ~33GB). Multimodal — image
|
||||
# + prompt in, a 5-dim verdict out (overall quality / prompt match / aesthetic /
|
||||
# LoRA activation / confidence). image-judge is a duplicate-entry alias.
|
||||
# ⚠️ arbo's hero-judge was bound to qwen3.6-35b-a3b — comfy-dev notified to
|
||||
# repoint (different I/O than the old general VL judge). ---
|
||||
# --- image-judge / qwen-image-bench — T2I quality JUDGE aliases. The dedicated
|
||||
# Qwen-Image-Bench NVFP4 backend (ana-ml2 GPU 1, :8014) was RETIRED 2026-07-15
|
||||
# (operator: reclaim ~30GB GPU1) after the arbo→gen hero-judge switch. Both
|
||||
# aliases now REPOINT to the gen backend (:8015, qwen3.6-35b-a3b-heretic,
|
||||
# vision-intact), held at deterministic judge sampling (temp 0 / top_k 1) with
|
||||
# enable_thinking:false (a reasoning preamble breaks json_object). Revert =
|
||||
# `docker compose start` stacks/qwen-image-bench on ana-ml2 + repoint api_base
|
||||
# back to :8014 + model hosted_vllm/qwen-image-bench. ---
|
||||
- model_name: qwen-image-bench
|
||||
litellm_params:
|
||||
model: hosted_vllm/qwen-image-bench
|
||||
api_base: http://10.250.50.54:8014/v1
|
||||
model: hosted_vllm/qwen3.6-35b-a3b-heretic
|
||||
api_base: http://10.250.50.54:8015/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
temperature: 0
|
||||
top_p: 1.0
|
||||
extra_body:
|
||||
top_k: 1
|
||||
repetition_penalty: 1.05
|
||||
chat_template_kwargs:
|
||||
enable_thinking: false
|
||||
model_info:
|
||||
mode: chat
|
||||
- model_name: image-judge
|
||||
litellm_params:
|
||||
model: hosted_vllm/qwen-image-bench
|
||||
api_base: http://10.250.50.54:8014/v1
|
||||
model: hosted_vllm/qwen3.6-35b-a3b-heretic
|
||||
api_base: http://10.250.50.54:8015/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
temperature: 0
|
||||
top_p: 1.0
|
||||
extra_body:
|
||||
top_k: 1
|
||||
repetition_penalty: 1.05
|
||||
chat_template_kwargs:
|
||||
enable_thinking: false
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
@@ -184,14 +189,15 @@ model_list:
|
||||
# char-rp (best-of-breed per seat) — see stacks/char-rp-gguf/README.md.
|
||||
- model_name: char-rp-reasoning
|
||||
litellm_params:
|
||||
model: hosted_vllm/deckard-pkd-27b
|
||||
model: hosted_vllm/char-rp-reasoning
|
||||
api_base: http://10.250.50.54:8018/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
temperature: 1.0
|
||||
top_p: 0.95
|
||||
extra_body:
|
||||
top_k: 40
|
||||
min_p: 0.05
|
||||
top_k: 20
|
||||
chat_template_kwargs:
|
||||
enable_thinking: true
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
|
||||
Reference in New Issue
Block a user