esh-pfi-infrastructure

1,206 Commits 2 Branches 5 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
vh	e124a2f233	tune(gpu1): grow selene 0.13→0.17 + qwen36 0.32→0.34 into the buffer Put GPU1's idle ~11 GB buffer to work on the two KV-bound models that gained live consumers from the worldtree migration (granite + the pooling models under-use their util, so growing them is wasted): - selene 0.13→0.17: KV 2.53→6.33 GiB, concurrency 1.27x→3.16x @32K (Domari judge) - qwen36 0.32→0.34: KV 7.73→9.63 GiB, concurrency 2.92x→3.64x @131K (arbo judge + worldtree actor/echo + gateway) GPU1 free now ~5.6 GB (safe floor for single-service recreates).	2026-06-15 20:36:04 -07:00
vh	c6d76051a4	feat(qwen36-vl): swap FP8→NVFP4 + GPU1 rebalance (granite restored) The nvidia ModelOpt NVFP4 MoE that failed on vLLM 0.19.1/0.22.0 (#44081) loads clean on 0.23.0. Cut prod qwen36 FP8→NVFP4: ~20.4 GiB weights vs ~34 (~40% lighter, ~13 GB reclaimed on GPU 1), faster single-stream on Blackwell FP4 cores, vision tower preserved (comfy-dev real anatomy-judge A/B on 16 prod images: PASS; brokkr text/speed: parity bar a minor multi-step-chained-reasoning slip that doesn't bite the judge role). - compose: pin image by 0.23.0 digest, drop --kv-cache-dtype fp8 (fp16 KV — the freed room buys full-precision KV), util 0.46→0.32. - GPU1 rebalance (pinned): granite restored 0.24→0.34 / 65536→131072 (undoes the FP8-era sacrifice); trio unchanged; total ~0.82, ~24 GB free. - gateway model name qwen3.6-35b-a3b unchanged (now NVFP4 behind it); thinking-split (enable_thinking=false default) intact — the judge needs it.	2026-06-15 17:28:15 -07:00
vh	a0fed13801	feat(ana-ml2): replace Qwen3.5-9B vision with Qwen3.6-35B-A3B FP8 on GPU 1 Retire qwen35-vl (Qwen3.5-9B); add qwen36-vl serving the official FP8 Qwen3.6-35B-A3B vision MoE on :8007 under its TRUE name only — no alias. qwen3.5-9b-fp8 is killed at vLLM AND the litellm gateway (404/400); a model is never served under a prior model's name. Consumer (comfy-dev/arbo) notified + migrated; arbo vkeys flipped to all-proxy-models; shared all-agents-local key repointed to qwen3.6-35b-a3b. GPU-1 rebalance for the heavier FP8 weights (~34 GB): granite 0.35->0.24 / 131K->64K, embed/rerank 0.05->0.03 (reclaimed util-reservation waste). Verified: vision correct, 20-concurrent/endpoint load test = no OOM (~7.5 GB headroom). Drop the llama-swap qwen3.5-9b GPU-0 pin (GPU 0 freed for the creative-writing hot-swap card). NVFP4 was the lighter fit (~21 GB) but its vLLM ModelOpt-MoE loader is broken (KeyError w2_input_scale / lm_head.input_scale, vllm #44081); revisit when fixed.	2026-06-14 14:41:49 -07:00

Author

SHA1

Message

Date

vh

e124a2f233

tune(gpu1): grow selene 0.13→0.17 + qwen36 0.32→0.34 into the buffer

Put GPU1's idle ~11 GB buffer to work on the two KV-bound models that gained
live consumers from the worldtree migration (granite + the pooling models
under-use their util, so growing them is wasted):
- selene 0.13→0.17: KV 2.53→6.33 GiB, concurrency 1.27x→3.16x @32K (Domari judge)
- qwen36 0.32→0.34: KV 7.73→9.63 GiB, concurrency 2.92x→3.64x @131K (arbo judge +
  worldtree actor/echo + gateway)
GPU1 free now ~5.6 GB (safe floor for single-service recreates).

2026-06-15 20:36:04 -07:00

vh

c6d76051a4

feat(qwen36-vl): swap FP8→NVFP4 + GPU1 rebalance (granite restored)

The nvidia ModelOpt NVFP4 MoE that failed on vLLM 0.19.1/0.22.0 (#44081)
loads clean on 0.23.0. Cut prod qwen36 FP8→NVFP4: ~20.4 GiB weights vs
~34 (~40% lighter, ~13 GB reclaimed on GPU 1), faster single-stream on
Blackwell FP4 cores, vision tower preserved (comfy-dev real anatomy-judge
A/B on 16 prod images: PASS; brokkr text/speed: parity bar a minor
multi-step-chained-reasoning slip that doesn't bite the judge role).

- compose: pin image by 0.23.0 digest, drop --kv-cache-dtype fp8 (fp16 KV
  — the freed room buys full-precision KV), util 0.46→0.32.
- GPU1 rebalance (pinned): granite restored 0.24→0.34 / 65536→131072
  (undoes the FP8-era sacrifice); trio unchanged; total ~0.82, ~24 GB free.
- gateway model name qwen3.6-35b-a3b unchanged (now NVFP4 behind it);
  thinking-split (enable_thinking=false default) intact — the judge needs it.

2026-06-15 17:28:15 -07:00

vh

a0fed13801

feat(ana-ml2): replace Qwen3.5-9B vision with Qwen3.6-35B-A3B FP8 on GPU 1

Retire qwen35-vl (Qwen3.5-9B); add qwen36-vl serving the official FP8
Qwen3.6-35B-A3B vision MoE on :8007 under its TRUE name only — no alias.
qwen3.5-9b-fp8 is killed at vLLM AND the litellm gateway (404/400); a model is
never served under a prior model's name. Consumer (comfy-dev/arbo) notified +
migrated; arbo vkeys flipped to all-proxy-models; shared all-agents-local key
repointed to qwen3.6-35b-a3b.

GPU-1 rebalance for the heavier FP8 weights (~34 GB): granite 0.35->0.24 /
131K->64K, embed/rerank 0.05->0.03 (reclaimed util-reservation waste). Verified:
vision correct, 20-concurrent/endpoint load test = no OOM (~7.5 GB headroom).

Drop the llama-swap qwen3.5-9b GPU-0 pin (GPU 0 freed for the creative-writing
hot-swap card). NVFP4 was the lighter fit (~21 GB) but its vLLM ModelOpt-MoE
loader is broken (KeyError w2_input_scale / lm_head.input_scale, vllm #44081);
revisit when fixed.

2026-06-14 14:41:49 -07:00