feat(h2o): temporary h2o-lightning-4b eval seat on fv-ml1 GPU 3 + the seat front (admission, bearer, /decide/shared translator)

Prime's call via brokkr: h2o-lightning-4b replaces the intern-decision seat once albok-dev re-validates Nemi's sensitivity judge on it. services/h2o-decision-front builds on infra-hermes's admission proxy (30f36f2): 32,000-token refuse-not-trim, the seat's bearer token, and intern-decision-serve's /decide/shared mapping onto /v1/systemone so the judge runs unchanged. stacks/h2o-eval serves it on GPU 3 :8034 exactly as benched (vLLM 0.30.0 + unmodified vendor shim; weights hashed against HF b43f759b). INC keeps serving production on GPU 1.
2026-10-12 13:01:53 -07:00
parent 30f36f2478
commit 2c3ef7a8af
7 changed files with 576 additions and 0 deletions
@@ -0,0 +1,56 @@
+# TEMPORARY h2o-lightning-4b decision seat on fv-ml1 GPU 3 (Prime 2026-10-12 via brokkr: "stand up the h2o copy on
+# gpu 3 if you need to test, it should replace the live seat as soon as possible and keep gpu3 only for transient
+# tasks"). It exists so albok-dev can re-validate Nemi's sensitivity judge on h2o before the GPU 1 cutover; REMOVE it
+# right after the cutover. INC (stacks/intern-decision) keeps serving production on GPU 1 meanwhile.
+# Serving exactly as benched (docs/pfi/decision-sweep-followon-bench-2026-10-12.md, Amendment 1): stock vLLM 0.30.0
+# + the vendor's unmodified shim + the admission front (services/h2o-decision-front, copied into conf/front).
+# Weights: /tank/aimodels/h2o-lightning-4b-b43f759b (hashes match HF at b43f759b; see the stack README).
+name: h2o-eval
+services:
+  vllm:
+    image: vllm/vllm-openai:v0.30.0
+    container_name: h2o-eval-vllm
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:18199:8000"
+    environment:
+      VLLM_USE_FLASHINFER_SAMPLER: "0"
+    volumes:
+      - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
+      - h2o-eval-vllm-cache:/root/.cache/vllm
+    command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b",
+              "--max-model-len", "40960", "--max-num-seqs", "16", "--gpu-memory-utilization", "0.1506",
+              "--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}",
+              "--mm-processor-kwargs", "{\"max_pixels\": 1605632}"]
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids: ["3"]
+              capabilities: [gpu]
+  front:
+    image: python:3.12-slim
+    container_name: h2o-eval-front
+    restart: unless-stopped
+    network_mode: host
+    depends_on: [vllm]
+    env_file: .env
+    environment:
+      SHIM_VLLM: http://127.0.0.1:18199
+      SHIM_PORT: "18191"
+      SHIM_NOUL_FLOOR: "0.801"
+      FRONT_SHIM: http://127.0.0.1:18191
+      FRONT_VLLM: http://127.0.0.1:18199
+      FRONT_BIND: 0.0.0.0
+      FRONT_PORT: "8034"
+      FRONT_CAP: "32000"
+      FRONT_SHIM_FILE: /model/h2o_lightning_shim.py
+      FRONT_SERVE_CONFIG: /model/serve_config.json
+      FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a
+    volumes:
+      - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
+      - /opt/docker/conf/h2o-eval/front:/front:ro
+    command: ["bash", "/front/run.sh"]
+volumes:
+  h2o-eval-vllm-cache: