feat(intern-decision): decision seat swapped to h2o-lightning-4b on fv-ml1 GPU 1; eval seat down

The :8033 seat now serves h2oai/h2o-lightning-4b @ b43f759b on vLLM 0.30.0 with byte-pinned KV (4.48 GiB, 135,558 tokens) behind services/h2o-decision-front: 32k refuse-not-trim admission, the seat's bearer token, and /decide + /decide/shared translated onto /v1/systemone for albok-nemi's judge. URL, port, token and stack name are unchanged. INC down 13:08:18, h2o healthy 13:11:27. Post-checks on the live seat: /health at the pinned revision; JevBench public 205/231 (= bench); 422 over the cap; 16q at the cap warm 2.0-2.16 s; seat peak 16,164 MiB under 4 concurrent 16q x 32k calls against a 16,847 budget. compose.inc.yaml + rollback-to-inc.sh restore Intern-Decision-4B in one step. The temporary GPU 3 eval seat (stacks/h2o-eval) is down.
2026-10-12 13:19:46 -07:00
parent 152137e41d
commit c08992f95f
8 changed files with 468 additions and 60 deletions
@@ -0,0 +1,74 @@
+# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on
+# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to
+# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls).
+# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now").
+#
+# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the
+# service keeps semif-serve's HTTP surface (/decide, /decide/shared, /health). Service code +
+# contract: services/intern-decision-serve/ (intern-decision-serve.contract.md). Image built on
+# fv-ml1 from that dir.
+#
+# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the
+#   container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside
+#   the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more
+#   gets 503 out_of_memory and the service stays up. See the README before changing it.
+#
+# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN
+#   (vault intern-decision/api-token, >= 32 chars).
+
+name: intern-decision
+
+services:
+  intern-decision:
+    image: ${IMAGE:?set IMAGE}
+    container_name: intern-decision
+    restart: unless-stopped
+    ports:
+      - "${PORT:-8033}:8000"
+    environment:
+      INTERN_DECISION_API_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
+      INTERN_DECISION_DEVICE: cuda
+      INTERN_DECISION_VRAM_CAP_GIB: ${VRAM_CAP_GIB:?set VRAM_CAP_GIB}
+      # Coupled to VRAM_CAP_GIB: the largest call measured to fit under the cap (README "VRAM").
+      INTERN_DECISION_MAX_TOKENS: ${MAX_TOKENS:?set MAX_TOKENS}
+      INTERN_DECISION_MAX_DECISIONS: ${MAX_DECISIONS:-64}
+      # POSTs in progress (queued + scoring) before new ones get 429 busy.
+      INTERN_DECISION_MAX_QUEUE: ${MAX_QUEUE:-32}
+    volumes:
+      # Pinned weights AND the checkpoint's inference.py, read offline. Never downloads.
+      - /tank/aimodels/huggingface:/hf:ro
+      # Triton/fla autotune cache (README "Cold-shape latency"): the ~6.5 s first-call-per-bucket
+      # autotune results survive RECREATES (deploys, image upgrades, .env edits), not just restarts.
+      # The mount point exists in the image owned by 10001, so the fresh volume is intern-writable.
+      - triton-cache:/tmp/triton-cache
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids: ["${GPU_ID:-1}"]
+              capabilities: [gpu]
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      # Startup loads ~9 GB of weights and scores the warm-up three times before it serves.
+      start_period: 300s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=AI - Eval & Retrieval
+      - homepage.name=Intern-Decision — typed decisions
+      - homepage.icon=mdi-scale-balance
+      - homepage.description=Typed decisions from one forward pass (Intern-Decision-4B, fv-ml1 GPU1)
+      - homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true
+
+volumes:
+  # -> intern-decision_triton-cache on the host
+  triton-cache: