fix(intern-decision): roll the decision seat back to Intern-Decision-4B; h2o staged with inline-images-only

albok-dev withdrew the judge sign-off on fleet-scale evidence and brokkr concurred: over 10,841 texts judged by both models, h2o flagged 242 adult at 0.5 against INC's 88, and no threshold reproduces INC's set. rollback-to-inc.sh ran 14:03:28, INC healthy 14:03:56, /decide/shared smoke 200 x3. compose.yaml is INC again; the h2o compose is kept as compose.h2o.yaml with switch-to-h2o.sh (GPU 1 free-memory gate, falls back to INC). It now carries Prime's inline-images-only setting: LIGHTNING_IMAGE_URLS=off on the front (verified live: an https image URL is refused before any fetch, inline images still answer) and vLLM --allowed-media-domains media.invalid as a backstop. rollback-to-inc.sh no longer leaves timestamped compose copies behind.
2026-10-12 14:05:44 -07:00
parent c08992f95f
commit f979cca1ea
10 changed files with 210 additions and 74 deletions
@@ -21,7 +21,8 @@ services:
    command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b",
              "--max-model-len", "40960", "--max-num-seqs", "16", "--gpu-memory-utilization", "0.1506",
              "--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}",
-              "--mm-processor-kwargs", "{\"max_pixels\": 1605632}"]
+              "--mm-processor-kwargs", "{\"max_pixels\": 1605632}",
+              "--allowed-media-domains", "media.invalid"]
    deploy:
      resources:
        reservations:
@@ -40,6 +41,7 @@ services:
      SHIM_VLLM: http://127.0.0.1:18199
      SHIM_PORT: "18191"
      SHIM_NOUL_FLOOR: "0.801"
+      LIGHTNING_IMAGE_URLS: "off"   # Prime 2026-10-12: inline images only, the seat never fetches an image URL
      FRONT_SHIM: http://127.0.0.1:18191
      FRONT_VLLM: http://127.0.0.1:18199
      FRONT_BIND: 0.0.0.0
@@ -1,6 +1,7 @@
 #!/bin/bash
 # One container: the unmodified vendor shim on loopback, then the front. If either dies, the container exits and
-# Docker restarts it (restart: unless-stopped). The shim's own SHIM_* env (SHIM_NOUL_FLOOR etc.) passes through.
+# Docker restarts it (restart: unless-stopped). Both inherit the container's whole environment, so the shim's own
+# settings (SHIM_*, e.g. SHIM_NOUL_FLOOR, and LIGHTNING_*, e.g. LIGHTNING_IMAGE_URLS=off) set in compose reach it.
 set -eu
 python3 /model/h2o_lightning_shim.py --config /model/serve_config.json --vllm "$SHIM_VLLM" \
  --host 127.0.0.1 --port "$SHIM_PORT" &
@@ -1,11 +1,21 @@
 # intern-decision — the decision seat (fv-ml1 GPU 1, :8033)

-> **Since 2026-10-12 1311 PT the seat serves h2o-lightning-4b**, not Intern-Decision-4B. Prime's order came via
-> brokkr-smithy-dev, gated on albok-dev re-validating Nemi's sensitivity judge (signed off at threshold 0.5 through
-> `/decide/shared`). INC went down at 130818 and h2o was healthy at 131127. The address, port, token and stack name
-> are unchanged on purpose (brokkr rule 8), so callers change nothing.
-> Everything from **"History: the Intern-Decision-4B seat"** down describes the previous seat. It is kept because
-> the rollback goes back to it.
+> **LIVE: Intern-Decision-4B again, since 2026-10-12 1403 PT** (`compose.yaml`, image `intern-decision-serve:0.1.3`).
+> Everything from **"The live seat: Intern-Decision-4B"** down describes it.
+>
+> **h2o-lightning-4b is STAGED, not live** (`compose.h2o.yaml`, `switch-to-h2o.sh`). It served 1311–1403 on 2026-10-12
+> and was rolled back on albok-dev's word, with brokkr concurring. Prime's plan had made the swap conditional on the
+> judge's sign-off, and albok-dev withdrew it on fleet-scale evidence:
+> - **adult:** over 10,841 texts judged by both models (same text, prompt and policy), h2o flagged 242 at 0.5 against
+>   INC's 88. That is all 88 of INC's plus 154 more, most of them at 0.80–0.90 where INC put them at 0.30–0.47. No
+>   threshold reproduces INC's set (0.70 still flags 166 and loses 6 of INC's).
+> - **personal:** INC flagged 5 and h2o 11, with only 2 in common.
+>
+> h2o comes back only after albok re-fits the judge for it (prompt and threshold against a labelled set large
+> enough to resolve a 2× change, as a 4.4 contract revision, on an eval seat). The h2o sections below describe the
+> staged seat, and its numbers are the 1311 cutover's.
+
+### The staged h2o seat (`compose.h2o.yaml`)

 | | |
 |---|---|
@@ -33,7 +43,7 @@
  returns 503 when the shim is not healthy.
 - `/v1/systemone` and `/api/alpha/decisions` pass through after admission, and `/v1/models` passes through as is.

-### Verification on the live seat (2026-10-12, 1311–1313 PT, on fv-ml1 loopback)
+### Verification at the 1311 cutover (2026-10-12, 1311–1318 PT, on fv-ml1 loopback)

 | check | result | expected |
 |---|---|---|
@@ -67,19 +77,42 @@ GPU 1 holds meromero-rp (~44.0 GB) and gen-small (~35.0 GB). Read the free memor
 `scripts/oneoff/fv-ml1-gpucheck.sh 1 0.1506 intern-decision-vllm`, and stop if it fails. The vLLM container takes about
 3 minutes to reach healthy (graph capture). The front waits on it, and the healthcheck has a 300 s start period.

-### Rollback to INC
+### Images: inline only (Prime 2026-10-12, via brokkr)

-On fv-ml1: `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh`. It runs `docker compose down`, keeps
-the h2o compose as `compose.yaml.h2o-<timestamp>`, copies `compose.inc.yaml` over `compose.yaml`, and starts INC
-(image `intern-decision-serve:0.1.3`, still on the host; its `intern-decision_triton-cache` volume survives the
-`down`). It waits up to 180 s for `/health`. INC's peak is 15,220 MiB, below what h2o frees. To return to h2o: put
-the saved h2o compose back as `compose.yaml` and `up -d`.
+The h2o seat must never fetch an image URL. `compose.h2o.yaml` sets `LIGHTNING_IMAGE_URLS=off` on the front, and the
+vendor shim then refuses every image URL with 422 "image URLs are not accepted here". Inline `data:` URIs and base64
+keep working. vLLM also runs with `--allowed-media-domains media.invalid` as a backstop. That is a reserved name that
+never resolves, and vLLM 0.30.0 checks every http(s) media URL against it before fetching, while `data:` URLs bypass
+the check.
+
+The shim switch was verified live at 1402, on the h2o seat with the front recreated alone:
+- the same `https://media.invalid/red.png` request went from "could not be fetched" (a fetch was attempted) to "not
+  accepted here" (refused before any fetch);
+- inline red still answered red at 0.984;
+- before the change, inline red/blue/green answered at 0.98+ and no-image came out near-uniform.
+
+The vLLM backstop has **not** been exercised: the seat was rolled back before vLLM was recreated. Verify it on the next
+switch, by posting an https image URL straight to vLLM's loopback port and expecting a refusal that names the allowed
+domains. INC needs neither setting, because its served checkpoint has no vision tower.
+
+### Switching between the two
+
+Both run ON fv-ml1 and share one `.env`.
+- **To h2o:** `sudo bash /opt/docker/compose/intern-decision/switch-to-h2o.sh`. It stops INC and checks that GPU 1 has
+  16,900 MiB free (h2o's measured budget). If not, it restarts INC and exits 2. Otherwise it copies
+  `compose.h2o.yaml` over `compose.yaml` and waits up to 300 s for `/health`. Only do this once albok has re-fitted
+  the judge.
+- **To INC:** `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh`. It stops h2o, copies `compose.inc.yaml`
+  over `compose.yaml` and waits up to 180 s. Image `intern-decision-serve:0.1.3` and its
+  `intern-decision_triton-cache` volume survive the `down`. Used 2026-10-12, 140328 → healthy 140356.
+
+After either switch, make `stacks/intern-decision/compose.yaml` match the host again (it is INC today).

 ---

-# History: the Intern-Decision-4B seat (2026-09-30 → 2026-10-12)
+# The live seat: Intern-Decision-4B (2026-09-30 → 2026-10-12 1308, and again since 1403)

-> **Was LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168.
+> **LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT, and again since 2026-10-12 1403 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168.
 > Acceptance on the live service is below. `semif` is stopped and kept as the rollback.

 **Intern-Decision-4B typed decisions** on **fv-ml1 GPU 1**, the utility card beside `vllm-coder`,
@@ -0,0 +1,75 @@
+# STAGED, NOT LIVE (since the 2026-10-12 1403 rollback): the h2o-lightning-4b decision seat, which switch-to-h2o.sh
+# copies over compose.yaml. It served 1311-1403 on 2026-10-12 (Prime via brokkr) and was rolled back when Nemi's judge
+# flagged 242 texts adult against INC's 88 at fleet scale. Address intern-decision.fv.internal:8033 (brokkr rule 8).
+# h2o-lightning-4b @ b43f759b. Stock vLLM 0.30.0 + the vendor's UNMODIFIED shim + services/h2o-decision-front (copied to
+# conf/front): 32,000-token REFUSE-not-trim admission, the seat's bearer token (the same INTERN_DECISION_API_TOKEN),
+# /decide + /decide/shared translated onto /v1/systemone, /health model {name, revision}.
+# Rollback to Intern-Decision-4B: compose.inc.yaml (the previous compose, same .env): see the stack README.
+# GPU 1 is shared (meromero-rp, gen-small): KV is pinned in BYTES (4.48 GiB = 135,070 tokens, the benched size);
+# util only gates the boot check. Gate on scripts/oneoff/fv-ml1-gpucheck.sh before any (re)create.
+# Images: INLINE ONLY (Prime 2026-10-12). LIGHTNING_IMAGE_URLS=off makes the shim refuse every image URL; vLLM's
+# --allowed-media-domains media.invalid (a reserved name that never resolves) is the backstop if the shim ever changes.
+name: intern-decision
+services:
+  vllm:
+    image: vllm/vllm-openai:v0.30.0
+    container_name: intern-decision-vllm
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:18299:8000"
+    environment:
+      VLLM_USE_FLASHINFER_SAMPLER: "0"
+    volumes:
+      - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
+      - vllm-cache:/root/.cache/vllm
+    command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b",
+              "--max-model-len", "40960", "--max-num-seqs", "16",
+              "--gpu-memory-utilization", "0.1506", "--kv-cache-memory", "4810363371",
+              "--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}",
+              "--mm-processor-kwargs", "{\"max_pixels\": 1605632}",
+              "--allowed-media-domains", "media.invalid"]
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids: ["${GPU_ID:-1}"]
+              capabilities: [gpu]
+  front:
+    image: python:3.12-slim
+    container_name: intern-decision
+    restart: unless-stopped
+    network_mode: host
+    depends_on: [vllm]
+    environment:
+      FRONT_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
+      SHIM_VLLM: http://127.0.0.1:18299
+      SHIM_PORT: "18291"
+      SHIM_NOUL_FLOOR: "0.801"
+      LIGHTNING_IMAGE_URLS: "off"   # Prime 2026-10-12: inline images only, the seat never fetches an image URL
+      FRONT_SHIM: http://127.0.0.1:18291
+      FRONT_VLLM: http://127.0.0.1:18299
+      FRONT_BIND: 0.0.0.0
+      FRONT_PORT: ${PORT:-8033}
+      FRONT_CAP: "32000"
+      FRONT_SHIM_FILE: /model/h2o_lightning_shim.py
+      FRONT_SERVE_CONFIG: /model/serve_config.json
+      FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a
+    volumes:
+      - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
+      - /opt/docker/conf/intern-decision/front:/front:ro
+    command: ["bash", "/front/run.sh"]
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8033/health', timeout=5)"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 300s
+    labels:
+      - homepage.group=AI - Eval & Retrieval
+      - homepage.name=Decision seat (h2o-lightning-4b)
+      - homepage.icon=mdi-scale-balance
+      - homepage.description=Typed decisions, 32k refusing (h2o-lightning-4b on vLLM, fv-ml1 GPU1; /decide/shared kept)
+      - homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health
+volumes:
+  vllm-cache:
@@ -1,29 +1,46 @@
-# The decision seat on fv-ml1 GPU 1, address intern-decision.fv.internal:8033 (name kept for continuity, brokkr rule 8).
-# Since 2026-10-12: h2o-lightning-4b @ b43f759b (Prime via brokkr; albok-dev signed off Nemi's judge at 0.5 through
-# /decide/shared). Stock vLLM 0.30.0 + the vendor's UNMODIFIED shim + services/h2o-decision-front (copied to
-# conf/front): 32,000-token REFUSE-not-trim admission, the seat's bearer token (the same INTERN_DECISION_API_TOKEN),
-# /decide + /decide/shared translated onto /v1/systemone, /health model {name, revision}.
-# Rollback to Intern-Decision-4B: compose.inc.yaml (the previous compose, same .env): see the stack README.
-# GPU 1 is shared (meromero-rp, gen-small): KV is pinned in BYTES (4.48 GiB = 135,070 tokens, the benched size);
-# util only gates the boot check. Gate on scripts/oneoff/fv-ml1-gpucheck.sh before any (re)create.
+# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on
+# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to
+# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls).
+# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now").
+#
+# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the
+# service keeps semif-serve's HTTP surface (/decide, /decide/shared, /health). Service code +
+# contract: services/intern-decision-serve/ (intern-decision-serve.contract.md). Image built on
+# fv-ml1 from that dir.
+#
+# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the
+#   container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside
+#   the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more
+#   gets 503 out_of_memory and the service stays up. See the README before changing it.
+#
+# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN
+#   (vault intern-decision/api-token, >= 32 chars).
+
 name: intern-decision
+
 services:
-  vllm:
-    image: vllm/vllm-openai:v0.30.0
-    container_name: intern-decision-vllm
+  intern-decision:
+    image: ${IMAGE:?set IMAGE}
+    container_name: intern-decision
    restart: unless-stopped
    ports:
-      - "127.0.0.1:18299:8000"
+      - "${PORT:-8033}:8000"
    environment:
-      VLLM_USE_FLASHINFER_SAMPLER: "0"
+      INTERN_DECISION_API_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
+      INTERN_DECISION_DEVICE: cuda
+      INTERN_DECISION_VRAM_CAP_GIB: ${VRAM_CAP_GIB:?set VRAM_CAP_GIB}
+      # Coupled to VRAM_CAP_GIB: the largest call measured to fit under the cap (README "VRAM").
+      INTERN_DECISION_MAX_TOKENS: ${MAX_TOKENS:?set MAX_TOKENS}
+      INTERN_DECISION_MAX_DECISIONS: ${MAX_DECISIONS:-64}
+      # POSTs in progress (queued + scoring) before new ones get 429 busy.
+      INTERN_DECISION_MAX_QUEUE: ${MAX_QUEUE:-32}
    volumes:
-      - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
-      - vllm-cache:/root/.cache/vllm
-    command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b",
-              "--max-model-len", "40960", "--max-num-seqs", "16",
-              "--gpu-memory-utilization", "0.1506", "--kv-cache-memory", "4810363371",
-              "--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}",
-              "--mm-processor-kwargs", "{\"max_pixels\": 1605632}"]
+      # Pinned weights AND the checkpoint's inference.py, read offline. Never downloads.
+      - /tank/aimodels/huggingface:/hf:ro
+      # Triton/fla autotune cache (README "Cold-shape latency"): the ~6.5 s first-call-per-bucket
+      # autotune results survive RECREATES (deploys, image upgrades, .env edits), not just restarts.
+      # The mount point exists in the image owned by 10001, so the fresh volume is intern-writable.
+      - triton-cache:/tmp/triton-cache
    deploy:
      resources:
        reservations:
@@ -31,40 +48,27 @@ services:
            - driver: nvidia
              device_ids: ["${GPU_ID:-1}"]
              capabilities: [gpu]
-  front:
-    image: python:3.12-slim
-    container_name: intern-decision
-    restart: unless-stopped
-    network_mode: host
-    depends_on: [vllm]
-    environment:
-      FRONT_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
-      SHIM_VLLM: http://127.0.0.1:18299
-      SHIM_PORT: "18291"
-      SHIM_NOUL_FLOOR: "0.801"
-      FRONT_SHIM: http://127.0.0.1:18291
-      FRONT_VLLM: http://127.0.0.1:18299
-      FRONT_BIND: 0.0.0.0
-      FRONT_PORT: ${PORT:-8033}
-      FRONT_CAP: "32000"
-      FRONT_SHIM_FILE: /model/h2o_lightning_shim.py
-      FRONT_SERVE_CONFIG: /model/serve_config.json
-      FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a
-    volumes:
-      - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
-      - /opt/docker/conf/intern-decision/front:/front:ro
-    command: ["bash", "/front/run.sh"]
    healthcheck:
-      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8033/health', timeout=5)"]
+      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"]
      interval: 30s
      timeout: 10s
      retries: 3
+      # Startup loads ~9 GB of weights and scores the warm-up three times before it serves.
      start_period: 300s
+    networks:
+      - tnet
    labels:
      - homepage.group=AI - Eval & Retrieval
-      - homepage.name=Decision seat (h2o-lightning-4b)
+      - homepage.name=Intern-Decision — typed decisions
      - homepage.icon=mdi-scale-balance
-      - homepage.description=Typed decisions, 32k refusing (h2o-lightning-4b on vLLM, fv-ml1 GPU1; /decide/shared kept)
+      - homepage.description=Typed decisions from one forward pass (Intern-Decision-4B, fv-ml1 GPU1)
      - homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true
+
 volumes:
-  vllm-cache:
+  # -> intern-decision_triton-cache on the host
+  triton-cache:
@@ -1,6 +1,7 @@
 #!/bin/bash
 # One container: the unmodified vendor shim on loopback, then the front. If either dies, the container exits and
-# Docker restarts it (restart: unless-stopped). The shim's own SHIM_* env (SHIM_NOUL_FLOOR etc.) passes through.
+# Docker restarts it (restart: unless-stopped). Both inherit the container's whole environment, so the shim's own
+# settings (SHIM_*, e.g. SHIM_NOUL_FLOOR, and LIGHTNING_*, e.g. LIGHTNING_IMAGE_URLS=off) set in compose reach it.
 set -eu
 python3 /model/h2o_lightning_shim.py --config /model/serve_config.json --vllm "$SHIM_VLLM" \
  --host 127.0.0.1 --port "$SHIM_PORT" &
@@ -2,10 +2,10 @@
 # One-command rollback of the decision seat to Intern-Decision-4B (the pre-2026-10-12 seat), run ON fv-ml1:
 #   sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh
 # It stops the h2o stack, puts the INC compose back (same .env) and starts it (~35 s to healthy).
+# Used 2026-10-12 1403. The h2o compose is kept as compose.h2o.yaml (canonical in eshpfi); switch-to-h2o.sh goes back.
 set -euo pipefail
 cd /opt/docker/compose/intern-decision
 docker compose down
-cp compose.yaml compose.yaml.h2o-$(date +%Y%m%d-%H%M)
 cp compose.inc.yaml compose.yaml
 docker compose up -d
 for i in $(seq 1 60); do curl -sf http://127.0.0.1:8033/health >/dev/null && { echo "INC healthy"; exit 0; }; sleep 3; done
@@ -0,0 +1,18 @@
+#!/bin/bash
+# Switch the decision seat to h2o-lightning-4b (the staged compose.h2o.yaml), run ON fv-ml1:
+#   sudo bash /opt/docker/compose/intern-decision/switch-to-h2o.sh
+# Only once albok-dev has re-fitted Nemi's judge for h2o (2026-10-12 rollback; see the stack README).
+# It stops INC, checks GPU 1 has room for h2o's measured peak, and either starts h2o or puts INC back.
+# h2o takes ~3 min to healthy (graph capture). Undo with rollback-to-inc.sh.
+set -euo pipefail
+cd /opt/docker/compose/intern-decision
+NEED_MIB=16900   # h2o seat budget 16,847 MiB (peak 16,164 under 4 x 16q x 32k on 2026-10-12)
+docker compose down
+FREE=$(nvidia-smi -i 1 --query-gpu=memory.free --format=csv,noheader,nounits | tr -d ' ')
+if [ "$FREE" -lt "$NEED_MIB" ]; then
+  echo "GPU 1 free ${FREE} MiB < ${NEED_MIB}: not starting h2o; restarting INC"; docker compose up -d; exit 2
+fi
+cp compose.h2o.yaml compose.yaml
+docker compose up -d
+for i in $(seq 1 100); do curl -sf http://127.0.0.1:8033/health >/dev/null && { echo "h2o healthy"; exit 0; }; sleep 3; done
+echo "h2o not healthy after 300 s: run rollback-to-inc.sh"; exit 1