diff --git a/stacks/intern-decision/README.md b/stacks/intern-decision/README.md index 6bbe460..07f784b 100644 --- a/stacks/intern-decision/README.md +++ b/stacks/intern-decision/README.md @@ -1,21 +1,22 @@ # intern-decision — the decision seat (fv-ml1 GPU 1, :8033) -> **LIVE: Intern-Decision-4B again, since 2026-10-12 1403 PT** (`compose.yaml`, image `intern-decision-serve:0.1.3`). -> Everything from **"The live seat: Intern-Decision-4B"** down describes it. +> **LIVE: h2o-lightning-4b since 2026-10-12 1704 PT** (Prime 1640, via albok-dev). `compose.yaml` = `compose.h2o.yaml` +> (`switch-to-h2o.sh` copies one over the other). Nemi's judge runs the **re-fitted v2 note question** +> (albok-nemi 0.1.18, contract 4.4 rev 1.15) at concurrency 4. > -> **h2o-lightning-4b is STAGED, not live** (`compose.h2o.yaml`, `switch-to-h2o.sh`). It served 1311–1403 on 2026-10-12 -> and was rolled back on albok-dev's word, with brokkr concurring. Prime's plan had made the swap conditional on the -> judge's sign-off, and albok-dev withdrew it on fleet-scale evidence: -> - **adult:** over 10,841 texts judged by both models (same text, prompt and policy), h2o flagged 242 at 0.5 against -> INC's 88. That is all 88 of INC's plus 154 more, most of them at 0.80–0.90 where INC put them at 0.30–0.47. No -> threshold reproduces INC's set (0.70 still flags 166 and loses 6 of INC's). -> - **personal:** INC flagged 5 and h2o 11, with only 2 in common. +> **What happened on 2026-10-12:** +> 1. **1311–1403, h2o's first stint, rolled back.** Under the judge's v1 question, h2o flagged 242 texts adult +> against INC's 88, over 10,841 identical texts. +> 2. **1428–1619, the investigation on GPU 3.** The v1 question asked two things at once: INC answered it +> inconsistently and h2o answered it as written. The shim's floor was ruled out, with identical verdicts on and +> off. Prime ruled "text itself". +> 3. **Under v2 the two seats agree:** work notes INC 0/22, h2o 1/22 at 0.52; nulls 0/18 on both. fleet/memory adult +> went 86 → 0 on INC. +> 4. **Rollback:** `rollback-to-inc.sh`, and set the judge's concurrency back to 1 (see "Switching"). > -> h2o comes back only after albok re-fits the judge for it (prompt and threshold against a labelled set large -> enough to resolve a 2× change, as a 4.4 contract revision, on an eval seat). The h2o sections below describe the -> staged seat, and its numbers are the 1311 cutover's. +> The section headed "The live seat: Intern-Decision-4B" further down describes INC, the rollback target. -### The staged h2o seat (`compose.h2o.yaml`) +### The h2o seat (`compose.h2o.yaml`, live since 1704) | | | |---|---| @@ -97,6 +98,10 @@ Repeat that check on this seat's loopback port (`:18299`) on the next switch. IN ### Switching between the two +The 1704 switch: `switch-to-h2o.sh` ran from 170223 (rc 0) and h2o was healthy at 170454. The 1705 checks: `/health` at the +pin, with floor 0.801; `/decide/shared` 200; a 422 over the cap; inline red → red at 0.984; an https image URL refused +by the shim; the vLLM backstop refusing example.com (400); GPU 1 resting at 14,430 MiB. + Both run ON fv-ml1 and share one `.env`. - **To h2o:** `sudo bash /opt/docker/compose/intern-decision/switch-to-h2o.sh`. It stops INC and checks that GPU 1 has 16,900 MiB free (h2o's measured budget). If not, it restarts INC and exits 2. Otherwise it copies @@ -106,7 +111,7 @@ Both run ON fv-ml1 and share one `.env`. over `compose.yaml` and waits up to 180 s. Image `intern-decision-serve:0.1.3` and its `intern-decision_triton-cache` volume survive the `down`. Used 2026-10-12, 140328 → healthy 140356. -After either switch, make `stacks/intern-decision/compose.yaml` match the host again (it is INC today). +After either switch, make `stacks/intern-decision/compose.yaml` match the host again (it is h2o since 1704). ⚠ **Tell albok-dev on either switch: Nemi's judge concurrency depends on the seat.** INC scores on one inference thread (INV-2, ~16 calls/s at most), so the judge runs with concurrency 1 against it (albok's `nemi.yaml`, since @@ -115,7 +120,7 @@ h2o, vLLM batches, so it goes back to 4. --- -# The live seat: Intern-Decision-4B (2026-09-30 → 2026-10-12 1308, and again since 1403) +# The live seat: Intern-Decision-4B (2026-09-30 → 2026-10-12 1308, and 1403–1702; now the rollback target) > **LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT, and again since 2026-10-12 1403 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168. > Acceptance on the live service is below. `semif` is stopped and kept as the rollback. diff --git a/stacks/intern-decision/compose.yaml b/stacks/intern-decision/compose.yaml index f4c9fb9..6a540af 100644 --- a/stacks/intern-decision/compose.yaml +++ b/stacks/intern-decision/compose.yaml @@ -1,46 +1,33 @@ -# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on -# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to -# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls). -# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now"). -# -# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the -# service keeps semif-serve's HTTP surface (/decide, /decide/shared, /health). Service code + -# contract: services/intern-decision-serve/ (intern-decision-serve.contract.md). Image built on -# fv-ml1 from that dir. -# -# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the -# container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside -# the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more -# gets 503 out_of_memory and the service stays up. See the README before changing it. -# -# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN -# (vault intern-decision/api-token, >= 32 chars). - +# STAGED, NOT LIVE (since the 2026-10-12 1403 rollback): the h2o-lightning-4b decision seat, which switch-to-h2o.sh +# copies over compose.yaml. It served 1311-1403 on 2026-10-12 (Prime via brokkr) and was rolled back when Nemi's judge +# flagged 242 texts adult against INC's 88 at fleet scale. Address intern-decision.fv.internal:8033 (brokkr rule 8). +# h2o-lightning-4b @ b43f759b. Stock vLLM 0.30.0 + the vendor's UNMODIFIED shim + services/h2o-decision-front (copied to +# conf/front): 32,000-token REFUSE-not-trim admission, the seat's bearer token (the same INTERN_DECISION_API_TOKEN), +# /decide + /decide/shared translated onto /v1/systemone, /health model {name, revision}. +# Rollback to Intern-Decision-4B: compose.inc.yaml (the previous compose, same .env): see the stack README. +# GPU 1 is shared (meromero-rp, gen-small): KV is pinned in BYTES (4.48 GiB = 135,070 tokens, the benched size); +# util only gates the boot check. Gate on scripts/oneoff/fv-ml1-gpucheck.sh before any (re)create. +# Images: INLINE ONLY (Prime 2026-10-12). LIGHTNING_IMAGE_URLS=off makes the shim refuse every image URL; vLLM's +# --allowed-media-domains media.invalid (a reserved name that never resolves) is the backstop if the shim ever changes. name: intern-decision - services: - intern-decision: - image: ${IMAGE:?set IMAGE} - container_name: intern-decision + vllm: + image: vllm/vllm-openai:v0.30.0 + container_name: intern-decision-vllm restart: unless-stopped ports: - - "${PORT:-8033}:8000" + - "127.0.0.1:18299:8000" environment: - INTERN_DECISION_API_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN} - INTERN_DECISION_DEVICE: cuda - INTERN_DECISION_VRAM_CAP_GIB: ${VRAM_CAP_GIB:?set VRAM_CAP_GIB} - # Coupled to VRAM_CAP_GIB: the largest call measured to fit under the cap (README "VRAM"). - INTERN_DECISION_MAX_TOKENS: ${MAX_TOKENS:?set MAX_TOKENS} - INTERN_DECISION_MAX_DECISIONS: ${MAX_DECISIONS:-64} - # POSTs in progress (queued + scoring) before new ones get 429 busy. - INTERN_DECISION_MAX_QUEUE: ${MAX_QUEUE:-32} + VLLM_USE_FLASHINFER_SAMPLER: "0" volumes: - # Pinned weights AND the checkpoint's inference.py, read offline. Never downloads. - - /tank/aimodels/huggingface:/hf:ro - # Triton/fla autotune cache (README "Cold-shape latency"): the ~6.5 s first-call-per-bucket - # autotune results survive RECREATES (deploys, image upgrades, .env edits), not just restarts. - # The mount point exists in the image owned by 10001, so the fresh volume is intern-writable. - - triton-cache:/tmp/triton-cache + - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro + - vllm-cache:/root/.cache/vllm + command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b", + "--max-model-len", "40960", "--max-num-seqs", "16", + "--gpu-memory-utilization", "0.1506", "--kv-cache-memory", "4810363371", + "--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}", + "--mm-processor-kwargs", "{\"max_pixels\": 1605632}", + "--allowed-media-domains", "media.invalid"] deploy: resources: reservations: @@ -48,27 +35,41 @@ services: - driver: nvidia device_ids: ["${GPU_ID:-1}"] capabilities: [gpu] + front: + image: python:3.12-slim + container_name: intern-decision + restart: unless-stopped + network_mode: host + depends_on: [vllm] + environment: + FRONT_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN} + SHIM_VLLM: http://127.0.0.1:18299 + SHIM_PORT: "18291" + SHIM_NOUL_FLOOR: "0.801" + LIGHTNING_IMAGE_URLS: "off" # Prime 2026-10-12: inline images only, the seat never fetches an image URL + FRONT_SHIM: http://127.0.0.1:18291 + FRONT_VLLM: http://127.0.0.1:18299 + FRONT_BIND: 0.0.0.0 + FRONT_PORT: ${PORT:-8033} + FRONT_CAP: "32000" + FRONT_SHIM_FILE: /model/h2o_lightning_shim.py + FRONT_SERVE_CONFIG: /model/serve_config.json + FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a + volumes: + - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro + - /opt/docker/conf/intern-decision/front:/front:ro + command: ["bash", "/front/run.sh"] healthcheck: - test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"] + test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8033/health', timeout=5)"] interval: 30s timeout: 10s retries: 3 - # Startup loads ~9 GB of weights and scores the warm-up three times before it serves. start_period: 300s - networks: - - tnet labels: - homepage.group=AI - Eval & Retrieval - - homepage.name=Intern-Decision — typed decisions + - homepage.name=Decision seat (h2o-lightning-4b) - homepage.icon=mdi-scale-balance - - homepage.description=Typed decisions from one forward pass (Intern-Decision-4B, fv-ml1 GPU1) + - homepage.description=Typed decisions, 32k refusing (h2o-lightning-4b on vLLM, fv-ml1 GPU1; /decide/shared kept) - homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health - -networks: - tnet: - name: traefik-net - external: true - volumes: - # -> intern-decision_triton-cache on the host - triton-cache: + vllm-cache: