diff --git a/persistent-memory.md b/persistent-memory.md index aaa63bb..302b6ad 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-10-12 ~1320 PT (decision seat swapped to h2o-lightning-4b on fv-ml1 GPU 1; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_ +_Last updated: 2026-10-12 ~1410 PT (decision seat: h2o 1311–1403, rolled back to INC, h2o staged; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -140,7 +140,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca | 0 | parakeet-nemo | audio | rest 2,108 MiB, cap 3,840 | — | — | | 1 | gen-small (+summarizer/classifier) | 262,144 | 5.5 GiB | 461,215 | 1.76× (default output cap 32,768) | | 1 | meromero-rp (char-rp) | 262,144 | 22 GiB | 346,436 | 1.32× | -| 1 | intern-decision = **h2o-lightning-4b** since 2026-10-12 | 40,960 (front admits 32,000/call) | 4.48 GiB (4,810,363,371 B) | 135,558 | 3.31× (seat peak 16,164 under 4×16q×32k, budget 16,847) | +| 1 | intern-decision = **INC** (h2o 1311–1403 on 10-12, rolled back; staged) | 32,768/call | cap 14.4 GiB, peak 15,220 (rest 8,812) | — | — | | 2 | flash-next (gen, gen-large, +6) | 262,144 | — | 344,155 | 1.31× | Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-memory`), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (`scripts/oneoff/fv-ml1-gpucheck.sh`), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory `feedback_idle_check_must_gate_the_restart`). @@ -166,11 +166,12 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- - **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file. - **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements"). -### The decision seat `intern-decision` :8033 (h2o-lightning-4b since 2026-10-12; INC 09-30 → 10-12) +### The decision seat `intern-decision` :8033 (Intern-Decision-4B; h2o staged) + +- **LIVE: Intern-Decision-4B** (`intern-decision-serve:0.1.3`), again since 2026-10-12 1403. The only production caller is albok-nemi's sensitivity judge (`/decide/shared`, threshold 0.5). Run `scripts/intern-decision-warmup` after an IMAGE change. → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md` +- **h2o-lightning-4b is STAGED** (`stacks/intern-decision/compose.h2o.yaml` + `switch-to-h2o.sh`, with a GPU 1 free-memory gate; weights `/tank/aimodels/h2o-lightning-4b-b43f759b`; front `services/h2o-decision-front`). Inline images only (`LIGHTNING_IMAGE_URLS=off`, verified; vLLM `--allowed-media-domains media.invalid` backstop NOT yet exercised). **It returns only after albok-dev re-fits the judge for it** (a 4.4 contract revision on an eval seat). ⚠ Its seat headroom under 4-way 32k load was 683 MiB. +- **Open:** label ~50 real Wyrd/Cicada turns before trusting INC in production; its card makes no contamination claim. -- **LIVE: h2o-lightning-4b @ b43f759b** on vLLM 0.30.0 (KV byte-pinned) behind `services/h2o-decision-front` (32k refuse-not-trim admission, bearer, `/decide` + `/decide/shared` translated onto `/v1/systemone`). Same URL, port, token and stack name. The only production caller is albok-nemi's sensitivity judge (`/decide/shared`, threshold 0.5). Rollback to INC: `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh` on fv-ml1. Everything else is in `stacks/intern-decision/README.md`. -- ⚠ **Seat headroom under 4-way 32k load is 683 MiB** (peak 16,164 against a 16,847 budget, infra-hermes 10-12). Re-measure before anything else lands on GPU 1. -- INC history (semif replacement, Triton warm-up cache): → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`. Its open item (label ~50 real Wyrd/Cicada turns) is moot with INC retired. ### nh3-pve + nh3-ml1 (2026-09-25/26) @@ -179,7 +180,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- ## Recent decisions -- `[2026-10-12]` **Decision seat swapped to h2o-lightning-4b @ b43f759b (Prime via brokkr; albok-dev signed off Nemi's judge at 0.5 on the GPU 3 eval seat). INC down 130818, h2o healthy on GPU 1 :8033 at 131127; eval seat `stacks/h2o-eval` down 1319.** Post-checks: /health at the pinned revision; JevBench public 205/231 (= bench); 422 over the cap; 16q@32k warm 2.0–2.16 s; 4×16q×32k seat peak 16,164 MiB against a 16,847 budget (infra-hermes). The swap was held at 1234 because the seat's only traffic is albok-nemi's `/decide/shared` judge (57,108 calls 10-05..10-10) and h2o's shim has no /decide, hence the front's translator. **Open:** albok-dev's hand re-judge walk report (started 1316, thread `01M4RVMXJQF1KXXQB6KJ34AYYX`); rollback = `rollback-to-inc.sh`. Brokkr thread `01M4RSFC9J2R0J07EEBD4N04W3`. +- `[2026-10-12]` **Decision seat: h2o-lightning-4b served 1311–1403, then ROLLED BACK to INC** on albok-dev's word, with brokkr concurring (Prime's plan made the swap conditional on the judge's sign-off). Every eval-set check had passed (JevBench 205/231, 422 refusal, 16q@32k ~2.1 s, seat peak 16,164/16,847), but albok's full walk over 10,841 same texts (both verdicts, same prompt and policy) found h2o flags 242 adult at 0.5 against INC's 88, with 154 extra at 0.80–0.90 that no threshold separates; for personal, 5 vs 11 with only 2 in common. **Lesson: a passing benchmark plus a small borderline set (9 entries, 2 → 5) could not resolve a 2× shift in a production classifier; only the same-text fleet comparison did.** albok re-stamps ~260 objects back. Prime's inline-images-only order (via brokkr) is carried in the staged h2o compose. h2o returns only after a judge re-fit. Threads `01M4RSFC9J2R0J07EEBD4N04W3` (brokkr), `01M4RYSMP3DV9HZHJ0KNBH2S96` (albok). - `[2026-10-12]` **U13 regression fixed 0943 (svos-dev's URGENT, Prime saw every app pane 403):** the browser frames a pane at the registry `base_url`, and U13 had set app-terminal's to the seat-host origin, which admits only nh3-svos. Fix: `base_url` back to `https://svos-terminal.phasefinal.com` (config_gen `acae45206bbc9fcb`), plus POST `/seats/{observe,check,launch,retire-launch}` allowed from the box on that origin. svos-dev's one-line fix alone would have 404'd the kernel's seat-host calls; the origin never carried them before. svos-dev owes a separate browser-facing origin field; then the seat-host goes back to being kernel-facing. - `[2026-10-12]` **Shared CI runner (ana-docker act_runner, capacity 2, `timeout: 30m`) is slow because pfi-pve's `ospool` is HDD mirrors with no SLOG:** 8 KiB `oflag=dsync` writes measured ~145 ms on ana-docker vs ~2.7 ms on nh3-docker (n=3 each; pfi-pve host on ospool 18–32 ms; corviduo-dev ~75 ms). Jobs past 30 min are killed with a truncated log and no footer (albok every run since 10-04; vastblue 10935). Worse since the 10-04 PVE 9 reboot, cause not pinned (drive write cache was the same before and after: sda WCE on, sdb–sdd off). vastblue fixed its side with PostgreSQL on tmpfs (run 11033 green, pytest 14m17s vs 7–8 min per 72-test batch); albok is adding pytest-timeout + durations in 0.1.19. **Prime's call, pending:** A opt-in fast runner on nh3-docker (recommended), B SLOG for ospool on the BOSS (check endurance), C leave it. ana-docker vCPU is x86-64-v2-AES (no AVX2), a separate lean-against. - `[2026-10-11]` **worldtree-dev is reading mail again (2219), answering four of our threads at once**: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread `01M4BDETDGMNVDTFXN3T3Y63DP`. diff --git a/services/h2o-decision-front/run.sh b/services/h2o-decision-front/run.sh index a902347..cb2c72d 100755 --- a/services/h2o-decision-front/run.sh +++ b/services/h2o-decision-front/run.sh @@ -1,6 +1,7 @@ #!/bin/bash # One container: the unmodified vendor shim on loopback, then the front. If either dies, the container exits and -# Docker restarts it (restart: unless-stopped). The shim's own SHIM_* env (SHIM_NOUL_FLOOR etc.) passes through. +# Docker restarts it (restart: unless-stopped). Both inherit the container's whole environment, so the shim's own +# settings (SHIM_*, e.g. SHIM_NOUL_FLOOR, and LIGHTNING_*, e.g. LIGHTNING_IMAGE_URLS=off) set in compose reach it. set -eu python3 /model/h2o_lightning_shim.py --config /model/serve_config.json --vllm "$SHIM_VLLM" \ --host 127.0.0.1 --port "$SHIM_PORT" & diff --git a/stacks/h2o-eval/compose.yaml b/stacks/h2o-eval/compose.yaml index 6136a39..d643e59 100644 --- a/stacks/h2o-eval/compose.yaml +++ b/stacks/h2o-eval/compose.yaml @@ -21,7 +21,8 @@ services: command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b", "--max-model-len", "40960", "--max-num-seqs", "16", "--gpu-memory-utilization", "0.1506", "--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}", - "--mm-processor-kwargs", "{\"max_pixels\": 1605632}"] + "--mm-processor-kwargs", "{\"max_pixels\": 1605632}", + "--allowed-media-domains", "media.invalid"] deploy: resources: reservations: @@ -40,6 +41,7 @@ services: SHIM_VLLM: http://127.0.0.1:18199 SHIM_PORT: "18191" SHIM_NOUL_FLOOR: "0.801" + LIGHTNING_IMAGE_URLS: "off" # Prime 2026-10-12: inline images only, the seat never fetches an image URL FRONT_SHIM: http://127.0.0.1:18191 FRONT_VLLM: http://127.0.0.1:18199 FRONT_BIND: 0.0.0.0 diff --git a/stacks/h2o-eval/conf/front/run.sh b/stacks/h2o-eval/conf/front/run.sh index a902347..cb2c72d 100755 --- a/stacks/h2o-eval/conf/front/run.sh +++ b/stacks/h2o-eval/conf/front/run.sh @@ -1,6 +1,7 @@ #!/bin/bash # One container: the unmodified vendor shim on loopback, then the front. If either dies, the container exits and -# Docker restarts it (restart: unless-stopped). The shim's own SHIM_* env (SHIM_NOUL_FLOOR etc.) passes through. +# Docker restarts it (restart: unless-stopped). Both inherit the container's whole environment, so the shim's own +# settings (SHIM_*, e.g. SHIM_NOUL_FLOOR, and LIGHTNING_*, e.g. LIGHTNING_IMAGE_URLS=off) set in compose reach it. set -eu python3 /model/h2o_lightning_shim.py --config /model/serve_config.json --vllm "$SHIM_VLLM" \ --host 127.0.0.1 --port "$SHIM_PORT" & diff --git a/stacks/intern-decision/README.md b/stacks/intern-decision/README.md index a12e3e1..0f1c148 100644 --- a/stacks/intern-decision/README.md +++ b/stacks/intern-decision/README.md @@ -1,11 +1,21 @@ # intern-decision — the decision seat (fv-ml1 GPU 1, :8033) -> **Since 2026-10-12 1311 PT the seat serves h2o-lightning-4b**, not Intern-Decision-4B. Prime's order came via -> brokkr-smithy-dev, gated on albok-dev re-validating Nemi's sensitivity judge (signed off at threshold 0.5 through -> `/decide/shared`). INC went down at 130818 and h2o was healthy at 131127. The address, port, token and stack name -> are unchanged on purpose (brokkr rule 8), so callers change nothing. -> Everything from **"History: the Intern-Decision-4B seat"** down describes the previous seat. It is kept because -> the rollback goes back to it. +> **LIVE: Intern-Decision-4B again, since 2026-10-12 1403 PT** (`compose.yaml`, image `intern-decision-serve:0.1.3`). +> Everything from **"The live seat: Intern-Decision-4B"** down describes it. +> +> **h2o-lightning-4b is STAGED, not live** (`compose.h2o.yaml`, `switch-to-h2o.sh`). It served 1311–1403 on 2026-10-12 +> and was rolled back on albok-dev's word, with brokkr concurring. Prime's plan had made the swap conditional on the +> judge's sign-off, and albok-dev withdrew it on fleet-scale evidence: +> - **adult:** over 10,841 texts judged by both models (same text, prompt and policy), h2o flagged 242 at 0.5 against +> INC's 88. That is all 88 of INC's plus 154 more, most of them at 0.80–0.90 where INC put them at 0.30–0.47. No +> threshold reproduces INC's set (0.70 still flags 166 and loses 6 of INC's). +> - **personal:** INC flagged 5 and h2o 11, with only 2 in common. +> +> h2o comes back only after albok re-fits the judge for it (prompt and threshold against a labelled set large +> enough to resolve a 2× change, as a 4.4 contract revision, on an eval seat). The h2o sections below describe the +> staged seat, and its numbers are the 1311 cutover's. + +### The staged h2o seat (`compose.h2o.yaml`) | | | |---|---| @@ -33,7 +43,7 @@ returns 503 when the shim is not healthy. - `/v1/systemone` and `/api/alpha/decisions` pass through after admission, and `/v1/models` passes through as is. -### Verification on the live seat (2026-10-12, 1311–1313 PT, on fv-ml1 loopback) +### Verification at the 1311 cutover (2026-10-12, 1311–1318 PT, on fv-ml1 loopback) | check | result | expected | |---|---|---| @@ -67,19 +77,42 @@ GPU 1 holds meromero-rp (~44.0 GB) and gen-small (~35.0 GB). Read the free memor `scripts/oneoff/fv-ml1-gpucheck.sh 1 0.1506 intern-decision-vllm`, and stop if it fails. The vLLM container takes about 3 minutes to reach healthy (graph capture). The front waits on it, and the healthcheck has a 300 s start period. -### Rollback to INC +### Images: inline only (Prime 2026-10-12, via brokkr) -On fv-ml1: `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh`. It runs `docker compose down`, keeps -the h2o compose as `compose.yaml.h2o-`, copies `compose.inc.yaml` over `compose.yaml`, and starts INC -(image `intern-decision-serve:0.1.3`, still on the host; its `intern-decision_triton-cache` volume survives the -`down`). It waits up to 180 s for `/health`. INC's peak is 15,220 MiB, below what h2o frees. To return to h2o: put -the saved h2o compose back as `compose.yaml` and `up -d`. +The h2o seat must never fetch an image URL. `compose.h2o.yaml` sets `LIGHTNING_IMAGE_URLS=off` on the front, and the +vendor shim then refuses every image URL with 422 "image URLs are not accepted here". Inline `data:` URIs and base64 +keep working. vLLM also runs with `--allowed-media-domains media.invalid` as a backstop. That is a reserved name that +never resolves, and vLLM 0.30.0 checks every http(s) media URL against it before fetching, while `data:` URLs bypass +the check. + +The shim switch was verified live at 1402, on the h2o seat with the front recreated alone: +- the same `https://media.invalid/red.png` request went from "could not be fetched" (a fetch was attempted) to "not + accepted here" (refused before any fetch); +- inline red still answered red at 0.984; +- before the change, inline red/blue/green answered at 0.98+ and no-image came out near-uniform. + +The vLLM backstop has **not** been exercised: the seat was rolled back before vLLM was recreated. Verify it on the next +switch, by posting an https image URL straight to vLLM's loopback port and expecting a refusal that names the allowed +domains. INC needs neither setting, because its served checkpoint has no vision tower. + +### Switching between the two + +Both run ON fv-ml1 and share one `.env`. +- **To h2o:** `sudo bash /opt/docker/compose/intern-decision/switch-to-h2o.sh`. It stops INC and checks that GPU 1 has + 16,900 MiB free (h2o's measured budget). If not, it restarts INC and exits 2. Otherwise it copies + `compose.h2o.yaml` over `compose.yaml` and waits up to 300 s for `/health`. Only do this once albok has re-fitted + the judge. +- **To INC:** `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh`. It stops h2o, copies `compose.inc.yaml` + over `compose.yaml` and waits up to 180 s. Image `intern-decision-serve:0.1.3` and its + `intern-decision_triton-cache` volume survive the `down`. Used 2026-10-12, 140328 → healthy 140356. + +After either switch, make `stacks/intern-decision/compose.yaml` match the host again (it is INC today). --- -# History: the Intern-Decision-4B seat (2026-09-30 → 2026-10-12) +# The live seat: Intern-Decision-4B (2026-09-30 → 2026-10-12 1308, and again since 1403) -> **Was LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168. +> **LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT, and again since 2026-10-12 1403 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168. > Acceptance on the live service is below. `semif` is stopped and kept as the rollback. **Intern-Decision-4B typed decisions** on **fv-ml1 GPU 1**, the utility card beside `vllm-coder`, diff --git a/stacks/intern-decision/compose.h2o.yaml b/stacks/intern-decision/compose.h2o.yaml new file mode 100644 index 0000000..6a540af --- /dev/null +++ b/stacks/intern-decision/compose.h2o.yaml @@ -0,0 +1,75 @@ +# STAGED, NOT LIVE (since the 2026-10-12 1403 rollback): the h2o-lightning-4b decision seat, which switch-to-h2o.sh +# copies over compose.yaml. It served 1311-1403 on 2026-10-12 (Prime via brokkr) and was rolled back when Nemi's judge +# flagged 242 texts adult against INC's 88 at fleet scale. Address intern-decision.fv.internal:8033 (brokkr rule 8). +# h2o-lightning-4b @ b43f759b. Stock vLLM 0.30.0 + the vendor's UNMODIFIED shim + services/h2o-decision-front (copied to +# conf/front): 32,000-token REFUSE-not-trim admission, the seat's bearer token (the same INTERN_DECISION_API_TOKEN), +# /decide + /decide/shared translated onto /v1/systemone, /health model {name, revision}. +# Rollback to Intern-Decision-4B: compose.inc.yaml (the previous compose, same .env): see the stack README. +# GPU 1 is shared (meromero-rp, gen-small): KV is pinned in BYTES (4.48 GiB = 135,070 tokens, the benched size); +# util only gates the boot check. Gate on scripts/oneoff/fv-ml1-gpucheck.sh before any (re)create. +# Images: INLINE ONLY (Prime 2026-10-12). LIGHTNING_IMAGE_URLS=off makes the shim refuse every image URL; vLLM's +# --allowed-media-domains media.invalid (a reserved name that never resolves) is the backstop if the shim ever changes. +name: intern-decision +services: + vllm: + image: vllm/vllm-openai:v0.30.0 + container_name: intern-decision-vllm + restart: unless-stopped + ports: + - "127.0.0.1:18299:8000" + environment: + VLLM_USE_FLASHINFER_SAMPLER: "0" + volumes: + - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro + - vllm-cache:/root/.cache/vllm + command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b", + "--max-model-len", "40960", "--max-num-seqs", "16", + "--gpu-memory-utilization", "0.1506", "--kv-cache-memory", "4810363371", + "--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}", + "--mm-processor-kwargs", "{\"max_pixels\": 1605632}", + "--allowed-media-domains", "media.invalid"] + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["${GPU_ID:-1}"] + capabilities: [gpu] + front: + image: python:3.12-slim + container_name: intern-decision + restart: unless-stopped + network_mode: host + depends_on: [vllm] + environment: + FRONT_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN} + SHIM_VLLM: http://127.0.0.1:18299 + SHIM_PORT: "18291" + SHIM_NOUL_FLOOR: "0.801" + LIGHTNING_IMAGE_URLS: "off" # Prime 2026-10-12: inline images only, the seat never fetches an image URL + FRONT_SHIM: http://127.0.0.1:18291 + FRONT_VLLM: http://127.0.0.1:18299 + FRONT_BIND: 0.0.0.0 + FRONT_PORT: ${PORT:-8033} + FRONT_CAP: "32000" + FRONT_SHIM_FILE: /model/h2o_lightning_shim.py + FRONT_SERVE_CONFIG: /model/serve_config.json + FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a + volumes: + - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro + - /opt/docker/conf/intern-decision/front:/front:ro + command: ["bash", "/front/run.sh"] + healthcheck: + test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8033/health', timeout=5)"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 300s + labels: + - homepage.group=AI - Eval & Retrieval + - homepage.name=Decision seat (h2o-lightning-4b) + - homepage.icon=mdi-scale-balance + - homepage.description=Typed decisions, 32k refusing (h2o-lightning-4b on vLLM, fv-ml1 GPU1; /decide/shared kept) + - homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health +volumes: + vllm-cache: diff --git a/stacks/intern-decision/compose.yaml b/stacks/intern-decision/compose.yaml index 9f72ee2..f4c9fb9 100644 --- a/stacks/intern-decision/compose.yaml +++ b/stacks/intern-decision/compose.yaml @@ -1,29 +1,46 @@ -# The decision seat on fv-ml1 GPU 1, address intern-decision.fv.internal:8033 (name kept for continuity, brokkr rule 8). -# Since 2026-10-12: h2o-lightning-4b @ b43f759b (Prime via brokkr; albok-dev signed off Nemi's judge at 0.5 through -# /decide/shared). Stock vLLM 0.30.0 + the vendor's UNMODIFIED shim + services/h2o-decision-front (copied to -# conf/front): 32,000-token REFUSE-not-trim admission, the seat's bearer token (the same INTERN_DECISION_API_TOKEN), -# /decide + /decide/shared translated onto /v1/systemone, /health model {name, revision}. -# Rollback to Intern-Decision-4B: compose.inc.yaml (the previous compose, same .env): see the stack README. -# GPU 1 is shared (meromero-rp, gen-small): KV is pinned in BYTES (4.48 GiB = 135,070 tokens, the benched size); -# util only gates the boot check. Gate on scripts/oneoff/fv-ml1-gpucheck.sh before any (re)create. +# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on +# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to +# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls). +# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now"). +# +# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the +# service keeps semif-serve's HTTP surface (/decide, /decide/shared, /health). Service code + +# contract: services/intern-decision-serve/ (intern-decision-serve.contract.md). Image built on +# fv-ml1 from that dir. +# +# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the +# container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside +# the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more +# gets 503 out_of_memory and the service stays up. See the README before changing it. +# +# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN +# (vault intern-decision/api-token, >= 32 chars). + name: intern-decision + services: - vllm: - image: vllm/vllm-openai:v0.30.0 - container_name: intern-decision-vllm + intern-decision: + image: ${IMAGE:?set IMAGE} + container_name: intern-decision restart: unless-stopped ports: - - "127.0.0.1:18299:8000" + - "${PORT:-8033}:8000" environment: - VLLM_USE_FLASHINFER_SAMPLER: "0" + INTERN_DECISION_API_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN} + INTERN_DECISION_DEVICE: cuda + INTERN_DECISION_VRAM_CAP_GIB: ${VRAM_CAP_GIB:?set VRAM_CAP_GIB} + # Coupled to VRAM_CAP_GIB: the largest call measured to fit under the cap (README "VRAM"). + INTERN_DECISION_MAX_TOKENS: ${MAX_TOKENS:?set MAX_TOKENS} + INTERN_DECISION_MAX_DECISIONS: ${MAX_DECISIONS:-64} + # POSTs in progress (queued + scoring) before new ones get 429 busy. + INTERN_DECISION_MAX_QUEUE: ${MAX_QUEUE:-32} volumes: - - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro - - vllm-cache:/root/.cache/vllm - command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b", - "--max-model-len", "40960", "--max-num-seqs", "16", - "--gpu-memory-utilization", "0.1506", "--kv-cache-memory", "4810363371", - "--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}", - "--mm-processor-kwargs", "{\"max_pixels\": 1605632}"] + # Pinned weights AND the checkpoint's inference.py, read offline. Never downloads. + - /tank/aimodels/huggingface:/hf:ro + # Triton/fla autotune cache (README "Cold-shape latency"): the ~6.5 s first-call-per-bucket + # autotune results survive RECREATES (deploys, image upgrades, .env edits), not just restarts. + # The mount point exists in the image owned by 10001, so the fresh volume is intern-writable. + - triton-cache:/tmp/triton-cache deploy: resources: reservations: @@ -31,40 +48,27 @@ services: - driver: nvidia device_ids: ["${GPU_ID:-1}"] capabilities: [gpu] - front: - image: python:3.12-slim - container_name: intern-decision - restart: unless-stopped - network_mode: host - depends_on: [vllm] - environment: - FRONT_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN} - SHIM_VLLM: http://127.0.0.1:18299 - SHIM_PORT: "18291" - SHIM_NOUL_FLOOR: "0.801" - FRONT_SHIM: http://127.0.0.1:18291 - FRONT_VLLM: http://127.0.0.1:18299 - FRONT_BIND: 0.0.0.0 - FRONT_PORT: ${PORT:-8033} - FRONT_CAP: "32000" - FRONT_SHIM_FILE: /model/h2o_lightning_shim.py - FRONT_SERVE_CONFIG: /model/serve_config.json - FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a - volumes: - - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro - - /opt/docker/conf/intern-decision/front:/front:ro - command: ["bash", "/front/run.sh"] healthcheck: - test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8033/health', timeout=5)"] + test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"] interval: 30s timeout: 10s retries: 3 + # Startup loads ~9 GB of weights and scores the warm-up three times before it serves. start_period: 300s + networks: + - tnet labels: - homepage.group=AI - Eval & Retrieval - - homepage.name=Decision seat (h2o-lightning-4b) + - homepage.name=Intern-Decision — typed decisions - homepage.icon=mdi-scale-balance - - homepage.description=Typed decisions, 32k refusing (h2o-lightning-4b on vLLM, fv-ml1 GPU1; /decide/shared kept) + - homepage.description=Typed decisions from one forward pass (Intern-Decision-4B, fv-ml1 GPU1) - homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health + +networks: + tnet: + name: traefik-net + external: true + volumes: - vllm-cache: + # -> intern-decision_triton-cache on the host + triton-cache: diff --git a/stacks/intern-decision/conf/front/run.sh b/stacks/intern-decision/conf/front/run.sh index a902347..cb2c72d 100755 --- a/stacks/intern-decision/conf/front/run.sh +++ b/stacks/intern-decision/conf/front/run.sh @@ -1,6 +1,7 @@ #!/bin/bash # One container: the unmodified vendor shim on loopback, then the front. If either dies, the container exits and -# Docker restarts it (restart: unless-stopped). The shim's own SHIM_* env (SHIM_NOUL_FLOOR etc.) passes through. +# Docker restarts it (restart: unless-stopped). Both inherit the container's whole environment, so the shim's own +# settings (SHIM_*, e.g. SHIM_NOUL_FLOOR, and LIGHTNING_*, e.g. LIGHTNING_IMAGE_URLS=off) set in compose reach it. set -eu python3 /model/h2o_lightning_shim.py --config /model/serve_config.json --vllm "$SHIM_VLLM" \ --host 127.0.0.1 --port "$SHIM_PORT" & diff --git a/stacks/intern-decision/rollback-to-inc.sh b/stacks/intern-decision/rollback-to-inc.sh index 093f52f..5d42288 100755 --- a/stacks/intern-decision/rollback-to-inc.sh +++ b/stacks/intern-decision/rollback-to-inc.sh @@ -2,10 +2,10 @@ # One-command rollback of the decision seat to Intern-Decision-4B (the pre-2026-10-12 seat), run ON fv-ml1: # sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh # It stops the h2o stack, puts the INC compose back (same .env) and starts it (~35 s to healthy). +# Used 2026-10-12 1403. The h2o compose is kept as compose.h2o.yaml (canonical in eshpfi); switch-to-h2o.sh goes back. set -euo pipefail cd /opt/docker/compose/intern-decision docker compose down -cp compose.yaml compose.yaml.h2o-$(date +%Y%m%d-%H%M) cp compose.inc.yaml compose.yaml docker compose up -d for i in $(seq 1 60); do curl -sf http://127.0.0.1:8033/health >/dev/null && { echo "INC healthy"; exit 0; }; sleep 3; done diff --git a/stacks/intern-decision/switch-to-h2o.sh b/stacks/intern-decision/switch-to-h2o.sh new file mode 100755 index 0000000..a484148 --- /dev/null +++ b/stacks/intern-decision/switch-to-h2o.sh @@ -0,0 +1,18 @@ +#!/bin/bash +# Switch the decision seat to h2o-lightning-4b (the staged compose.h2o.yaml), run ON fv-ml1: +# sudo bash /opt/docker/compose/intern-decision/switch-to-h2o.sh +# Only once albok-dev has re-fitted Nemi's judge for h2o (2026-10-12 rollback; see the stack README). +# It stops INC, checks GPU 1 has room for h2o's measured peak, and either starts h2o or puts INC back. +# h2o takes ~3 min to healthy (graph capture). Undo with rollback-to-inc.sh. +set -euo pipefail +cd /opt/docker/compose/intern-decision +NEED_MIB=16900 # h2o seat budget 16,847 MiB (peak 16,164 under 4 x 16q x 32k on 2026-10-12) +docker compose down +FREE=$(nvidia-smi -i 1 --query-gpu=memory.free --format=csv,noheader,nounits | tr -d ' ') +if [ "$FREE" -lt "$NEED_MIB" ]; then + echo "GPU 1 free ${FREE} MiB < ${NEED_MIB}: not starting h2o; restarting INC"; docker compose up -d; exit 2 +fi +cp compose.h2o.yaml compose.yaml +docker compose up -d +for i in $(seq 1 100); do curl -sf http://127.0.0.1:8033/health >/dev/null && { echo "h2o healthy"; exit 0; }; sleep 3; done +echo "h2o not healthy after 300 s: run rollback-to-inc.sh"; exit 1