diff --git a/persistent-memory.md b/persistent-memory.md index 6bfa935..f2eda99 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-10-12 ~1750 PT (decision seat: h2o tried twice, 1311–1403 and 1704–1747, rolled back both times, INC live, h2o staged; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_ +_Last updated: 2026-10-12 ~1810 PT (decision seat: h2o production since 1807 on Prime's fix-forward ruling, after two rollbacks; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -140,7 +140,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca | 0 | parakeet-nemo | audio | rest 2,108 MiB, cap 3,840 | — | — | | 1 | gen-small (+summarizer/classifier) | 262,144 | 5.5 GiB | 461,215 | 1.76× (default output cap 32,768) | | 1 | meromero-rp (char-rp) | 262,144 | 22 GiB | 346,436 | 1.32× | -| 1 | intern-decision = **INC** since 10-12 1748 (h2o staged; live 1311–1403 and 1704–1747) | 32,768/call | cap 14.4 GiB, peak 15,220 (rest 8,812) | — | — | +| 1 | intern-decision = **h2o-lightning-4b** since 10-12 1807 (production; no INC fallback per Prime) | 40,960 (front admits 32,000/call) | 4.48 GiB (4,810,363,371 B) | 135,558 | 3.31× (seat peak 16,164 under 4×16q×32k, budget 16,847) | | 2 | flash-next (gen, gen-large, +6) | 262,144 | — | 344,155 | 1.31× | Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-memory`), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (`scripts/oneoff/fv-ml1-gpucheck.sh`), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory `feedback_idle_check_must_gate_the_restart`). @@ -166,11 +166,12 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- - **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file. - **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements"). -### The decision seat `intern-decision` :8033 (Intern-Decision-4B since 2026-10-12 1748; h2o staged) +### The decision seat `intern-decision` :8033 (h2o-lightning-4b, production since 2026-10-12 1807) -- **LIVE: Intern-Decision-4B** (`intern-decision-serve:0.1.3`). Nemi's judge runs the v2 note question (albok-nemi 0.1.18) at **concurrency 1** (INC scores on one thread, INV-2). Run `scripts/intern-decision-warmup` after an IMAGE change. → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md` -- **h2o-lightning-4b is STAGED** (`compose.h2o.yaml` + `switch-to-h2o.sh` with a GPU 1 free-memory gate; weights `/tank/aimodels/h2o-lightning-4b-b43f759b`; front `services/h2o-decision-front`; inline images only, both layers verified). **Rolled back twice on 10-12, each time on albok-dev's word against a pre-named line.** A third attempt waits on albok-dev's two offline fixes going to Prime: (1) the book-profile margin (the Schell chapter at 0.531 vs INC 0.438, via a per-profile threshold or `visibility: public` on published titles); (2) h2o's reading of the personal question (6 → 18 memory entries). ⚠ Seat headroom under 4-way 32k load was 683 MiB. -- On any switch, albok-dev sets judge concurrency: 1 for INC, 4 for h2o. +- **LIVE: h2o-lightning-4b @ b43f759b** (`compose.yaml` = `compose.h2o.yaml`) behind `services/h2o-decision-front`: 32k refuse-not-trim admission, bearer, `/decide/shared` translator. Inline images only (`LIGHTNING_IMAGE_URLS=off` + vLLM `--allowed-media-domains media.invalid`, both verified). Nemi's judge runs v2 (albok-nemi 0.1.18) at **concurrency 4**. +- ⚠ **Prime 1805 (via albok-dev): "go to h2o, and fix forward. do not fall back to inc."** `rollback-to-inc.sh` + `compose.inc.yaml` remain on the host as artifacts only. Seat problems are fixed forward in the judge (wording, thresholds, `visibility` declarations); INC again needs Prime's explicit word. +- ⚠ Seat headroom under 4-way 32k load is 683 MiB (peak 16,164 against 16,847). Re-measure before anything else lands on GPU 1. +- INC history: → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`. ### nh3-pve + nh3-ml1 (2026-09-25/26) @@ -180,7 +181,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- ## Recent decisions - `[2026-10-12]` **Prime's rulings ~1715:** CI fix A (fast runner on nh3-docker); push eshpfi main (done, `3aa715c..6a01da7`); deny old card `05cf119e` (moot: the kernel DB shows Prime approved it 10-11 1354:25, so there is nothing to deny); **no rotation of `albok/judge-token`** (it was printed only into a local transcript). The ana-docker vCPU-type change is dropped. -- `[2026-10-12]` **Decision seat: h2o-lightning-4b served 1311–1403, then ROLLED BACK to INC** on albok-dev's word, with brokkr concurring (Prime's plan made the swap conditional on the judge's sign-off). Every eval-set check had passed (JevBench 205/231, 422 refusal, 16q@32k ~2.1 s, seat peak 16,164/16,847), but albok's full walk over 10,841 same texts (both verdicts, same prompt and policy) found h2o flags 242 adult at 0.5 against INC's 88, with 154 extra at 0.80–0.90 that no threshold separates; for personal, 5 vs 11 with only 2 in common. **Lesson: a passing benchmark plus a small borderline set (9 entries, 2 → 5) could not resolve a 2× shift in a production classifier; only the same-text fleet comparison did.** albok's re-stamp walk on INC (1406–1408) put the hidden set back exactly to its pre-cutover counts (fleet/memory 86, agent-feedback 21, nonfiction 0). Prime's inline-images-only order (via brokkr) is carried in the staged h2o compose. h2o returns only after a judge re-fit. Threads `01M4RSFC9J2R0J07EEBD4N04W3` (brokkr), `01M4RYSMP3DV9HZHJ0KNBH2S96` (albok). **Then:** the round 2 eval seat on GPU 3 (1428–1619, Prime 1420) found the cause. The judge's v1 note question asked two things at once: INC answered it inconsistently, h2o literally. The shim floor is ruled out (identical verdicts on and off). Prime ruled "text itself", and albok-nemi 0.1.18 (judge v2) re-fitted the question. Both seats agree under v2 (work notes INC 0/22, h2o 1/22 at 0.52). **Prime ruled the cutover at 1640 ("do it", via albok-dev): h2o LIVE again since 1704** (`switch-to-h2o.sh` 170223, healthy 170454, probes clean). **The v2 walk on h2o (1705–1745) moved two margins past albok-dev's named line**: nonfiction adult 0 → 79 (the Schell chapter at 0.531 on the book profile) and memory personal 6 → 18. Notes held at 0 adult. **Rolled back to INC at 1748 on albok-dev's word.** The 1749–1751 re-stamp walk restored the counts (memory 0 adult / 6 personal, feedback 0 / 2, nonfiction 0 / 0), and the timer was back at 1751. h2o stays staged until the two fixes reach Prime; albok-dev will ask for the GPU 3 eval seat when that work needs it. Threads `01M4S026GWS354WZGXZ5E34F31`, `01M4S91WPPPN0B17JBNX98K5CY`. +- `[2026-10-12]` **Decision seat: h2o-lightning-4b served 1311–1403, then ROLLED BACK to INC** on albok-dev's word, with brokkr concurring (Prime's plan made the swap conditional on the judge's sign-off). Every eval-set check had passed (JevBench 205/231, 422 refusal, 16q@32k ~2.1 s, seat peak 16,164/16,847), but albok's full walk over 10,841 same texts (both verdicts, same prompt and policy) found h2o flags 242 adult at 0.5 against INC's 88, with 154 extra at 0.80–0.90 that no threshold separates; for personal, 5 vs 11 with only 2 in common. **Lesson: a passing benchmark plus a small borderline set (9 entries, 2 → 5) could not resolve a 2× shift in a production classifier; only the same-text fleet comparison did.** albok's re-stamp walk on INC (1406–1408) put the hidden set back exactly to its pre-cutover counts (fleet/memory 86, agent-feedback 21, nonfiction 0). Prime's inline-images-only order (via brokkr) is carried in the staged h2o compose. h2o returns only after a judge re-fit. Threads `01M4RSFC9J2R0J07EEBD4N04W3` (brokkr), `01M4RYSMP3DV9HZHJ0KNBH2S96` (albok). **Then:** the round 2 eval seat on GPU 3 (1428–1619, Prime 1420) found the cause. The judge's v1 note question asked two things at once: INC answered it inconsistently, h2o literally. The shim floor is ruled out (identical verdicts on and off). Prime ruled "text itself", and albok-nemi 0.1.18 (judge v2) re-fitted the question. Both seats agree under v2 (work notes INC 0/22, h2o 1/22 at 0.52). **Prime ruled the cutover at 1640 ("do it", via albok-dev): h2o LIVE again since 1704** (`switch-to-h2o.sh` 170223, healthy 170454, probes clean). **The v2 walk on h2o (1705–1745) moved two margins past albok-dev's named line**: nonfiction adult 0 → 79 (the Schell chapter at 0.531 on the book profile) and memory personal 6 → 18. Notes held at 0 adult. **Rolled back to INC at 1748 on albok-dev's word.** The 1749–1751 re-stamp walk restored the counts (memory 0 adult / 6 personal, feedback 0 / 2, nonfiction 0 / 0), and the timer was back at 1751. **Then Prime, 1805 (via albok-dev): "go to h2o, and fix forward. do not fall back to inc."** albok-dev declared the two published nonfiction titles `visibility: public` (the chapter margin cannot recur). **h2o has been production since 1807** (`switch-to-h2o.sh` 180513, healthy 180740, all checks clean). The personal-question re-fit continues forward on the live seat. Threads `01M4S026GWS354WZGXZ5E34F31`, `01M4S91WPPPN0B17JBNX98K5CY`. - `[2026-10-12]` **U13 regression fixed 0943 (svos-dev's URGENT, Prime saw every app pane 403):** the browser frames a pane at the registry `base_url`, and U13 had set app-terminal's to the seat-host origin, which admits only nh3-svos. Fix: `base_url` back to `https://svos-terminal.phasefinal.com` (config_gen `acae45206bbc9fcb`), plus POST `/seats/{observe,check,launch,retire-launch}` allowed from the box on that origin. svos-dev's one-line fix alone would have 404'd the kernel's seat-host calls; the origin never carried them before. svos-dev owes a separate browser-facing origin field; then the seat-host goes back to being kernel-facing. - `[2026-10-12]` **Shared CI runner (ana-docker act_runner, capacity 2, `timeout: 30m`) is slow because pfi-pve's `ospool` is HDD mirrors with no SLOG:** 8 KiB `oflag=dsync` writes measured ~145 ms on ana-docker vs ~2.7 ms on nh3-docker (n=3 each; pfi-pve host on ospool 18–32 ms; corviduo-dev ~75 ms). Jobs past 30 min are killed with a truncated log and no footer (albok every run since 10-04; vastblue 10935). Worse since the 10-04 PVE 9 reboot, cause not pinned (drive write cache was the same before and after: sda WCE on, sdb–sdd off). vastblue fixed its side with PostgreSQL on tmpfs (run 11033 green, pytest 14m17s vs 7–8 min per 72-test batch); albok is adding pytest-timeout + durations in 0.1.19. **Prime ruled A (2026-10-12 ~1715): an opt-in fast runner on nh3-docker.** The ana-docker vCPU-type question (x86-64-v2-AES, no AVX2) is DROPPED: the measured bottleneck was disk fsync plus a hanging network test, not CPU, so nothing rides on it. - `[2026-10-11]` **worldtree-dev is reading mail again (2219), answering four of our threads at once**: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread `01M4BDETDGMNVDTFXN3T3Y63DP`. diff --git a/stacks/intern-decision/README.md b/stacks/intern-decision/README.md index f2a9fd1..213c126 100644 --- a/stacks/intern-decision/README.md +++ b/stacks/intern-decision/README.md @@ -1,25 +1,25 @@ # intern-decision — the decision seat (fv-ml1 GPU 1, :8033) -> **LIVE: Intern-Decision-4B, since 2026-10-12 1748 PT** (`compose.yaml` = `compose.inc.yaml`). Nemi's judge -> runs at concurrency 1 against it (INC is single-threaded, INV-2). -> **h2o-lightning-4b is STAGED** (`compose.h2o.yaml`, `switch-to-h2o.sh`). It has been rolled back twice, each time on -> albok-dev's word against a rollback line named before the switch. +> **LIVE: h2o-lightning-4b, since 2026-10-12 1807 PT: the production seat.** Prime, 1805, via albok-dev: "okay, go +> to h2o, and fix forward. do not fall back to inc." `compose.yaml` = `compose.h2o.yaml`. Nemi's judge runs the v2 +> note question at concurrency 4. +> +> ⚠ **No fallback to INC.** `rollback-to-inc.sh` and `compose.inc.yaml` stay on the host as an artifact, but problems +> on this seat are fixed forward (judge wording, thresholds, `visibility` declarations), not by switching back. Using +> INC again would need Prime's explicit word. > > **What happened on 2026-10-12:** -> 1. **h2o 1311–1403, judge v1, rolled back.** Adult flags went from INC's 88 to 242 over 10,841 identical texts. -> 2. **The GPU 3 investigation, 1428–1619.** The v1 question asked two things at once, and the floor was ruled out. -> Prime ruled "text itself", and albok-nemi 0.1.18 shipped the v2 question. -> 3. **h2o 1704–1747, judge v2, rolled back.** Adult on notes held at 0 as validated, but two margins moved past -> the named line: -> - library/nonfiction adult 0 → 79: the Schell game-design chapter at 0.531 on h2o's book profile (INC 0.438), -> with its 77 concepts inheriting; -> - fleet/memory personal 6 → 18 on an unchanged personal question. -> -> albok-dev calls both fixable offline: a per-profile threshold or a `visibility: public` declaration on published -> titles, and a measured look at how h2o reads the personal question. They bring those to Prime before any third -> attempt. "The model is not the problem, the two margins are." +> 1. **h2o 1311–1403 (judge v1), rolled back.** Adult flags went from 88 to 242. The v1 question asked two things at +> once. +> 2. **The GPU 3 investigation.** The floor was ruled out, and the v2 question was shipped (albok-nemi 0.1.18). +> 3. **h2o 1704–1747 (v2), rolled back.** Two margins moved: +> - nonfiction adult 0 → 79, from one textbook chapter at 0.531; +> - memory personal 6 → 18. +> 4. **h2o for good from 1807.** albok-dev fixed the chapter margin first, by declaring the two published nonfiction +> titles `visibility: public`, so the judge is no longer consulted for them. The personal-question reading is +> being re-fitted forward on the live seat. -### The staged h2o seat (`compose.h2o.yaml`; live 1311–1403 and 1704–1747 on 2026-10-12) +### The h2o seat (`compose.h2o.yaml`; production since 2026-10-12 1807) | | | |---|---| @@ -101,7 +101,7 @@ Repeat that check on this seat's loopback port (`:18299`) on the next switch. IN ### Switching between the two -The 1704 switch: `switch-to-h2o.sh` ran from 170223 (rc 0) and h2o was healthy at 170454. The 1705 checks: `/health` at the +The final switch: `switch-to-h2o.sh` ran from 180513, h2o healthy at 180740, with the same checks all passing. The 1704 switch: `switch-to-h2o.sh` ran from 170223 (rc 0) and h2o was healthy at 170454. The 1705 checks: `/health` at the pin, with floor 0.801; `/decide/shared` 200; a 422 over the cap; inline red → red at 0.984; an https image URL refused by the shim; the vLLM backstop refusing example.com (400); GPU 1 resting at 14,430 MiB. @@ -115,7 +115,7 @@ Both run ON fv-ml1 and share one `.env`. `intern-decision_triton-cache` volume survive the `down`. Used 2026-10-12, 140328 → healthy 140356, and again 174746 → healthy 174811. -After either switch, make `stacks/intern-decision/compose.yaml` match the host again (it is INC since 1748). +After either switch, make `stacks/intern-decision/compose.yaml` match the host again (it is h2o since 1807). ⚠ **Tell albok-dev on either switch: Nemi's judge concurrency depends on the seat.** INC scores on one inference thread (INV-2, ~16 calls/s at most), so the judge runs with concurrency 1 against it (albok's `nemi.yaml`, since @@ -124,7 +124,7 @@ h2o, vLLM batches, so it goes back to 4. --- -# The live seat: Intern-Decision-4B (2026-09-30 → 10-12 1308, 1403–1702, and again since 1748) +# History: the Intern-Decision-4B seat (2026-09-30 → 10-12 1308, 1403–1702, 1748–1805; retired, no fallback per Prime) > **LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT, and again since 2026-10-12 1403 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168. > Acceptance on the live service is below. `semif` is stopped and kept as the rollback. diff --git a/stacks/intern-decision/compose.yaml b/stacks/intern-decision/compose.yaml index f4c9fb9..6a540af 100644 --- a/stacks/intern-decision/compose.yaml +++ b/stacks/intern-decision/compose.yaml @@ -1,46 +1,33 @@ -# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on -# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to -# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls). -# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now"). -# -# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the -# service keeps semif-serve's HTTP surface (/decide, /decide/shared, /health). Service code + -# contract: services/intern-decision-serve/ (intern-decision-serve.contract.md). Image built on -# fv-ml1 from that dir. -# -# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the -# container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside -# the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more -# gets 503 out_of_memory and the service stays up. See the README before changing it. -# -# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN -# (vault intern-decision/api-token, >= 32 chars). - +# STAGED, NOT LIVE (since the 2026-10-12 1403 rollback): the h2o-lightning-4b decision seat, which switch-to-h2o.sh +# copies over compose.yaml. It served 1311-1403 on 2026-10-12 (Prime via brokkr) and was rolled back when Nemi's judge +# flagged 242 texts adult against INC's 88 at fleet scale. Address intern-decision.fv.internal:8033 (brokkr rule 8). +# h2o-lightning-4b @ b43f759b. Stock vLLM 0.30.0 + the vendor's UNMODIFIED shim + services/h2o-decision-front (copied to +# conf/front): 32,000-token REFUSE-not-trim admission, the seat's bearer token (the same INTERN_DECISION_API_TOKEN), +# /decide + /decide/shared translated onto /v1/systemone, /health model {name, revision}. +# Rollback to Intern-Decision-4B: compose.inc.yaml (the previous compose, same .env): see the stack README. +# GPU 1 is shared (meromero-rp, gen-small): KV is pinned in BYTES (4.48 GiB = 135,070 tokens, the benched size); +# util only gates the boot check. Gate on scripts/oneoff/fv-ml1-gpucheck.sh before any (re)create. +# Images: INLINE ONLY (Prime 2026-10-12). LIGHTNING_IMAGE_URLS=off makes the shim refuse every image URL; vLLM's +# --allowed-media-domains media.invalid (a reserved name that never resolves) is the backstop if the shim ever changes. name: intern-decision - services: - intern-decision: - image: ${IMAGE:?set IMAGE} - container_name: intern-decision + vllm: + image: vllm/vllm-openai:v0.30.0 + container_name: intern-decision-vllm restart: unless-stopped ports: - - "${PORT:-8033}:8000" + - "127.0.0.1:18299:8000" environment: - INTERN_DECISION_API_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN} - INTERN_DECISION_DEVICE: cuda - INTERN_DECISION_VRAM_CAP_GIB: ${VRAM_CAP_GIB:?set VRAM_CAP_GIB} - # Coupled to VRAM_CAP_GIB: the largest call measured to fit under the cap (README "VRAM"). - INTERN_DECISION_MAX_TOKENS: ${MAX_TOKENS:?set MAX_TOKENS} - INTERN_DECISION_MAX_DECISIONS: ${MAX_DECISIONS:-64} - # POSTs in progress (queued + scoring) before new ones get 429 busy. - INTERN_DECISION_MAX_QUEUE: ${MAX_QUEUE:-32} + VLLM_USE_FLASHINFER_SAMPLER: "0" volumes: - # Pinned weights AND the checkpoint's inference.py, read offline. Never downloads. - - /tank/aimodels/huggingface:/hf:ro - # Triton/fla autotune cache (README "Cold-shape latency"): the ~6.5 s first-call-per-bucket - # autotune results survive RECREATES (deploys, image upgrades, .env edits), not just restarts. - # The mount point exists in the image owned by 10001, so the fresh volume is intern-writable. - - triton-cache:/tmp/triton-cache + - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro + - vllm-cache:/root/.cache/vllm + command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b", + "--max-model-len", "40960", "--max-num-seqs", "16", + "--gpu-memory-utilization", "0.1506", "--kv-cache-memory", "4810363371", + "--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}", + "--mm-processor-kwargs", "{\"max_pixels\": 1605632}", + "--allowed-media-domains", "media.invalid"] deploy: resources: reservations: @@ -48,27 +35,41 @@ services: - driver: nvidia device_ids: ["${GPU_ID:-1}"] capabilities: [gpu] + front: + image: python:3.12-slim + container_name: intern-decision + restart: unless-stopped + network_mode: host + depends_on: [vllm] + environment: + FRONT_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN} + SHIM_VLLM: http://127.0.0.1:18299 + SHIM_PORT: "18291" + SHIM_NOUL_FLOOR: "0.801" + LIGHTNING_IMAGE_URLS: "off" # Prime 2026-10-12: inline images only, the seat never fetches an image URL + FRONT_SHIM: http://127.0.0.1:18291 + FRONT_VLLM: http://127.0.0.1:18299 + FRONT_BIND: 0.0.0.0 + FRONT_PORT: ${PORT:-8033} + FRONT_CAP: "32000" + FRONT_SHIM_FILE: /model/h2o_lightning_shim.py + FRONT_SERVE_CONFIG: /model/serve_config.json + FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a + volumes: + - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro + - /opt/docker/conf/intern-decision/front:/front:ro + command: ["bash", "/front/run.sh"] healthcheck: - test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"] + test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8033/health', timeout=5)"] interval: 30s timeout: 10s retries: 3 - # Startup loads ~9 GB of weights and scores the warm-up three times before it serves. start_period: 300s - networks: - - tnet labels: - homepage.group=AI - Eval & Retrieval - - homepage.name=Intern-Decision — typed decisions + - homepage.name=Decision seat (h2o-lightning-4b) - homepage.icon=mdi-scale-balance - - homepage.description=Typed decisions from one forward pass (Intern-Decision-4B, fv-ml1 GPU1) + - homepage.description=Typed decisions, 32k refusing (h2o-lightning-4b on vLLM, fv-ml1 GPU1; /decide/shared kept) - homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health - -networks: - tnet: - name: traefik-net - external: true - volumes: - # -> intern-decision_triton-cache on the host - triton-cache: + vllm-cache: