vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(gemma4-erp-tune): whole-card placement — gen moves to GPU1, sec stands down

Browse Source 
Operator chose a third placement over the two the sizing offered: rather
than train beside gen on GPU0 or on GPU1 in mog-sec's slot, move gen to
GPU1 and empty GPU0 completely. The tune gets 95.60 GiB with no
co-tenant and gen never goes dark beyond its own restart.

Revised run parameters, since a whole card changes them:

- micro-batch 8 (71.8 GiB of 95.60) rather than 4, grad-accum 1, giving
  888 optimizer steps instead of 444. At one epoch the step count is
  worth having, and 8 x 8192 tokens puts ~4,096 rows through each expert
  per step against ~512 at micro-batch 1 — a far healthier GEMM on
  704-wide experts.
- Gradient checkpointing stays ON. Dropping it takes ~17% off wall-clock
  but pushes activations to ~24 GiB per sequence, which forces
  micro-batch 1 and costs 8x on MoE efficiency. Wide beats shallow.
- Scriberr stays on GPU1. The previous revision suggested moving it to
  GPU0, which was correct only while training was going to live on GPU1.

Records the ordering constraint in both directions, the elway identity
requirement, and that sec's aliases should be allowed to fail at the
gateway rather than be substituted with another model.
This commit is contained in:
vh
2026-08-24 18:40:42 -07:00
parent 9d70100867
commit c9943b1507
2 changed files with 67 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/pfi/gemma4-erp-tune-sizing.md
+66 -27
View File
@@ -163,23 +163,33 @@ Card total: 97,887 MiB = **95.60 GiB** each.
| | GPU0 | GPU1 |
|---|---|---|
| resident | `vllm-gen` 42,508 MiB (up 3 h) | `vllm-mog-sec` 56,624 MiB + `embed` 3,304 + `reward` 9,512 + `coder` 6,158 + `rerank-a3` 2,170; Scriberr pinned here, loads on demand |
| resident before the window | `vllm-gen` 42,508 MiB (up 3 h) | `vllm-mog-sec` 56,624 MiB + `embed` 3,304 + `reward` 9,512 + `coder` 6,158 + `rerank-a3` 2,170; Scriberr pinned here, loads on demand |
| free | 54,741 MiB = **53.46 GiB** | 19,446 MiB = **18.99 GiB** |
Three placements were on the table:
- **GPU0 beside `gen`: does not fit.** 53.46 GiB free against ~57.6 GiB needed —
short by ~4 GiB. And `gen` is only three hours old: measured footprint runs
38.5 GiB fresh → 42.5 GiB at 3 h → 45.6 GiB at 3 days. Budgeting against the
current number is budgeting against a moving one.
- **GPU1 as-is: nowhere near.** 18.99 GiB.
- **GPU1 with `vllm-mog-sec` stopped: 76,070 MiB = 74.29 GiB free.** Fits
micro-batch 4 (61.8 GiB) with ~12 GiB clear even after reserving ~6 GiB for
Scriberr's on-demand whisper load.
- **GPU1 with `mog-sec` stopped: 76,070 MiB free.** Fits, but shares a card with
four small seats and Scriberr.
- **GPU0 with `gen` MOVED OFF: the whole card.** ← what was chosen.
---
## 4. Recommendation
## 4. The window, as executed
**Run on GPU1. Stand down `sec`, not `gen`.**
**Operator call, 2026-08-24: move `gen` to GPU1 and stand `sec` down, so GPU0 is
emptied completely rather than shared.** This is strictly better than training
beside `gen`: the tune gets 95.60 GiB with no co-tenant, and the fleet's general
seat never goes dark beyond its own ~5-minute restart.
before: GPU0 [ gen 42.5 ] GPU1 [ sec 55.3 | small seats 20.7 ]
after: GPU0 [ ---- empty, 95.60 GiB ---- ] GPU1 [ gen ~41 | small seats 20.7 | ~33 free ]
`sec` is genuinely in use and this is not free — but it is the smaller blast
radius by a wide margin:
| | `gen` | `sec` |
|---|---|---|
@@ -187,31 +197,60 @@ Card total: 97,887 MiB = **95.60 GiB** each.
| standing role | the fleet's general seat; a documented always-available dependency in global `CLAUDE.md` | M.O.G.-SEC, niche |
| measured traffic | 765 busy-engine log lines in 24 h — continuously in use | bursty; peak 8 concurrent, **last request ~5 h ago** |
`sec` is genuinely in use and this is not free — but it is the smaller blast
radius by a wide margin, and it is the difference between a 4–10 hour window
that nobody outside the security work notices and one that takes the fleet's
default model offline for a working day.
Traffic to `sec` arrives from `10.250.50.70` (the LiteLLM gateway), so the
aliases will fail at the gateway for the duration. Per the standing rule, let
them fail — **do not route `sec` to another model as a stand-in.**
Also fold in, cheaply and reversibly:
Both directions are playbooks, and **the order in each is load-bearing**:
- **Move Scriberr to GPU0 for the window** (`device_ids: ["0"]`, one compose
edit + `up -d`). GPU0 will be sitting on ~53 GiB free with `gen` up, and it
removes contention from the training card entirely.
- **Micro-batch 4, grad-accum 4** (effective 16, 444 steps). Micro-batch 6 is
the measured ceiling; the gap is deliberate margin. A seat crash-looped
earlier the same day on 0.6 GiB of assumed headroom — this document does not
repeat that.
scripts/elway infra-ops@10.250.50.54 --playbook playbooks/ana-ml2-training-window-open.yaml
scripts/elway infra-ops@10.250.50.54 --playbook playbooks/ana-ml2-training-window-close.yaml
⚠ `gen` runs at `--gpu-memory-utilization 0.43`, which vLLM reads as a fraction
of **total** card memory: 42,091 MiB must be *free at startup* or the engine
refuses to boot. GPU1 has 19,446 MiB free while `mog-sec` is up. **Recreating
`gen` onto GPU1 before stopping `mog-sec` takes the fleet's main seat down and
leaves it down.** The open playbook stops `mog-sec` first and hard-gates on the
freed memory; the close playbook mirrors it, because `mog-sec` needs 50,901 MiB
of its own and cannot start until `gen` has vacated GPU1.
⚠ Invoke elway as `infra-ops@10.250.50.54`, not the `ana-ml2` ssh-target — that
resolves to `lkraven`, which has no NOPASSWD sudo, and elway aborts at its sudo
probe.
### Revised run parameters, now that it is a whole card
| micro-batch | GiB of 95.60 | |
|---:|---:|---|
| 4 | 61.8 | (the shared-card recommendation, now superseded) |
| **8** | **71.8** | **recommended — ~24 GiB clear** |
| 12 | 81.8 | ~14 GiB clear |
| 16 | 91.8 | too tight |
**Micro-batch 8, grad-accum 1 → 888 optimizer steps** (up from 444 on the shared
card). Two independent wins: the step count doubles, which matters at only one
epoch, and 8 × 8192 tokens gives ~4,096 rows per expert per step against ~512 at
micro-batch 1 — a far healthier GEMM on 704-wide experts, so MFU should land at
the top of the 10–25% band rather than the bottom.
⚠ **Keep gradient checkpointing ON.** With a whole card it is tempting to drop
it (~17% off wall-clock by removing the recompute forward), but activations then
run ~24 GiB per sequence, which forces micro-batch 1 — and micro-batch 1 craters
MoE GEMM efficiency by 8×. Going wide beats going shallow here. Do not
"optimize" this later without re-reading this paragraph.
Also fold in:
- **Scriberr STAYS on GPU1.** (An earlier draft of this document suggested moving
it to GPU0; that was written when training was going to live on GPU1, and it is
now exactly backwards. GPU0 is the training card and wants no co-tenant.)
- **Package as a `uv` venv on `/tank`, not a Docker image.** Root is at **91%
(36 GB free)** and `/var/lib/docker` lives on it; a PyTorch training image
would come close to filling it. `/tank` has 4.0 TB.
### If `sec` may not be stood down
Fallback is GPU0 with `gen` stopped — the run fits with ~38 GiB to spare, which
buys micro-batch 8 and a shorter wall-clock. Make it **resumable and split** in
that case: INV-T7 plus wall-clock checkpointing already allows the window to be
broken into two or three shorter stretches with `gen` restored between them,
rather than one long outage.
- **The run is still resumable-by-design** (INV-T7 + wall-clock checkpointing).
Nothing about a dedicated card removes that requirement — a 4–10 hour window
is long enough that an unresumable run is a bad bet regardless of who owns the
GPU.
---