vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(ana-ml2): elway playbooks to open and close the ERP/RP tune window

Browse Source 
Operator call: rather than train beside gen on GPU0, move gen to GPU1 and
stand sec down for the night, so the tune gets a whole 95.60 GiB card and
the fleet's general seat never goes dark beyond its own restart.

Order is load-bearing in both directions and the playbooks enforce it.
gen runs at --gpu-memory-utilization 0.43, which vLLM reads as a fraction
of TOTAL card memory: 42,091 MiB must be FREE at startup or the engine
refuses to boot. GPU1 has 19,446 MiB free while mog-sec is up, so
recreating gen onto GPU1 first would take the main seat down and leave it
down. mog-sec stops first and a hard gate checks the freed memory before
gen is touched. The close playbook mirrors it: gen must vacate GPU1
before mog-sec starts, since mog-sec needs 50,901 MiB of its own.

Close opens with a gate that refuses to run while a process is still
resident on GPU0, so it cannot evict a training run mid-flight.
Override with --var allow_busy_gpu0=true.

Three defects found and fixed while landing this, all worth keeping:

- Verifying GPU residency via `docker inspect --format {{.State.Pid}}`
  never matches. vLLM V1 runs EngineCore as a child of the container's
  pid 1, and it is the child that holds the memory and that nvidia-smi
  reports. Match by cgroup instead.
- A step's `sudo: true` does not extend to its when/creates/changed_when
  guards, which run as the login user. The root-only .env made an
  unsudo'd grep exit 2, so the GPU-id flip SILENTLY SKIPPED. The
  effective-value assert is what caught it.
- That assert originally grepped the config YAML for -\s*'?1'? and failed
  against compose's double-quoted `- "1"`. Parse the JSON with jq; an
  assert that fails for the wrong reason is worse than no assert.

elway must be invoked as infra-ops@10.250.50.54 rather than the ana-ml2
ssh-target, which resolves to lkraven and has no NOPASSWD sudo.
This commit is contained in:
Vuong Hoang
2026-08-24 18:39:13 -07:00
parent c507db9ac0
commit 9d70100867
2 changed files with 357 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
playbooks/ana-ml2-training-window-close.yaml
+175
View File
@@ -0,0 +1,175 @@
# ana-ml2 — CLOSE the ERP/RP tune window: put the fleet back the way it was.
#
# gen GPU1 -> GPU0 -> start mog-sec back onto GPU1
#
# The exact inverse of playbooks/ana-ml2-training-window-open.yaml.
#
# ⚠⚠ ORDER IS LOAD-BEARING, AND IT IS THE MIRROR OF THE OPEN ORDER.
# `gen` must vacate GPU1 BEFORE mog-sec is started. mog-sec runs at
# --gpu-memory-utilization 0.52 = 50,901 MiB that must be free at startup. With
# gen still resident on GPU1 only ~30,000 MiB is free, so mog-sec would fail to
# boot. gen moves back to the (empty) GPU0 first; step 4 waits for GPU1 to
# actually release before mog-sec is started at all.
#
# ⚠ FIRST STEP IS A GATE, NOT A COURTESY. If a training process is still
# resident on GPU0 this playbook REFUSES to run — moving gen back would either
# OOM the run or OOM gen. Override only when you have confirmed the run is
# finished or deliberately abandoned:
#
# scripts/elway ana-ml2 --playbook playbooks/ana-ml2-training-window-close.yaml \
# --var allow_busy_gpu0=true
#
# scripts/elway ana-ml2 --playbook playbooks/ana-ml2-training-window-close.yaml
vars:
gen_dir: /opt/docker/compose/gen-seat
mog_dir: /opt/docker/compose/mog-sec
gen_port: "8015"
mog_port: "8019"
# mog-sec's startup requirement: 0.52 x 97,887 MiB, rounded up.
mog_required_free_mib: "50950"
# Set to "true" to close the window even with a process still on GPU0.
allow_busy_gpu0: "false"
steps:
- name: "GATE — GPU0 is idle (refuses to evict a training run mid-flight)"
sudo: true
shell: |
set -e
gpu0_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 0)
n=$(nvidia-smi --query-compute-apps=gpu_uuid,pid --format=csv,noheader | grep -c "$gpu0_uuid" || true)
echo "GPU0 compute procs: $n"
if [ "$n" -eq 0 ]; then exit 0; fi
if [ "{{ allow_busy_gpu0 }}" = "true" ]; then
echo "GPU0 still busy but allow_busy_gpu0=true — proceeding under override"
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory,process_name --format=csv | grep "$gpu0_uuid" || true
exit 0
fi
echo "REFUSING: a process is still resident on GPU0. Confirm the tune has"
echo "finished, then rerun with --var allow_busy_gpu0=true"
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory,process_name --format=csv | grep "$gpu0_uuid" || true
exit 1
changed_when: "false"
- name: "PREFLIGHT — record gen's current container id (proves the recreate later)"
sudo: true
shell: docker inspect vllm-gen --format '{{.Id}}' | tee /tmp/gen-container-id-before.txt
changed_when: "false"
- name: "Point gen back at GPU0 in its .env (GEN_GPU_ID 1 -> 0)"
sudo: true
# ⚠ `sudo` INSIDE the when: expression — a step's `sudo: true` does NOT
# cover its guards, and the root-only .env makes an unsudo'd grep exit 2,
# which silently skips the step. See the open playbook for the incident.
shell: sed -i 's/^GEN_GPU_ID=1$/GEN_GPU_ID=0/' {{ gen_dir }}/.env
when: "sudo grep -qx 'GEN_GPU_ID=1' {{ gen_dir }}/.env"
- name: "Assert the EFFECTIVE device id, not the .env line"
sudo: true
# ⚠ Parse the JSON, do not regex the YAML — compose emits `- "0"` with
# DOUBLE quotes. See the open playbook for the incident.
shell: |
docker compose --project-directory {{ gen_dir }} config --format json \
| jq -e '.services["vllm-gen"].deploy.resources.reservations.devices[0].device_ids == ["0"]'
changed_when: "false"
- name: "Recreate gen onto GPU0"
sudo: true
shell: docker compose --project-directory {{ gen_dir }} up -d vllm-gen
- name: "Wait for gen to serve /health on GPU0"
sudo: true
shell: |
for i in $(seq 1 180); do
if curl -sf -o /dev/null http://127.0.0.1:{{ gen_port }}/health; then
echo "gen healthy after $((i*5))s"; exit 0
fi
sleep 5
done
echo "TIMEOUT: gen did not become healthy in 900s"; exit 1
changed_when: "false"
- name: "Wait for GPU1 to release gen's memory before mog-sec is started"
sudo: true
shell: |
for i in $(seq 1 60); do
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 1)
if [ "$free" -ge {{ mog_required_free_mib }} ]; then
echo "GPU1 free: ${free} MiB"; exit 0
fi
sleep 2
done
echo "TIMEOUT: GPU1 free is ${free} MiB, need >= {{ mog_required_free_mib }}"; exit 1
changed_when: "false"
- name: "Start mog-sec back up on GPU1 (`sec` / `sec-reasoning`)"
sudo: true
shell: docker compose --project-directory {{ mog_dir }} start vllm-mog-sec
when: "! docker inspect -f '{{.State.Running}}' vllm-mog-sec 2>/dev/null | grep -q true"
- name: "Wait for mog-sec to serve /health"
sudo: true
shell: |
for i in $(seq 1 180); do
if curl -sf -o /dev/null http://127.0.0.1:{{ mog_port }}/health; then
echo "mog-sec healthy after $((i*5))s"; exit 0
fi
sleep 5
done
echo "TIMEOUT: mog-sec did not become healthy in 900s"; exit 1
changed_when: "false"
verify:
- name: "gen was genuinely RECREATED (container id changed)"
sudo: true
shell: |
before=$(cat /tmp/gen-container-id-before.txt)
after=$(docker inspect vllm-gen --format '{{.Id}}')
echo "before=${before:0:12} after=${after:0:12}"
test "$before" != "$after"
changed_when: "false"
- name: "gen's process is resident on GPU0 again"
sudo: true
# ⚠ Match by CGROUP, not by `.State.Pid` — vLLM V1's EngineCore is a CHILD
# of the container's pid 1, and it is the child nvidia-smi reports.
shell: |
cid=$(docker inspect vllm-gen --format '{{.Id}}')
gpu0_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 0)
found=0
for p in $(nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory --format=csv,noheader \
| grep "$gpu0_uuid" | cut -d, -f2 | tr -d ' '); do
if grep -q "$cid" /proc/$p/cgroup 2>/dev/null; then
echo "gen pid $p resident on GPU0: $(nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader | grep "^$p,")"
found=1
fi
done
test "$found" -eq 1
changed_when: "false"
- name: "gen answers a real completion"
sudo: true
shell: |
. {{ gen_dir }}/.env
curl -sf -m 120 http://127.0.0.1:{{ gen_port }}/v1/chat/completions \
-H "Authorization: Bearer ${API_KEY}" -H 'Content-Type: application/json' \
-d '{"model":"'"${GEN_SERVED_NAME}"'","messages":[{"role":"user","content":"reply with the single word: ok"}],"max_tokens":16}' \
| grep -q '"content"'
changed_when: "false"
- name: "sec answers a real completion"
sudo: true
shell: |
. {{ mog_dir }}/.env
curl -sf -m 120 http://127.0.0.1:{{ mog_port }}/v1/chat/completions \
-H "Authorization: Bearer ${API_KEY}" -H 'Content-Type: application/json' \
-d '{"model":"'"${MOG_SERVED_NAME}"'","messages":[{"role":"user","content":"reply with the single word: ok"}],"max_tokens":16}' \
| grep -q '"content"'
changed_when: "false"
- name: "Both cards are back to their normal tenancy"
sudo: true
shell: |
nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory,process_name --format=csv
changed_when: "false"
playbooks/ana-ml2-training-window-open.yaml
+182
View File
@@ -0,0 +1,182 @@
# ana-ml2 — OPEN the ERP/RP tune window: clear GPU0 completely.
#
# stop mog-sec (GPU1) -> move gen GPU0 -> GPU1 -> GPU0 empty for training
#
# Operator call 2026-08-24: rather than train beside `gen`, move `gen` off GPU0
# entirely and stand `sec` down for the night. Training then gets a whole card
# (95.60 GiB) instead of a shared one, and the fleet's general seat never goes
# dark beyond its own restart.
#
# ⚠⚠ ORDER IS LOAD-BEARING — DO NOT REORDER THE STEPS.
# `gen` runs at --gpu-memory-utilization 0.43, which vLLM reads as a fraction of
# TOTAL card memory: 0.43 x 97,887 MiB = 42,091 MiB that must be FREE at startup
# or the engine refuses to boot. GPU1 has only 19,446 MiB free while mog-sec is
# up. Recreating `gen` onto GPU1 first would take the fleet's main seat down and
# leave it down. mog-sec stops FIRST, and step 3 hard-gates on the freed memory
# before `gen` is touched at all.
#
# ⚠ `stop`, never `down`. `down` removes the container; `stop` leaves it in
# place so the close playbook can `start` it. Both seats are `restart:
# unless-stopped`, which does NOT resurrect a deliberately-stopped container.
#
# ⚠ device_ids vs nvidia-smi ordering was VERIFIED on this host, not assumed:
# gen (GEN_GPU_ID=0) reports under the GPU nvidia-smi indexes 0, mog-sec
# (MOG_GPU_ID=1) under index 1. They agree here. (They do NOT on irv-ml1 —
# never carry that assumption between boxes.)
#
# Restore with: playbooks/ana-ml2-training-window-close.yaml
#
# scripts/elway ana-ml2 --playbook playbooks/ana-ml2-training-window-open.yaml
vars:
gen_dir: /opt/docker/compose/gen-seat
mog_dir: /opt/docker/compose/mog-sec
gen_port: "8015"
# gen's startup requirement: 0.43 x 97,887 MiB, rounded up. If GPU1 has less
# than this free, gen will not boot and the window must not proceed.
gen_required_free_mib: "42100"
steps:
- name: "PREFLIGHT — GPU0 holds vllm-gen and nothing else unexpected"
sudo: true
shell: |
set -e
procs=$(nvidia-smi --query-compute-apps=gpu_uuid,pid --format=csv,noheader | wc -l)
gpu0_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 0)
gpu0_procs=$(nvidia-smi --query-compute-apps=gpu_uuid,pid --format=csv,noheader | grep -c "$gpu0_uuid" || true)
echo "GPU0 compute procs: $gpu0_procs (total on box: $procs)"
test "$gpu0_procs" -le 1
changed_when: "false"
- name: "PREFLIGHT — record gen's current container id (proves the recreate later)"
sudo: true
shell: docker inspect vllm-gen --format '{{.Id}}' | tee /tmp/gen-container-id-before.txt
changed_when: "false"
- name: "Stop mog-sec (the `sec` / `sec-reasoning` seat) — frees ~55.3 GiB on GPU1"
sudo: true
shell: docker compose --project-directory {{ mog_dir }} stop vllm-mog-sec
# Skip if already stopped, so the playbook is rerunnable.
when: "docker inspect -f '{{.State.Running}}' vllm-mog-sec 2>/dev/null | grep -q true"
- name: "Wait for GPU1 memory to actually release (teardown is not instant)"
sudo: true
shell: |
for i in $(seq 1 60); do
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 1)
if [ "$free" -ge {{ gen_required_free_mib }} ]; then
echo "GPU1 free: ${free} MiB"; exit 0
fi
sleep 2
done
echo "TIMEOUT: GPU1 free is ${free} MiB, need >= {{ gen_required_free_mib }}"; exit 1
changed_when: "false"
- name: "HARD GATE — GPU1 has room for gen's 0.43 budget before we touch gen"
sudo: true
shell: |
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 1)
echo "GPU1 free ${free} MiB vs required {{ gen_required_free_mib }} MiB"
test "$free" -ge {{ gen_required_free_mib }}
changed_when: "false"
- name: "Point gen at GPU1 in its .env (GEN_GPU_ID 0 -> 1)"
sudo: true
# ⚠ `sudo` INSIDE the when: expression. A step's `sudo: true` covers the
# shell, NOT its when/creates/changed_when guards — those run as the login
# user. The .env is root-only 0600, so an unsudo'd grep exits 2
# (permission denied), which is not 0, so the step SILENTLY SKIPS and the
# flip never happens. Caught 2026-08-24 by the effective-value assert below.
shell: sed -i 's/^GEN_GPU_ID=0$/GEN_GPU_ID=1/' {{ gen_dir }}/.env
when: "sudo grep -qx 'GEN_GPU_ID=0' {{ gen_dir }}/.env"
- name: "Assert the EFFECTIVE device id, not the .env line"
sudo: true
# grep on the .env proves a substring is present; only `compose config`
# proves what the container will actually be created with.
# ⚠ Parse the JSON, do not regex the YAML. The first version of this grepped
# for -\s*'?1'? and failed against compose's DOUBLE-quoted `- "1"` — an
# assert that fails for the wrong reason is worse than no assert.
shell: |
docker compose --project-directory {{ gen_dir }} config --format json \
| jq -e '.services["vllm-gen"].deploy.resources.reservations.devices[0].device_ids == ["1"]'
changed_when: "false"
- name: "Recreate gen onto GPU1 (a device change needs up -d, not restart)"
sudo: true
shell: docker compose --project-directory {{ gen_dir }} up -d vllm-gen
- name: "Wait for gen to serve /health (cold start: weights + CUDA graphs + MTP)"
sudo: true
shell: |
for i in $(seq 1 180); do
if curl -sf -o /dev/null http://127.0.0.1:{{ gen_port }}/health; then
echo "gen healthy after $((i*5))s"; exit 0
fi
sleep 5
done
echo "TIMEOUT: gen did not become healthy in 900s"; exit 1
changed_when: "false"
verify:
- name: "gen was genuinely RECREATED (container id changed)"
sudo: true
shell: |
before=$(cat /tmp/gen-container-id-before.txt)
after=$(docker inspect vllm-gen --format '{{.Id}}')
echo "before=${before:0:12} after=${after:0:12}"
test "$before" != "$after"
changed_when: "false"
- name: "gen's process is resident on GPU1"
sudo: true
# ⚠ Match by CGROUP, not by `.State.Pid`. vLLM V1 runs EngineCore as a CHILD
# of the container's pid 1, and it is the child that holds the GPU memory —
# nvidia-smi never reports `.State.Pid`, so comparing against it always fails.
shell: |
cid=$(docker inspect vllm-gen --format '{{.Id}}')
gpu1_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 1)
found=0
for p in $(nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory --format=csv,noheader \
| grep "$gpu1_uuid" | cut -d, -f2 | tr -d ' '); do
if grep -q "$cid" /proc/$p/cgroup 2>/dev/null; then
echo "gen pid $p resident on GPU1: $(nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader | grep "^$p,")"
found=1
fi
done
test "$found" -eq 1
changed_when: "false"
- name: "gen answers a real completion, not just /health"
sudo: true
shell: |
. {{ gen_dir }}/.env
curl -sf -m 120 http://127.0.0.1:{{ gen_port }}/v1/chat/completions \
-H "Authorization: Bearer ${API_KEY}" -H 'Content-Type: application/json' \
-d '{"model":"'"${GEN_SERVED_NAME}"'","messages":[{"role":"user","content":"reply with the single word: ok"}],"max_tokens":16}' \
| grep -q '"content"'
changed_when: "false"
- name: "GPU0 IS EMPTY — zero compute processes"
sudo: true
shell: |
gpu0_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 0)
n=$(nvidia-smi --query-compute-apps=gpu_uuid,pid --format=csv,noheader | grep -c "$gpu0_uuid" || true)
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 0)
echo "GPU0 compute procs=${n} free=${free} MiB"
test "$n" -eq 0 && test "$free" -ge 95000
changed_when: "false"
- name: "mog-sec is stopped (not removed — close depends on `start` working)"
sudo: true
shell: |
docker inspect -f '{{.State.Status}}' vllm-mog-sec | tee /dev/stderr | grep -qx exited
changed_when: "false"
- name: "GPU1 still has headroom for Scriberr's on-demand load"
sudo: true
shell: |
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 1)
echo "GPU1 free after gen landed: ${free} MiB"
test "$free" -ge 12000
changed_when: "false"