vh
e58360668e
Two operator-authorised changes on the same afternoon.
## althing v3 (U9b flag day, one-way, no rollback)
The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.
73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.
Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.
nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.
The v2 database is left inert on disk. There is no import path and none
was improvised.
## sec onto GPU0
GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.
Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
81 lines
4.0 KiB
YAML
81 lines
4.0 KiB
YAML
# Move the `sec` pen-test seat (M.O.G.-SEC-27B) from ana-ml2 GPU 1 to GPU 0 and bring it up.
|
|
#
|
|
# Why: GPU 1 carries the five resident fleet seats (gen 46 GB + embed 9.8 + coder 8.4 +
|
|
# rerank 3.5 + reward 2.1 = ~69.9 GB of 97.9), leaving ~28 GB. This seat reserves
|
|
# MOG_GPU_MEM_UTIL=0.52 -> ~51 GB, so it could not start on GPU 1 at all. GPU 0 has been
|
|
# idle since run 3c was stopped on 2026-08-26. Operator-directed 2026-08-28.
|
|
#
|
|
# ⚠ POWER. This re-arms the two-GPU load condition that tripped the Anaheim rack breaker
|
|
# on 2026-08-26. One circuit feeds the whole rack including ana-gw and ana-wg, so a trip
|
|
# costs the site AND the way back in. Idle draw is negligible; the risk materialises when
|
|
# sec and gen are under concurrent load. Operator accepted this with the constraint stated.
|
|
#
|
|
# Labels only apply at container CREATION, so this uses `up -d`, never `restart` --
|
|
# the homepage description carries the GPU number and would otherwise stay stale.
|
|
#
|
|
# Run: scripts/elway infra-ops@ana-ml2 --playbook playbooks/mog-sec-move-to-gpu0.yaml
|
|
# Model load is slow (22 GB + 262K ctx + MTP graft); the verify phase polls rather than
|
|
# assuming readiness, and the compose healthcheck allows a 900s start_period.
|
|
|
|
vars:
|
|
stack_dir: /opt/docker/compose/mog-sec
|
|
container: vllm-mog-sec
|
|
service: vllm-mog-sec
|
|
gpu_id: "0"
|
|
port: "8019"
|
|
staging: /tmp/mog-sec-compose.yaml
|
|
|
|
steps:
|
|
- name: Stage the updated compose (GPU pin default + label now say GPU 0)
|
|
upload:
|
|
src: stacks/mog-sec/compose.yaml
|
|
dest: "{{ staging }}"
|
|
mode: "0644"
|
|
|
|
- name: Install it over the deployed copy
|
|
# /opt/docker/compose is root-owned, so the scp above lands in /tmp and this
|
|
# promotes it. Verified byte-identical against the deployed file beforehand:
|
|
# the only diff was these edits, so nothing on the host is being clobbered.
|
|
shell: sudo install -o root -g root -m 0644 {{ staging }} {{ stack_dir }}/compose.yaml
|
|
changed_when: "! sudo cmp -s {{ staging }} {{ stack_dir }}/compose.yaml"
|
|
|
|
- name: Pin the seat to GPU {{ gpu_id }} in the host .env
|
|
# The .env is the tunable surface and is NOT in git (secrets//tunables are
|
|
# excluded both directions). The compose default now matches, but the .env
|
|
# is what actually decides, so set it explicitly rather than relying on the
|
|
# default resolving.
|
|
shell: sudo sed -i 's/^MOG_GPU_ID=.*/MOG_GPU_ID={{ gpu_id }}/' {{ stack_dir }}/.env
|
|
when: "! sudo grep -qxF 'MOG_GPU_ID={{ gpu_id }}' {{ stack_dir }}/.env"
|
|
|
|
- name: Bring the seat up (up -d, not restart — labels apply at creation)
|
|
shell: cd {{ stack_dir }} && sudo docker compose up -d {{ service }}
|
|
|
|
verify:
|
|
- name: Container exists and is running
|
|
# ⚠ No `docker inspect -f` here. Go templates use {{ }} and so does elway's own
|
|
# variable substitution, so an inspect format string gets eaten before it reaches
|
|
# the host -- these two checks reported FAILED on a deploy that had in fact
|
|
# succeeded. Filter-and-grep has no such collision.
|
|
shell: sudo docker ps --filter name={{ container }} --filter status=running --quiet | grep -q .
|
|
changed_when: "false"
|
|
|
|
- name: The container is actually pinned to GPU {{ gpu_id }}
|
|
# Assert the EFFECTIVE device reservation on the running container, not the
|
|
# .env string we wrote -- the .env is an input, this is the outcome.
|
|
shell: sudo docker inspect {{ container }} | tr -d ' \n' | grep -q '"DeviceIDs":\["{{ gpu_id }}"\]'
|
|
changed_when: "false"
|
|
|
|
- name: GPU 0 now holds a vLLM process (the seat really loaded onto that card)
|
|
shell: nvidia-smi --id={{ gpu_id }} --query-compute-apps=pid,used_memory --format=csv,noheader | grep -qE '[0-9]'
|
|
changed_when: "false"
|
|
|
|
- name: Health endpoint answers
|
|
shell: curl -fsS --max-time 10 http://127.0.0.1:{{ port }}/health >/dev/null
|
|
changed_when: "false"
|
|
|
|
- name: Both served names are advertised (base + thinking)
|
|
shell: |
|
|
MODELS=$(curl -fsS --max-time 10 http://127.0.0.1:{{ port }}/v1/models)
|
|
echo "$MODELS" | grep -q 'mog-sec-27b' && echo "$MODELS" | grep -q 'mog-sec-27b-thinking'
|
|
changed_when: "false"
|