fix(mtp-bench): stop_bench raced docker name release, silently dropping arms

`docker rm -f` returns once removal is INITIATED, not once complete. The
bench container holds ~92 GB of GPU memory plus the offloaded PLE mapping, so
the name stays reserved for several seconds while the device is released. The
next `docker run --name` then dies with a name Conflict.

That failure was near-invisible: run_arm treats a failed boot as
`arm SKIPPED`, the campaign continues, and it still prints CAMPAIGN DONE.
Observed 2026-09-13 -- arm k1 lost the race after off_A's loaded container,
while k2 won it only because k1 had never started a container to tear down.
Every arm that follows one which actually ran is exposed, so k3 and off_B were
both on track to vanish from a run that would have reported success.

stop_bench now polls until the name is released (120s ceiling, warns and
continues). Added a completeness gate: the campaign asserts every expected
res-<arm>-rep<n>.json exists and reports CAMPAIGN INCOMPLETE naming each
missing result rather than DONE, so a gutted run cannot look like a clean one.

Verified the docker ps format string empirically -- an earlier draft nested
quotes so the template rendered as '{{.Names}}' with literal quotes, which
grep -qx could never match, making the wait a no-op that passed bash -n.
This commit is contained in:
2026-09-13 13:29:17 -07:00
parent 0a9cf58e19
commit 67a7dc539e
+28 -2
View File
@@ -124,7 +124,23 @@ IMAGE=$(python3 -c "
import json; print(json.load(open('$OUT/resolved-compose.json'))['services']['vllm-flash-next']['image'])")
log "image=$IMAGE model=$MODEL_DIR"
stop_bench(){ sudo -n docker rm -f "$NAME" >/dev/null 2>&1 || true; }
stop_bench(){
sudo -n docker rm -f "$NAME" >/dev/null 2>&1 || true
# `docker rm -f` returns once removal is INITIATED, not once it is done. A
# container holding ~92 GB of GPU memory plus the offloaded PLE mapping keeps
# the name reserved for several seconds while the device is released, so the
# next `docker run --name` loses a race and dies with a name Conflict. That
# surfaces as `arm SKIPPED (boot failed)` -- an entire arm silently missing
# from a campaign that still reports DONE. Observed 2026-09-13 on arm k1.
local t0=$SECONDS
while sudo -n docker ps -a --format "{{.Names}}" | grep -qx "$NAME"; do
if [ $((SECONDS-t0)) -ge 120 ]; then
log " WARN: $NAME still present 120s after rm -f; continuing anyway"
break
fi
sleep 2
done
}
cleanup(){ stop_bench; [ -n "${POWER_PID:-}" ] && kill "$POWER_PID" 2>/dev/null; }
trap 'cleanup' EXIT
@@ -177,7 +193,17 @@ run_arm k2 "{\"method\":\"$METHOD\",\"num_speculative_tokens\":2}"
run_arm k3 "{\"method\":\"$METHOD\",\"num_speculative_tokens\":3}"
run_arm off_B ""
stop_bench
log "########## CAMPAIGN DONE -- results in $OUT ##########"
missing=0
for a in off_A k1 k2 k3 off_B; do
for r in $(seq 1 "$REPS"); do
[ -s "$OUT/res-$a-rep$r.json" ] || { log " MISSING RESULT: $a rep$r"; missing=$((missing+1)); }
done
done
if [ "$missing" -gt 0 ]; then
log "########## CAMPAIGN INCOMPLETE -- $missing missing result(s); DO NOT treat as a finished run ##########"
else
log "########## CAMPAIGN DONE -- results in $OUT ##########"
fi
# Deliberately does NOT touch the production seat. It is live on GPU 2 / :8022 and
# serving the operator's `gen-large` traffic throughout; this campaign only ever
# created and removed `fn-mtp-bench` on GPU 3.