fix(mtp-bench): give each arm its own container name instead of tuning teardown waits
Container teardown latency here is unpredictable, not merely slow. Measured 2026-09-13 on the same container in the same session: once ~55s, once 0s. Removing a container holding ~92 GB of GPU memory plus the offloaded PLE mapping leaves an Exited record owning its name for that whole window, so any fixed wait or retry budget is a guess that will eventually be wrong -- a 12x5s retry lost arm k1 by roughly two attempts. Arms now use $NAME_BASE-$arm, set in boot() after stop_bench has torn down the previous arm. Names are never reused, so the collision is impossible by construction and teardown latency stops being load-bearing. cleanup() reaps every fn-mtp-bench-* container at exit. stop_bench keeps the GPU-memory wait -- the next container genuinely needs the device, and unlike the container record nvidia-smi clears promptly and reports truthfully. It no longer waits on the container listing at all. Three earlier hypotheses were wrong and are recorded so they are not retried: the name is not held by a phantom (the container is present and Exited), the removal does not fail (rm -f succeeds; it is asynchronous), and GPU memory release does not track name release (memory frees first, by a wide margin).
This commit is contained in:
@@ -70,7 +70,8 @@ MAXTOK=${MAXTOK:-400}
|
|||||||
RPS=${RPS:-4} # requests per stream, per concbench
|
RPS=${RPS:-4} # requests per stream, per concbench
|
||||||
METHOD=${METHOD:-qwen4_exp_mtp}
|
METHOD=${METHOD:-qwen4_exp_mtp}
|
||||||
MODEL=${MODEL:-qwen3.8-flash-next-uncensored}
|
MODEL=${MODEL:-qwen3.8-flash-next-uncensored}
|
||||||
NAME=fn-mtp-bench
|
NAME_BASE=fn-mtp-bench
|
||||||
|
NAME="" # set per-arm by boot(); never reused, so the name can never collide
|
||||||
BOOT_TIMEOUT=${BOOT_TIMEOUT:-2400}
|
BOOT_TIMEOUT=${BOOT_TIMEOUT:-2400}
|
||||||
|
|
||||||
mkdir -p "$OUT"
|
mkdir -p "$OUT"
|
||||||
@@ -125,34 +126,39 @@ import json; print(json.load(open('$OUT/resolved-compose.json'))['services']['vl
|
|||||||
log "image=$IMAGE model=$MODEL_DIR"
|
log "image=$IMAGE model=$MODEL_DIR"
|
||||||
|
|
||||||
stop_bench(){
|
stop_bench(){
|
||||||
sudo -n docker rm -f "$NAME" >/dev/null 2>&1 || true
|
# Remove the PREVIOUS arm's container (if any). Deliberately does NOT wait for
|
||||||
# `docker rm -f` returns once removal is INITIATED, not once it is done. A
|
# the container record to disappear: measured 2026-09-13, tearing down a
|
||||||
# container holding ~92 GB of GPU memory plus the offloaded PLE mapping keeps
|
# container holding ~92 GB of GPU memory plus the offloaded PLE mapping leaves
|
||||||
# the name reserved for several seconds while the device is released, so the
|
# an Exited record holding its name for anywhere between 0s and ~60s. That
|
||||||
# next `docker run --name` loses a race and dies with a name Conflict. That
|
# latency is unpredictable, so arms get unique names (see boot) and the record
|
||||||
# surfaces as `arm SKIPPED (boot failed)` -- an entire arm silently missing
|
# is simply left to be reaped by cleanup at the end.
|
||||||
# from a campaign that still reports DONE. Observed 2026-09-13 on arm k1.
|
[ -n "${NAME:-}" ] && sudo -n docker rm -f "$NAME" >/dev/null 2>&1 || true
|
||||||
# NOTE: polling `docker ps -a` for the name is USELESS -- verified 2026-09-13.
|
# Do wait on GPU memory: that IS what the next container needs, and unlike the
|
||||||
# The container stops being listed while the daemon still holds the name
|
# container record it clears promptly and reports truthfully.
|
||||||
# reservation, so the poll goes false early and the next `docker run` still
|
|
||||||
# hits a Conflict against an ID that no longer exists. Wait instead on the
|
|
||||||
# resource that actually has to be free: the GPU's memory.
|
|
||||||
local t0=$SECONDS
|
local t0=$SECONDS
|
||||||
while [ "$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i "$GPU" 2>/dev/null || echo 0)" -gt 1000 ]; do
|
while [ "$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i "$GPU" 2>/dev/null || echo 0)" -gt 1000 ]; do
|
||||||
if [ $((SECONDS-t0)) -ge 120 ]; then
|
if [ $((SECONDS-t0)) -ge 180 ]; then
|
||||||
log " WARN: GPU $GPU still holding memory 120s after rm -f; continuing anyway"
|
log " WARN: GPU $GPU still holding memory after 180s; continuing anyway"
|
||||||
break
|
break
|
||||||
fi
|
fi
|
||||||
sleep 2
|
sleep 2
|
||||||
done
|
done
|
||||||
}
|
}
|
||||||
cleanup(){ stop_bench; [ -n "${POWER_PID:-}" ] && kill "$POWER_PID" 2>/dev/null; }
|
cleanup(){
|
||||||
|
stop_bench
|
||||||
|
# reap every per-arm container this campaign created
|
||||||
|
for n in $(sudo -n docker ps -a --format '{{.Names}}' 2>/dev/null | grep "^${NAME_BASE}-" || true); do
|
||||||
|
sudo -n docker rm -f "$n" >/dev/null 2>&1 || true
|
||||||
|
done
|
||||||
|
[ -n "${POWER_PID:-}" ] && kill "$POWER_PID" 2>/dev/null
|
||||||
|
}
|
||||||
trap 'cleanup' EXIT
|
trap 'cleanup' EXIT
|
||||||
|
|
||||||
boot(){ # boot <arm-label> [spec-json]
|
boot(){ # boot <arm-label> [spec-json]
|
||||||
local arm="$1"; shift
|
local arm="$1"; shift
|
||||||
local extra=(); [ $# -gt 0 ] && [ -n "${1:-}" ] && extra=(--speculative-config "$1")
|
local extra=(); [ $# -gt 0 ] && [ -n "${1:-}" ] && extra=(--speculative-config "$1")
|
||||||
stop_bench
|
stop_bench
|
||||||
|
NAME="${NAME_BASE}-${arm}" # unique per arm: collision is impossible by construction
|
||||||
log "=== boot arm=$arm ${extra[*]:-(no spec-config)}"
|
log "=== boot arm=$arm ${extra[*]:-(no spec-config)}"
|
||||||
# The daemon can still hold the name after the container is gone from
|
# The daemon can still hold the name after the container is gone from
|
||||||
# `docker ps -a`, so a Conflict here is transient -- retry rather than
|
# `docker ps -a`, so a Conflict here is transient -- retry rather than
|
||||||
|
|||||||
Reference in New Issue
Block a user