The previous fix polled `docker ps -a` until the bench container's name
disappeared. That probe is useless here and the fix was a no-op: measured
2026-09-13, the container stops being listed while the daemon still holds the
name reservation, so the poll returns false early and `docker run` hits a
Conflict naming a container ID that `docker inspect` already reports as
'no such object'. Arm k1 was lost twice this way.
Two changes, each aimed at something actually observed:
- stop_bench waits on GPU memory falling below 1000 MiB rather than on the
container listing. That is the resource the next container genuinely needs,
and nvidia-smi reports it truthfully.
- boot() retries `docker run` while stderr matches 'already in use', up to 12
attempts at 5s. The daemon's name lag is transient, and forfeiting an arm
over it is silent data loss -- run_arm turns a failed boot into a skipped
arm that still lets the campaign report success.
Both probes verified against real artifacts before deploy: grep -c
'already in use' on the captured k1.runerr returns 1, and the nvidia-smi
query returns a bare integer that compares correctly under [ -gt ]. The
earlier fix passed bash -n and was still inert, so syntax is not the check
that matters here.
`docker rm -f` returns once removal is INITIATED, not once complete. The
bench container holds ~92 GB of GPU memory plus the offloaded PLE mapping, so
the name stays reserved for several seconds while the device is released. The
next `docker run --name` then dies with a name Conflict.
That failure was near-invisible: run_arm treats a failed boot as
`arm SKIPPED`, the campaign continues, and it still prints CAMPAIGN DONE.
Observed 2026-09-13 -- arm k1 lost the race after off_A's loaded container,
while k2 won it only because k1 had never started a container to tear down.
Every arm that follows one which actually ran is exposed, so k3 and off_B were
both on track to vanish from a run that would have reported success.
stop_bench now polls until the name is released (120s ceiling, warns and
continues). Added a completeness gate: the campaign asserts every expected
res-<arm>-rep<n>.json exists and reports CAMPAIGN INCOMPLETE naming each
missing result rather than DONE, so a gutted run cannot look like a clean one.
Verified the docker ps format string empirically -- an earlier draft nested
quotes so the template rendered as '{{.Names}}' with literal quotes, which
grep -qx could never match, making the wait a no-op that passed bash -n.
power.log lines begin '<HH:MM:SS> 0, <W> W, ...', so splitting the first
pipe-field on ', ' yields a[1] = '20:18:00 0' rather than '0'. GPU 0 was
therefore keyed by sample timestamp, emitting one bogus row per sample and
no recoverable peak, while GPUs 1-3 aggregated correctly.
Take the GPU index as the last whitespace-separated token of a[1]. Verified
against a two-sample fixture: the old parser emits a spurious row per
timestamp, the patched one reports 0/1/2/3 once each at the correct peaks.
The end-of-campaign summary is the GPU-side input to the fv-ml1 circuit
budget, so a silently wrong GPU 0 row is a measurement fault, not cosmetic.
The in-flight campaign still runs the old on-host copy (editing a running
bash script corrupts execution by byte offset); its summary will be
recomputed from the raw power.log.
Operator-directed so the campaign could run in parallel with live gen-large testing.
Strips the production-restore tail entirely -- the only container the driver can create
or remove is fn-mtp-bench, and the sole remaining compose call is a read-only
config --format json to derive argv. Adds per-card power+memory sampling every 10 s to
power.log, because two cards under load was the risk being accepted and a record beats
an argument.
That power log is now the only surviving measurement of what the load drew before
Fountain Valley went dark; it lives on /tank rather than in the container.
Operator-directed: raise context to the model's native maximum and take as much KV
as the card safely allows, and expose the seat through LiteLLM as `gen-large`.
max_model_len 131,072 -> 262,144
KV cache 8.76 -> 14.00 GiB (332,721 -> 560,654 tokens)
concurrency 2.54x@128K -> 2.14x@262K
⚠ 16.00 GiB WAS TRIED FIRST AND IS TOO AGGRESSIVE. A 155,497-token non-repeating
prefill drove GPU 2 to 97,074 of 97,887 MiB and the caching allocator logged "OOM on
device 0 while trying to allocate 488636416 bytes (free: 422117376)" -- 466 MiB wanted
against 403 MiB free. The request completed, so nothing failed visibly; that is one
step before the shape that crashed stacks/mog-sec twice on 2026-09-10 (~1.04 GiB
wanted, ~600 MB free). Backed off to 14.00 GiB, which re-probes clean: zero allocator
warnings, a 155,557-token prefill in 14.2 s, and 2,085 MiB still free at peak.
The reason the first estimate was wrong is worth keeping, because it is not obvious
and it inverts the usual advice: --kv-cache-memory makes vLLM SKIP MEMORY PROFILING
ENTIRELY and ignore --gpu-memory-utilization. The profiler was the thing accounting
for deep-prefill activation, so pinning bytes switched off the protection that the
pin was supposed to formalise. vLLM's own "--kv-cache-memory=18745235968 (17.46 GiB)
to fully utilize gpu memory" line is computed from a profile measured at
max-num-batched-tokens depth and sits 3.5 GiB above what a 150K-token request
survives; open #54764 compounds it, since PLE short-conv prefill pads every request
in a batch to the batch-MAX query length.
max-num-batched-tokens stays at 8192 -- it is what bounds the activation peak, and
doubling max_model_len left the profiled peak unchanged at 1.65 GiB precisely because
the peak tracks chunk size, not context length.
Gateway: `gen-large` added to the LiteLLM model_list, pointing at fv-ml1:8022. One
alias on purpose -- a single alias cannot trip the shared-config enable_thinking
mutation footgun, which needs two over the same (model, api_base). Sampling is the
checkpoint's own declared set (temp 1.0 / top_p 0.95 / top_k 20); presence_penalty,
min_p and repetition_penalty are left unset because the checkpoint declares no
canonical value for them. Verified registered for both the infra-ops admin key and
the shared all-agents key, since a new model behind a scoped allowlist 403s silently.
Also adds services/flash-next-mtp-bench/ -- the MTP measurement campaign and its
rationale. MTP stays off, but on "not yet measured here" rather than on vLLM's
4xH100 recipe number, which is a cross-harness comparison and not evidence about a
TP=1 Blackwell seat.
vh