58 lines
4.2 KiB
Markdown
58 lines
4.2 KiB
Markdown
# `[2026-09-15]` A client timeout SOMETIMES cancels a vLLM generation and sometimes does not — the boundary is unknown
|
||
|
||
⚠⚠ **DO NOT carry "a client-side timeout is not a cancellation" as a rule. It is FALSE as
|
||
stated, and it was disproved by the peer who coined it, on our own seat, within the hour.**
|
||
`tts-dev` orphaned six unbounded generations on `vllm-erp-seat` (fv-ml1 GPU 1) by firing
|
||
`char-rp-fast` probes with no `max_tokens` and letting clients time out at 110 s / 115 s /
|
||
600 s. They wrote the lesson up, then **controlled their own detector and the POSITIVE
|
||
CONTROL FAILED** — chasing it produced this, measured against the live seat:
|
||
|
||
t+1.6s running=1 kv=0.4% request reaches the engine
|
||
client gave up (urlopen timeout=2)
|
||
t+3.1s running=1 kv=0.8% still generating
|
||
t+7.8s running=0 kv=0.0% CANCELLED, unprompted, ~6s after the client left
|
||
|
||
**A clean client abandon DOES propagate.** Yet six requests genuinely orphaned — I observed
|
||
that independently. **So some abandons propagate and some do not, and nobody has isolated
|
||
the boundary.** Unseparated candidates: SIGTERM'd process vs clean client-side timeout;
|
||
multi-minute unbounded generation vs short one; several stacked at once. ⭐ **That unknown
|
||
is the argument FOR a detector and AGAINST a rule — a rule needs the boundary, a detector
|
||
just looks.** tts-dev holds a standing request: if we ever isolate what makes an abandon
|
||
stick, tell them; it is the input that would let them build a real positive control (theirs
|
||
is SYNTHETIC and their file says so in place — detection logic proven, reproduction of the
|
||
underlying bug not).
|
||
|
||
⭐⭐ **THE DISCRIMINATOR, and it is the durable artifact of the day: a serving engine's KV
|
||
cache CYCLES; an orphaned one only CLIMBS.** Request count and throughput are **ambiguous**
|
||
between a loaded seat and a wedged one — I read `vllm-erp-seat` twice off those signals and
|
||
called it healthy both times, correctly on the evidence (39 completions/hour, 210–290 tok/s,
|
||
`Running: 3 / Waiting: 3`, KV cycling 70→99→70%). The traffic was genuinely real; it then
|
||
*ended*, and what remained were orphans. The tell was `prompt throughput 0.0` sustained,
|
||
`Waiting: 0`, and KV **monotonic** 87.4 → 87.9 → 88.4 → 88.9 → 89.4. Now implemented in
|
||
`tts-stack tools/engine_guard.py --watch` (`db9d847`). vLLM serves `/metrics`
|
||
**unauthenticated** on the seat ports, so `num_requests_running`, `num_requests_waiting` and
|
||
`kv_cache_usage_perc` are directly pollable — no gateway, no auth. ⚠ Its `settle` defaults
|
||
to 20 s so normal cancellation lag is not reported as a leak: a guard that cries wolf gets
|
||
disabled, and then you are back to a docstring.
|
||
|
||
⚠ **A `max_tokens` ceiling would NOT have prevented this.** tts-dev's worst offender ran
|
||
with `max_tokens=16384` **explicitly set**, hit it exactly, and returned 24,594 characters
|
||
of whitespace wrapping a correct three-field answer. **A ceiling bounds how long you wait
|
||
for the failure, not whether it happens.** Escalated to the operator anyway as a two-layer
|
||
choice (gateway-side LiteLLM default — one blast radius, misses direct-to-seat callers;
|
||
vs per-seat limits — catches everything, nine seats to touch); gateway first and measure
|
||
what it breaks is the right order. Related: [[feedback_detector_after_reflex_beats_reminder_before]].
|
||
|
||
**Remediation**: `docker restart vllm-erp-seat` 23:36:31 UTC, healthy in ~1 min, GPU 1
|
||
100% / 275 W (at the cap) / 74°C → 0% / 4.8 W / 42°C. The five other tenants on that card
|
||
(`vllm-reward`, `vllm-rerank-a3`, `vllm-embed`, `vllm-coder`, `vllm-meromero-rp`) were
|
||
untouched. Restarted rather than waiting — they DO self-terminate at the context limit and
|
||
one dropped off mid-diagnosis (6→5, KV 89.4→86.8) — because KV at 89% and climbing starts
|
||
costing the co-tenants through preemption.
|
||
|
||
⚠ **Noticed in passing, unresolved: `vllm-erp-seat` and `vllm-meromero-rp` advertise the
|
||
SAME `--served-model-name`** (`G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16`). Fine
|
||
if it is deliberate replication for throughput; it is also the exact shape that makes
|
||
gateway routing ambiguous and "which seat served this?" unanswerable after the fact.
|
||
Surfaced to the operator, not yet answered.
|