From 7cc847f33c86a3f7382423cc14f45e9669aa4cd8 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 15 Sep 2026 16:49:08 -0700 Subject: [PATCH] memory: client abandon sometimes cancels and sometimes does not; the KV trajectory is the tell --- ...15-client-abandon-cancellation-boundary.md | 57 +++++++++++++++++++ persistent-memory.md | 1 + 2 files changed, 58 insertions(+) create mode 100644 persistent-memory.d/2026-09-15-client-abandon-cancellation-boundary.md diff --git a/persistent-memory.d/2026-09-15-client-abandon-cancellation-boundary.md b/persistent-memory.d/2026-09-15-client-abandon-cancellation-boundary.md new file mode 100644 index 0000000..18091e5 --- /dev/null +++ b/persistent-memory.d/2026-09-15-client-abandon-cancellation-boundary.md @@ -0,0 +1,57 @@ +# `[2026-09-15]` A client timeout SOMETIMES cancels a vLLM generation and sometimes does not — the boundary is unknown + +⚠⚠ **DO NOT carry "a client-side timeout is not a cancellation" as a rule. It is FALSE as +stated, and it was disproved by the peer who coined it, on our own seat, within the hour.** +`tts-dev` orphaned six unbounded generations on `vllm-erp-seat` (fv-ml1 GPU 1) by firing +`char-rp-fast` probes with no `max_tokens` and letting clients time out at 110 s / 115 s / +600 s. They wrote the lesson up, then **controlled their own detector and the POSITIVE +CONTROL FAILED** — chasing it produced this, measured against the live seat: + + t+1.6s running=1 kv=0.4% request reaches the engine + client gave up (urlopen timeout=2) + t+3.1s running=1 kv=0.8% still generating + t+7.8s running=0 kv=0.0% CANCELLED, unprompted, ~6s after the client left + +**A clean client abandon DOES propagate.** Yet six requests genuinely orphaned — I observed +that independently. **So some abandons propagate and some do not, and nobody has isolated +the boundary.** Unseparated candidates: SIGTERM'd process vs clean client-side timeout; +multi-minute unbounded generation vs short one; several stacked at once. ⭐ **That unknown +is the argument FOR a detector and AGAINST a rule — a rule needs the boundary, a detector +just looks.** tts-dev holds a standing request: if we ever isolate what makes an abandon +stick, tell them; it is the input that would let them build a real positive control (theirs +is SYNTHETIC and their file says so in place — detection logic proven, reproduction of the +underlying bug not). + +⭐⭐ **THE DISCRIMINATOR, and it is the durable artifact of the day: a serving engine's KV +cache CYCLES; an orphaned one only CLIMBS.** Request count and throughput are **ambiguous** +between a loaded seat and a wedged one — I read `vllm-erp-seat` twice off those signals and +called it healthy both times, correctly on the evidence (39 completions/hour, 210–290 tok/s, +`Running: 3 / Waiting: 3`, KV cycling 70→99→70%). The traffic was genuinely real; it then +*ended*, and what remained were orphans. The tell was `prompt throughput 0.0` sustained, +`Waiting: 0`, and KV **monotonic** 87.4 → 87.9 → 88.4 → 88.9 → 89.4. Now implemented in +`tts-stack tools/engine_guard.py --watch` (`db9d847`). vLLM serves `/metrics` +**unauthenticated** on the seat ports, so `num_requests_running`, `num_requests_waiting` and +`kv_cache_usage_perc` are directly pollable — no gateway, no auth. ⚠ Its `settle` defaults +to 20 s so normal cancellation lag is not reported as a leak: a guard that cries wolf gets +disabled, and then you are back to a docstring. + +⚠ **A `max_tokens` ceiling would NOT have prevented this.** tts-dev's worst offender ran +with `max_tokens=16384` **explicitly set**, hit it exactly, and returned 24,594 characters +of whitespace wrapping a correct three-field answer. **A ceiling bounds how long you wait +for the failure, not whether it happens.** Escalated to the operator anyway as a two-layer +choice (gateway-side LiteLLM default — one blast radius, misses direct-to-seat callers; +vs per-seat limits — catches everything, nine seats to touch); gateway first and measure +what it breaks is the right order. Related: [[feedback_detector_after_reflex_beats_reminder_before]]. + +**Remediation**: `docker restart vllm-erp-seat` 23:36:31 UTC, healthy in ~1 min, GPU 1 +100% / 275 W (at the cap) / 74°C → 0% / 4.8 W / 42°C. The five other tenants on that card +(`vllm-reward`, `vllm-rerank-a3`, `vllm-embed`, `vllm-coder`, `vllm-meromero-rp`) were +untouched. Restarted rather than waiting — they DO self-terminate at the context limit and +one dropped off mid-diagnosis (6→5, KV 89.4→86.8) — because KV at 89% and climbing starts +costing the co-tenants through preemption. + +⚠ **Noticed in passing, unresolved: `vllm-erp-seat` and `vllm-meromero-rp` advertise the +SAME `--served-model-name`** (`G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16`). Fine +if it is deliberate replication for throughput; it is also the exact shape that makes +gateway routing ambiguous and "which seat served this?" unanswerable after the fact. +Surfaced to the operator, not yet answered. diff --git a/persistent-memory.md b/persistent-memory.md index fd42b29..74b2a09 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -135,6 +135,7 @@ _As of 2026-09-15 ~09:45 PT._ ## Recent decisions +- `[2026-09-15]` ⚠⚠ **DO NOT carry "a client-side timeout is not a cancellation" as a rule — it is FALSE as stated.** A clean abandon cancels itself ~6 s later (measured); yet six requests genuinely orphaned on `vllm-erp-seat`. Some propagate, some do not, **boundary unknown** — which argues for a detector, not a rule. ⭐⭐ The durable artifact: **a serving engine's KV cache CYCLES, an orphaned one only CLIMBS** — request count and throughput are ambiguous between loaded and wedged, and I called the seat healthy twice off them (correctly, on the evidence). ⚠ A `max_tokens` ceiling would NOT have prevented it: the worst offender had 16384 set, hit it, and returned 24,594 chars of whitespace. → `persistent-memory.d/2026-09-15-client-abandon-cancellation-boundary.md` - `[2026-09-15]` ⚠⚠ **`--gpu-memory-utilization` DOES NOT PREDICT RESIDENT VRAM — measure it, never compute it.** Wrong in **both** directions on fv-ml1: `vllm-cyberprev` util 0.40 (expect ~39,155 MiB) holds **47,124** (+8 GB over); `vllm-gen-small` util 0.48 (expect ~46,986) holds **36,942** (−10 GB under). Planning a placement off the fractions would have been 8 GB wrong. Read `nvidia-smi --query-compute-apps`. Full per-seat residency table + the breeze shuffle arithmetic → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md` - `[2026-09-15]` **breeze-tts stays on irv-ml1; the TTS-stack move to fv-ml1 is PARKED (park id 75, `move-the-tts-stack-breeze-tts-bragi-tts-gateway`), triggered on evacuating embed/rerank/reward.** ⚠ Trigger as stated says "gpu0" but those three are on **GPU 1** (~0.16 util, ~15.7 GB; GPU 1 is the tight card at 0.975 / 4,336 MiB free) — confirm which he meant before executing. All three services move together because only `breeze-tts` is GPU-resident (~10.3 GiB, **growing**) while `bragi` and `tts-gateway` are CPU proxies, and co-location is what avoids a cross-site hop per TTS call. **breeze-tts sizing — original recommendation NOT to move it.** ~**10.3 GiB** measured under load at 53 min uptime, **up from 9.2 GiB** shortly after warm-up (it grows; n=2, plateau unmeasured) — so GPU 0's 11,982 MiB free is a **1.7 GB margin and shrinking**, on the live chat serving path. ⚠ Two measurement traps: it reports **nothing at idle on the wrong card** (`BREEZE_GPU_DEVICES=0` = the **3090**, not the A6000), and an early reading understates it. ⭐ The real objection is **topology**: `tts-gateway` is on irv-ml1 and reaches it same-box, so moving breeze alone adds a cross-site hop to every TTS call against a 478 ms first-sample budget. GPU 3 would fit it but spends the reserve. → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`