vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(parakeet-nemo): rollback is compose up of the kept sherpa stack (old container removed to clear a stale Homepage entry)

Browse Source
This commit is contained in:
vh
2026-10-17 00:10:18 -07:00
parent b0235a1e46
commit 4ddd72f22e
2 changed files with 4 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+1 -1
View File
@@ -161,7 +161,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01) ### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)
- **Incident 2026-10-01 04:21, fixed by nemo-0.1.1** (infra-hermes 963f9ed). gen-small OOM'd on a lazily allocated workspace while the seat held its window cache. The fix: `empty_cache` around every window, `MEM_CAP_MIB=3840` (a hard ceiling), `CUDA_GRAPHS=0` (empty_cache poisons the graph pool). The seat rests at 2,108 MiB and peaks at 3,028 on a 12-min file. **Lesson: a shared-card tenant must carry a HARD cap, whether or not it returns memory.** Prime kept graphs OFF. His 10-01 "no insurance trim" on gen-small was superseded on 10-05 (KV 8 → 5.5 GiB) once the GPU 0 plateau turned out to be ~60 MiB free, not the ~1.05 GB read just after a restart. - **Incident 2026-10-01 04:21, fixed by nemo-0.1.1** (infra-hermes 963f9ed). gen-small OOM'd on a lazily allocated workspace while the seat held its window cache. The fix: `empty_cache` around every window, `MEM_CAP_MIB=3840` (a hard ceiling), `CUDA_GRAPHS=0` (empty_cache poisons the graph pool). The seat rests at 2,108 MiB and peaks at 3,028 on a 12-min file. **Lesson: a shared-card tenant must carry a HARD cap, whether or not it returns memory.** Prime kept graphs OFF. His 10-01 "no insurance trim" on gen-small was superseded on 10-05 (KV 8 → 5.5 GiB) once the GPU 0 plateau turned out to be ~60 MiB free, not the ~1.05 GB read just after a restart.
- **Rollback:** `docker stop parakeet-nemo && docker start parakeet` (old container and image kept). Seat invariants (in its README): cast to bf16 AFTER change_attention_model; uvicorn pinned with `--http h11` (httptools 0.8.0 emits `HTTP/1.1 200\x00OK`, which LiteLLM/httpx rejects). NVIDIA Open Model License accepted for internal use. → `persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md` - **Rollback:** `docker stop parakeet-nemo`, then `docker compose up -d` in `/opt/docker/compose/parakeet` (the old container was removed 2026-10-17 to clear a stale Homepage entry; the image and compose dir are kept). Seat invariants (in its README): cast to bf16 AFTER change_attention_model; uvicorn pinned with `--http h11` (httptools 0.8.0 emits `HTTP/1.1 200\x00OK`, which LiteLLM/httpx rejects). NVIDIA Open Model License accepted for internal use. → `persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md`
### Worldtree U11 memory cutover (demo + personal) ### Worldtree U11 memory cutover (demo + personal)
stacks/parakeet-nemo/README.md
+3 -2
View File
@@ -69,8 +69,9 @@ headroom.
## Rollback ## Rollback
The old seat was STOPPED, not removed: `docker stop parakeet-nemo && docker start parakeet` `docker stop parakeet-nemo && (cd /opt/docker/compose/parakeet && docker compose up -d)` restores the sherpa
restores the sherpa seat on :8300 exactly as before (container and image both kept). seat on :8300. Its container was REMOVED 2026-10-17, because its labels kept a stale "Parakeet ASR" on Homepage.
The image `local/parakeet:sherpa-onnx-v4` and the compose dir are kept, so `compose up` recreates it as it was.
## Deploy ## Deploy