fish-cpp: switch to resident s2 server + proxy shim — fix per-request CUDA init dominating wall time
Subprocess-per-request architecture forced CUDA + model load on every /v1/tts call (~10-20s init, then 5-15s generation). Even though CUDA is now actually being used (`-c 0` fix landed), 32s for "Verify." proved per-request init was the bottleneck. s2.cpp ships a built-in HTTP server (`--server -H -P`) that keeps the model resident on the GPU. Refactor: * entrypoint.sh — backgrounds `s2 --server -P 3030 -c 0 -m ... -t ...`, waits for it to bind 3030, then foregrounds uvicorn. tini supervises via `wait -n` so either child dying takes down the container. * server.py — drops subprocess.run; instead httpx-POSTs Fish-shaped /v1/tts JSON to s2's localhost:3030/generate (multipart form: text + optional prompt_text/prompt_audio for cloning). Model load + CUDA init now happen once at container start, not per-request. * Dockerfile — added httpx (shim dep), curl (entrypoint readiness probe), and the entrypoint.sh COPY+chmod. CMD now invokes entrypoint.sh instead of uvicorn directly. * deploy-fish-cpp.yaml — uploads entrypoint.sh alongside server.py.
This commit is contained in:
@@ -66,6 +66,12 @@ steps:
|
||||
dest: "{{ compose_dir }}/server.py"
|
||||
mode: "0644"
|
||||
|
||||
- name: Upload entrypoint.sh (starts s2 server + uvicorn shim)
|
||||
upload:
|
||||
src: stacks/fish-cpp/entrypoint.sh
|
||||
dest: "{{ compose_dir }}/entrypoint.sh"
|
||||
mode: "0755"
|
||||
|
||||
- name: Seed .env from template (only if absent)
|
||||
upload:
|
||||
src: stacks/fish-cpp/.env.example
|
||||
|
||||
Reference in New Issue
Block a user