Three deploy iterations + four backend attempts (subprocess CUDA,
resident-server CUDA, Vulkan rebuild) all failed to deliver speedup
over fish-s2:
* CUDA path: ggml_cuda_init succeeded, weights loaded onto GPU per
s2's logs, but nvidia-smi showed 0% utilization during synthesis.
Wall time 20s/long phrase vs fish-s2's 7.5s. The "CUDA get_rows
unsupported for type q6_K" warning hints at incomplete op coverage
in s2.cpp's alpha CUDA backend for fish-speech architecture.
* Vulkan path: vk::IncompatibleDriverError on container init. NVIDIA
Vulkan ICD not accessible inside the container despite
NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics. Would need
host-side nvidia-utils-vulkan installation or manual ICD bind
mount. Didn't pursue.
Both are fixable — CUDA needs op coverage upstream (author actively
working on it; "selective embedding dequant" commit landed 16 days
ago), Vulkan needs host-side ICD setup. Neither is a config-flip,
both are real work for marginal-or-zero return. Better to delete the
stack and revisit when s2.cpp matures or when we tackle FP8
quantization on ana-ml2's RTX 6000 Ada (sm_89, native FP8 hardware).
Local image rmi'd, /opt/docker/compose/fish-cpp removed on irv-ml1.
/worktank/fish-cpp left for user-side sudo cleanup.
Future Fish acceleration paths (in order of decreasing certainty):
1. Wait for s2.cpp CUDA op coverage to mature (track upstream commits).
2. Quantize Fish BF16 → FP8 via TransformerEngine, deploy on
ana-ml2's RTX 6000 Ada (Ada has native FP8 tensor cores, A6000
doesn't). ~2x speedup if it works.
3. vLLM port of Fish (no upstream support today).
vh
0ba41e02ea