feat(nh3-ml1): LFM2.5-VL-3B (llama.cpp) + VibeVoice-ASR-Streaming-1.5B (audio.cpp) utility seats
For brokkr's dataset foundry (operator-approved 2026-09-26, relayed). - stacks/lfm-vl-seat: llama.cpp server-cuda b11176 (digest-pinned), Q5_K_M + mmproj Q8_0, :8030; gateway alias lfm25-vl-3b (LiteLLM restarted, 36 s). Positive control exact; null control shows it describes a missing image. - stacks/vibevoice-asr-seat: audio.cpp v0.8.2-audio8-perf-hotfix (the GGUF's own runtime, not vibevoice.cpp) on cuda 12.8 runtime + libgomp + libsoxr, sha256-pinned; :8031 direct. LibriSpeech WER 3/69, RTF 0.07-0.14; ~31 s cold first request.
This commit is contained in:
@@ -0,0 +1,14 @@
|
||||
# lfm-vl-seat tunables (nh3-ml1). Copy to `.env` on the server.
|
||||
|
||||
# llama.cpp CUDA server, pinned by digest: build b11176 (commit f805c57a2),
|
||||
# published 2026-09-25. Ada sm_89 is in the CUDA build's target set.
|
||||
LLAMACPP_IMAGE=ghcr.io/ggml-org/llama.cpp@sha256:1f4b9cf58982dd4d7cc497aea31b1a456ca9a3a1f94f527d317d3fdee0d60ab6
|
||||
|
||||
VL_PORT=8030
|
||||
VL_ALIAS=lfm25-vl-3b
|
||||
VL_MODEL_FILE=LFM2.5-VL-3B-Q5_K_M.gguf
|
||||
VL_MMPROJ_FILE=mmproj-LFM2.5-VL-3B-Q8_0.gguf
|
||||
|
||||
# 2 parallel slots sharing 16k context (8k each): room for a few images plus text.
|
||||
VL_CTX=16384
|
||||
VL_PARALLEL=2
|
||||
@@ -0,0 +1,23 @@
|
||||
# lfm-vl-seat
|
||||
|
||||
**LiquidAI LFM2.5-VL-3B**, a small vision-language model, on **nh3-ml1**, served
|
||||
by llama.cpp (`ghcr.io/ggml-org/llama.cpp` server-cuda, build b11176, pinned by
|
||||
digest) on `:8030`. A utility seat for brokkr's dataset foundry (image
|
||||
understanding). Operator-approved 2026-09-26, relayed by brokkr-smithy-dev.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| direct | `http://10.100.50.80:8030/v1/chat/completions`, model `lfm25-vl-3b` |
|
||||
| gateway | `lfm25-vl-3b` (ana-docker:4000, `hosted_vllm/`, `supports_vision`) |
|
||||
| files | `LFM2.5-VL-3B-Q5_K_M.gguf` + **`mmproj-LFM2.5-VL-3B-Q8_0.gguf` (required for images)** from `LiquidAI/LFM2.5-VL-3B-GGUF` @ `6f730e9a2c45` |
|
||||
| VRAM | ~2.9 GB (2 slots × 8k ctx) |
|
||||
|
||||
**Checks, 2026-09-26.**
|
||||
- Positive control: a synthetic image (red square, blue circle, the text
|
||||
"NH3 42") was described exactly at T=0, both direct and through the gateway.
|
||||
- Speed: prefill ~800 tok/s, decode ~92 tok/s.
|
||||
- ⚠ **Null control: with no image attached it confidently describes one anyway**
|
||||
("a solid black background"). Callers must make sure the image actually went in.
|
||||
|
||||
Not wired yet: `LiquidAI/LFM2.5-VL-3B-DSpark-GGUF`, a speculative-decoding
|
||||
drafter (brokkr's "optional later").
|
||||
@@ -0,0 +1,62 @@
|
||||
# lfm-vl-seat — LiquidAI LFM2.5-VL-3B (vision-language) on nh3-ml1 (CT 109 on
|
||||
# nh3-pve, RTX 2000E Ada 16 GB), llama.cpp server. Utility seat for brokkr's
|
||||
# dataset foundry (image understanding), operator-approved 2026-09-26 (relayed
|
||||
# by brokkr-smithy-dev).
|
||||
#
|
||||
# llama-server LFM2.5-VL-3B Q5_K_M + mmproj Q8_0 → /v1/chat/completions
|
||||
# (OpenAI, with image_url content parts) :8030
|
||||
#
|
||||
# ⚠ --mmproj is REQUIRED for image input; without it the model is text-only and
|
||||
# says nothing about it.
|
||||
# Batch work, not latency-critical: this card decodes ~5x slower than an fv-ml1
|
||||
# Blackwell (servers/nh3-ml1/README.md), which is fine for a foundry.
|
||||
#
|
||||
# Files (not in git): /opt/aimodels/gguf/lfm25-vl-3b/ from
|
||||
# LiquidAI/LFM2.5-VL-3B-GGUF @ 6f730e9a2c45.
|
||||
name: lfm-vl-seat
|
||||
services:
|
||||
llama-server:
|
||||
image: ${LLAMACPP_IMAGE}
|
||||
container_name: lfm-vl
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "${VL_PORT}:8080"
|
||||
volumes:
|
||||
- /opt/aimodels/gguf/lfm25-vl-3b:/models:ro
|
||||
command:
|
||||
- -m
|
||||
- /models/${VL_MODEL_FILE}
|
||||
- --mmproj
|
||||
- /models/${VL_MMPROJ_FILE}
|
||||
- --alias
|
||||
- ${VL_ALIAS}
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8080"
|
||||
- -ngl
|
||||
- "999"
|
||||
- -c
|
||||
- "${VL_CTX}"
|
||||
- -np
|
||||
- "${VL_PARALLEL}"
|
||||
- --jinja
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ["0"]
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 120s
|
||||
labels:
|
||||
- homepage.group=AI - Eval & Retrieval
|
||||
- homepage.name=VL — LFM2.5-VL-3B (llama.cpp, nh3-ml1)
|
||||
- homepage.icon=mdi-image-search
|
||||
- homepage.description=Image understanding for the dataset foundry (gateway lfm25-vl-3b)
|
||||
- homepage.href=http://10.100.50.80:${VL_PORT}
|
||||
@@ -512,6 +512,20 @@ model_list:
|
||||
model_info:
|
||||
mode: rerank
|
||||
|
||||
# --- lfm25-vl-3b → LiquidAI LFM2.5-VL-3B (Q5_K_M + mmproj Q8_0), llama.cpp server on
|
||||
# nh3-ml1 :8030 (stacks/lfm-vl-seat, 2026-09-26). Image understanding for
|
||||
# brokkr's dataset foundry; send images as OpenAI image_url parts. A small VLM:
|
||||
# with NO image attached it confidently describes one anyway (measured), so
|
||||
# callers must check the image actually went in. Batch-grade speed (~90 tok/s). ---
|
||||
- model_name: lfm25-vl-3b
|
||||
litellm_params:
|
||||
model: hosted_vllm/lfm25-vl-3b
|
||||
api_base: http://10.100.50.80:8030/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
model_info:
|
||||
mode: chat
|
||||
supports_vision: true
|
||||
|
||||
# --- coder-fast → Qwen2.5-Coder-1.5B (BASE), FIM code-completion seat (ana-ml2
|
||||
# GPU1 :8020, vLLM; deep-research pick 2026-07-27). For Zed editor inline
|
||||
# edit-predictions via the LEGACY /v1/completions endpoint with Qwen FIM
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
# vibevoice-asr-seat tunables (nh3-ml1). Copy to `.env` on the server.
|
||||
ASR_PORT=8031
|
||||
@@ -0,0 +1,16 @@
|
||||
# audio.cpp server (0xShug0/audio.cpp) for VibeVoice-ASR-Streaming on nh3-ml1.
|
||||
# The release's Linux CUDA build ("cuda12.8-colab") is a bare binary that needs the
|
||||
# CUDA 12 runtime libs (cudart, cublas, cufft, nccl) plus libgomp. This image
|
||||
# supplies them, plus libsoxr (without it audio.cpp falls back to linear
|
||||
# resampling for the 16k→24k step). Built locally on the host; not pushed.
|
||||
FROM nvidia/cuda:12.8.1-runtime-ubuntu22.04
|
||||
ARG AUDIOCPP_TAG=v0.8.2-audio8-perf-hotfix
|
||||
ARG AUDIOCPP_SHA256=ccfb35869d67e520801981392189a1ab8b6c829151093e0816ecd6a2d828d8c8
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends libgomp1 libsoxr0 curl ca-certificates \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
RUN curl -fsSL -o /tmp/a.tgz "https://github.com/0xShug0/audio.cpp/releases/download/${AUDIOCPP_TAG}/audio-${AUDIOCPP_TAG}-bin-ubuntu-x64-cuda12.8-colab.tar.gz" \
|
||||
&& echo "${AUDIOCPP_SHA256} /tmp/a.tgz" | sha256sum -c - \
|
||||
&& mkdir -p /app && tar xzf /tmp/a.tgz -C /app --no-same-owner && rm /tmp/a.tgz \
|
||||
&& chmod +x /app/audiocpp_server
|
||||
WORKDIR /app
|
||||
ENTRYPOINT ["/app/audiocpp_server"]
|
||||
@@ -0,0 +1,30 @@
|
||||
# vibevoice-asr-seat
|
||||
|
||||
**Microsoft VibeVoice-ASR-Streaming-1.5B** (Q4_K) on **nh3-ml1**, served by
|
||||
**audio.cpp** (`0xShug0/audio.cpp` v0.8.2-audio8-perf-hotfix, Linux CUDA 12.8
|
||||
build) on `:8031`, with direct access only. A utility seat for brokkr's dataset
|
||||
foundry (speech → text with inline speaker labels). Operator-approved 2026-09-26,
|
||||
relayed by brokkr-smithy-dev.
|
||||
|
||||
- `POST /v1/audio/transcriptions`: multipart `file=@x.wav`, `model=vibevoice-asr-streaming-1.5b`.
|
||||
Returns `{"text": " \n Speaker 0:...", "timing": {"rtf": ...}}`.
|
||||
- `POST /v1/audio/transcriptions/live?model=…&sample_rate=16000&channels=1&sample_format=s16le`
|
||||
takes chunked raw PCM.
|
||||
- `GET /v1/models`, `GET /health`.
|
||||
|
||||
**Why audio.cpp and not vibevoice.cpp or llama.cpp.** The
|
||||
`christopherthompson81/VibeVoice-ASR-Streaming-1.5B-GGUF` files are audio.cpp
|
||||
packages with their sidecars embedded, per the model card, so no separate
|
||||
tokenizer is needed. The release binary expects the CUDA 12 runtime libraries,
|
||||
so `Dockerfile` puts it on `nvidia/cuda:12.8.1-runtime` with a SHA-256-pinned
|
||||
download, plus `libgomp1` and **`libsoxr0`**. Without libsoxr it falls back to
|
||||
linear resampling (16k→24k), and on one clip that turned "cutter" into "country".
|
||||
|
||||
**Checks, 2026-09-26.** On the 4 LibriSpeech clips shipped with audio.cpp, WER
|
||||
is 3/69 = 4.35%, identical across 3 reps. Two of the three errors are "I'm" vs
|
||||
"I am" normalization. RTF is 0.07–0.14. ⚠ **The first request after a start
|
||||
takes ~31 s** (CUDA graph warmup); later ones take 0.3–1 s. VRAM ~2.0 GB.
|
||||
|
||||
The model card recommends Q8_0 (3.3 GB; its CUDA WER is 5.80% vs Q4_K's 7.25%,
|
||||
on 69 words). To switch, change `path` in `conf/server.json`, download the file,
|
||||
and recreate the container.
|
||||
@@ -0,0 +1,50 @@
|
||||
# vibevoice-asr-seat — Microsoft VibeVoice-ASR-Streaming-1.5B on nh3-ml1 (CT 109,
|
||||
# RTX 2000E Ada 16 GB), served by audio.cpp. Utility seat for brokkr's dataset
|
||||
# foundry (speech → text, speaker-attributed), operator-approved 2026-09-26
|
||||
# (relayed by brokkr-smithy-dev).
|
||||
#
|
||||
# audiocpp_server vibevoice-asr-streaming-1.5b (Q4_K) :8031
|
||||
# POST /v1/audio/transcriptions (OpenAI-style, whole file)
|
||||
# POST /v1/audio/transcriptions/live?... (chunked 16 kHz s16le PCM)
|
||||
# GET /v1/models, /health
|
||||
#
|
||||
# Runtime is audio.cpp, per the GGUF's own model card (the GGUF is an audio.cpp
|
||||
# package with its sidecars embedded, so no separate tokenizer file). It is NOT
|
||||
# vanilla llama.cpp and NOT mudler's vibevoice.cpp.
|
||||
# Direct access only; no gateway entry.
|
||||
#
|
||||
# Files (not in git): /opt/aimodels/gguf/vibevoice-asr-streaming-1.5b/ from
|
||||
# christopherthompson81/VibeVoice-ASR-Streaming-1.5B-GGUF @ ff9615110299.
|
||||
name: vibevoice-asr-seat
|
||||
services:
|
||||
audiocpp:
|
||||
build:
|
||||
context: .
|
||||
image: local/audiocpp:v0.8.2-audio8-perf-hotfix
|
||||
container_name: vibevoice-asr
|
||||
restart: unless-stopped
|
||||
command: ["--config", "/config/server.json", "--no-ui", "--log"]
|
||||
ports:
|
||||
- "${ASR_PORT}:8080"
|
||||
volumes:
|
||||
- /opt/aimodels/gguf/vibevoice-asr-streaming-1.5b:/models:ro
|
||||
- /opt/docker/conf/vibevoice-asr-seat/server.json:/config/server.json:ro
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ["0"]
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 120s
|
||||
labels:
|
||||
- homepage.group=AI - Eval & Retrieval
|
||||
- homepage.name=ASR — VibeVoice Streaming 1.5B (audio.cpp, nh3-ml1)
|
||||
- homepage.icon=mdi-microphone-message
|
||||
- homepage.description=Streaming speech-to-text for the dataset foundry (direct, no gateway)
|
||||
- homepage.href=http://10.100.50.80:${ASR_PORT}/v1/models
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"host": "0.0.0.0",
|
||||
"port": 8080,
|
||||
"backend": "cuda",
|
||||
"threads": 6,
|
||||
"models": [
|
||||
{
|
||||
"id": "vibevoice-asr-streaming-1.5b",
|
||||
"family": "vibevoice_asr_streaming",
|
||||
"path": "/models/vibevoice-asr-streaming-1.5b-q4_k.gguf",
|
||||
"task": "asr",
|
||||
"mode": "streaming"
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user