50 lines
2.2 KiB
Markdown
50 lines
2.2 KiB
Markdown
# Direct inference seats
|
||
|
||
Most seats are vLLM behind the LiteLLM gateway — prefer the gateway
|
||
(`docs/fleettools/litellm.md`) unless you need something it does not expose.
|
||
|
||
## ⚠ Query the live inventory, never a written list
|
||
|
||
```sh
|
||
# every fleet service, curated, with hrefs and descriptions
|
||
curl -s http://10.0.50.45:5100/api/services | python3 -m json.tool
|
||
# the LLM roster
|
||
curl -s -H "Authorization: Bearer $KEY" http://10.250.50.70:4000/v1/models
|
||
```
|
||
|
||
Seats move between hosts and GPUs often enough that any transcribed table here would
|
||
mislead within a month.
|
||
|
||
## Asset Engine — RETIRED 2026-10-17
|
||
|
||
The control plane at `http://10.250.50.70:8200` is gone (Prime). Its last job was 2026-08-02, and every
|
||
catalog endpoint pointed at irv-ml1's dead wg0 address after the 09-06 mesh cutover. `docker compose
|
||
down` on ana-docker; the DB, image, repo (`vh/asset-engine`) and catalog (`docs/asset-engine/`) are kept, so
|
||
`docker compose up -d` in `/opt/docker/compose/asset-engine` brings it back. Start/stop a seat by hand
|
||
(`docker compose up -d` / `stop` in its compose dir on the GPU host).
|
||
|
||
## GPU hosts
|
||
|
||
| host | GPUs | notes |
|
||
|---|---|---|
|
||
| fv-ml1 `10.251.50.54` | 4× RTX PRO 6000 Blackwell Max-Q, 96 GB ea (391 GB) | primary; native FP8/NVFP4 |
|
||
| ana-ml2 `10.250.50.54` | see live inventory | |
|
||
| irv-ml1 `irv-ml1.nh3.internal` | RTX 3090 + RTX A6000 (72 GB) | **Ampere sm_86: no native FP8/NVFP4** |
|
||
|
||
Every seat is FastAPI — **`GET /openapi.json` on any of them is the live contract.**
|
||
Do not transcribe endpoint shapes; fetch them.
|
||
|
||
## Traps
|
||
|
||
- **Idle VRAM is reserved, not waste.** Single-user dev fleet. Never propose raising
|
||
`gpu-memory-utilization` to fill free VRAM.
|
||
- **vLLM temp 0 is not deterministic under concurrency** — ~3.12pp replicate spread at
|
||
`jobs=8`. Any eval delta under ~3.1pp is instrument floor, not signal. Only
|
||
`--jobs 1` shrinks it.
|
||
- **Never alias a model under a stale name.** Serve under the TRUE served-name only;
|
||
let the old name 404 and notify consumers. No silent substitution.
|
||
- **Ollama is banned fleet-wide.** Never stand one up; tear down any found. Use
|
||
llama-swap or vLLM.
|
||
- **`docker compose up -d` recreates the WHOLE stack** when a shared `.env` changes.
|
||
Target one service: `up -d <service>`.
|