esh-pfi-infrastructure/docs/fleettools/inference.md

Direct inference seats

Most seats are vLLM behind the LiteLLM gateway — prefer the gateway (docs/fleettools/litellm.md) unless you need something it does not expose.

⚠ Query the live inventory, never a written list

# every fleet service, curated, with hrefs and descriptions
curl -s http://10.0.50.45:5100/api/services | python3 -m json.tool
# the LLM roster
curl -s -H "Authorization: Bearer $KEY" http://10.250.50.70:4000/v1/models

Seats move between hosts and GPUs often enough that any transcribed table here would mislead within a month.

Asset Engine — RETIRED 2026-10-17

The control plane at http://10.250.50.70:8200 is gone (Prime). Its last job was 2026-08-02, and every catalog endpoint pointed at irv-ml1's dead wg0 address after the 09-06 mesh cutover. docker compose down on ana-docker; the DB, image, repo (vh/asset-engine) and catalog (docs/asset-engine/) are kept, so docker compose up -d in /opt/docker/compose/asset-engine brings it back. Start/stop a seat by hand (docker compose up -d / stop in its compose dir on the GPU host).

GPU hosts

host	GPUs	notes
fv-ml1 10.251.50.54	4× RTX PRO 6000 Blackwell Max-Q, 96 GB ea (391 GB)	primary; native FP8/NVFP4
ana-ml2 10.250.50.54	see live inventory
irv-ml1 irv-ml1.nh3.internal	RTX 3090 + RTX A6000 (72 GB)	Ampere sm_86: no native FP8/NVFP4

Every seat is FastAPI — GET /openapi.json on any of them is the live contract. Do not transcribe endpoint shapes; fetch them.

Traps

• Idle VRAM is reserved, not waste. Single-user dev fleet. Never propose raising gpu-memory-utilization to fill free VRAM.
• vLLM temp 0 is not deterministic under concurrency — ~3.12pp replicate spread at jobs=8. Any eval delta under ~3.1pp is instrument floor, not signal. Only --jobs 1 shrinks it.
• Never alias a model under a stale name. Serve under the TRUE served-name only; let the old name 404 and notify consumers. No silent substitution.
• Ollama is banned fleet-wide. Never stand one up; tear down any found. Use llama-swap or vLLM.
• docker compose up -d recreates the WHOLE stack when a shared .env changes. Target one service: up -d <service>.