vh
53c3e8000e
Every agent on nh3-dev — Claude, Codex, Grok, Aider — needs the same answers: what runs here, how do I call it, what will bite me. Until now that lived in ~/.claude/CLAUDE.md, which only Claude sessions load, and it was interleaved with operator preferences that other families have no use for. Two-tier by design, matching the persistent-memory split: FLEETTOOLS.md is a 135-line index an agent reads whole, and each entry links to a detail file it opens only when it actually needs that tool. Reading the index costs about a fifth of reading the tree. Detail paths are absolute so they resolve from any working directory, since a non-Claude agent will cat the path rather than follow a markdown link. ~/FLEETTOOLS.md symlinks to the index for discovery. Rule zero is that live inventories get queried, not transcribed: Homepage /api/services, asset-engine /api/v1/services, LiteLLM /v1/models, and every FastAPI seat's /openapi.json. A copied service table would be stale within a month and this repo already has a standing rule against second copies that drift. Contents verified against the running fleet rather than copied from existing docs: binaries resolved on PATH, seven endpoints probed live, the LiteLLM roster counted at 40 models where the old note said ~30. No credentials are included; the vault and its CLI are pointed at instead.
2.4 KiB
2.4 KiB
Direct inference seats + the Asset Engine control plane
Most seats are vLLM behind the LiteLLM gateway — prefer the gateway
(docs/fleettools/litellm.md) unless you need something it does not expose.
⚠ Query the live inventory, never a written list
# every fleet service, curated, with hrefs and descriptions
curl -s http://10.0.50.45:5100/api/services | python3 -m json.tool
# the inference control plane: what exists, what is up, GPU state
curl -s http://10.250.50.70:8200/api/v1/services
curl -s http://10.250.50.70:8200/api/v1/gpus
Seats move between hosts and GPUs often enough that any transcribed table here would mislead within a month.
Asset Engine — http://10.250.50.70:8200
Control plane over the PFI inference fleet. Can enumerate services, read GPU state, start and stop seats, submit jobs and fetch assets.
GET /api/v1/services GET /api/v1/services/{id}/state
POST /api/v1/services/{id}/start POST /api/v1/services/{id}/stop
GET /api/v1/gpus POST /api/v1/jobs
GET /api/v1/assets/{id}/blob GET /api/v1/presets
⚠ Liveness is keyed on the compose PROJECT name. A seat reads OFFLINE unless the
compose project name equals the catalog's lifecycle.stack. Pin name: <stack> in
the compose file.
GPU hosts
| host | GPUs | notes |
|---|---|---|
fv-ml1 10.251.50.54 |
4× RTX PRO 6000 Blackwell Max-Q, 96 GB ea (391 GB) | primary; native FP8/NVFP4 |
ana-ml2 10.250.50.54 |
see live inventory | |
irv-ml1 irv-ml1.nh3.internal |
RTX 3090 + RTX A6000 (72 GB) | Ampere sm_86: no native FP8/NVFP4 |
Every seat is FastAPI — GET /openapi.json on any of them is the live contract.
Do not transcribe endpoint shapes; fetch them.
Traps
- Idle VRAM is reserved, not waste. Single-user dev fleet. Never propose raising
gpu-memory-utilizationto fill free VRAM. - vLLM temp 0 is not deterministic under concurrency — ~3.12pp replicate spread at
jobs=8. Any eval delta under ~3.1pp is instrument floor, not signal. Only--jobs 1shrinks it. - Never alias a model under a stale name. Serve under the TRUE served-name only; let the old name 404 and notify consumers. No silent substitution.
- Ollama is banned fleet-wide. Never stand one up; tear down any found. Use llama-swap or vLLM.
docker compose up -drecreates the WHOLE stack when a shared.envchanges. Target one service:up -d <service>.