vh
53c3e8000e
Every agent on nh3-dev — Claude, Codex, Grok, Aider — needs the same answers: what runs here, how do I call it, what will bite me. Until now that lived in ~/.claude/CLAUDE.md, which only Claude sessions load, and it was interleaved with operator preferences that other families have no use for. Two-tier by design, matching the persistent-memory split: FLEETTOOLS.md is a 135-line index an agent reads whole, and each entry links to a detail file it opens only when it actually needs that tool. Reading the index costs about a fifth of reading the tree. Detail paths are absolute so they resolve from any working directory, since a non-Claude agent will cat the path rather than follow a markdown link. ~/FLEETTOOLS.md symlinks to the index for discovery. Rule zero is that live inventories get queried, not transcribed: Homepage /api/services, asset-engine /api/v1/services, LiteLLM /v1/models, and every FastAPI seat's /openapi.json. A copied service table would be stale within a month and this repo already has a standing rule against second copies that drift. Contents verified against the running fleet rather than copied from existing docs: binaries resolved on PATH, seven endpoints probed live, the LiteLLM roster counted at 40 models where the old note said ~30. No credentials are included; the vault and its CLI are pointed at instead.
60 lines
2.4 KiB
Markdown
60 lines
2.4 KiB
Markdown
# Direct inference seats + the Asset Engine control plane
|
||
|
||
Most seats are vLLM behind the LiteLLM gateway — prefer the gateway
|
||
(`docs/fleettools/litellm.md`) unless you need something it does not expose.
|
||
|
||
## ⚠ Query the live inventory, never a written list
|
||
|
||
```sh
|
||
# every fleet service, curated, with hrefs and descriptions
|
||
curl -s http://10.0.50.45:5100/api/services | python3 -m json.tool
|
||
|
||
# the inference control plane: what exists, what is up, GPU state
|
||
curl -s http://10.250.50.70:8200/api/v1/services
|
||
curl -s http://10.250.50.70:8200/api/v1/gpus
|
||
```
|
||
|
||
Seats move between hosts and GPUs often enough that any transcribed table here would
|
||
mislead within a month.
|
||
|
||
## Asset Engine — `http://10.250.50.70:8200`
|
||
|
||
Control plane over the PFI inference fleet. Can enumerate services, read GPU state,
|
||
**start and stop seats**, submit jobs and fetch assets.
|
||
|
||
```
|
||
GET /api/v1/services GET /api/v1/services/{id}/state
|
||
POST /api/v1/services/{id}/start POST /api/v1/services/{id}/stop
|
||
GET /api/v1/gpus POST /api/v1/jobs
|
||
GET /api/v1/assets/{id}/blob GET /api/v1/presets
|
||
```
|
||
|
||
⚠ **Liveness is keyed on the compose PROJECT name.** A seat reads OFFLINE unless the
|
||
compose project name equals the catalog's `lifecycle.stack`. Pin `name: <stack>` in
|
||
the compose file.
|
||
|
||
## GPU hosts
|
||
|
||
| host | GPUs | notes |
|
||
|---|---|---|
|
||
| fv-ml1 `10.251.50.54` | 4× RTX PRO 6000 Blackwell Max-Q, 96 GB ea (391 GB) | primary; native FP8/NVFP4 |
|
||
| ana-ml2 `10.250.50.54` | see live inventory | |
|
||
| irv-ml1 `irv-ml1.nh3.internal` | RTX 3090 + RTX A6000 (72 GB) | **Ampere sm_86: no native FP8/NVFP4** |
|
||
|
||
Every seat is FastAPI — **`GET /openapi.json` on any of them is the live contract.**
|
||
Do not transcribe endpoint shapes; fetch them.
|
||
|
||
## Traps
|
||
|
||
- **Idle VRAM is reserved, not waste.** Single-user dev fleet. Never propose raising
|
||
`gpu-memory-utilization` to fill free VRAM.
|
||
- **vLLM temp 0 is not deterministic under concurrency** — ~3.12pp replicate spread at
|
||
`jobs=8`. Any eval delta under ~3.1pp is instrument floor, not signal. Only
|
||
`--jobs 1` shrinks it.
|
||
- **Never alias a model under a stale name.** Serve under the TRUE served-name only;
|
||
let the old name 404 and notify consumers. No silent substitution.
|
||
- **Ollama is banned fleet-wide.** Never stand one up; tear down any found. Use
|
||
llama-swap or vLLM.
|
||
- **`docker compose up -d` recreates the WHOLE stack** when a shared `.env` changes.
|
||
Target one service: `up -d <service>`.
|