docs: add FleetTools — an agent-family-agnostic index of fleet capability

Every agent on nh3-dev — Claude, Codex, Grok, Aider — needs the same
answers: what runs here, how do I call it, what will bite me. Until now
that lived in ~/.claude/CLAUDE.md, which only Claude sessions load, and
it was interleaved with operator preferences that other families have
no use for.

Two-tier by design, matching the persistent-memory split: FLEETTOOLS.md
is a 135-line index an agent reads whole, and each entry links to a
detail file it opens only when it actually needs that tool. Reading the
index costs about a fifth of reading the tree.

Detail paths are absolute so they resolve from any working directory,
since a non-Claude agent will cat the path rather than follow a
markdown link. ~/FLEETTOOLS.md symlinks to the index for discovery.

Rule zero is that live inventories get queried, not transcribed:
Homepage /api/services, asset-engine /api/v1/services, LiteLLM
/v1/models, and every FastAPI seat's /openapi.json. A copied service
table would be stale within a month and this repo already has a
standing rule against second copies that drift.

Contents verified against the running fleet rather than copied from
existing docs: binaries resolved on PATH, seven endpoints probed live,
the LiteLLM roster counted at 40 models where the old note said ~30.
No credentials are included; the vault and its CLI are pointed at
instead.
This commit is contained in:
vh
2026-09-18 07:35:28 -07:00
parent d6a9d70b9e
commit 53c3e8000e
15 changed files with 737 additions and 0 deletions
+32
View File
@@ -0,0 +1,32 @@
# Observability — where to look when something is wrong
| surface | URL | what it answers |
|---|---|---|
| **Homepage** | `http://10.0.50.45:5100` | the curated fleet service map; `/api/services` is machine-readable |
| **Beszel** | `http://10.250.50.70:8090` | host + container up/down, resource history; alerts land in althing |
| **Dozzle** | `http://10.250.50.70:8088` | container logs, ana-docker + ana-ml2 |
| **Uptime Kuma** | `http://10.0.50.45:3001` | ESH-side service monitoring |
| **Backrest** | `http://10.250.50.70:9898` | restic snapshot browse / restore |
| **task-board** | `http://10.250.50.70:7878` | what an assistant session is working on right now |
| Dockge (per host) | `<host>:5001` | compose UI on ana-docker, ana-ml2, esh-docker-vm, irv-ml1, nh3-docker |
| PBS | `10.250.50.90:8007` (primary) · `10.100.50.90:8007` (DR) | VM/LXC image backups |
**Homepage's `/api/services` is the single best machine-readable inventory of the
fleet** and it is maintained as a side effect of normal work. Query it rather than
keeping a list.
## Diagnostic reflexes
- **All hosts at one site flapping together = a WAN/mesh event, not host failure.**
Check uptimes first; they will show weeks. Then read the site's subnet-router
tailscaled log.
- **ana-nas is a SPOF** for postgres, rest-server-ana, PBS-ANA and cross-site restic.
Check it first on any multi-failure.
- **After a power loss, crowdsec fails closed**: it exits 255 and the Traefik bouncer
403s ALL HTTP. Restart crowdsec THEN traefik. This is a recurring recovery step.
- **A filter can silently narrow your window.** `docker logs --since HH:MM` returning 0
while the line demonstrably exists, a `find -maxdepth` undercount, a character class
that omits digits — re-run unfiltered before concluding absence. **Absence of a
signal is not a safe reading of it.**
- **Timestamp every state report.** An untimestamped claim is a claim about "now" and
has manufactured phantom events in peer artifacts before.