vh
53c3e8000e
Every agent on nh3-dev — Claude, Codex, Grok, Aider — needs the same answers: what runs here, how do I call it, what will bite me. Until now that lived in ~/.claude/CLAUDE.md, which only Claude sessions load, and it was interleaved with operator preferences that other families have no use for. Two-tier by design, matching the persistent-memory split: FLEETTOOLS.md is a 135-line index an agent reads whole, and each entry links to a detail file it opens only when it actually needs that tool. Reading the index costs about a fifth of reading the tree. Detail paths are absolute so they resolve from any working directory, since a non-Claude agent will cat the path rather than follow a markdown link. ~/FLEETTOOLS.md symlinks to the index for discovery. Rule zero is that live inventories get queried, not transcribed: Homepage /api/services, asset-engine /api/v1/services, LiteLLM /v1/models, and every FastAPI seat's /openapi.json. A copied service table would be stale within a month and this repo already has a standing rule against second copies that drift. Contents verified against the running fleet rather than copied from existing docs: binaries resolved on PATH, seven endpoints probed live, the LiteLLM roster counted at 40 models where the old note said ~30. No credentials are included; the vault and its CLI are pointed at instead.
2.0 KiB
2.0 KiB
Observability — where to look when something is wrong
| surface | URL | what it answers |
|---|---|---|
| Homepage | http://10.0.50.45:5100 |
the curated fleet service map; /api/services is machine-readable |
| Beszel | http://10.250.50.70:8090 |
host + container up/down, resource history; alerts land in althing |
| Dozzle | http://10.250.50.70:8088 |
container logs, ana-docker + ana-ml2 |
| Uptime Kuma | http://10.0.50.45:3001 |
ESH-side service monitoring |
| Backrest | http://10.250.50.70:9898 |
restic snapshot browse / restore |
| task-board | http://10.250.50.70:7878 |
what an assistant session is working on right now |
| Dockge (per host) | <host>:5001 |
compose UI on ana-docker, ana-ml2, esh-docker-vm, irv-ml1, nh3-docker |
| PBS | 10.250.50.90:8007 (primary) · 10.100.50.90:8007 (DR) |
VM/LXC image backups |
Homepage's /api/services is the single best machine-readable inventory of the
fleet and it is maintained as a side effect of normal work. Query it rather than
keeping a list.
Diagnostic reflexes
- All hosts at one site flapping together = a WAN/mesh event, not host failure. Check uptimes first; they will show weeks. Then read the site's subnet-router tailscaled log.
- ana-nas is a SPOF for postgres, rest-server-ana, PBS-ANA and cross-site restic. Check it first on any multi-failure.
- After a power loss, crowdsec fails closed: it exits 255 and the Traefik bouncer 403s ALL HTTP. Restart crowdsec THEN traefik. This is a recurring recovery step.
- A filter can silently narrow your window.
docker logs --since HH:MMreturning 0 while the line demonstrably exists, afind -maxdepthundercount, a character class that omits digits — re-run unfiltered before concluding absence. Absence of a signal is not a safe reading of it. - Timestamp every state report. An untimestamped claim is a claim about "now" and has manufactured phantom events in peer artifacts before.