docs: add FleetTools — an agent-family-agnostic index of fleet capability
Every agent on nh3-dev — Claude, Codex, Grok, Aider — needs the same answers: what runs here, how do I call it, what will bite me. Until now that lived in ~/.claude/CLAUDE.md, which only Claude sessions load, and it was interleaved with operator preferences that other families have no use for. Two-tier by design, matching the persistent-memory split: FLEETTOOLS.md is a 135-line index an agent reads whole, and each entry links to a detail file it opens only when it actually needs that tool. Reading the index costs about a fifth of reading the tree. Detail paths are absolute so they resolve from any working directory, since a non-Claude agent will cat the path rather than follow a markdown link. ~/FLEETTOOLS.md symlinks to the index for discovery. Rule zero is that live inventories get queried, not transcribed: Homepage /api/services, asset-engine /api/v1/services, LiteLLM /v1/models, and every FastAPI seat's /openapi.json. A copied service table would be stale within a month and this repo already has a standing rule against second copies that drift. Contents verified against the running fleet rather than copied from existing docs: binaries resolved on PATH, seven endpoints probed live, the LiteLLM roster counted at 40 models where the old note said ~30. No credentials are included; the vault and its CLI are pointed at instead.
This commit is contained in:
@@ -0,0 +1,135 @@
|
||||
# FleetTools — what this system can do, and where the details live
|
||||
|
||||
**Audience: any agent working on this box (nh3-dev), Claude or not.**
|
||||
This file is an **index**, deliberately. Read it whole — it is cheap. Then open *only*
|
||||
the detail file for the tool you actually need. Do not read the whole `fleettools/`
|
||||
directory; that defeats the point.
|
||||
|
||||
- **Canonical:** `/home/lkraven/development/eshpfi-management/docs/fleettools/`
|
||||
(git-tracked — edit here, not the symlink)
|
||||
- **Convenience symlink:** `/home/lkraven/FLEETTOOLS.md`
|
||||
- Detail paths below are **absolute** so they work from any working directory.
|
||||
|
||||
---
|
||||
|
||||
## Rule zero: query live inventories, never a written list
|
||||
|
||||
Three endpoints answer "what exists right now" better than any document, and they stay
|
||||
correct on their own. **Reach for these before asking anyone:**
|
||||
|
||||
```sh
|
||||
curl -s http://10.0.50.45:5100/api/services # every fleet service, curated
|
||||
curl -s http://10.250.50.70:8200/api/v1/services # inference seats + GPU state
|
||||
curl -s -H "Authorization: Bearer $KEY" \
|
||||
http://10.250.50.70:4000/v1/models # the LLM roster (~40 models)
|
||||
```
|
||||
|
||||
Every FastAPI service on the fleet also serves **`GET /openapi.json`** — that is the
|
||||
live contract for its API. Fetch it rather than trusting a transcription.
|
||||
|
||||
---
|
||||
|
||||
## Talking to people and other agents
|
||||
|
||||
- **althing** — the inter-agent message bus. `postbox` CLI against the post office at
|
||||
`10.100.50.40:8390`.
|
||||
*When:* you need to ask another agent something, or answer one.
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/althing.md`
|
||||
|
||||
- **The Booth** — ephemeral media board at `10.100.10.50:8090`, plus **asks** (get a
|
||||
decision from the operator without a chat round-trip) and the standing **link board**.
|
||||
*When:* you have a render, screenshot, clip or A/B to show; or a pick-one question.
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/booth.md`
|
||||
|
||||
- **The Henge** — durable idea parking at `park.phasefinal.com:8420`, `park` CLI.
|
||||
*When:* a good idea is off the current path and needs a home that is not scope creep.
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/henge.md`
|
||||
|
||||
## Credentials
|
||||
|
||||
- **secret** — Vaultwarden-backed credential store. `secret get|put|list|backfill`.
|
||||
*When:* you need a token/key, or you just created one worth keeping.
|
||||
*Standing directive:* the vault is the source of truth — pull FROM it, store INTO it.
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/secret.md`
|
||||
|
||||
## LLM and inference
|
||||
|
||||
- **LiteLLM gateway** — `http://10.250.50.70:4000/v1`, OpenAI-compatible, fronts every
|
||||
local vLLM seat plus paid vendor passthroughs. Free local `summarizer` / `classifier`
|
||||
/ embedding / reranker aliases.
|
||||
*When:* any LLM call. Prefer it over hitting a seat directly.
|
||||
⚠ *The shared key CAN spend real vendor credits.*
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/litellm.md`
|
||||
|
||||
- **Direct inference seats + Asset Engine** — the control plane at
|
||||
`http://10.250.50.70:8200` enumerates seats, reads GPU state, and starts/stops them.
|
||||
*When:* the gateway does not expose what you need, or a seat is down.
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/inference.md`
|
||||
|
||||
- **Speech (TTS/ASR)** — Breeze-TTS-2 is the fleet primary; Chatterbox-fast is the
|
||||
low-latency chat front end; Speaches and Parakeet do ASR.
|
||||
*When:* you need audio out of text, or text out of audio.
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/speech.md`
|
||||
|
||||
- **Arbo** — catalog-driven image generation at `irv-ml1.nh3.internal:8201`, including
|
||||
the Character Studio dataset→LoRA pipeline.
|
||||
*When:* you need images rendered, or a character LoRA trained.
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/arbo.md`
|
||||
|
||||
## Working on the fleet itself
|
||||
|
||||
- **elway** — SSH playbook runner for **CHANGING** things.
|
||||
*When:* any multi-step remote change. Do not chain `ssh -t host 'sudo …'`.
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/elway.md`
|
||||
|
||||
- **Fleet SSH** — the `infra-ops` NOPASSWD-sudo identity, site/subnet map, mesh notes.
|
||||
*When:* you need to reach a host, or wonder whether you are allowed to.
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/ssh-fleet.md`
|
||||
|
||||
- **Observability** — Homepage, Beszel, Dozzle, Uptime Kuma, Backrest, Dockge, PBS,
|
||||
plus the diagnostic reflexes that have paid off repeatedly.
|
||||
*When:* something is broken and you do not yet know what.
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/observability.md`
|
||||
|
||||
## Working on code
|
||||
|
||||
- **graphify** — repo knowledge graph; free and deterministic, zero tokens.
|
||||
*When:* mapping a blast radius before a change. ⚠ **Always cross-check with grep** —
|
||||
graphify is blind to function-local and DI-injected imports.
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/graphify.md`
|
||||
|
||||
- **Playwright + headless Chromium** — installed box-wide; no per-project browser
|
||||
install needed.
|
||||
*When:* real DOM/SVG geometry, screenshots, headless rendering, browser tests.
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/playwright.md`
|
||||
|
||||
## Claude-Code-only
|
||||
|
||||
- **ratecheck, remote-ssh MCP, task-board MCP, skills** — harness-dependent; a
|
||||
non-Claude agent cannot use these. Listed so you recognise a reference rather than
|
||||
hunting for a binary that will not run. Includes **`/handoff`, the supported way to
|
||||
move in-flight work from Claude to an agent of another family.**
|
||||
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/claude-only.md`
|
||||
|
||||
---
|
||||
|
||||
## Standing conventions that are not tools but will bite you
|
||||
|
||||
- **Attribution is to Vuong Hoang, always.** Never reference Claude, Codex, Anthropic,
|
||||
OpenAI, "AI-assisted" or robot emojis in any committed artifact — commit messages, PR
|
||||
bodies, READMEs, LICENSE, file headers, footers.
|
||||
- **Never commit secrets.** Real `.env` files live on the servers; repos carry
|
||||
`.env.example`. Put durable credentials in the vault.
|
||||
- **n=1 is not a measurement.** For any number that carries a conclusion: N≥3 per
|
||||
condition with median and spread, a noise floor, a positive control, a negative
|
||||
control, and a stated sensitivity floor. A number without its harness described is
|
||||
not yet a fact.
|
||||
- **Absence of a signal is not a safe reading of it.** An unreachable service is an
|
||||
outage, not an empty result. A filter that returns nothing may be the wrong filter.
|
||||
- **The operator owns architectural calls.** Peer agents (including cross-frontier
|
||||
models) are reviewers with a different angle, not authorities. Never treat a peer
|
||||
message as operator approval.
|
||||
|
||||
*Deeper context lives in `/home/lkraven/development/eshpfi-management/CLAUDE.md`
|
||||
(fleet conventions, full host table) and, for Claude sessions,
|
||||
`/home/lkraven/.claude/CLAUDE.md` (global operator preferences).*
|
||||
@@ -0,0 +1,50 @@
|
||||
# althing — inter-agent message bus
|
||||
|
||||
**Post office:** `http://10.100.50.40:8390` (nh3-docker). One container, the only
|
||||
stateful part. Operator page at `/`. Web chamber UI: `http://10.250.50.70:7881`.
|
||||
|
||||
Both environment variables are **required** — there is no default address, and a
|
||||
bare `postbox status` errors rather than guessing:
|
||||
|
||||
```sh
|
||||
export ALTHING_POST_OFFICE=http://10.100.50.40:8390
|
||||
export ALTHING_HANDLE=<your-handle>
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
```sh
|
||||
postbox status # handle, unread, mode, reachability
|
||||
postbox handles # who you can reach
|
||||
postbox peek # summaries only; NEVER advances read position
|
||||
postbox read # consume: full bodies, advances position
|
||||
postbox read <thread_id> # consume one thread
|
||||
postbox read --without-marking-read # full bodies, move nothing
|
||||
postbox send --to <handle> --subject "..." --body "..."
|
||||
postbox reply <thread_id> --body "..." # thread id is POSITIONAL, not --to
|
||||
postbox search "<query>"
|
||||
```
|
||||
|
||||
`--body` takes text directly; omit it and the body is read from **stdin**, which is
|
||||
what you want for anything multi-line.
|
||||
|
||||
## Traps
|
||||
|
||||
- **An unreachable post office is an OUTAGE, never an empty inbox.** If postbox says
|
||||
it could not reach the post office, that is the fault. Do not read it as "no mail".
|
||||
- **There is no outbox.** A send that cannot reach the post office fails loudly and
|
||||
is dropped. Check the output says it was accepted.
|
||||
- **Every althing v2 command was DELETED, not deprecated.** `althing-cli` → `postbox`.
|
||||
`althing-wake-listener` → `althing-listen`. `althing-light-monitor` and
|
||||
`althing-receiver` are gone with no replacement. A script calling one gets
|
||||
"command not found".
|
||||
- **Delivery to Claude Code is PUSH.** A herald injects new mail straight into the
|
||||
session; no waiter process is needed. `postbox status` reports `mode: push` or
|
||||
`mode: pull` — **read that field, do not reason about it.**
|
||||
- `mode: pull` means a missing **route**, not a missing herald. The herald is
|
||||
per-machine (one only; a second refuses). Check `~/.althing/routes/<handle>.route`.
|
||||
- Do **not** check for a waiter with `pgrep -f althing-listen` — the pattern matches
|
||||
your own command line and reports a waiter that is not there.
|
||||
- Closing the loop is the whole point: **read → act → answer the peer.** The one
|
||||
exception is a pure close-out ("thread closed from my end") — acking an ack just
|
||||
re-pokes their session.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Arbo — catalog-driven image generation engine
|
||||
|
||||
**Endpoint:** `http://irv-ml1.nh3.internal:8201` (title *ImageGen Engine*, v1.16.2)
|
||||
**Bare ComfyUI underneath:** `http://irv-ml1.nh3.internal:8188`
|
||||
**Repo:** gitea `vh/arbo` · deploy is `deploy-engine.sh` + a gitea Actions webhook on push.
|
||||
|
||||
Runs on CPU on irv-ml1 and dispatches to ComfyUI. `GET /openapi.json` is the live
|
||||
contract — the paths below are the map, not the spec.
|
||||
|
||||
## Jobs
|
||||
|
||||
```
|
||||
POST /workflows/{workflow_id}/run -> 202, a job id
|
||||
GET /jobs GET /jobs/{job_id}
|
||||
POST /jobs/{job_id}/cancel
|
||||
GET /artifacts/{job_id}/{index} # the rendered output
|
||||
```
|
||||
|
||||
## Character Studio — dataset → LoRA pipeline
|
||||
|
||||
```
|
||||
POST,GET /character-studio
|
||||
POST /character-studio/{job_id}/curate | /reframe | /recaption
|
||||
POST /character-studio/{job_id}/dataset/upload | /dataset/from-gallery
|
||||
GET /character-studio/trains GET /character-studio/trained
|
||||
DELETE /character-studio/trained/{lora_id}
|
||||
POST /character-studio/trained/{lora_id}/identity
|
||||
```
|
||||
|
||||
## Also
|
||||
|
||||
`POST /typography/compose` · `POST /ideogram/gallery`
|
||||
|
||||
## Traps
|
||||
|
||||
- **A new Python dependency needs `docker exec arbo pip install --user <pkg>`** — the
|
||||
deploy script does not rebuild the image.
|
||||
- **Triton backend is a HARD NO on the Ampere ComfyUI seat.** It crashes *every* render
|
||||
on the A6000 (`fp8e4nv` unsupported on sm_86); a cu130 bump does not fix it. Deferred
|
||||
until an Ada/Blackwell card is in that slot.
|
||||
- The mmartial ComfyUI image crash-loops if anything runs `pip` as root inside it —
|
||||
that breaks the venv. Fix: `chown -R 1000:1000 venv`. Set `DISABLE_UPGRADES=true` to
|
||||
pin torch.
|
||||
- Render results belong in a booth (`docs/fleettools/booth.md`), not described in prose.
|
||||
@@ -0,0 +1,66 @@
|
||||
# The Booth — ephemeral media board + operator decisions
|
||||
|
||||
**URL:** `http://10.100.10.50:8090/` · **CLI:** `booth` (on PATH on nh3-dev)
|
||||
**Storage:** a booth is *just a folder* — `~/booth-data/<name>/` on nh3-dev. No upload API.
|
||||
|
||||
Wiped **24h after last activity** unless kept. No auth (LAN/WG-internal), so never
|
||||
drop secrets in a booth.
|
||||
|
||||
## Show the operator something
|
||||
|
||||
```sh
|
||||
booth new <name>; booth add <name> <file>... # or just cp/rsync into ~/booth-data/<name>/
|
||||
rsync -a ./out/ nh3-dev:booth-data/<name>/ # from another host
|
||||
booth url <name> # -> http://10.100.10.50:8090/b/<name>/
|
||||
booth keep <name> / booth unkeep <name> # exempt from / restore the 24h sweep
|
||||
```
|
||||
|
||||
Auto-renders a gallery of images, webm+mp4 video and audio. A `<file>.txt` sidecar
|
||||
becomes that item's caption (this is how you label an A/B pair). A folder's own
|
||||
`index.html` is served **verbatim** — build a custom report when you want one.
|
||||
|
||||
## Ask the operator to decide (no chat round-trip)
|
||||
|
||||
```sh
|
||||
booth ask <booth> <stem> "<prompt>" "<option A>" "<option B>" [...] [--no-notes]
|
||||
booth answer <booth> <stem> --wait # blocks until answered (default 1h)
|
||||
booth answer <booth> <stem> # non-blocking; exit 1 while unanswered
|
||||
booth asks <booth>
|
||||
```
|
||||
|
||||
Open asks are flagged amber on the Booth front page, so a waiting question is visible
|
||||
without pinging anyone. Files are the state: `<stem>.ask.json` (yours),
|
||||
`<stem>.answer.json` (theirs: `choice`, `choice_index`, `label`, `notes`,
|
||||
`answered_at`, `answered_by`). Re-answering overwrites — the sidecar is the current
|
||||
answer, not a log. Radio only, no multi-select.
|
||||
|
||||
**Put the ask WHERE THE ARTIFACT IS.** If the booth has its own `index.html`, the
|
||||
Booth substitutes placeholders: `<div data-booth-ask="<stem>"></div>` for the whole
|
||||
ask, `data-booth-ask="<stem>:<question-key>"` for one question's radios,
|
||||
`data-booth-ask-submit="<stem>"` for notes + submit. Per-question fragments bind to
|
||||
one form via the HTML5 `form=` attribute, so a five-clip audition can put each radio
|
||||
group under its own audio and still submit in a single POST.
|
||||
⚠ Keep the placeholder OUTSIDE any grid/flex container or it becomes a cell in it.
|
||||
|
||||
⚠ **A partial answer is legal and you must handle it.** The operator can leave any
|
||||
question blank; the sidecar then carries `complete: false` and the blank keys in
|
||||
`unanswered`. **Check `complete` before acting**, and read a key in `unanswered` as
|
||||
"not decided yet", never as "declined".
|
||||
|
||||
## Standing convention — the link board
|
||||
|
||||
**Every operator-facing link worth more than the next thirty seconds goes on the
|
||||
standing link board, in the same turn you mention it.**
|
||||
|
||||
```sh
|
||||
booth link <url> "<short description>" # on nh3-dev
|
||||
ssh nh3-dev 'booth link <url> "<desc>"' # from anywhere else
|
||||
booth links # read it
|
||||
```
|
||||
|
||||
Still put the link in your reply — this is *in addition to*, not instead of. Links in
|
||||
chat drown in scrollback; the board is a kept booth rendered at the top of the Booth
|
||||
front page. Don't post noise: if the operator would not click it a week from now, it
|
||||
does not belong there.
|
||||
|
||||
**Full schema + rules:** `~/development/eshpfi-management/services/booth/README.md`
|
||||
@@ -0,0 +1,61 @@
|
||||
# Claude-Code-only tooling
|
||||
|
||||
Everything here depends on the Claude Code harness. **A Codex, Grok, Aider or other
|
||||
non-Claude agent cannot use these** — they are listed so you recognise a reference to
|
||||
them rather than hunting for a binary that will not work.
|
||||
|
||||
## ratecheck — how close this session is to its rate limits
|
||||
|
||||
`~/.claude/bin/ratecheck` (NOT on PATH). Exit `0` GO · `1` PAUSE · `2` UNKNOWN. Prints
|
||||
both windows, the file's age and the reset time.
|
||||
|
||||
The numbers are **not otherwise visible to an agent**: no CLI subcommand exposes them
|
||||
and nothing else on disk carries them. The harness pipes them to the statusline only,
|
||||
so `~/.claude/statusline-command.sh` tees them to `~/.claude/rate-limits.json` on every
|
||||
render. **If that tee is removed, everything built on it silently stops working**,
|
||||
which is why a stale file reports UNKNOWN rather than healthy.
|
||||
|
||||
- under ~85% — proceed normally
|
||||
- ~85–93% — prefer the cheap path; resume an existing subagent rather than spawning one
|
||||
- at/over the project threshold — stop starting new expensive work; let in-flight finish
|
||||
- **UNKNOWN — proceed, loudly.** A broken instrument that halts everything is a
|
||||
self-inflicted outage worse than the overspend it guards. Say plainly that limits
|
||||
were unreadable and you continued blind.
|
||||
|
||||
⚠ A GO is margin, not permission for an arbitrarily large action — one consult subagent
|
||||
has measured at 270k–650k tokens, so a round cleared at 92% can finish past 100%.
|
||||
|
||||
## remote-ssh MCP
|
||||
|
||||
Project-scoped in `eshpfi-management/.mcp.json`; tools `ssh_hosts` / `ssh_open` /
|
||||
`ssh_run` / `ssh_peek` / `ssh_interrupt` / `ssh_list` / `ssh_close`. Holds a
|
||||
**persistent shell**, so `cd` and exports survive between calls at ~6 ms per command.
|
||||
For reconnaissance; it deliberately has no file transfer and no idempotency (that is
|
||||
elway's half).
|
||||
|
||||
⚠⚠ **A bare `sudo` HANGS the session — always pipe it: `sudo -n <cmd> | cat`.** Cause:
|
||||
sudo ≥1.9.14 defaults `use_pty` on; with stdin on `/dev/null` and stdout on the session
|
||||
PTY the relay never completes and the completion marker never arrives. A pipe makes
|
||||
stdout a non-tty and sudo skips the PTY.
|
||||
|
||||
⚠ The **host allowlist is the only real containment boundary** — the denylist is three
|
||||
regexes and `bash -c` walks past it. `SSH_MCP_STRICT_ALLOWLIST=1` makes
|
||||
`~/.config/remote-ssh-mcp/config.json`'s `allowedHosts` authoritative.
|
||||
|
||||
## task-board MCP
|
||||
|
||||
`task_start` / `task_update` / `task_wait` / `task_complete` to surface work in
|
||||
progress at `http://10.250.50.70:7878/`. When you launch background shells, call
|
||||
`task_set_shells` with one short description each (and again with `[]` when they
|
||||
finish) — the board turns orange so the operator can tell the session is parked on
|
||||
background work rather than stalled on them.
|
||||
|
||||
## Skills
|
||||
|
||||
Claude Code skills live in `~/.claude/skills/`. Notable ones with fleet meaning:
|
||||
`/snapshot` (durable repo memory), `/handoff` (transfer active work to an agent of
|
||||
another family — **this is the cross-family bridge**), `/heid` (cross-frontier consult),
|
||||
`/park`, `/elitk`, `/vor`, `/tdd`.
|
||||
|
||||
**`/handoff` is the supported way to move in-flight work from Claude to Codex/Grok** —
|
||||
it writes an agent-agnostic brief to `/tmp` naming the repo's convention files.
|
||||
@@ -0,0 +1,33 @@
|
||||
# elway — SSH playbook runner (CHANGING things)
|
||||
|
||||
**Path:** `~/development/eshpfi-management/scripts/elway` (not on PATH)
|
||||
|
||||
```sh
|
||||
scripts/elway <host> --playbook playbooks/<name>.yaml
|
||||
```
|
||||
|
||||
Write a playbook under `playbooks/` instead of chaining `ssh -t host 'sudo …'`.
|
||||
Handles sudo once lazily, gives structured pass/change/fail reporting, supports
|
||||
idempotency via `creates:` / `when:` / `changed_when:`, and does file upload.
|
||||
|
||||
**Template:** `~/development/eshpfi-management/playbooks/elway-smoke.yaml`
|
||||
|
||||
## The split — pick by what you are doing
|
||||
|
||||
| | tool | why |
|
||||
|---|---|---|
|
||||
| **CHANGING** something | `scripts/elway` | idempotency, sudo handling, file transfer, structured results |
|
||||
| **LOOKING at** something | `remote-ssh` MCP (Claude) or plain `ssh` | no ceremony; a persistent shell where `cd` and exports survive |
|
||||
|
||||
Neither replaces the other. Reconnaissance — the `docker ps` / `grep` / `curl`
|
||||
questions — does not warrant authoring a playbook.
|
||||
|
||||
## Traps
|
||||
|
||||
- **Deploys must converge, not accrete.** The staging directory must be OUTSIDE the
|
||||
rsync target; inside, it becomes a silently frozen source. Verify CONTENT, not
|
||||
liveness.
|
||||
- **`pkill -f <pattern>` over ssh kills your own shell** when the pattern matches your
|
||||
own argv (exit 255). Kill by explicit PID or pidfile.
|
||||
- **Never run destructive fs operations on a variable path.** `rm` / `shred` /
|
||||
`mv`-over take LITERAL paths, one per line. No loops, no variables.
|
||||
@@ -0,0 +1,33 @@
|
||||
# graphify — repo knowledge graph
|
||||
|
||||
**CLI:** `graphify` (on PATH; `uv tool install graphifyy`) · **MCP:** `graphify-mcp`
|
||||
|
||||
```sh
|
||||
graphify update <path> # build/refresh: writes graphify-out/{graph.json,GRAPH_REPORT.md}
|
||||
graphify explain <symbol> # callers + neighbours
|
||||
graphify query <...> # traversal over EXTRACTED call edges
|
||||
graphify affected <symbol> # blast radius
|
||||
graphify path <a> <b> # reach between two symbols
|
||||
```
|
||||
|
||||
`graphify update` is the **free, deterministic path**: tree-sitter AST extraction plus
|
||||
Leiden clustering, **zero LLM, zero tokens**, roughly 25 s for ~900 files. Highest-value
|
||||
free output is the **God Nodes** list (core-abstraction surfacing).
|
||||
|
||||
The LLM layer is optional and points at any backend via `~/.graphify/providers.json` —
|
||||
pair community *labeling* (low-caliber) with the free LiteLLM `summarizer` endpoint.
|
||||
Leave deep semantic extraction **off** unless you need it: its INFERRED/AMBIGUOUS edges
|
||||
fight the explicit-over-implicit floor.
|
||||
|
||||
## ⚠ Graphify alone under-counts a blast radius — always cross-check with grep
|
||||
|
||||
Graphify is blind to **function-local and DI-injected imports** (`from .x import y`
|
||||
inside a function body, store objects passed through constructors) — exactly the
|
||||
pattern that hides cross-module callers. Graphify confirms call structure; grep catches
|
||||
what its AST extraction misses; **neither alone is sufficient** (graphify-only
|
||||
under-counts, grep-only misses transitive reach).
|
||||
|
||||
Run both before writing any contract that touches existing code. A contract that names
|
||||
3 files when the rip touches 6 is a scope miss the up-front pass would have caught.
|
||||
|
||||
Complements a hand-authored `docs/CODEBASE.md` (curated geography); does not replace it.
|
||||
@@ -0,0 +1,23 @@
|
||||
# The Henge — durable idea parking
|
||||
|
||||
**URL:** `http://park.phasefinal.com:8420/` (stonehenge-park, ana-docker)
|
||||
**CLI:** `park`
|
||||
|
||||
Replaces per-repo ROADMAP parking lots for deferred ideas. Park an idea with its
|
||||
originating repo as the source; retrieve or resolve it later.
|
||||
|
||||
```sh
|
||||
park <idea> # write, tagged with the originating repo
|
||||
park get <slug>
|
||||
park promote <slug> # it is happening
|
||||
park drop <slug> # it is not
|
||||
park snooze <slug>
|
||||
park list | park search <q> # triage; the due-queue lives in the browse UI
|
||||
```
|
||||
|
||||
Claude Code sessions have a `/park` skill wrapping the same thing.
|
||||
|
||||
**Why it exists:** projects sprawl at v0 when features land with no v1 target to gate
|
||||
them against — every good idea, lacking a home, silently becomes v0 scope. The gate is:
|
||||
*on the v1 path? → in; else → park it*, and **the default is park**. When both are
|
||||
defensible, park it.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Direct inference seats + the Asset Engine control plane
|
||||
|
||||
Most seats are vLLM behind the LiteLLM gateway — prefer the gateway
|
||||
(`docs/fleettools/litellm.md`) unless you need something it does not expose.
|
||||
|
||||
## ⚠ Query the live inventory, never a written list
|
||||
|
||||
```sh
|
||||
# every fleet service, curated, with hrefs and descriptions
|
||||
curl -s http://10.0.50.45:5100/api/services | python3 -m json.tool
|
||||
|
||||
# the inference control plane: what exists, what is up, GPU state
|
||||
curl -s http://10.250.50.70:8200/api/v1/services
|
||||
curl -s http://10.250.50.70:8200/api/v1/gpus
|
||||
```
|
||||
|
||||
Seats move between hosts and GPUs often enough that any transcribed table here would
|
||||
mislead within a month.
|
||||
|
||||
## Asset Engine — `http://10.250.50.70:8200`
|
||||
|
||||
Control plane over the PFI inference fleet. Can enumerate services, read GPU state,
|
||||
**start and stop seats**, submit jobs and fetch assets.
|
||||
|
||||
```
|
||||
GET /api/v1/services GET /api/v1/services/{id}/state
|
||||
POST /api/v1/services/{id}/start POST /api/v1/services/{id}/stop
|
||||
GET /api/v1/gpus POST /api/v1/jobs
|
||||
GET /api/v1/assets/{id}/blob GET /api/v1/presets
|
||||
```
|
||||
|
||||
⚠ **Liveness is keyed on the compose PROJECT name.** A seat reads OFFLINE unless the
|
||||
compose project name equals the catalog's `lifecycle.stack`. Pin `name: <stack>` in
|
||||
the compose file.
|
||||
|
||||
## GPU hosts
|
||||
|
||||
| host | GPUs | notes |
|
||||
|---|---|---|
|
||||
| fv-ml1 `10.251.50.54` | 4× RTX PRO 6000 Blackwell Max-Q, 96 GB ea (391 GB) | primary; native FP8/NVFP4 |
|
||||
| ana-ml2 `10.250.50.54` | see live inventory | |
|
||||
| irv-ml1 `irv-ml1.nh3.internal` | RTX 3090 + RTX A6000 (72 GB) | **Ampere sm_86: no native FP8/NVFP4** |
|
||||
|
||||
Every seat is FastAPI — **`GET /openapi.json` on any of them is the live contract.**
|
||||
Do not transcribe endpoint shapes; fetch them.
|
||||
|
||||
## Traps
|
||||
|
||||
- **Idle VRAM is reserved, not waste.** Single-user dev fleet. Never propose raising
|
||||
`gpu-memory-utilization` to fill free VRAM.
|
||||
- **vLLM temp 0 is not deterministic under concurrency** — ~3.12pp replicate spread at
|
||||
`jobs=8`. Any eval delta under ~3.1pp is instrument floor, not signal. Only
|
||||
`--jobs 1` shrinks it.
|
||||
- **Never alias a model under a stale name.** Serve under the TRUE served-name only;
|
||||
let the old name 404 and notify consumers. No silent substitution.
|
||||
- **Ollama is banned fleet-wide.** Never stand one up; tear down any found. Use
|
||||
llama-swap or vLLM.
|
||||
- **`docker compose up -d` recreates the WHOLE stack** when a shared `.env` changes.
|
||||
Target one service: `up -d <service>`.
|
||||
@@ -0,0 +1,52 @@
|
||||
# LiteLLM gateway — the fleet's OpenAI-compatible front door
|
||||
|
||||
**Base URL:** `http://10.250.50.70:4000/v1` (ana-docker) · **Logs UI:** `:4000/ui`
|
||||
Fronts the ana-ml2 / fv-ml1 / irv-ml1 vLLM seats plus paid vendor passthroughs.
|
||||
|
||||
## ⚠ Never hardcode the model roster — query it
|
||||
|
||||
```sh
|
||||
curl -s -H "Authorization: Bearer $KEY" http://10.250.50.70:4000/v1/models \
|
||||
| python3 -c "import sys,json;[print(m['id']) for m in json.load(sys.stdin)['data']]"
|
||||
```
|
||||
|
||||
The roster drifts constantly (40 entries as of 2026-09-18; a doc that names them goes
|
||||
stale within weeks). Any list written down here would be a second copy that rots.
|
||||
|
||||
## Keys
|
||||
|
||||
The gateway **401s without a virtual API key**. Three tiers:
|
||||
|
||||
| key | where | reach |
|
||||
|---|---|---|
|
||||
| shared all-agents (`all-agents-local`) | `~/.claude/CLAUDE.md` § Global tools | **every** model, paid passthroughs included |
|
||||
| infra-ops admin | `~/.config/litellm/infra-ops-key` (0600) | admin operations |
|
||||
| project-scoped | request from infra-ops via althing | isolated, auditable spend |
|
||||
|
||||
⚠ **The shared key is NOT free-local-only.** It reaches paid GLM/z.ai and Kimi
|
||||
passthroughs, and **a paid call on it spends real vendor credits shared across every
|
||||
session and project holding it.** There is no isolation. For anything where "shows up
|
||||
on someone else's tab" matters, get a project-scoped key.
|
||||
|
||||
## Stable aliases worth knowing
|
||||
|
||||
`summarizer` / `summarizer-large` / `classifier` — the always-available local seat for
|
||||
summarization, naming/labeling, classification and triage. Essentially free compute;
|
||||
reach for it before spending premium API tokens on work a small local model handles.
|
||||
Embeddings and rerankers (`qwen3-embedding`, `qwen3-reranker`, `reranker`) are local
|
||||
and free too. Anything named `*-frontier`, `glm-*` or `kimi-*` is **PAID**.
|
||||
|
||||
## Traps
|
||||
|
||||
- **LiteLLM mutates shared deployment config in-place.** Per-request sampler params
|
||||
bleed into the SHARED `litellm_params` for that deployment. Fix: give each variant a
|
||||
distinct served-model-name rather than reusing one with different params.
|
||||
- **Retiring a model orphans scoped keys.** A stale allowlist produces a silent
|
||||
per-endpoint 403. Audit `/key/list` + `/key/info` on every repoint.
|
||||
- **The `:4000/ui` playground cannot test vLLM backends** — it sends `tools: []`, which
|
||||
vLLM rejects with a 400. Use a chat frontend (Open WebUI, Gateway Chat) instead.
|
||||
- **"Random model calls" in the logs are usually AUTH FAILURES** — empty `api_key` or a
|
||||
missing `Bearer ` prefix, showing as zero-spend 401s.
|
||||
- **Kimi models 400 on any `temperature` other than 1** — omit the field entirely
|
||||
rather than sending 0. They answer in `content` with chain-of-thought in
|
||||
`reasoning_content`.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Observability — where to look when something is wrong
|
||||
|
||||
| surface | URL | what it answers |
|
||||
|---|---|---|
|
||||
| **Homepage** | `http://10.0.50.45:5100` | the curated fleet service map; `/api/services` is machine-readable |
|
||||
| **Beszel** | `http://10.250.50.70:8090` | host + container up/down, resource history; alerts land in althing |
|
||||
| **Dozzle** | `http://10.250.50.70:8088` | container logs, ana-docker + ana-ml2 |
|
||||
| **Uptime Kuma** | `http://10.0.50.45:3001` | ESH-side service monitoring |
|
||||
| **Backrest** | `http://10.250.50.70:9898` | restic snapshot browse / restore |
|
||||
| **task-board** | `http://10.250.50.70:7878` | what an assistant session is working on right now |
|
||||
| Dockge (per host) | `<host>:5001` | compose UI on ana-docker, ana-ml2, esh-docker-vm, irv-ml1, nh3-docker |
|
||||
| PBS | `10.250.50.90:8007` (primary) · `10.100.50.90:8007` (DR) | VM/LXC image backups |
|
||||
|
||||
**Homepage's `/api/services` is the single best machine-readable inventory of the
|
||||
fleet** and it is maintained as a side effect of normal work. Query it rather than
|
||||
keeping a list.
|
||||
|
||||
## Diagnostic reflexes
|
||||
|
||||
- **All hosts at one site flapping together = a WAN/mesh event, not host failure.**
|
||||
Check uptimes first; they will show weeks. Then read the site's subnet-router
|
||||
tailscaled log.
|
||||
- **ana-nas is a SPOF** for postgres, rest-server-ana, PBS-ANA and cross-site restic.
|
||||
Check it first on any multi-failure.
|
||||
- **After a power loss, crowdsec fails closed**: it exits 255 and the Traefik bouncer
|
||||
403s ALL HTTP. Restart crowdsec THEN traefik. This is a recurring recovery step.
|
||||
- **A filter can silently narrow your window.** `docker logs --since HH:MM` returning 0
|
||||
while the line demonstrably exists, a `find -maxdepth` undercount, a character class
|
||||
that omits digits — re-run unfiltered before concluding absence. **Absence of a
|
||||
signal is not a safe reading of it.**
|
||||
- **Timestamp every state report.** An untimestamped claim is a claim about "now" and
|
||||
has manufactured phantom events in peer artifacts before.
|
||||
@@ -0,0 +1,21 @@
|
||||
# Playwright + headless Chromium
|
||||
|
||||
Installed **box-wide on nh3-dev**. Use for anything needing a real browser engine:
|
||||
true DOM/SVG layout geometry, screenshots, headless rendering, browser-based tests or
|
||||
scraping.
|
||||
|
||||
- System shared-libs installed via apt (`playwright install-deps`)
|
||||
- Browser binaries in shared **`/opt/ms-playwright`** (root-owned, world-readable)
|
||||
- `PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright` wired globally
|
||||
(`/etc/environment` + `/etc/profile.d/`)
|
||||
|
||||
A project needs only the `playwright` module — `npm i playwright`. **The browser
|
||||
download is skippable and there is no per-project `playwright install`**; it resolves
|
||||
the shared binary.
|
||||
|
||||
Add a browser or bump versions:
|
||||
|
||||
```sh
|
||||
ssh infra-ops@10.100.10.50 \
|
||||
'sudo env PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright npx -y playwright install <browser>'
|
||||
```
|
||||
@@ -0,0 +1,35 @@
|
||||
# secret — Vaultwarden credential store
|
||||
|
||||
**CLI:** `secret` (on PATH on nh3-dev; real path
|
||||
`~/development/eshpfi-management/services/secrets-broker/secret`)
|
||||
**Backend:** `vaultwarden.phasefinal.com`, the `infra-ops` org.
|
||||
**Auth bootstrap:** `~/.config/secrets-broker/bootstrap.env` (mode 0600).
|
||||
|
||||
```sh
|
||||
secret list [--prefix P]
|
||||
secret get <name> [--field F] [--file OUT] # the only command that prints a value
|
||||
secret put <name> (--file P | --stdin) [--folder C] [--field k=v]
|
||||
secret backfill [--dry-run] # scan THIS box's env.sh/.env/~/.config creds
|
||||
```
|
||||
|
||||
Items are hostname-namespaced (`<host>/…`). Small text goes to the item note, small
|
||||
binary to a base64 hidden field, anything over 6000 B becomes a bw attachment.
|
||||
|
||||
## Standing directive (operator, 2026-08-11)
|
||||
|
||||
**The vault is the credential source of truth — store durable secrets in it AND pull
|
||||
the creds you need FROM it.** Any token, API key, cert or env-secret that is not
|
||||
ephemeral and does not belong in git should be `secret put` into the vault so it
|
||||
survives and is findable, never left single-copy on a box. When a task needs a
|
||||
credential, `secret get` it rather than reading an on-disk copy where practical.
|
||||
|
||||
The one exception is `bootstrap.env` itself — it unlocks the vault, so it cannot live
|
||||
inside it. Keep it 0600.
|
||||
|
||||
## Traps
|
||||
|
||||
- **Not a fleet service, no daemon.** Each dev box duplicates this stack against its
|
||||
own local secrets. Setup steps in the service README.
|
||||
- **`secret get` output shape varies by item.** Some items store a bare value, some
|
||||
store `key: <value>` fields. A blind `grep '^key:'` returns empty on a bare-value
|
||||
item — read the raw output first, then parse.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Speech — TTS and ASR endpoints
|
||||
|
||||
All on irv-ml1 unless noted. Every one is FastAPI: **`GET /openapi.json` is the live
|
||||
contract**, `/health` is the liveness probe.
|
||||
|
||||
## TTS
|
||||
|
||||
| service | endpoint | what it is |
|
||||
|---|---|---|
|
||||
| **Breeze-TTS-2** | `http://irv-ml1.nh3.internal:8204` | clone + direction, **fleet primary** |
|
||||
| TTS fleet gateway | `http://irv-ml1.nh3.internal:8198` | switchboard; engines sit behind it |
|
||||
| Chatterbox Fast | `http://irv-ml1.nh3.internal:8197` | streaming, sub-second first-audio — the live-chat front end |
|
||||
| dots.tts | `http://irv-ml1.nh3.internal:8202` | continuous-AR 48 kHz zero-shot clone |
|
||||
| Kokoro | `http://irv-ml1.nh3.internal:8193` | low-latency English, streaming |
|
||||
|
||||
**Breeze** (`/v1/voices`, `/v1/voices/detail`, `/v1/audio/speech`, `/v1/audio/design`)
|
||||
— `/v1/voices` matches dots byte-for-byte so they are drop-in swappable;
|
||||
`/v1/voices/detail` exposes the registry with its directions visible.
|
||||
|
||||
**Gateway** (`/v1/audio/speech`, `/v1/voices`) is also reachable through LiteLLM as
|
||||
the `ext-tts` alias. Voices: donut / emmie / glados / miranda / sindra plus emotion
|
||||
variants, or the OpenAI aliases nova / alloy. ⚠ **An unknown voice 404s and can trip
|
||||
the router cooldown** — list voices first.
|
||||
|
||||
## ASR
|
||||
|
||||
| service | endpoint | what it is |
|
||||
|---|---|---|
|
||||
| Speaches | `http://irv-ml1.nh3.internal:8204` | OpenAI-compatible faster-whisper, `verbose_json` |
|
||||
| Parakeet | `http://10.251.50.54:8300` | Parakeet-TDT via sherpa-onnx (fv-ml1 GPU 3) |
|
||||
| Scriberr | `http://10.250.50.54:8080` | transcription + diarization UI (ana-ml2 GPU 1) |
|
||||
|
||||
## Traps
|
||||
|
||||
- **OmniVoice-style diffusion TTS has a TTFA floor** — roughly fixed per-call cost,
|
||||
~0.7 s first-audio, real RTF 2.8–5.6× (not the advertised 40×). Chatterbox-fast
|
||||
stays the live-chat front end for that reason.
|
||||
- **Never expose repetition-penalty dials on Zonos-lineage engines** — it is a silence
|
||||
footgun.
|
||||
- Several engines listed on Homepage under *AI - Dormant* are down by design and take
|
||||
~90 s to respin. Check `/health` before assuming a service is broken.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Fleet SSH — the infra-ops identity and host map
|
||||
|
||||
## Identity
|
||||
|
||||
`ssh infra-ops@<host>` — a dedicated NOPASSWD-sudo user on PFI-owned boxes, key
|
||||
`~/.ssh/infra-ops_ed25519`. Covers **every** fleet host including all four Proxmox
|
||||
hypervisors, corviduo-dev and esh-docker-vm.
|
||||
|
||||
**Excludes the SureFire tenant hosts only** (`sf-*`, `sfsrv-ana`).
|
||||
|
||||
⚠ **Test with `sudo -n -v` first** rather than assuming. Not every box has been
|
||||
bootstrapped — `pbs-nh3` (10.100.50.90), for instance, has no infra-ops user and is
|
||||
reachable only as `lkraven@`.
|
||||
|
||||
⚠ **Remote sudo needs `ssh -t`.** Read-only probes can use plain ssh.
|
||||
|
||||
## Sites and hosts
|
||||
|
||||
| site | subnet | gateway |
|
||||
|---|---|---|
|
||||
| Anaheim (colo) | `10.250.0.0/16` | FortiGate `10.250.0.1` |
|
||||
| NH3 | `10.100.0.0/16` | UDM-SE `10.100.0.1` |
|
||||
| ESH home lab | `10.0.50.0/24` (`esteban.net`) | UDM-PM `10.0.0.1` |
|
||||
| Fountain Valley | `10.251.0.0/16` | |
|
||||
| Irvine | via headscale mesh | |
|
||||
|
||||
Full host table with roles: `~/development/eshpfi-management/CLAUDE.md` § Servers.
|
||||
Live per-host snapshots: `servers/<name>/system-details.txt`.
|
||||
Internal DNS names are `<host>.<site>.internal`, source of truth `dns/internal.yaml`.
|
||||
|
||||
## Mesh
|
||||
|
||||
Headscale is primary since 2026-09-06 (IPsec tunnels dormant). Subnet routers are
|
||||
`nh3-scale` / `ana-scale` / `esh-scale` at `100.64.0.1` / `.3` / `.2`.
|
||||
|
||||
⚠ **Disable a site-to-site tunnel BEFORE adding a mesh route for the same prefix** or
|
||||
they fight.
|
||||
⚠ **`accept-routes` needs a return path FIRST.** On a dev box it can black-hole its own
|
||||
LAN (route table 52 beats main). Gateway routes or SNAT before, never on your own box.
|
||||
|
||||
## Traps
|
||||
|
||||
- **SureFire tenant hosts are client property** under the hosting agreement. Coordinate
|
||||
before touching anything that affects data. This is a contractual posture, not a
|
||||
security one.
|
||||
- **Confirm a reboot by observing DOWN**, then return. Poll for disappearance and
|
||||
cross-check the boot timestamp — "never rebooted" and "rebooted very fast" look
|
||||
identical otherwise.
|
||||
- **`chroot` rbind needs `--make-rslave`.** Without it, `umount -R` unmounts the LIVE
|
||||
host's cgroup/devpts, which looks exactly like disk failure.
|
||||
- **`pct snapshot` exits 0 when it refuses** (bind-mounted guest). Use `zfs snapshot`
|
||||
and read it back.
|
||||
Reference in New Issue
Block a user