diff --git a/docs/asset-engine/CATALOG-CONTRACT.md b/docs/asset-engine/CATALOG-CONTRACT.md index 52d74a5..fb12f16 100644 --- a/docs/asset-engine/CATALOG-CONTRACT.md +++ b/docs/asset-engine/CATALOG-CONTRACT.md @@ -1,5 +1,10 @@ # services.yaml — contract for downstream consumers +> **RETIRED 2026-10-17 (Prime).** asset-engine, this catalog's only consumer, is `compose down` on ana-docker. +> The catalog is NOT maintained: every irv-ml1 endpoint in `services.yaml` still points at `10.100.79.3`, the wg0 +> lifeline that died at the 09-06 mesh cutover, and several of those ports now belong to other services. Kept as +> the record and for a bring-back, which would need a full per-entry refresh first. + `services.yaml` in this directory is the **canonical, versioned** description of the inference services this fleet exposes. It is a first-class artifact: external tools (the asset-engine UI is the diff --git a/docs/fleettools/AGENT-BOOTSTRAP.md b/docs/fleettools/AGENT-BOOTSTRAP.md index 90bedc9..bb114bc 100644 --- a/docs/fleettools/AGENT-BOOTSTRAP.md +++ b/docs/fleettools/AGENT-BOOTSTRAP.md @@ -17,7 +17,6 @@ their own: ```sh curl -s http://10.0.50.45:5100/api/services # every fleet service, curated -curl -s http://10.250.50.70:8200/api/v1/services # inference seats + GPU state curl -s http://10.250.50.70:4000/v1/models # the LLM roster (needs a key) ``` diff --git a/docs/fleettools/FLEETTOOLS.md b/docs/fleettools/FLEETTOOLS.md index 740a80a..2ce4627 100644 --- a/docs/fleettools/FLEETTOOLS.md +++ b/docs/fleettools/FLEETTOOLS.md @@ -14,12 +14,11 @@ directory; that defeats the point. ## Rule zero: query live inventories, never a written list -Three endpoints answer "what exists right now" better than any document, and they stay +Two endpoints answer "what exists right now" better than any document, and they stay correct on their own. **Reach for these before asking anyone:** ```sh curl -s http://10.0.50.45:5100/api/services # every fleet service, curated -curl -s http://10.250.50.70:8200/api/v1/services # inference seats + GPU state curl -s -H "Authorization: Bearer $KEY" \ http://10.250.50.70:4000/v1/models # the LLM roster (~40 models) ``` @@ -70,13 +69,13 @@ live contract for its API. Fetch it rather than trusting a transcription. ⚠ *The shared key CAN spend real vendor credits.* *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/litellm.md` -- **Direct inference seats + Asset Engine** — the control plane at - `http://10.250.50.70:8200` enumerates seats, reads GPU state, and starts/stops them. +- **Direct inference seats** — most sit behind the gateway; the GPU hosts and their traps. + (The Asset Engine control plane at `:8200` was RETIRED 2026-10-17.) *When:* the gateway does not expose what you need, or a seat is down. *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/inference.md` - **Speech (TTS/ASR)** — Breeze-TTS-2 is the fleet primary; Chatterbox-fast is the - low-latency chat front end; Speaches and Parakeet do ASR. + low-latency chat front end; Parakeet does ASR (`whisper-1` / `ext-stt` on the gateway). *When:* you need audio out of text, or text out of audio. *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/speech.md` diff --git a/docs/fleettools/inference.md b/docs/fleettools/inference.md index a086187..90934ea 100644 --- a/docs/fleettools/inference.md +++ b/docs/fleettools/inference.md @@ -1,4 +1,4 @@ -# Direct inference seats + the Asset Engine control plane +# Direct inference seats Most seats are vLLM behind the LiteLLM gateway — prefer the gateway (`docs/fleettools/litellm.md`) unless you need something it does not expose. @@ -8,30 +8,20 @@ Most seats are vLLM behind the LiteLLM gateway — prefer the gateway ```sh # every fleet service, curated, with hrefs and descriptions curl -s http://10.0.50.45:5100/api/services | python3 -m json.tool - -# the inference control plane: what exists, what is up, GPU state -curl -s http://10.250.50.70:8200/api/v1/services -curl -s http://10.250.50.70:8200/api/v1/gpus +# the LLM roster +curl -s -H "Authorization: Bearer $KEY" http://10.250.50.70:4000/v1/models ``` Seats move between hosts and GPUs often enough that any transcribed table here would mislead within a month. -## Asset Engine — `http://10.250.50.70:8200` +## Asset Engine — RETIRED 2026-10-17 -Control plane over the PFI inference fleet. Can enumerate services, read GPU state, -**start and stop seats**, submit jobs and fetch assets. - -``` -GET /api/v1/services GET /api/v1/services/{id}/state -POST /api/v1/services/{id}/start POST /api/v1/services/{id}/stop -GET /api/v1/gpus POST /api/v1/jobs -GET /api/v1/assets/{id}/blob GET /api/v1/presets -``` - -⚠ **Liveness is keyed on the compose PROJECT name.** A seat reads OFFLINE unless the -compose project name equals the catalog's `lifecycle.stack`. Pin `name: ` in -the compose file. +The control plane at `http://10.250.50.70:8200` is gone (Prime). Its last job was 2026-08-02, and every +catalog endpoint pointed at irv-ml1's dead wg0 address after the 09-06 mesh cutover. `docker compose +down` on ana-docker; the DB, image, repo (`vh/asset-engine`) and catalog (`docs/asset-engine/`) are kept, so +`docker compose up -d` in `/opt/docker/compose/asset-engine` brings it back. Start/stop a seat by hand +(`docker compose up -d` / `stop` in its compose dir on the GPU host). ## GPU hosts diff --git a/docs/fleettools/speech.md b/docs/fleettools/speech.md index eef96cb..acba563 100644 --- a/docs/fleettools/speech.md +++ b/docs/fleettools/speech.md @@ -33,7 +33,7 @@ the router cooldown** — list voices first. Retired, verified 2026-10-17: **Speaches** (irv-ml1 :8204 is now Breeze-TTS-2), the sherpa-onnx `parakeet` container on fv-ml1 (stopped; NeMo replaced it), and the asset-engine `parakeet` entry at `10.100.79.3:8765` (the dead wg0 -lifeline address). The first two's leftover containers were removed 2026-10-17 (images and compose dirs kept), so +lifeline address; asset-engine itself was retired 2026-10-17). The first two's leftover containers were removed 2026-10-17 (images and compose dirs kept), so Homepage no longer lists them. ## Traps diff --git a/persistent-memory.md b/persistent-memory.md index 8f331be..1beec0c 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -30,7 +30,7 @@ Sister repos (separate gitea repos, deployed by playbooks here): | `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) — **[2026-09-24] MOTHBALLED** by Prime (superseded by the High Seat + ledger); container removed on ana-docker, data/image/compose kept (`e6da607`) | push-to-main → CI deploys (2026-04-29) | | `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) | | `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) | -| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) | +| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) — **[2026-10-17] RETIRED** by Prime: `compose down` on ana-docker (DB in `/opt/docker/conf/asset-engine/db` incl. its unmerged WAL, image `asset-engine:local`, repo all kept; bring-back = `docker compose up -d`). Last job 2026-08-02, and every catalog endpoint had been dead since the 09-06 mesh cutover. ⚠ CI still deploys on a push to main, so a push would resurrect it | push-to-main → CI deploys (2026-05-12) | | `vh/althing` | Lean trusted inter-agent message bus — **v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback)**: ONE container on nh3-dev at `http://10.100.50.40:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. **Every v2 command was DELETED, not deprecated** — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` | per-box install (NOT CI-deploy); **nh3-dev** = container host + repo; **nh3-extdev** = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`) | | `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) | | `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) | @@ -185,6 +185,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- ## Recent decisions +- `[2026-10-17]` **asset-engine RETIRED (Prime)**, along with the stale speech entries: the Homepage Speaches (irv-ml1) and sherpa Parakeet (fv-ml1) ghosts were removed via `docker rm` of their dead containers (images and compose dirs kept); the Uptime Kuma monitor was deleted (`scripts/kuma delete`, new); asset-engine is `compose down` (data kept). Evidence: last job 08-02; all 21 catalog endpoints on the dead 10.100.79.3. FleetTools, the agent bootstrap and global CLAUDE.md (dotfiles f909dcf, unpushed) no longer list `:8200`. - `[2026-10-16]` **albok unit 6: the first encrypted wing `personal/sealed` provisioned on nh3-albok** (Prime's goal "land all of u6", via albok-dev): a 20 GiB LUKS2 file + ext4, key ONLY in vault `albok/wing-key-personal-sealed`, gid 1513, mountpoint `/srv/albok/sealed/personal.sealed` (immutable while unmounted), the parent bound rslave. The lock/unlock lifecycle was tested with the container running. It is not in albok.yaml until albok-dev's release. Runbook: `stacks/albok-service/README.md` § Sealed. After an nh3-albok reboot the wing is LOCKED until someone unlocks it. - `[2026-10-13]` **pewpewstudio LIVE on nh3-docker :8185** (Prime via pewpew-dev): api + Caddy front; the Host allowlist must sit in a `route` (foreign Host 400); DB, assets and shop config on named volumes. → `stacks/pewpewstudio/README.md` - `[2026-10-13]` **infra-ops on esh-nas (CT 103) + per-host satellite keys (Prime)** — bootstrapped via pct from esh-pve-nas; satellite-1 authorized on esh-nas only (`playbooks/files/infra-ops-satellite-keys`); its login test + name pending. diff --git a/scripts/kuma b/scripts/kuma index 1272910..427c232 100755 --- a/scripts/kuma +++ b/scripts/kuma @@ -421,6 +421,25 @@ def cmd_add(args): k.__exit__() +def cmd_delete(args): + """Remove ONE monitor by its exact name. `seed` never prunes (a missing row in + monitors.yaml is not a delete), so retiring a service needs this as well as the + yaml edit. Exits 1 if the name is not on the board.""" + k = connected_and_logged_in() + try: + mons = k.monitors_by_name() + if args.name not in mons: + print(f"no monitor named {args.name!r}", file=sys.stderr) + sys.exit(1) + k.delete(mons[args.name]["id"]) + if args.name in k.monitors_by_name(): + print(f"delete of {args.name!r} did not take", file=sys.stderr) + sys.exit(1) + print(f"deleted {args.name}") + finally: + k.__exit__() + + def main(): p = argparse.ArgumentParser(prog="kuma", description=__doc__.split("\n")[0]) sub = p.add_subparsers(dest="cmd", required=True) @@ -432,6 +451,7 @@ def main(): a.add_argument("--type", default="http", choices=["http", "ping", "port"]) a.add_argument("--port", type=int) a.set_defaults(fn=cmd_add) + d = sub.add_parser("delete"); d.add_argument("name"); d.set_defaults(fn=cmd_delete) sub.add_parser("dedupe").set_defaults(fn=cmd_dedupe) sub.add_parser("channels").set_defaults(fn=cmd_channels) args = p.parse_args() diff --git a/stacks/uptimekuma/monitors.yaml b/stacks/uptimekuma/monitors.yaml index 5ebbcfc..bb458b4 100644 --- a/stacks/uptimekuma/monitors.yaml +++ b/stacks/uptimekuma/monitors.yaml @@ -89,8 +89,7 @@ monitors: url: http://10.250.50.70:4000/health/liveliness description: liveliness endpoint, so a model outage does not read as a gateway outage - - name: Asset Engine - url: http://10.250.50.70:8200/api/v1/services + # Asset Engine: RETIRED 2026-10-17 (Prime), monitor deleted with `scripts/kuma delete`. # ---- fleet embed/rerank: the one EXCEPTION to "seats are OUT" ---- # Since 2026-09-25 these back the gateway's `qwen3-embedding` and `reranker`