vh
66c860d6c1
Operator-directed. The wg0 lifeline retired at the 2026-09-06 headscale cutover is on no interface anywhere, so anything pointing at it gets no route at all. Homepage went from 9 dead cards to 0 of 112. The load-bearing part is that there is no single right target: it depends on who resolves it. The operator's browser and the Homepage and open-webui containers on esh-docker-vm all resolve nh3.internal, so those get the name and survive the next renumber. Containers on irv-ml1 and ana-docker cannot resolve it at all, so those get the IP. litellm on ana-docker looked like a counterexample and is not: it resolves the name only through its own extra_hosts entry, while asset-engine on the same host fails on it. Test from the container you are about to change, never from a neighbour. Before committing to the name I confirmed the Homepage container actually fetches ytvc's healthz through it in production rather than assuming resolution implies reach. On irv-ml1, 24 files swept and 14 comment-only hits left as port-allocation history. Seven running containers recreated so the labels took. Seven dormant ones carried stale labels because editing a compose file does not touch an existing container object - fixed with compose create --force-recreate, which rebuilds the container without starting it, the right tool for a deliberately dormant stack. The sweep's real find was off irv-ml1 entirely: four live values on two other hosts, silently dead for nine days and alerting nobody. Open WebUI's read-aloud TTS, asset-engine's inference host, and two skaldsong TTS URLs. Both running services were recreated and verified reaching their targets afterwards rather than merely carrying the new string. One self-inflicted outage worth recording: I recreated breeze-tts for a cosmetic label change and took ext-tts down for its ~90s CUDA-graph warm-up, returning 500. I caught it only because I had taken a baseline before touching it. A label-only edit still costs a full model reload on a GPU container.
asset-engine
Control plane over the PFI inference fleet — FastAPI + HTMX/Shoelace
UI that exposes the catalog at docs/asset-engine/services.yaml
as a web app, routes generation requests to inference hosts (irv-ml1
over WG by default), and persists generated assets to a local SQLite
DB + content-addressed blob store.
Server: ana-docker
URL: http://10.250.50.70:8200 (configurable via .env)
Upstream repo: vh/asset-engine
Image: asset-engine:local — built on the host from the git repo by
the deploy playbook. Not pulled from a registry.
Deploy
Two paths — automated (preferred) and manual (escape hatch / first-time).
Automated (Gitea Actions, push-to-main)
The asset-engine repo ships .gitea/workflows/{ci,deploy}.yaml. CI runs
on PRs (uv sync, pytest, catalog drift check against this repo's
docs/asset-engine/services.yaml); the deploy workflow runs on push to
main and just calls the elway playbook below pinned to the triggering
commit SHA. Drift check is a BLOCKING gate — a PR that vendors a
services.yaml mismatched against this repo fails CI and can't merge.
A reference copy of the deploy workflow lives next to this README at
gitea-workflow-deploy.yaml.example;
the canonical source is in the asset-engine repo. The example header
lists the two repo secrets required (DEPLOY_SSH_KEY, MGMT_REPO_TOKEN).
Manual (elway from a workstation)
The playbook owns the full flow: clone/update the source repo,
docker build, install compose + seed .env, bring up, verify health.
# First deploy (or update to latest main)
scripts/elway ana-docker --playbook playbooks/deploy-asset-engine.yaml
# Pin to a specific ref (tag, branch, or commit SHA)
scripts/elway ana-docker --playbook playbooks/deploy-asset-engine.yaml --var ref=v0.1.0
Path layout (on ana-docker)
| Host path | Container path | Purpose | Restic? |
|---|---|---|---|
/opt/docker/build/asset-engine/ |
— | git checkout used as docker build context | excluded |
/opt/docker/compose/asset-engine/ |
— | compose.yaml + .env | included (via /opt/docker) |
/opt/docker/conf/asset-engine/db/ |
/app/runtime/db |
SQLite (asset_engine.db + WAL) |
included |
/opt/docker/conf/asset-engine/outputs/ |
/app/runtime/outputs |
content-addressed blob store | included |
Network model
Internal tooling, LAN-only. Container port 8000 is published on the host
at 0.0.0.0:8200 (configurable via ASSET_ENGINE_BIND /
ASSET_ENGINE_PORT); access is direct via http://10.250.50.70:8200.
No Traefik, no TLS terminator, no public hostname. If we later need
TLS or external access, that's a separate decision.
INFERENCE_HOST defaults to 10.100.79.3 (irv-ml1 over WG). Override
in .env if the fleet's inference topology moves.
Catalog drift
docs/asset-engine/services.yaml in this repo is the canonical
catalog. The asset-engine repo vendors a copy at data/services.yaml
and re-vendors via uv run scripts/sync_catalog.py after upstream
changes; CI fails any PR where the vendored copy diverges from this
one. Pattern is: edit catalog here → asset-engine re-vendors → both
sides commit on the same merge window.
Outputs directory growth
outputs/ grows unbounded in v1 — Asset.retention exists in the
schema but the GC sweep isn't wired yet. The plan: Beszel alert when
du -sh /opt/docker/conf/asset-engine/outputs crosses ~50 GB,
revisit the threshold once we have real growth data. Tracking issue
in the asset-engine repo.