vh
6d66bc2f30
Operator's call: keep the arbo stack in eshpfi and version its deploy machinery alongside the compose (was host-only on irv-ml1 = recoverability foot-gun). - arbo-webhook.py: :9009 HMAC listener (secret externalized to host file, not git) - arbo-deploy.sh: internal-route fetch + catalog-only targeted restart Document both in the README Q5 section + the internal-gitea-route gotcha.
97 lines
5.2 KiB
Markdown
97 lines
5.2 KiB
Markdown
# arbo — catalog-driven ComfyUI engine
|
|
|
|
Arbo's `comfy_engine` (FastAPI) reads a versioned workflow catalog and drives
|
|
**ComfyUI** to generate images/video, serving an SPA + `/workflows` + `/run`.
|
|
Owned by **comfy-dev** (engine code + catalog + image build); this stack is the
|
|
**infra side** — compose, deploy, secrets, volumes, pull automation. See
|
|
`vh/arbo` `docs/adr/0001-engine-production-deployment.md` for the architecture.
|
|
|
|
## Placement (ADR-0001)
|
|
|
|
- **Host:** irv-ml1, **co-located beside the `comfyui` stack** (D1/D3).
|
|
- **engine ↔ ComfyUI:** `http://comfyui:8188` over the shared `traefik-net`
|
|
(the "localhost-equivalent" — no scp, no WireGuard hop). File ops use the
|
|
**same host dirs** comfyui mounts (`/worktank/comfyui/basedir/{input,output}`),
|
|
bound into the engine — so `ARBO_COMFY_SSH_HOST` stays **unset** and the
|
|
engine's local-cp path (comfy-dev's D3 follow-on) handles file ops.
|
|
|
|
## Deploy
|
|
|
|
```bash
|
|
# 1. comfy-dev builds + pushes the image; pin the tag in .env (ARBO_IMAGE).
|
|
# 2. On irv-ml1, clone the catalog checkout (the git-pull target, D2):
|
|
# git clone git@gitea.phasefinal.com:vh/arbo.git /worktank/arbo/repo
|
|
# 3. Create stacks/arbo/.env from .env.example on the host, fill secrets.
|
|
# 4. From this workspace:
|
|
scripts/deploy-stack.sh irv-ml1 arbo # diffs vs live, prompts y/N
|
|
```
|
|
|
|
## Q3 — Secrets
|
|
|
|
Both injected via the on-host **`.env`** (gitignored; never the althing bus),
|
|
the standard fleet pattern:
|
|
|
|
- **`ENGINE_TOKEN`** — bearer auth is **intentionally OFF** (operator decision
|
|
2026-06-13). Prod is WireGuard-only; the perimeter is the boundary, so the
|
|
per-browser token was friction with little added security. The compose
|
|
`ENGINE_TOKEN` injection is commented out **and** the `.env` line removed — the
|
|
app's protected-gate no-ops only when the var is **absent** (an empty string
|
|
still gates). To re-lock: un-comment the compose line, set a bearer in `.env`
|
|
(`openssl rand -hex 32`), `compose up -d`. Reverses ADR-0001 (comfy-dev owns
|
|
that ADR's update). See `playbooks/arbo-disable-engine-token.yaml`.
|
|
- **`GRANITE_KEY`** — the LiteLLM **virtual key scoped to arbo**. The
|
|
`arbo-prompt-enhance` vkey (comfy-dev, issued 2026-06-09) was **extended**
|
|
to reach **both** `granite-4.1-8b` (prompt-craft / `gen` step) **and**
|
|
`qwen3.5-9b-fp8` (vision / the hero auto-`judge` step, v0.11.3+) — `/v1/models`
|
|
for this key lists exactly those two. Reuse or rotate it; never the master
|
|
`sk-corvid`.
|
|
|
|
## Q4 — Volumes + backup
|
|
|
|
| Volume | Holds | Durability |
|
|
|---|---|---|
|
|
| `arbo_db` (named, local disk) | gallery/history SQLite (`ARBO_DB`) | **backed up** — restic/Backrest daily |
|
|
| `arbo_heroes` (named, local disk) | runtime hero images | persisted; regenerable, lower backup priority |
|
|
| `/worktank/arbo/repo` (bind, ro) | catalog checkout | reproducible from git — not backed up |
|
|
|
|
`ARBO_DB` is a **named volume on local disk, NOT NFS** (the DB-off-NFS rule;
|
|
SQLite-on-NFS locking is a foot-gun). The gallery DB is the durable asset → it
|
|
joins the restic file backup. (Adds irv-ml1's `arbo_db` path to a Backrest
|
|
repo — small; flag if irv-ml1 isn't yet a restic source.)
|
|
|
|
## Q5 — Catalog-pull automation (IMPLEMENTED)
|
|
|
|
A gitea push-webhook → HMAC listener → `git pull --reset` reaches prod in one
|
|
action. The two host-side scripts are tracked here (they live on irv-ml1 at
|
|
`~/arbo-webhook.py` + `~/arbo-deploy.sh`; copy them back if rebuilding the host):
|
|
|
|
- **[`arbo-webhook.py`](arbo-webhook.py)** — HTTP listener on `:9009`. Validates
|
|
the gitea `X-Gitea-Signature` HMAC-SHA256 against `~/.config/arbo/webhook-secret`
|
|
(secret stays on the host, NOT in git), and on a verified push to `refs/heads/main`
|
|
fires `arbo-deploy.sh` in a daemon thread.
|
|
- **[`arbo-deploy.sh`](arbo-deploy.sh)** — `git fetch` + `reset --hard origin/main`
|
|
in `/worktank/arbo/repo`, then a **targeted** restart:
|
|
- **`catalog/`** changed → `compose restart engine` (catalog is loaded once at
|
|
startup into `app.state.cat`).
|
|
- **`graphs/` / `frontend/`** → no restart (read per-request).
|
|
- **`src/` or `Dockerfile`** → warns "NEW IMAGE required" + skips (baked code
|
|
needs a rebuild, not a restart; comfy-dev pins a tag, infra-ops redeploys).
|
|
- **`pyproject.toml` / `uv.lock`** → ignored (they bump on every commit via
|
|
SemVer etiquette, so they're not image signals).
|
|
|
|
**Gotcha:** the `gitea-arbo` ssh remote on irv-ml1 must point at the **internal**
|
|
gitea route `10.250.50.70:222` (the ana-docker container's git-SSH), NOT the public
|
|
`gitea.phasefinal.com:22` — the public path fail2bans the host's egress IP and
|
|
silently wedges the webhook fetch. See `docs/orientation.md` → Git / gitea.
|
|
|
|
## Items needing comfy-dev's image (jointly owned)
|
|
|
|
1. **`ARBO_IMAGE` tag** — the built `gitea.phasefinal.com/vh/arbo:<tag>`.
|
|
2. **`/healthz`** — the healthcheck assumes it; comfy-dev is adding it.
|
|
3. **Catalog in-container path** — compose mounts the checkout's `catalog/` +
|
|
`graphs/` to `/app/catalog` + `/app/graphs` (ro). Confirm against the image's
|
|
expected layout (or expose a catalog-dir env), so the engine reloads from the
|
|
mount on restart.
|
|
4. **Run-as UID** — container runs `1000:1000` to write comfyui's basedir +
|
|
volumes cleanly; confirm the image tolerates a non-root user.
|