esh-pfi-infrastructure/stacks/arbo

arbo — catalog-driven ComfyUI engine

Arbo's comfy_engine (FastAPI) reads a versioned workflow catalog and drives ComfyUI to generate images/video, serving an SPA + /workflows + /run. Owned by comfy-dev (engine code + catalog + image build); this stack is the infra side — compose, deploy, secrets, volumes, pull automation. See vh/arbo docs/adr/0001-engine-production-deployment.md for the architecture.

Placement (ADR-0001)

• Host: irv-ml1, co-located beside the comfyui stack (D1/D3).
• engine ↔ ComfyUI: http://comfyui:8188 over the shared traefik-net (the "localhost-equivalent" — no scp, no WireGuard hop). File ops use the same host dirs comfyui mounts (/worktank/comfyui/basedir/{input,output}), bound into the engine — so ARBO_COMFY_SSH_HOST stays unset and the engine's local-cp path (comfy-dev's D3 follow-on) handles file ops.

Deploy

# 1. comfy-dev builds + pushes the image; pin the tag in .env (ARBO_IMAGE).
# 2. On irv-ml1, clone the catalog checkout (the git-pull target, D2):
#    git clone git@gitea.phasefinal.com:vh/arbo.git /worktank/arbo/repo
# 3. Create stacks/arbo/.env from .env.example on the host, fill secrets.
# 4. From this workspace:
scripts/deploy-stack.sh irv-ml1 arbo          # diffs vs live, prompts y/N

Q3 — Secrets

Both injected via the on-host .env (gitignored; never the althing bus), the standard fleet pattern:

• ENGINE_TOKEN — bearer auth is intentionally OFF (operator decision 2026-06-13). Prod is WireGuard-only; the perimeter is the boundary, so the per-browser token was friction with little added security. The compose ENGINE_TOKEN injection is commented out and the .env line removed — the app's protected-gate no-ops only when the var is absent (an empty string still gates). To re-lock: un-comment the compose line, set a bearer in .env (openssl rand -hex 32), compose up -d. Reverses ADR-0001 (comfy-dev owns that ADR's update). See playbooks/arbo-disable-engine-token.yaml.
• GRANITE_KEY — the LiteLLM virtual key scoped to arbo. The arbo-prompt-enhance vkey (comfy-dev, issued 2026-06-09) was extended to reach both granite-4.1-8b (prompt-craft / gen step) and qwen3.5-9b-fp8 (vision / the hero auto-judge step, v0.11.3+) — /v1/models for this key lists exactly those two. Reuse or rotate it; never the master sk-corvid.

Q4 — Volumes + backup

Volume	Holds	Durability
arbo_db (named, local disk)	gallery/history SQLite (ARBO_DB)	backed up — restic/Backrest daily
arbo_heroes (named, local disk)	runtime hero images	persisted; regenerable, lower backup priority
/worktank/arbo/repo (bind, ro)	catalog checkout	reproducible from git — not backed up

ARBO_DB is a named volume on local disk, NOT NFS (the DB-off-NFS rule; SQLite-on-NFS locking is a foot-gun). The gallery DB is the durable asset → it joins the restic file backup. (Adds irv-ml1's arbo_db path to a Backrest repo — small; flag if irv-ml1 isn't yet a restic source.)

Q5 — Catalog-pull automation (IMPLEMENTED)

A gitea push-webhook → HMAC listener → git pull --reset reaches prod in one action. The two host-side scripts are tracked here (they live on irv-ml1 at ~/arbo-webhook.py + ~/arbo-deploy.sh; copy them back if rebuilding the host):

• arbo-webhook.py — HTTP listener on :9009. Validates the gitea X-Gitea-Signature HMAC-SHA256 against ~/.config/arbo/webhook-secret (secret stays on the host, NOT in git), and on a verified push to refs/heads/main fires arbo-deploy.sh in a daemon thread.
• arbo-deploy.sh — git fetch + reset --hard origin/main in /worktank/arbo/repo, then a targeted restart:
  • catalog/ changed → compose restart engine (catalog is loaded once at startup into app.state.cat).
  • graphs/ / frontend/ → no restart (read per-request).
  • src/ or Dockerfile → warns "NEW IMAGE required" + skips (baked code needs a rebuild, not a restart; comfy-dev pins a tag, infra-ops redeploys).
  • pyproject.toml / uv.lock → ignored (they bump on every commit via SemVer etiquette, so they're not image signals).

Gotcha: the gitea-arbo ssh remote on irv-ml1 must point at the internal gitea route 10.250.50.70:222 (the ana-docker container's git-SSH), NOT the public gitea.phasefinal.com:22 — the public path fail2bans the host's egress IP and silently wedges the webhook fetch. See docs/orientation.md → Git / gitea.

Items needing comfy-dev's image (jointly owned)

1. ARBO_IMAGE tag — the built gitea.phasefinal.com/vh/arbo:<tag>.
2. /healthz — the healthcheck assumes it; comfy-dev is adding it.
3. Catalog in-container path — compose mounts the checkout's catalog/ + graphs/ to /app/catalog + /app/graphs (ro). Confirm against the image's expected layout (or expose a catalog-dir env), so the engine reloads from the mount on restart.
4. Run-as UID — container runs 1000:1000 to write comfyui's basedir + volumes cleanly; confirm the image tolerates a non-root user.