vh
39da1d4a97
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.
Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.
A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.
The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.
28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.
The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.
Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.
Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
arbo — catalog-driven ComfyUI engine
Arbo's comfy_engine (FastAPI) reads a versioned workflow catalog and drives
ComfyUI to generate images/video, serving an SPA + /workflows + /run.
Owned by comfy-dev (engine code + catalog + image build); this stack is the
infra side — compose, deploy, secrets, volumes, pull automation. See
vh/arbo docs/adr/0001-engine-production-deployment.md for the architecture.
Placement (ADR-0001)
- Host: irv-ml1, co-located beside the
comfyuistack (D1/D3). - engine ↔ ComfyUI:
http://comfyui:8188over the sharedtraefik-net(the "localhost-equivalent" — no scp, no WireGuard hop). File ops use the same host dirs comfyui mounts (/worktank/comfyui/basedir/{input,output}), bound into the engine — soARBO_COMFY_SSH_HOSTstays unset and the engine's local-cp path (comfy-dev's D3 follow-on) handles file ops.
Deploy
# 1. comfy-dev builds + pushes the image; pin the tag in .env (ARBO_IMAGE).
# 2. On irv-ml1, clone the catalog checkout (the git-pull target, D2):
# git clone git@gitea.phasefinal.com:vh/arbo.git /worktank/arbo/repo
# 3. Create stacks/arbo/.env from .env.example on the host, fill secrets.
# 4. From this workspace:
scripts/deploy-stack.sh irv-ml1 arbo # diffs vs live, prompts y/N
Q3 — Secrets
Both injected via the on-host .env (gitignored; never the althing bus),
the standard fleet pattern:
ENGINE_TOKEN— bearer auth is intentionally OFF (operator decision 2026-06-13). Prod is WireGuard-only; the perimeter is the boundary, so the per-browser token was friction with little added security. The composeENGINE_TOKENinjection is commented out and the.envline removed — the app's protected-gate no-ops only when the var is absent (an empty string still gates). To re-lock: un-comment the compose line, set a bearer in.env(openssl rand -hex 32),compose up -d. Reverses ADR-0001 (comfy-dev owns that ADR's update). Seeplaybooks/arbo-disable-engine-token.yaml.GRANITE_KEY— the LiteLLM virtual key scoped to arbo. Thearbo-prompt-enhancevkey (comfy-dev, issued 2026-06-09) was extended to reach bothgranite-4.1-8b(prompt-craft /genstep) andqwen3.5-9b-fp8(vision / the hero auto-judgestep, v0.11.3+) —/v1/modelsfor this key lists exactly those two. Reuse or rotate it; never the mastersk-corvid.
Q4 — Volumes + backup
| Volume | Holds | Durability |
|---|---|---|
arbo_db (named, local disk) |
gallery/history SQLite (ARBO_DB) |
backed up — restic/Backrest daily |
arbo_heroes (named, local disk) |
runtime hero images | persisted; regenerable, lower backup priority |
/worktank/arbo/repo (bind, ro) |
catalog checkout | reproducible from git — not backed up |
ARBO_DB is a named volume on local disk, NOT NFS (the DB-off-NFS rule;
SQLite-on-NFS locking is a foot-gun). The gallery DB is the durable asset → it
joins the restic file backup. (Adds irv-ml1's arbo_db path to a Backrest
repo — small; flag if irv-ml1 isn't yet a restic source.)
Q5 — Catalog-pull automation (IMPLEMENTED)
A gitea push-webhook → HMAC listener → git pull --reset reaches prod in one
action. The two host-side scripts are tracked here (they live on irv-ml1 at
~/arbo-webhook.py + ~/arbo-deploy.sh; copy them back if rebuilding the host):
arbo-webhook.py— HTTP listener on:9009. Validates the giteaX-Gitea-SignatureHMAC-SHA256 against~/.config/arbo/webhook-secret(secret stays on the host, NOT in git), and on a verified push torefs/heads/mainfiresarbo-deploy.shin a daemon thread.arbo-deploy.sh—git fetch+reset --hard origin/mainin/worktank/arbo/repo, then a targeted restart:catalog/changed →compose restart engine(catalog is loaded once at startup intoapp.state.cat).graphs//frontend/→ no restart (read per-request).src/orDockerfile→ warns "NEW IMAGE required" + skips (baked code needs a rebuild, not a restart; comfy-dev pins a tag, infra-ops redeploys).pyproject.toml/uv.lock→ ignored (they bump on every commit via SemVer etiquette, so they're not image signals).
Gotcha: the gitea-arbo ssh remote on irv-ml1 must point at the internal
gitea route 10.250.50.70:222 (the ana-docker container's git-SSH), NOT the public
gitea.phasefinal.com:22 — the public path fail2bans the host's egress IP and
silently wedges the webhook fetch. See docs/orientation.md → Git / gitea.
Items needing comfy-dev's image (jointly owned)
ARBO_IMAGEtag — the builtgitea.phasefinal.com/vh/arbo:<tag>./healthz— the healthcheck assumes it; comfy-dev is adding it.- Catalog in-container path — compose mounts the checkout's
catalog/+graphs/to/app/catalog+/app/graphs(ro). Confirm against the image's expected layout (or expose a catalog-dir env), so the engine reloads from the mount on restart. - Run-as UID — container runs
1000:1000to write comfyui's basedir + volumes cleanly; confirm the image tolerates a non-root user.