vh
39da1d4a97
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.
Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.
A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.
The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.
28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.
The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.
Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.
Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
gitea-runner
Self-hosted Gitea Actions
runner. Polls gitea.phasefinal.com for jobs from any repo that has a
.gitea/workflows/ directory and runs them in ephemeral docker
containers on this host.
Server: ana-docker (single central runner — see "Topology" below
for when to add more)
Image: gitea/act_runner:latest
Outbound only — no host port published; the runner connects out to
gitea, gitea never connects in.
Topology
We run one central runner on ana-docker. Reasoning:
- gitea is on ana-docker, so runner→API is local
- existing fleet tooling (
elway,sync-stacks,deploy-stack,refresh-server-info) already SSHes from one origin to all hosts; the runner inherits that pattern - single point to manage SSH keys, secrets, and runner upgrades
The deploy playbook (playbooks/deploy-gitea-runner.yaml) is
parameterized by host / runner-name / labels, so spinning up
nh3-docker-runner or an ESH runner later is a one-line elway
invocation — not a copy-pasted playbook.
When to add a site-local runner:
- Cross-site SSH from ana-docker to that site has become unreliable
- A workflow needs LAN access to something only reachable from inside that site's network segment
- You want failure isolation (NH3 can deploy itself when Anaheim is down)
Until one of those bites, one runner is enough.
Prereqs
Before running the deploy playbook:
-
Verify Gitea Actions is enabled. In gitea 1.21+ Actions ships on by default, but check
/-/admin/actionsresolves. If not, addGITEA__actions__ENABLED=trueto the gitea stack env and bounce. -
Generate a registration token. Pick the scope:
Scope URL Use when Global (admin) https://gitea.phasefinal.com/-/admin/actions/runnersrunner serves any repo on the instance (recommended for the central PFI runner) Org/user https://gitea.phasefinal.com/<owner>/-/actions/runnersrunner serves all repos under one owner Repo https://gitea.phasefinal.com/<owner>/<repo>/settings/actions/runnersrunner serves one repo Register-as-admin is right for our use case: one runner, fleet-wide.
-
Create an SSH deploy key for the runner that lets it execute the elway playbooks against fleet hosts. The key lives only on ana-docker (passed in as a workflow secret per repo, or mounted into the runner via volume — see "Wiring deploys" below).
-
(Optional) Create a Gitea PAT with
read:repositoryscope onvh/esh-pfi-infrastructure. Workflows need to check out the management repo to get at the playbooks; the auto-injectedGITHUB_TOKENonly works for the triggering repo.
Deploy
# Edit .env first if not using defaults — at minimum paste the registration token
$EDITOR stacks/gitea-runner/.env.example # template
# First-time deploy
scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml
# Site-local runner later (NH3 or ESH)
scripts/elway nh3-docker --playbook playbooks/deploy-gitea-runner.yaml \
--var runner_name=nh3-docker-runner --var runner_labels=pfi-fleet,nh3-docker
The playbook seeds .env from .env.example only if absent; for the
first run, copy .env.example to /opt/docker/compose/gitea-runner/.env
on the host and paste the registration token in before running, OR
let the playbook seed it and edit on the host before the
docker compose up -d step (it's idempotent — second run will pick up
the edited token).
After successful registration, the token is consumed (it's one-time
use). You can clear GITEA_RUNNER_REGISTRATION_TOKEN from .env;
the runner reads its permanent credentials from ${DATA_DIR}/.runner
on subsequent starts.
Wiring deploys
A workflow that runs on the central runner needs three things:
runs-on:matching a runner label —pfi-fleet(cross-fleet) orana-docker(pin to that host).- An SSH key to reach the deploy target. Stored as a repo or
org-level Actions secret named e.g.
DEPLOY_SSH_KEY. The corresponding public key must be in~lkraven/.ssh/authorized_keyson every host the workflow targets. - A token to clone
vh/esh-pfi-infrastructureif the workflow wants to invoke an elway playbook from this repo. Stored asMGMT_REPO_TOKEN(Gitea PAT,read:repositoryscope).
See stacks/task-board/gitea-workflow-deploy.yaml.example for a
complete deploy workflow that consumes all three.
Path layout (on ana-docker)
| Host path | Container path | Purpose | Restic? |
|---|---|---|---|
/opt/docker/compose/gitea-runner/ |
— | compose.yaml + .env | included (via /opt/docker) |
/opt/docker/conf/gitea-runner/data/ |
/data |
.runner creds, cache, job workspaces |
excluded (regenerable; nothing irreplaceable) |
Operations
# Tail runner logs
ssh ana-docker docker logs -f gitea-runner
# List currently registered runners (admin)
# https://gitea.phasefinal.com/-/admin/actions/runners
# Re-register (lost the .runner file? regenerate token, then:)
ssh ana-docker docker compose -f /opt/docker/compose/gitea-runner/compose.yaml down
ssh ana-docker rm /opt/docker/conf/gitea-runner/data/.runner
# paste new GITEA_RUNNER_REGISTRATION_TOKEN into .env
scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml
# Pin to a specific act_runner version
# Edit RUNNER_IMAGE in /opt/docker/compose/gitea-runner/.env, then:
scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml