esh-pfi-infrastructure/stacks/gitea-runner

gitea-runner

Self-hosted Gitea Actions runner. Polls gitea.phasefinal.com for jobs from any repo that has a .gitea/workflows/ directory and runs them in ephemeral docker containers on this host.

Server: ana-docker (the central runner), plus the opt-in fast runner on nh3-docker (see "Topology") Image: gitea/act_runner:latest Outbound only — no host port published; the runner connects out to gitea, gitea never connects in.

Topology

We run one central runner on ana-docker. Reasoning:

• gitea is on ana-docker, so runner→API is local
• existing fleet tooling (elway, sync-stacks, deploy-stack, refresh-server-info) already SSHes from one origin to all hosts; the runner inherits that pattern
• single point to manage SSH keys, secrets, and runner upgrades

The deploy playbook (playbooks/deploy-gitea-runner.yaml) is parameterized by host / runner-name / labels, so spinning up nh3-docker-runner or an ESH runner later is a one-line elway invocation — not a copy-pasted playbook.

When to add a site-local runner:

• Cross-site SSH from ana-docker to that site has become unreliable
• A workflow needs LAN access to something only reachable from inside that site's network segment
• You want failure isolation (NH3 can deploy itself when Anaheim is down)

Until one of those bites, one runner is enough.

The opt-in fast runner on nh3-docker (Prime's CI fix A, 2026-10-12)

nh3-docker-runner (gitea runner id 4, act_runner v0.6.1) carries one label, nh3-docker, and NOT pfi-fleet. Nothing lands on it unless a workflow says runs-on: nh3-docker. It exists because ana-docker's disk is slow to fsync: pfi-pve's ospool is HDD mirrors with no SLOG. Jobs that fsync a lot (a Postgres service, pytest temp dirs) crawled into the 30 m timeout there.

Measured 2026-10-12 1725 in a throwaway repo (vh/ci-probe-nh3, deleted after). One run had a job on each runner with the same steps (8 KiB dd oflag=dsync × 200, n = 3 each):

	nh3-docker-runner	ana-docker-runner
200 synced 8 KiB writes	0.51 / 0.54 / 0.57 s (~2.7 ms each)	6.36 / 6.82 / 7.21 s (~34 ms each)
docker socket in the job	absent	present (root over the host)
job memory / CPU limit	4 GiB / 6 CPUs	none
services: container	Postgres reachable after 1 s	(not tested here)

Its config is conf/config.nh3-docker.yaml, tighter than ana-docker's because nh3-docker also runs NH3's DNS (adguardhome) and the althing post office on 8 vCPU / 7 GiB:

• docker_host: "-" keeps the docker socket out of jobs (the probe above proves it). services: still work.
• capacity: 1, and each job container is capped at --memory=4g --memory-swap=4g --cpus=6.
• network: "" gives every job its own network instead of traefik-net.
• The timeout stays at 30 m.

A job that needs docker inside it (buildx and the like) cannot run here; keep those on pfi-fleet. ⚠ nh3-docker's vCPU has no AVX2 either, so CPU-bound torch tests are no faster here. This runner fixes the disk, not the CPU.

Deployed with:

scripts/elway infra-ops@10.100.50.40 --playbook playbooks/deploy-gitea-runner.yaml \
  --var runner_name=nh3-docker-runner --var runner_labels=nh3-docker:docker://node:20-bookworm-slim \
  --var runner_config=stacks/gitea-runner/conf/config.nh3-docker.yaml

Use infra-ops@<ip>, not the nh3-docker alias: the alias logs in as lkraven, and the first attempt left a lkraven-owned data dir that infra-ops could not write. The registration token came from the admin API (POST /api/v1/admin/actions/runners/registration-token) and was cleared from .env once the runner registered.

What a job on this runner can actually do

Measured 2026-09-02 on pfi-fleet (throwaway repo, three jobs, since deleted). Recorded because two of these are commonly assumed the other way.

capability	result
services: containers	yes — Postgres 16 answered on the service name as hostname after ~6 s; wait on pg_isready, not on ordering
host docker daemon	yes, root-equivalent — /var/run/docker.sock is in every job container, docker ps showed all 49 host containers, docker compose v2.33.0 on PATH
uses: from the local mirrors	yes — uses: https://gitea.phasefinal.com/actions/checkout@v4 resolves and runs, with no DEFAULT_ACTIONS_URL change

⚠ container.valid_volumes: [] does not keep docker out of jobs. act_runner mounts the daemon socket itself, independently of that list, so a tight-looking valid_volumes is not containment. Any repo the runner serves — it is registered instance-wide — can control everything on ana-docker, gitea included. Four repos (vh/Worldtree, vh/soong-lab, vh/skaldsong, vh/wt-matrix-bridge) drive buildx through it, so it is load-bearing and closing it would break their CI. Isolate sensitive builds onto a dedicated runner rather than tightening this one.

⚠ Job images need a node binary. JS actions execute as node /var/run/act/..., so python:3-slim fails on the first uses:. Use node:20-bookworm, or docker:cli plus apk add --no-cache git nodejs when the job also builds images.

Full-URL uses: is the un-parked half of the github-independence work. The global DEFAULT_ACTIONS_URL=self flip is still blocked on act_runner's action-fetch auth, but a per-workflow full-URL ref needs neither the flip nor the auth path. Mirrors live under the actions and astral-sh orgs, all public: checkout, cache, upload-artifact, download-artifact, setup-node, setup-python, setup-uv.

Polling a run from the API: use /actions/runs, not /actions/tasks — on gitea 1.26.1 tasks returned an empty workflow_runs for a run that runs listed and executed.

Prereqs

Before running the deploy playbook:

1. Verify Gitea Actions is enabled. In gitea 1.21+ Actions ships on by default, but check /-/admin/actions resolves. If not, add GITEA__actions__ENABLED=true to the gitea stack env and bounce.

2. Generate a registration token. Pick the scope:

   Scope	URL	Use when
   Global (admin)	https://gitea.phasefinal.com/-/admin/actions/runners	runner serves any repo on the instance (recommended for the central PFI runner)
   Org/user	https://gitea.phasefinal.com/<owner>/-/actions/runners	runner serves all repos under one owner
   Repo	https://gitea.phasefinal.com/<owner>/<repo>/settings/actions/runners	runner serves one repo

   Register-as-admin is right for our use case: one runner, fleet-wide.

3. Create an SSH deploy key for the runner that lets it execute the elway playbooks against fleet hosts. The key lives only on ana-docker (passed in as a workflow secret per repo, or mounted into the runner via volume — see "Wiring deploys" below).

4. (Optional) Create a Gitea PAT with read:repository scope on vh/esh-pfi-infrastructure. Workflows need to check out the management repo to get at the playbooks; the auto-injected GITHUB_TOKEN only works for the triggering repo.

Deploy

# Edit .env first if not using defaults — at minimum paste the registration token
$EDITOR stacks/gitea-runner/.env.example   # template

# First-time deploy
scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml

# Site-local runner later (NH3 or ESH)
scripts/elway nh3-docker --playbook playbooks/deploy-gitea-runner.yaml \
  --var runner_name=nh3-docker-runner --var runner_labels=pfi-fleet,nh3-docker

The playbook seeds .env from .env.example only if absent; for the first run, copy .env.example to /opt/docker/compose/gitea-runner/.env on the host and paste the registration token in before running, OR let the playbook seed it and edit on the host before the docker compose up -d step (it's idempotent — second run will pick up the edited token).

After successful registration, the token is consumed (it's one-time use). You can clear GITEA_RUNNER_REGISTRATION_TOKEN from .env; the runner reads its permanent credentials from ${DATA_DIR}/.runner on subsequent starts.

Wiring deploys

A workflow that runs on the central runner needs three things:

1. runs-on: matching a runner label — pfi-fleet (cross-fleet) or ana-docker (pin to that host).

   ⚠ A label LIST is AND, not fallback. runs-on: [vastblue, pfi-fleet] requires a runner carrying both labels — it does not try the first and fall back to the second. A list naming a label no runner has matches nothing and the job queues forever rather than failing, which is the worst of the three outcomes because the board looks busy rather than broken. Use one label, and when a workflow should move to a new runner, edit the label deliberately at that moment. (Caught by eitri-smithy-dev on 2026-09-02 against vastblue's U1 contract, after infra-ops wrote "one-line fallback to pfi-fleet" meaning a manual edit and was read as scheduler behaviour. Standard Actions semantics, not a gitea quirk — the ambiguity was in the prose, not the product.)

2. An SSH key to reach the deploy target. Stored as a repo or org-level Actions secret named e.g. DEPLOY_SSH_KEY. The corresponding public key must be in ~lkraven/.ssh/authorized_keys on every host the workflow targets.

3. A token to clone vh/esh-pfi-infrastructure if the workflow wants to invoke an elway playbook from this repo. Stored as MGMT_REPO_TOKEN (Gitea PAT, read:repository scope).

See stacks/task-board/gitea-workflow-deploy.yaml.example for a complete deploy workflow that consumes all three.

Path layout (on ana-docker)

Host path	Container path	Purpose	Restic?
/opt/docker/compose/gitea-runner/	—	compose.yaml + .env	included (via /opt/docker)
/opt/docker/conf/gitea-runner/data/	/data	.runner creds, cache, job workspaces	excluded (regenerable; nothing irreplaceable)

Operations

# Tail runner logs
ssh ana-docker docker logs -f gitea-runner

# List currently registered runners (admin)
# https://gitea.phasefinal.com/-/admin/actions/runners

# Re-register (lost the .runner file? regenerate token, then:)
ssh ana-docker docker compose -f /opt/docker/compose/gitea-runner/compose.yaml down
ssh ana-docker rm /opt/docker/conf/gitea-runner/data/.runner
# paste new GITEA_RUNNER_REGISTRATION_TOKEN into .env
scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml

# Pin to a specific act_runner version
# Edit RUNNER_IMAGE in /opt/docker/compose/gitea-runner/.env, then:
scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml