esh-pfi-infrastructure/stacks/gitea-runner/README.md

# gitea-runner

Self-hosted [Gitea Actions](https://docs.gitea.com/usage/actions/overview)
runner. Polls `gitea.phasefinal.com` for jobs from any repo that has a
`.gitea/workflows/` directory and runs them in ephemeral docker
containers on this host.

**Server:** ana-docker (the central runner), plus the opt-in fast runner on nh3-docker (see "Topology")
**Image:** `gitea/act_runner:latest`
**Outbound only** — no host port published; the runner connects out to
gitea, gitea never connects in.

## Topology

We run **one central runner on ana-docker**. Reasoning:

- gitea is on ana-docker, so runner→API is local
- existing fleet tooling (`elway`, `sync-stacks`, `deploy-stack`,
  `refresh-server-info`) already SSHes from one origin to all hosts;
  the runner inherits that pattern
- single point to manage SSH keys, secrets, and runner upgrades

The deploy playbook (`playbooks/deploy-gitea-runner.yaml`) is
parameterized by host / runner-name / labels, so spinning up
`nh3-docker-runner` or an ESH runner later is a one-line elway
invocation — not a copy-pasted playbook.

**When to add a site-local runner:**

- Cross-site SSH from ana-docker to that site has become unreliable
- A workflow needs LAN access to something only reachable from inside
  that site's network segment
- You want failure isolation (NH3 can deploy itself when Anaheim is down)

Until one of those bites, one runner is enough.

### The opt-in fast runner on nh3-docker (Prime's CI fix A, 2026-10-12)

`nh3-docker-runner` (gitea runner id 4, act_runner v0.6.1) carries **one label, `nh3-docker`**, and NOT `pfi-fleet`.
Nothing lands on it unless a workflow says `runs-on: nh3-docker`. It exists because ana-docker's disk is slow to
fsync: pfi-pve's `ospool` is HDD mirrors with no SLOG. Jobs that fsync a lot (a Postgres service, pytest temp dirs)
crawled into the 30 m timeout there.

Measured 2026-10-12 1725 in a throwaway repo (`vh/ci-probe-nh3`, deleted after). One run had a job on each runner
with the same steps (8 KiB `dd oflag=dsync` × 200, n = 3 each):

| | nh3-docker-runner | ana-docker-runner |
|---|---|---|
| 200 synced 8 KiB writes | 0.51 / 0.54 / 0.57 s (~2.7 ms each) | 6.36 / 6.82 / 7.21 s (~34 ms each) |
| docker socket in the job | **absent** | present (root over the host) |
| job memory / CPU limit | 4 GiB / 6 CPUs | none |
| `services:` container | Postgres reachable after 1 s | (not tested here) |

Its config is `conf/config.nh3-docker.yaml`, tighter than ana-docker's because nh3-docker also runs NH3's DNS
(adguardhome) and the althing post office on 8 vCPU / 7 GiB:
- `docker_host: "-"` keeps the docker socket out of jobs (the probe above proves it). `services:` still work.
- `capacity: 1`, and each job container is capped at `--memory=4g --memory-swap=4g --cpus=6`.
- `network: ""` gives every job its own network instead of traefik-net.
- The timeout stays at 30 m.

**First real use:** vastblue/platform's python job moved here at c02048c. Run 11042 passed in 5 m; pytest
(~1,234 tests) took 4m45s, against 14m17s on ana-docker in run 11033. That is one run each, at different times of
day, so read it as a ballpark. The Postgres service, apt and npm all worked without the socket.

**Second repo: vh/albok** (test job moved at 27d2bb5). Green as of run 11051 (2026-10-12 2312): 1,415 passed in
9m43s, a job of 11m29s. ⚠ **Correction to infra-ops's 10-12 diagnosis:** albok's CI was never a hanging test. The 169
service tests fsync their journal and stores on every test: they take 57 s on a laptop and took over 25 min on
ana-docker, and pytest-timeout never fired. The 4 GiB job cap here then exposed a real bug: a chromadb thread leak,
26,710 threads / 3.2 GB at peak, and EAGAIN at test 1,273. albok fixed it with a conftest. Note also that job
containers run as root, so tests relying on `chmod 0` must skip.

A job that needs docker inside it (buildx and the like) cannot run here; keep those on `pfi-fleet`. ⚠ nh3-docker's
vCPU has no AVX2 either, so CPU-bound torch tests are no faster here. This runner fixes the disk, not the CPU.

Deployed with:
```bash
scripts/elway infra-ops@10.100.50.40 --playbook playbooks/deploy-gitea-runner.yaml \
  --var runner_name=nh3-docker-runner --var runner_labels=nh3-docker:docker://node:20-bookworm-slim \
  --var runner_config=stacks/gitea-runner/conf/config.nh3-docker.yaml
```
Use `infra-ops@<ip>`, not the `nh3-docker` alias: the alias logs in as lkraven, and the first attempt left a
lkraven-owned data dir that infra-ops could not write. The registration token came from the admin API
(`POST /api/v1/admin/actions/runners/registration-token`) and was cleared from `.env` once the runner registered.

## What a job on this runner can actually do

Measured 2026-09-02 on `pfi-fleet` (throwaway repo, three jobs, since deleted).
Recorded because two of these are commonly assumed the other way.

| capability | result |
|---|---|
| `services:` containers | **yes** — Postgres 16 answered on the service name as hostname after ~6 s; wait on `pg_isready`, not on ordering |
| host docker daemon | **yes, root-equivalent** — `/var/run/docker.sock` is in every job container, `docker ps` showed all 49 host containers, `docker compose v2.33.0` on PATH |
| `uses:` from the local mirrors | **yes** — `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves and runs, with no `DEFAULT_ACTIONS_URL` change |

⚠ **`container.valid_volumes: []` does not keep docker out of jobs.** act_runner
mounts the daemon socket itself, independently of that list, so a tight-looking
`valid_volumes` is not containment. Any repo the runner serves — it is registered
instance-wide — can control everything on ana-docker, gitea included. Four repos
(`vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge`) drive
buildx through it, so it is load-bearing and closing it would break their CI.
**Isolate sensitive builds onto a dedicated runner rather than tightening this one.**

⚠ **Job images need a node binary.** JS actions execute as `node /var/run/act/...`,
so `python:3-slim` fails on the first `uses:`. Use `node:20-bookworm`, or
`docker:cli` plus `apk add --no-cache git nodejs` when the job also builds images.

**Full-URL `uses:` is the un-parked half of the github-independence work.** The
global `DEFAULT_ACTIONS_URL=self` flip is still blocked on act_runner's
action-fetch auth, but a per-workflow full-URL ref needs neither the flip nor the
auth path. Mirrors live under the `actions` and `astral-sh` orgs, all public:
checkout, cache, upload-artifact, download-artifact, setup-node, setup-python,
setup-uv.

**Polling a run from the API:** use `/actions/runs`, not `/actions/tasks` — on
gitea 1.26.1 `tasks` returned an empty `workflow_runs` for a run that `runs` listed
and executed.

## Prereqs

Before running the deploy playbook:

1. **Verify Gitea Actions is enabled.** In gitea 1.21+ Actions ships
   on by default, but check `/-/admin/actions` resolves. If not, add
   `GITEA__actions__ENABLED=true` to the gitea stack env and bounce.

2. **Generate a registration token.** Pick the scope:

   | Scope | URL | Use when |
   |---|---|---|
   | Global (admin) | `https://gitea.phasefinal.com/-/admin/actions/runners` | runner serves any repo on the instance (recommended for the central PFI runner) |
   | Org/user | `https://gitea.phasefinal.com/<owner>/-/actions/runners` | runner serves all repos under one owner |
   | Repo | `https://gitea.phasefinal.com/<owner>/<repo>/settings/actions/runners` | runner serves one repo |

   Register-as-admin is right for our use case: one runner, fleet-wide.

3. **Create an SSH deploy key for the runner** that lets it execute
   the elway playbooks against fleet hosts. The key lives only on
   ana-docker (passed in as a workflow secret per repo, or mounted
   into the runner via volume — see "Wiring deploys" below).

4. **(Optional) Create a Gitea PAT** with `read:repository` scope on
   `vh/esh-pfi-infrastructure`. Workflows need to check out the
   management repo to get at the playbooks; the auto-injected
   `GITHUB_TOKEN` only works for the triggering repo.

## Deploy

```bash
# Edit .env first if not using defaults — at minimum paste the registration token
$EDITOR stacks/gitea-runner/.env.example   # template

# First-time deploy
scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml

# Site-local runner later (NH3 or ESH)
scripts/elway nh3-docker --playbook playbooks/deploy-gitea-runner.yaml \
  --var runner_name=nh3-docker-runner --var runner_labels=pfi-fleet,nh3-docker
```

The playbook seeds `.env` from `.env.example` only if absent; for the
first run, copy `.env.example` to `/opt/docker/compose/gitea-runner/.env`
on the host and paste the registration token in before running, OR
let the playbook seed it and edit on the host before the
`docker compose up -d` step (it's idempotent — second run will pick up
the edited token).

After successful registration, the token is consumed (it's one-time
use). You can clear `GITEA_RUNNER_REGISTRATION_TOKEN` from `.env`;
the runner reads its permanent credentials from `${DATA_DIR}/.runner`
on subsequent starts.

## Wiring deploys

A workflow that runs on the central runner needs three things:

1. **`runs-on:`** matching a runner label — `pfi-fleet` (cross-fleet)
   or `ana-docker` (pin to that host).

   ⚠ **A label LIST is AND, not fallback.** `runs-on: [vastblue, pfi-fleet]`
   requires a runner carrying *both* labels — it does not try the first and
   fall back to the second. A list naming a label no runner has matches
   nothing and the job queues forever rather than failing, which is the worst
   of the three outcomes because the board looks busy rather than broken.
   Use **one** label, and when a workflow should move to a new runner, edit
   the label deliberately at that moment. (Caught by eitri-smithy-dev on
   2026-09-02 against vastblue's U1 contract, after infra-ops wrote "one-line
   fallback to `pfi-fleet`" meaning a manual edit and was read as scheduler
   behaviour. Standard Actions semantics, not a gitea quirk — the ambiguity
   was in the prose, not the product.)
2. **An SSH key** to reach the deploy target. Stored as a repo or
   org-level Actions secret named e.g. `DEPLOY_SSH_KEY`. The
   corresponding public key must be in `~lkraven/.ssh/authorized_keys`
   on every host the workflow targets.
3. **A token to clone `vh/esh-pfi-infrastructure`** if the workflow
   wants to invoke an elway playbook from this repo. Stored as
   `MGMT_REPO_TOKEN` (Gitea PAT, `read:repository` scope).

See `stacks/task-board/gitea-workflow-deploy.yaml.example` for a
complete deploy workflow that consumes all three.

## Path layout (on ana-docker)

| Host path | Container path | Purpose | Restic? |
|---|---|---|---|
| `/opt/docker/compose/gitea-runner/` | — | compose.yaml + .env | included (via `/opt/docker`) |
| `/opt/docker/conf/gitea-runner/data/` | `/data` | `.runner` creds, cache, job workspaces | excluded (regenerable; nothing irreplaceable) |

## Operations

```bash
# Tail runner logs
ssh ana-docker docker logs -f gitea-runner

# List currently registered runners (admin)
# https://gitea.phasefinal.com/-/admin/actions/runners

# Re-register (lost the .runner file? regenerate token, then:)
ssh ana-docker docker compose -f /opt/docker/compose/gitea-runner/compose.yaml down
ssh ana-docker rm /opt/docker/conf/gitea-runner/data/.runner
# paste new GITEA_RUNNER_REGISTRATION_TOKEN into .env
scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml

# Pin to a specific act_runner version
# Edit RUNNER_IMAGE in /opt/docker/compose/gitea-runner/.env, then:
scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml
```