esh-pfi-infrastructure/stacks/gitea-runner/README.md

# gitea-runner

Self-hosted [Gitea Actions](https://docs.gitea.com/usage/actions/overview)
runner. Polls `gitea.phasefinal.com` for jobs from any repo that has a
`.gitea/workflows/` directory and runs them in ephemeral docker
containers on this host.

**Server:** ana-docker (single central runner — see "Topology" below
for when to add more)
**Image:** `gitea/act_runner:latest`
**Outbound only** — no host port published; the runner connects out to
gitea, gitea never connects in.

## Topology

We run **one central runner on ana-docker**. Reasoning:

- gitea is on ana-docker, so runner→API is local
- existing fleet tooling (`elway`, `sync-stacks`, `deploy-stack`,
  `refresh-server-info`) already SSHes from one origin to all hosts;
  the runner inherits that pattern
- single point to manage SSH keys, secrets, and runner upgrades

The deploy playbook (`playbooks/deploy-gitea-runner.yaml`) is
parameterized by host / runner-name / labels, so spinning up
`nh3-docker-runner` or an ESH runner later is a one-line elway
invocation — not a copy-pasted playbook.

**When to add a site-local runner:**

- Cross-site SSH from ana-docker to that site has become unreliable
- A workflow needs LAN access to something only reachable from inside
  that site's network segment
- You want failure isolation (NH3 can deploy itself when Anaheim is down)

Until one of those bites, one runner is enough.

## What a job on this runner can actually do

Measured 2026-09-02 on `pfi-fleet` (throwaway repo, three jobs, since deleted).
Recorded because two of these are commonly assumed the other way.

| capability | result |
|---|---|
| `services:` containers | **yes** — Postgres 16 answered on the service name as hostname after ~6 s; wait on `pg_isready`, not on ordering |
| host docker daemon | **yes, root-equivalent** — `/var/run/docker.sock` is in every job container, `docker ps` showed all 49 host containers, `docker compose v2.33.0` on PATH |
| `uses:` from the local mirrors | **yes** — `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves and runs, with no `DEFAULT_ACTIONS_URL` change |

⚠ **`container.valid_volumes: []` does not keep docker out of jobs.** act_runner
mounts the daemon socket itself, independently of that list, so a tight-looking
`valid_volumes` is not containment. Any repo the runner serves — it is registered
instance-wide — can control everything on ana-docker, gitea included. Four repos
(`vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge`) drive
buildx through it, so it is load-bearing and closing it would break their CI.
**Isolate sensitive builds onto a dedicated runner rather than tightening this one.**

⚠ **Job images need a node binary.** JS actions execute as `node /var/run/act/...`,
so `python:3-slim` fails on the first `uses:`. Use `node:20-bookworm`, or
`docker:cli` plus `apk add --no-cache git nodejs` when the job also builds images.

**Full-URL `uses:` is the un-parked half of the github-independence work.** The
global `DEFAULT_ACTIONS_URL=self` flip is still blocked on act_runner's
action-fetch auth, but a per-workflow full-URL ref needs neither the flip nor the
auth path. Mirrors live under the `actions` and `astral-sh` orgs, all public:
checkout, cache, upload-artifact, download-artifact, setup-node, setup-python,
setup-uv.

**Polling a run from the API:** use `/actions/runs`, not `/actions/tasks` — on
gitea 1.26.1 `tasks` returned an empty `workflow_runs` for a run that `runs` listed
and executed.

## Prereqs

Before running the deploy playbook:

1. **Verify Gitea Actions is enabled.** In gitea 1.21+ Actions ships
   on by default, but check `/-/admin/actions` resolves. If not, add
   `GITEA__actions__ENABLED=true` to the gitea stack env and bounce.

2. **Generate a registration token.** Pick the scope:

   | Scope | URL | Use when |
   |---|---|---|
   | Global (admin) | `https://gitea.phasefinal.com/-/admin/actions/runners` | runner serves any repo on the instance (recommended for the central PFI runner) |
   | Org/user | `https://gitea.phasefinal.com/<owner>/-/actions/runners` | runner serves all repos under one owner |
   | Repo | `https://gitea.phasefinal.com/<owner>/<repo>/settings/actions/runners` | runner serves one repo |

   Register-as-admin is right for our use case: one runner, fleet-wide.

3. **Create an SSH deploy key for the runner** that lets it execute
   the elway playbooks against fleet hosts. The key lives only on
   ana-docker (passed in as a workflow secret per repo, or mounted
   into the runner via volume — see "Wiring deploys" below).

4. **(Optional) Create a Gitea PAT** with `read:repository` scope on
   `vh/esh-pfi-infrastructure`. Workflows need to check out the
   management repo to get at the playbooks; the auto-injected
   `GITHUB_TOKEN` only works for the triggering repo.

## Deploy

```bash
# Edit .env first if not using defaults — at minimum paste the registration token
$EDITOR stacks/gitea-runner/.env.example   # template

# First-time deploy
scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml

# Site-local runner later (NH3 or ESH)
scripts/elway nh3-docker --playbook playbooks/deploy-gitea-runner.yaml \
  --var runner_name=nh3-docker-runner --var runner_labels=pfi-fleet,nh3-docker
```

The playbook seeds `.env` from `.env.example` only if absent; for the
first run, copy `.env.example` to `/opt/docker/compose/gitea-runner/.env`
on the host and paste the registration token in before running, OR
let the playbook seed it and edit on the host before the
`docker compose up -d` step (it's idempotent — second run will pick up
the edited token).

After successful registration, the token is consumed (it's one-time
use). You can clear `GITEA_RUNNER_REGISTRATION_TOKEN` from `.env`;
the runner reads its permanent credentials from `${DATA_DIR}/.runner`
on subsequent starts.

## Wiring deploys

A workflow that runs on the central runner needs three things:

1. **`runs-on:`** matching a runner label — `pfi-fleet` (cross-fleet)
   or `ana-docker` (pin to that host).

   ⚠ **A label LIST is AND, not fallback.** `runs-on: [vastblue, pfi-fleet]`
   requires a runner carrying *both* labels — it does not try the first and
   fall back to the second. A list naming a label no runner has matches
   nothing and the job queues forever rather than failing, which is the worst
   of the three outcomes because the board looks busy rather than broken.
   Use **one** label, and when a workflow should move to a new runner, edit
   the label deliberately at that moment. (Caught by eitri-smithy-dev on
   2026-09-02 against vastblue's U1 contract, after infra-ops wrote "one-line
   fallback to `pfi-fleet`" meaning a manual edit and was read as scheduler
   behaviour. Standard Actions semantics, not a gitea quirk — the ambiguity
   was in the prose, not the product.)
2. **An SSH key** to reach the deploy target. Stored as a repo or
   org-level Actions secret named e.g. `DEPLOY_SSH_KEY`. The
   corresponding public key must be in `~lkraven/.ssh/authorized_keys`
   on every host the workflow targets.
3. **A token to clone `vh/esh-pfi-infrastructure`** if the workflow
   wants to invoke an elway playbook from this repo. Stored as
   `MGMT_REPO_TOKEN` (Gitea PAT, `read:repository` scope).

See `stacks/task-board/gitea-workflow-deploy.yaml.example` for a
complete deploy workflow that consumes all three.

## Path layout (on ana-docker)

| Host path | Container path | Purpose | Restic? |
|---|---|---|---|
| `/opt/docker/compose/gitea-runner/` | — | compose.yaml + .env | included (via `/opt/docker`) |
| `/opt/docker/conf/gitea-runner/data/` | `/data` | `.runner` creds, cache, job workspaces | excluded (regenerable; nothing irreplaceable) |

## Operations

```bash
# Tail runner logs
ssh ana-docker docker logs -f gitea-runner

# List currently registered runners (admin)
# https://gitea.phasefinal.com/-/admin/actions/runners

# Re-register (lost the .runner file? regenerate token, then:)
ssh ana-docker docker compose -f /opt/docker/compose/gitea-runner/compose.yaml down
ssh ana-docker rm /opt/docker/conf/gitea-runner/data/.runner
# paste new GITEA_RUNNER_REGISTRATION_TOKEN into .env
scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml

# Pin to a specific act_runner version
# Edit RUNNER_IMAGE in /opt/docker/compose/gitea-runner/.env, then:
scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml
```