feat(gitea-runner): opt-in fast runner on nh3-docker (label nh3-docker), no docker socket in jobs

Prime's CI fix A. ana-docker's runner sits on HDD mirrors with no SLOG; synced
8 KiB writes measured ~34 ms there vs ~2.7 ms on nh3-docker (same probe, one run,
n=3 each). Repos opt in with runs-on: nh3-docker; the runner carries no
pfi-fleet label. Its config is tighter because nh3-docker also runs NH3 DNS and
the post office: docker_host '-' (verified absent in the job, present on the
ana-docker baseline), capacity 1, jobs capped at 4 GiB / 6 CPUs, a network per
job. The deploy playbook takes runner_config; each runner's homepage name is
its runner name.
This commit is contained in:
vh
2026-10-12 17:27:46 -07:00
parent 0da7df7534
commit cd3c4b8be9
4 changed files with 90 additions and 7 deletions
+38 -2
View File
@@ -5,8 +5,7 @@ runner. Polls `gitea.phasefinal.com` for jobs from any repo that has a
`.gitea/workflows/` directory and runs them in ephemeral docker
containers on this host.
**Server:** ana-docker (single central runner — see "Topology" below
for when to add more)
**Server:** ana-docker (the central runner), plus the opt-in fast runner on nh3-docker (see "Topology")
**Image:** `gitea/act_runner:latest`
**Outbound only** — no host port published; the runner connects out to
gitea, gitea never connects in.
@@ -35,6 +34,43 @@ invocation — not a copy-pasted playbook.
Until one of those bites, one runner is enough.
### The opt-in fast runner on nh3-docker (Prime's CI fix A, 2026-10-12)
`nh3-docker-runner` (gitea runner id 4, act_runner v0.6.1) carries **one label, `nh3-docker`**, and NOT `pfi-fleet`.
Nothing lands on it unless a workflow says `runs-on: nh3-docker`. It exists because ana-docker's disk is slow to
fsync: pfi-pve's `ospool` is HDD mirrors with no SLOG. Jobs that fsync a lot (a Postgres service, pytest temp dirs)
crawled into the 30 m timeout there.
Measured 2026-10-12 1725 in a throwaway repo (`vh/ci-probe-nh3`, deleted after). One run had a job on each runner
with the same steps (8 KiB `dd oflag=dsync` × 200, n = 3 each):
| | nh3-docker-runner | ana-docker-runner |
|---|---|---|
| 200 synced 8 KiB writes | 0.51 / 0.54 / 0.57 s (~2.7 ms each) | 6.36 / 6.82 / 7.21 s (~34 ms each) |
| docker socket in the job | **absent** | present (root over the host) |
| job memory / CPU limit | 4 GiB / 6 CPUs | none |
| `services:` container | Postgres reachable after 1 s | (not tested here) |
Its config is `conf/config.nh3-docker.yaml`, tighter than ana-docker's because nh3-docker also runs NH3's DNS
(adguardhome) and the althing post office on 8 vCPU / 7 GiB:
- `docker_host: "-"` keeps the docker socket out of jobs (the probe above proves it). `services:` still work.
- `capacity: 1`, and each job container is capped at `--memory=4g --memory-swap=4g --cpus=6`.
- `network: ""` gives every job its own network instead of traefik-net.
- The timeout stays at 30 m.
A job that needs docker inside it (buildx and the like) cannot run here; keep those on `pfi-fleet`. ⚠ nh3-docker's
vCPU has no AVX2 either, so CPU-bound torch tests are no faster here. This runner fixes the disk, not the CPU.
Deployed with:
```bash
scripts/elway infra-ops@10.100.50.40 --playbook playbooks/deploy-gitea-runner.yaml \
--var runner_name=nh3-docker-runner --var runner_labels=nh3-docker:docker://node:20-bookworm-slim \
--var runner_config=stacks/gitea-runner/conf/config.nh3-docker.yaml
```
Use `infra-ops@<ip>`, not the `nh3-docker` alias: the alias logs in as lkraven, and the first attempt left a
lkraven-owned data dir that infra-ops could not write. The registration token came from the admin API
(`POST /api/v1/admin/actions/runners/registration-token`) and was cleared from `.env` once the runner registered.
## What a job on this runner can actually do
Measured 2026-09-02 on `pfi-fleet` (throwaway repo, three jobs, since deleted).