feat(gitea-runner): opt-in fast runner on nh3-docker (label nh3-docker), no docker socket in jobs
Prime's CI fix A. ana-docker's runner sits on HDD mirrors with no SLOG; synced 8 KiB writes measured ~34 ms there vs ~2.7 ms on nh3-docker (same probe, one run, n=3 each). Repos opt in with runs-on: nh3-docker; the runner carries no pfi-fleet label. Its config is tighter because nh3-docker also runs NH3 DNS and the post office: docker_host '-' (verified absent in the job, present on the ana-docker baseline), capacity 1, jobs capped at 4 GiB / 6 CPUs, a network per job. The deploy playbook takes runner_config; each runner's homepage name is its runner name.
This commit is contained in:
@@ -5,8 +5,7 @@ runner. Polls `gitea.phasefinal.com` for jobs from any repo that has a
|
||||
`.gitea/workflows/` directory and runs them in ephemeral docker
|
||||
containers on this host.
|
||||
|
||||
**Server:** ana-docker (single central runner — see "Topology" below
|
||||
for when to add more)
|
||||
**Server:** ana-docker (the central runner), plus the opt-in fast runner on nh3-docker (see "Topology")
|
||||
**Image:** `gitea/act_runner:latest`
|
||||
**Outbound only** — no host port published; the runner connects out to
|
||||
gitea, gitea never connects in.
|
||||
@@ -35,6 +34,43 @@ invocation — not a copy-pasted playbook.
|
||||
|
||||
Until one of those bites, one runner is enough.
|
||||
|
||||
### The opt-in fast runner on nh3-docker (Prime's CI fix A, 2026-10-12)
|
||||
|
||||
`nh3-docker-runner` (gitea runner id 4, act_runner v0.6.1) carries **one label, `nh3-docker`**, and NOT `pfi-fleet`.
|
||||
Nothing lands on it unless a workflow says `runs-on: nh3-docker`. It exists because ana-docker's disk is slow to
|
||||
fsync: pfi-pve's `ospool` is HDD mirrors with no SLOG. Jobs that fsync a lot (a Postgres service, pytest temp dirs)
|
||||
crawled into the 30 m timeout there.
|
||||
|
||||
Measured 2026-10-12 1725 in a throwaway repo (`vh/ci-probe-nh3`, deleted after). One run had a job on each runner
|
||||
with the same steps (8 KiB `dd oflag=dsync` × 200, n = 3 each):
|
||||
|
||||
| | nh3-docker-runner | ana-docker-runner |
|
||||
|---|---|---|
|
||||
| 200 synced 8 KiB writes | 0.51 / 0.54 / 0.57 s (~2.7 ms each) | 6.36 / 6.82 / 7.21 s (~34 ms each) |
|
||||
| docker socket in the job | **absent** | present (root over the host) |
|
||||
| job memory / CPU limit | 4 GiB / 6 CPUs | none |
|
||||
| `services:` container | Postgres reachable after 1 s | (not tested here) |
|
||||
|
||||
Its config is `conf/config.nh3-docker.yaml`, tighter than ana-docker's because nh3-docker also runs NH3's DNS
|
||||
(adguardhome) and the althing post office on 8 vCPU / 7 GiB:
|
||||
- `docker_host: "-"` keeps the docker socket out of jobs (the probe above proves it). `services:` still work.
|
||||
- `capacity: 1`, and each job container is capped at `--memory=4g --memory-swap=4g --cpus=6`.
|
||||
- `network: ""` gives every job its own network instead of traefik-net.
|
||||
- The timeout stays at 30 m.
|
||||
|
||||
A job that needs docker inside it (buildx and the like) cannot run here; keep those on `pfi-fleet`. ⚠ nh3-docker's
|
||||
vCPU has no AVX2 either, so CPU-bound torch tests are no faster here. This runner fixes the disk, not the CPU.
|
||||
|
||||
Deployed with:
|
||||
```bash
|
||||
scripts/elway infra-ops@10.100.50.40 --playbook playbooks/deploy-gitea-runner.yaml \
|
||||
--var runner_name=nh3-docker-runner --var runner_labels=nh3-docker:docker://node:20-bookworm-slim \
|
||||
--var runner_config=stacks/gitea-runner/conf/config.nh3-docker.yaml
|
||||
```
|
||||
Use `infra-ops@<ip>`, not the `nh3-docker` alias: the alias logs in as lkraven, and the first attempt left a
|
||||
lkraven-owned data dir that infra-ops could not write. The registration token came from the admin API
|
||||
(`POST /api/v1/admin/actions/runners/registration-token`) and was cleared from `.env` once the runner registered.
|
||||
|
||||
## What a job on this runner can actually do
|
||||
|
||||
Measured 2026-09-02 on `pfi-fleet` (throwaway repo, three jobs, since deleted).
|
||||
|
||||
@@ -28,7 +28,7 @@ services:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=Agents (no UI)
|
||||
- homepage.name=gitea-runner
|
||||
- homepage.name=${GITEA_RUNNER_NAME}
|
||||
- homepage.icon=mdi-cog-play
|
||||
- homepage.description=Gitea Actions self-hosted runner (${GITEA_RUNNER_NAME})
|
||||
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
# act_runner config for the OPT-IN fast runner on nh3-docker (Prime's CI fix A, 2026-10-12).
|
||||
# Mounted at /data/config.yaml. Deployed by playbooks/deploy-gitea-runner.yaml with
|
||||
# --var runner_config=stacks/gitea-runner/conf/config.nh3-docker.yaml
|
||||
#
|
||||
# Why it exists: the central runner on ana-docker sits on pfi-pve's `ospool` (HDD mirrors, no SLOG); 8 KiB fsync
|
||||
# writes measured ~145 ms there vs ~2.7 ms on nh3-docker. Jobs that fsync a lot (a Postgres service, pytest temp
|
||||
# dirs) ran past the 30 m job timeout. Repos opt in with `runs-on: nh3-docker`. The runner carries ONLY that label,
|
||||
# never `pfi-fleet`, so nothing lands here by accident.
|
||||
#
|
||||
# Why it is tighter than ana-docker's config: nh3-docker also runs NH3's DNS (adguardhome) and the althing post
|
||||
# office, on 8 vCPU / 7 GiB. So:
|
||||
# - docker_host "-": the job containers do NOT get the host docker socket (ana-docker's runner hands every job root
|
||||
# over its host). `services:` still work, because the runner creates them through its own socket.
|
||||
# - capacity 1, and each job container is capped at 4 GiB / 6 CPUs, leaving room for DNS and the bus.
|
||||
# - network "": every job gets its own throwaway network, not traefik-net.
|
||||
|
||||
log:
|
||||
level: info
|
||||
|
||||
runner:
|
||||
file: /data/.runner
|
||||
capacity: 1
|
||||
timeout: 30m
|
||||
shutdown_timeout: 1m
|
||||
insecure: false
|
||||
fetch_timeout: 5s
|
||||
fetch_interval: 2s
|
||||
|
||||
cache:
|
||||
enabled: true
|
||||
dir: /data/cache
|
||||
|
||||
container:
|
||||
network: ""
|
||||
privileged: false
|
||||
options: "--memory=4g --memory-swap=4g --cpus=6"
|
||||
workdir_parent: /data/workspace
|
||||
valid_volumes: []
|
||||
docker_host: "-"
|
||||
force_pull: false
|
||||
|
||||
host:
|
||||
workdir_parent: /data/host-workspace
|
||||
Reference in New Issue
Block a user