Files

T

vh cc0e3af87d feat(soong-lab-ci): red-run althing relay (nh3-dev poll -> ping soong-dev)

Per operator call (no gitea write token on the Worldtree-team VM): a 2-min systemd
--user timer on nh3-dev polls corviduo's last-deploy.json and pings soong-dev via
althing on a NEW red deploy (green stays silent). Delivers soong-dev's red-run
visibility without a credential on corviduo. Tested (red detect+format DRY, green quiet).

2026-07-13 14:28:39 -07:00

3.3 KiB

Raw Blame History

soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated)

Green-gated CI/CD for the soong-lab studio: push to main → run the test suite → redeploy the studio ONLY if tests pass (running studio is never touched on a red run). Built 2026-07-13 (Vuong-directed). Adapts the ytvc-autodeploy webhook pattern.

Flow

push→main → gitea webhook (POST, HMAC) → soong-webhook listener :9010 on corviduo-dev
  → ~/soong-lab-deploy.sh:
       git clone (read-only deploy key, internal SSH :222)
       uv sync ; uv run pytest        ── RED → abort, studio UNTOUCHED, status=red
       rsync backend/ → studio dir ; uv sync --no-dev ; systemctl restart
       → status=green, studio healthy

Components (all on corviduo-dev, user `infra-ops`)

~/soong-lab-deploy.sh — clone → test → deploy-on-green. Logs to ~/soong-lab-deploy.log; writes ~/.config/soong/last-deploy.json ({result: green|red, stage, sha, at}).
~/soong-webhook.py — HTTP listener on :9010. HMAC-SHA256 (X-Gitea-Signature) vs ~/.config/soong/webhook-secret (mode 600); fires the deploy only on ref == refs/heads/main. GET / returns ok | last: <status>.
soong-webhook.service (system unit, enabled) — runs the listener.
Read-only deploy key ~/.ssh/soong-deploy_ed25519 → gitea repo key id 5 on vh/soong-lab (read_only). Clone via ssh://git@10.250.50.70:222/vh/soong-lab.git.
Studio unit soong-lab-studio.service (WD /home/infra-ops/soong-lab/backend); restart needs infra-ops NOPASSWD sudo (present).
Gitea webhook: repo vh/soong-lab hook id 3 → http://10.250.50.152:9010/, JSON, Push events, the shared secret.

Verify / operate

ssh corviduo-dev 'systemctl is-active soong-webhook.service; curl -s localhost:9010/'
ssh corviduo-dev 'tail -30 ~/soong-lab-deploy.log'          # deploy history
# manual deploy (same as the webhook does):
ssh corviduo-dev 'bash ~/soong-lab-deploy.sh'

Notes / gotchas

Green-gated by construction: pytest || fail runs BEFORE any studio touch, so a red suite aborts with the studio still on the old version. Validated 2026-07-13 (a mid-deploy rsync failure left the studio untouched/active).
rsync is required on corviduo-dev (apt install rsync — installed 2026-07-13; it wasn't present initially).
bifrost dep resolves from the internal Gitea PyPI via ~/.netrc (already present on corviduo-dev); no extra auth in the deploy script.
SSRF: gitea reached corviduo-dev 10.250.50.152 fine (test-delivery 204) — no ALLOWED_HOST_LIST relax needed (unlike the ytvc/WG case).
Red-run push-notify via an althing relay on nh3-dev (soong-ci-relay.timer, 2-min poll of corviduo's last-deploy.json → pings soong-dev via althing on a NEW red run; green runs stay silent = fire-and-forget). corviduo itself has no althing, so the relay lives on nh3-dev (which does), needing no gitea write token on the Worldtree-team VM. Files: services/soong-lab-ci/soong-ci-relay.{sh,service,timer}; state ~/.local/state/soong-ci-relay/last-at.txt. (A gitea commit-status was the alternative but needs a write token gitea won't mint without basic-auth.)
Test suite: uv run pytest in backend/ (242 tests as of v0.3.6).

3.3 KiB Raw Blame History