esh-pfi-infrastructure/docs/runbooks/soong-lab-autodeploy.md

soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated)

Green-gated CI/CD for the soong-lab studio: push to main → run the test suite → redeploy the studio ONLY if tests pass (running studio is never touched on a red run). Built 2026-07-13 (Vuong-directed). Adapts the ytvc-autodeploy webhook pattern.

Flow

push→main → gitea webhook (POST, HMAC) → soong-webhook listener :9010 on corviduo-dev
  → ~/soong-lab-deploy.sh:
       git clone (read-only deploy key, internal SSH :222)
       uv sync ; uv run pytest        ── RED → abort, studio UNTOUCHED, status=red
       rsync backend/ → studio dir + web/ → SOONG_LAB_WEB_DIR ; uv sync --no-dev ; restart
       → status=green, studio healthy

Components (all on corviduo-dev, user infra-ops)

• ~/soong-lab-deploy.sh — clone → test → deploy-on-green. Logs to ~/soong-lab-deploy.log; writes ~/.config/soong/last-deploy.json ({result: green|red, stage, sha, at}).
• ~/soong-webhook.py — HTTP listener on :9010. HMAC-SHA256 (X-Gitea-Signature) vs ~/.config/soong/webhook-secret (mode 600); fires the deploy only on ref == refs/heads/main. GET / returns ok | last: <status>.
• soong-webhook.service (system unit, enabled) — runs the listener.
• Read-only deploy key ~/.ssh/soong-deploy_ed25519 → gitea repo key id 5 on vh/soong-lab (read_only). Clone via ssh://git@10.250.50.70:222/vh/soong-lab.git.
• Studio unit soong-lab-studio.service (WD /home/infra-ops/soong-lab/backend); restart needs infra-ops NOPASSWD sudo (present).
• Gitea webhook: repo vh/soong-lab hook id 3 → http://10.250.50.152:9010/, JSON, Push events, the shared secret.

Verify / operate

ssh corviduo-dev 'systemctl is-active soong-webhook.service; curl -s localhost:9010/'
ssh corviduo-dev 'tail -30 ~/soong-lab-deploy.log'          # deploy history
# manual deploy (same as the webhook does):
ssh corviduo-dev 'bash ~/soong-lab-deploy.sh'

Notes / gotchas

• Frontend (web/) sync IS part of the deploy: the studio serves web/ from SOONG_LAB_WEB_DIR (/home/infra-ops/soong-lab/web), separate from the backend WorkingDirectory. The deploy rsyncs BOTH backend/→studio and web/→SOONG_LAB_WEB_DIR. (Added 2026-07-13 after soong-dev caught the served frontend silently rotting — the backend was updating while web/ stayed pinned to the initial manual copy; a bounce alone re-serves the same stale file.)
• Green-gated by construction: pytest || fail runs BEFORE any studio touch, so a red suite aborts with the studio still on the old version. Validated 2026-07-13 (a mid-deploy rsync failure left the studio untouched/active).
• rsync is required on corviduo-dev (apt install rsync — installed 2026-07-13; it wasn't present initially).
• bifrost dep resolves from the internal Gitea PyPI via ~/.netrc (already present on corviduo-dev); no extra auth in the deploy script.
• ⚠️ Auto-deploy silently never worked until 2026-07-14 — TWO compounding blockers. The LISTENER binds 0.0.0.0:9010 and works, but nothing gitea sent ever reached it, so every push was a no-op (v0.3.6 was manual; v0.3.7–v0.3.15 never auto-deployed until fixed). Two separate, both-real blockers:
  1. corviduo-dev ufw — default-deny, only 22 + 8080 allowed, so a direct TCP to :9010 from ana-docker DROP-timed-out. Fix: ufw allow from 10.0.0.0/8 (operator-directed — "that footgun happens a lot", accept the fleet).
  2. ★ gitea webhook.ALLOWED_HOST_LIST (the DECISIVE one) — was external, 10.100.0.0/16 (NH3 only); corviduo-dev is 10.250.50.152 (Anaheim), so gitea refused to deliver: webhook can only call allowed HTTP servers ... deny '10.250.50.152' — it never even opens the TCP connection, so the ufw fix alone did nothing. Fix: ALLOWED_HOST_LIST = external, 10.0.0.0/8 in gitea app.ini (/data/gitea/conf/app.ini, [webhook]) + docker restart gitea (~8s blip). The HMAC secret was already correct (once delivery arrives, hmac_ok=True). RED HERRINGS that cost two diagnosis rounds: (a) "test-delivery 204" is gitea queuing, NOT delivering — never proves the round-trip; (b) a proxy test signing with the listener's own secret (bypassing gitea) proves the listener but NOT gitea's real delivery. Diagnose from BOTH ends: the SENDER (docker logs gitea --since 5m | grep webhook → the deny '<ip>' line) AND an instrumented RECEIVER — the listener now ships with delivery logging (journalctl -u soong-webhook.service | grep '\[webhook\]' shows source-IP / hmac_ok / ref / action; the old log_message=pass silence hid all of it). Proof of fix: a real gitea delivery logs POST from 10.250.50.70 ... hmac_ok=True, ref='refs/heads/main', -> 202 deploying → green deploy of the latest main SHA.
• Red-run push-notify via an althing relay on nh3-dev (soong-ci-relay.timer, 2-min poll of corviduo's last-deploy.json → pings soong-dev via althing on a NEW red run; green runs stay silent = fire-and-forget). corviduo itself has no althing, so the relay lives on nh3-dev (which does), needing no gitea write token on the Worldtree-team VM. Files: services/soong-lab-ci/soong-ci-relay.{sh,service,timer}; state ~/.local/state/soong-ci-relay/last-at.txt. (A gitea commit-status was the alternative but needs a write token gitea won't mint without basic-auth.)
• Test suite: uv run pytest in backend/ (242 tests as of v0.3.6).