# soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated)

Green-gated CI/CD for the soong-lab studio: **push to `main` → run the test
suite → redeploy the studio ONLY if tests pass** (running studio is never
touched on a red run). Built 2026-07-13 (Vuong-directed). Adapts the
[ytvc-autodeploy](./ytvc-autodeploy.md) webhook pattern.

## Flow

```
push→main → gitea webhook (POST, HMAC) → soong-webhook listener :9010 on corviduo-dev
  → ~/soong-lab-deploy.sh:
       git clone (read-only deploy key, internal SSH :222)
       uv sync ; uv run pytest        ── RED → abort, studio UNTOUCHED, status=red
       rsync backend/ → studio dir + web/ → SOONG_LAB_WEB_DIR ; uv sync --no-dev ; restart
       → status=green, studio healthy
```

## Components (all on corviduo-dev, user `infra-ops`)

- `~/soong-lab-deploy.sh` — clone → test → deploy-on-green. Logs to
  `~/soong-lab-deploy.log`; writes `~/.config/soong/last-deploy.json`
  (`{result: green|red, stage, sha, at}`).
- `~/soong-webhook.py` — HTTP listener on `:9010`. HMAC-SHA256 (`X-Gitea-Signature`)
  vs `~/.config/soong/webhook-secret` (mode 600); fires the deploy only on
  `ref == refs/heads/main`. `GET /` returns `ok | last: <status>`.
- `soong-webhook.service` (system unit, enabled) — runs the listener.
- Read-only deploy key `~/.ssh/soong-deploy_ed25519` → gitea repo key id 5 on
  `vh/soong-lab` (read_only). Clone via `ssh://git@10.250.50.70:222/vh/soong-lab.git`.
- Studio unit `soong-lab-studio.service` (WD `/home/infra-ops/soong-lab/backend`);
  restart needs infra-ops NOPASSWD sudo (present).
- Gitea webhook: repo `vh/soong-lab` hook id 3 → `http://10.250.50.152:9010/`,
  JSON, Push events, the shared secret.

## Verify / operate

```bash
ssh corviduo-dev 'systemctl is-active soong-webhook.service; curl -s localhost:9010/'
ssh corviduo-dev 'tail -30 ~/soong-lab-deploy.log'          # deploy history
# manual deploy (same as the webhook does):
ssh corviduo-dev 'bash ~/soong-lab-deploy.sh'
```

## Notes / gotchas

- **Frontend (`web/`) sync IS part of the deploy**: the studio serves `web/` from
  `SOONG_LAB_WEB_DIR` (`/home/infra-ops/soong-lab/web`), *separate* from the backend
  `WorkingDirectory`. The deploy rsyncs BOTH `backend/`→studio and `web/`→`SOONG_LAB_WEB_DIR`.
  (Added 2026-07-13 after soong-dev caught the served frontend silently rotting — the backend
  was updating while `web/` stayed pinned to the initial manual copy; a bounce alone re-serves
  the same stale file.)
- **Green-gated by construction**: `pytest || fail` runs BEFORE any studio touch,
  so a red suite aborts with the studio still on the old version. Validated
  2026-07-13 (a mid-deploy rsync failure left the studio untouched/active).
- **rsync is required** on corviduo-dev (`apt install rsync` — installed 2026-07-13;
  it wasn't present initially).
- **bifrost dep** resolves from the internal Gitea PyPI via `~/.netrc` (already
  present on corviduo-dev); no extra auth in the deploy script.
- **⚠️ Auto-deploy silently never worked until 2026-07-14 — TWO compounding blockers.**
  The LISTENER binds `0.0.0.0:9010` and works, but nothing gitea sent ever reached
  it, so every push was a no-op (v0.3.6 was manual; v0.3.7–v0.3.15 never
  auto-deployed until fixed). Two separate, both-real blockers:
  1. **corviduo-dev ufw** — `default-deny`, only 22 + 8080 allowed, so a *direct*
     TCP to :9010 from ana-docker DROP-timed-out. Fix: `ufw allow from 10.0.0.0/8`
     (operator-directed — "that footgun happens a lot", accept the fleet).
  2. **★ gitea `webhook.ALLOWED_HOST_LIST` (the DECISIVE one)** — was
     `external, 10.100.0.0/16` (NH3 only); corviduo-dev is `10.250.50.152`
     (Anaheim), so gitea **refused to deliver**: `webhook can only call allowed HTTP
     servers ... deny '10.250.50.152'` — it never even opens the TCP connection, so
     the ufw fix alone did nothing. Fix: `ALLOWED_HOST_LIST = external, 10.0.0.0/8`
     in gitea `app.ini` (`/data/gitea/conf/app.ini`, `[webhook]`) + `docker restart
     gitea` (~8s blip). The HMAC secret was already correct (once delivery arrives,
     `hmac_ok=True`).
  **RED HERRINGS that cost two diagnosis rounds:** (a) "test-delivery 204" is gitea
  *queuing*, NOT delivering — never proves the round-trip; (b) a proxy test signing
  with the *listener's own* secret (bypassing gitea) proves the listener but NOT
  gitea's real delivery. **Diagnose from BOTH ends:** the SENDER (`docker logs gitea
  --since 5m | grep webhook` → the `deny '<ip>'` line) AND an instrumented RECEIVER
  — the listener now ships with delivery logging (`journalctl -u soong-webhook.service
  | grep '\[webhook\]'` shows source-IP / hmac_ok / ref / action; the old
  `log_message=pass` silence hid all of it). **Proof of fix:** a real gitea delivery
  logs `POST from 10.250.50.70 ... hmac_ok=True`, `ref='refs/heads/main'`,
  `-> 202 deploying` → green deploy of the latest main SHA.
- **Red-run push-notify** via an **althing relay on nh3-dev** (`soong-ci-relay.timer`,
  2-min poll of corviduo's `last-deploy.json` → pings **soong-dev** via althing on a
  NEW red run; green runs stay silent = fire-and-forget). corviduo itself has no
  althing, so the relay lives on nh3-dev (which does), needing no gitea write token
  on the Worldtree-team VM. Files: `services/soong-lab-ci/soong-ci-relay.{sh,service,timer}`;
  state `~/.local/state/soong-ci-relay/last-at.txt`. (A gitea commit-status was the
  alternative but needs a write token gitea won't mint without basic-auth.)
- Test suite: `uv run pytest` in `backend/` (242 tests as of v0.3.6).