vh
462d528bef
The ufw fix (prior commit) was necessary but insufficient. The DECISIVE blocker
was gitea webhook.ALLOWED_HOST_LIST = 'external, 10.100.0.0/16' (NH3 only) —
corviduo-dev is 10.250.50.152 (Anaheim), so gitea refused to deliver ('deny
10.250.50.152') and never opened the TCP connection. Fixed to 'external,
10.0.0.0/8' (whole fleet, matches the ufw choice) + gitea restart.
Listener now logs every delivery (source-IP/hmac_ok/ref/action) — the old
log_message=pass silence hid the whole failure. Proven end-to-end: real gitea
delivery -> hmac_ok=True, ref=main, 202 deploying -> green deploy.
92 lines
5.5 KiB
Markdown
92 lines
5.5 KiB
Markdown
# soong-lab push-to-deploy (gitea webhook → corviduo-dev, test-gated)
|
||
|
||
Green-gated CI/CD for the soong-lab studio: **push to `main` → run the test
|
||
suite → redeploy the studio ONLY if tests pass** (running studio is never
|
||
touched on a red run). Built 2026-07-13 (Vuong-directed). Adapts the
|
||
[ytvc-autodeploy](./ytvc-autodeploy.md) webhook pattern.
|
||
|
||
## Flow
|
||
|
||
```
|
||
push→main → gitea webhook (POST, HMAC) → soong-webhook listener :9010 on corviduo-dev
|
||
→ ~/soong-lab-deploy.sh:
|
||
git clone (read-only deploy key, internal SSH :222)
|
||
uv sync ; uv run pytest ── RED → abort, studio UNTOUCHED, status=red
|
||
rsync backend/ → studio dir + web/ → SOONG_LAB_WEB_DIR ; uv sync --no-dev ; restart
|
||
→ status=green, studio healthy
|
||
```
|
||
|
||
## Components (all on corviduo-dev, user `infra-ops`)
|
||
|
||
- `~/soong-lab-deploy.sh` — clone → test → deploy-on-green. Logs to
|
||
`~/soong-lab-deploy.log`; writes `~/.config/soong/last-deploy.json`
|
||
(`{result: green|red, stage, sha, at}`).
|
||
- `~/soong-webhook.py` — HTTP listener on `:9010`. HMAC-SHA256 (`X-Gitea-Signature`)
|
||
vs `~/.config/soong/webhook-secret` (mode 600); fires the deploy only on
|
||
`ref == refs/heads/main`. `GET /` returns `ok | last: <status>`.
|
||
- `soong-webhook.service` (system unit, enabled) — runs the listener.
|
||
- Read-only deploy key `~/.ssh/soong-deploy_ed25519` → gitea repo key id 5 on
|
||
`vh/soong-lab` (read_only). Clone via `ssh://git@10.250.50.70:222/vh/soong-lab.git`.
|
||
- Studio unit `soong-lab-studio.service` (WD `/home/infra-ops/soong-lab/backend`);
|
||
restart needs infra-ops NOPASSWD sudo (present).
|
||
- Gitea webhook: repo `vh/soong-lab` hook id 3 → `http://10.250.50.152:9010/`,
|
||
JSON, Push events, the shared secret.
|
||
|
||
## Verify / operate
|
||
|
||
```bash
|
||
ssh corviduo-dev 'systemctl is-active soong-webhook.service; curl -s localhost:9010/'
|
||
ssh corviduo-dev 'tail -30 ~/soong-lab-deploy.log' # deploy history
|
||
# manual deploy (same as the webhook does):
|
||
ssh corviduo-dev 'bash ~/soong-lab-deploy.sh'
|
||
```
|
||
|
||
## Notes / gotchas
|
||
|
||
- **Frontend (`web/`) sync IS part of the deploy**: the studio serves `web/` from
|
||
`SOONG_LAB_WEB_DIR` (`/home/infra-ops/soong-lab/web`), *separate* from the backend
|
||
`WorkingDirectory`. The deploy rsyncs BOTH `backend/`→studio and `web/`→`SOONG_LAB_WEB_DIR`.
|
||
(Added 2026-07-13 after soong-dev caught the served frontend silently rotting — the backend
|
||
was updating while `web/` stayed pinned to the initial manual copy; a bounce alone re-serves
|
||
the same stale file.)
|
||
- **Green-gated by construction**: `pytest || fail` runs BEFORE any studio touch,
|
||
so a red suite aborts with the studio still on the old version. Validated
|
||
2026-07-13 (a mid-deploy rsync failure left the studio untouched/active).
|
||
- **rsync is required** on corviduo-dev (`apt install rsync` — installed 2026-07-13;
|
||
it wasn't present initially).
|
||
- **bifrost dep** resolves from the internal Gitea PyPI via `~/.netrc` (already
|
||
present on corviduo-dev); no extra auth in the deploy script.
|
||
- **⚠️ Auto-deploy silently never worked until 2026-07-14 — TWO compounding blockers.**
|
||
The LISTENER binds `0.0.0.0:9010` and works, but nothing gitea sent ever reached
|
||
it, so every push was a no-op (v0.3.6 was manual; v0.3.7–v0.3.15 never
|
||
auto-deployed until fixed). Two separate, both-real blockers:
|
||
1. **corviduo-dev ufw** — `default-deny`, only 22 + 8080 allowed, so a *direct*
|
||
TCP to :9010 from ana-docker DROP-timed-out. Fix: `ufw allow from 10.0.0.0/8`
|
||
(operator-directed — "that footgun happens a lot", accept the fleet).
|
||
2. **★ gitea `webhook.ALLOWED_HOST_LIST` (the DECISIVE one)** — was
|
||
`external, 10.100.0.0/16` (NH3 only); corviduo-dev is `10.250.50.152`
|
||
(Anaheim), so gitea **refused to deliver**: `webhook can only call allowed HTTP
|
||
servers ... deny '10.250.50.152'` — it never even opens the TCP connection, so
|
||
the ufw fix alone did nothing. Fix: `ALLOWED_HOST_LIST = external, 10.0.0.0/8`
|
||
in gitea `app.ini` (`/data/gitea/conf/app.ini`, `[webhook]`) + `docker restart
|
||
gitea` (~8s blip). The HMAC secret was already correct (once delivery arrives,
|
||
`hmac_ok=True`).
|
||
**RED HERRINGS that cost two diagnosis rounds:** (a) "test-delivery 204" is gitea
|
||
*queuing*, NOT delivering — never proves the round-trip; (b) a proxy test signing
|
||
with the *listener's own* secret (bypassing gitea) proves the listener but NOT
|
||
gitea's real delivery. **Diagnose from BOTH ends:** the SENDER (`docker logs gitea
|
||
--since 5m | grep webhook` → the `deny '<ip>'` line) AND an instrumented RECEIVER
|
||
— the listener now ships with delivery logging (`journalctl -u soong-webhook.service
|
||
| grep '\[webhook\]'` shows source-IP / hmac_ok / ref / action; the old
|
||
`log_message=pass` silence hid all of it). **Proof of fix:** a real gitea delivery
|
||
logs `POST from 10.250.50.70 ... hmac_ok=True`, `ref='refs/heads/main'`,
|
||
`-> 202 deploying` → green deploy of the latest main SHA.
|
||
- **Red-run push-notify** via an **althing relay on nh3-dev** (`soong-ci-relay.timer`,
|
||
2-min poll of corviduo's `last-deploy.json` → pings **soong-dev** via althing on a
|
||
NEW red run; green runs stay silent = fire-and-forget). corviduo itself has no
|
||
althing, so the relay lives on nh3-dev (which does), needing no gitea write token
|
||
on the Worldtree-team VM. Files: `services/soong-lab-ci/soong-ci-relay.{sh,service,timer}`;
|
||
state `~/.local/state/soong-ci-relay/last-at.txt`. (A gitea commit-status was the
|
||
alternative but needs a write token gitea won't mint without basic-auth.)
|
||
- Test suite: `uv run pytest` in `backend/` (242 tests as of v0.3.6).
|