diff --git a/docs/runbooks/soong-lab-autodeploy.md b/docs/runbooks/soong-lab-autodeploy.md index 0d2db16..3caa70a 100644 --- a/docs/runbooks/soong-lab-autodeploy.md +++ b/docs/runbooks/soong-lab-autodeploy.md @@ -56,22 +56,31 @@ ssh corviduo-dev 'bash ~/soong-lab-deploy.sh' it wasn't present initially). - **bifrost dep** resolves from the internal Gitea PyPI via `~/.netrc` (already present on corviduo-dev); no extra auth in the deploy script. -- **⚠️ ufw firewall — the auto-deploy silently never worked until 2026-07-14.** - The webhook LISTENER binds `0.0.0.0:9010` fine, but corviduo-dev's ufw is - `default-deny` (only 22 + 8080 were allowed), so gitea's deliveries from - ana-docker **silently timed out (DROP)** — every push landed as a no-op and - the studio drifted (v0.3.6 was a manual deploy; v0.3.7–v0.3.13 never - auto-deployed). The setup-time "test-delivery 204" was a RED HERRING: 204 is - gitea *queuing* the delivery, NOT the listener receiving it — it never proved - reachability. **Diagnosis signal**: `ssh ana-docker 'curl -m8 http://10.250.50.152:9010/'` - → HTTP 000 timeout while `ssh corviduo-dev 'curl localhost:9010'` → 200 = a - firewall/bind gap, not a listener bug. **Fix**: `sudo ufw allow from 10.0.0.0/8` - on corviduo (operator-directed 2026-07-14 — "that footgun happens a lot", so - accept the whole internal fleet rather than per-port). Confirmed end-to-end: - gitea→:9010 = 200, a signed `refs/heads/main` push → listener 202 → green - deploy; gitea hook secret force-synced to the listener's. The listener's HMAC - secret is the auth layer, so 10/8 exposure is fine. (NOT an `ALLOWED_HOST_LIST` - / SSRF issue — gitea's egress allowlist was never the blocker here.) +- **⚠️ Auto-deploy silently never worked until 2026-07-14 — TWO compounding blockers.** + The LISTENER binds `0.0.0.0:9010` and works, but nothing gitea sent ever reached + it, so every push was a no-op (v0.3.6 was manual; v0.3.7–v0.3.15 never + auto-deployed until fixed). Two separate, both-real blockers: + 1. **corviduo-dev ufw** — `default-deny`, only 22 + 8080 allowed, so a *direct* + TCP to :9010 from ana-docker DROP-timed-out. Fix: `ufw allow from 10.0.0.0/8` + (operator-directed — "that footgun happens a lot", accept the fleet). + 2. **★ gitea `webhook.ALLOWED_HOST_LIST` (the DECISIVE one)** — was + `external, 10.100.0.0/16` (NH3 only); corviduo-dev is `10.250.50.152` + (Anaheim), so gitea **refused to deliver**: `webhook can only call allowed HTTP + servers ... deny '10.250.50.152'` — it never even opens the TCP connection, so + the ufw fix alone did nothing. Fix: `ALLOWED_HOST_LIST = external, 10.0.0.0/8` + in gitea `app.ini` (`/data/gitea/conf/app.ini`, `[webhook]`) + `docker restart + gitea` (~8s blip). The HMAC secret was already correct (once delivery arrives, + `hmac_ok=True`). + **RED HERRINGS that cost two diagnosis rounds:** (a) "test-delivery 204" is gitea + *queuing*, NOT delivering — never proves the round-trip; (b) a proxy test signing + with the *listener's own* secret (bypassing gitea) proves the listener but NOT + gitea's real delivery. **Diagnose from BOTH ends:** the SENDER (`docker logs gitea + --since 5m | grep webhook` → the `deny ''` line) AND an instrumented RECEIVER + — the listener now ships with delivery logging (`journalctl -u soong-webhook.service + | grep '\[webhook\]'` shows source-IP / hmac_ok / ref / action; the old + `log_message=pass` silence hid all of it). **Proof of fix:** a real gitea delivery + logs `POST from 10.250.50.70 ... hmac_ok=True`, `ref='refs/heads/main'`, + `-> 202 deploying` → green deploy of the latest main SHA. - **Red-run push-notify** via an **althing relay on nh3-dev** (`soong-ci-relay.timer`, 2-min poll of corviduo's `last-deploy.json` → pings **soong-dev** via althing on a NEW red run; green runs stay silent = fire-and-forget). corviduo itself has no diff --git a/services/soong-lab-ci/soong-webhook.py b/services/soong-lab-ci/soong-webhook.py index 12ec1de..4b54d08 100644 --- a/services/soong-lab-ci/soong-webhook.py +++ b/services/soong-lab-ci/soong-webhook.py @@ -1,27 +1,58 @@ #!/usr/bin/env python3 """Gitea push-webhook listener for soong-lab: on a verified push to main, run -~/soong-lab-deploy.sh. HMAC-SHA256 (X-Gitea-Signature) vs ~/.config/soong/webhook-secret.""" -import hashlib, hmac, json, os, subprocess, threading +~/soong-lab-deploy.sh. HMAC-SHA256 (X-Gitea-Signature) vs ~/.config/soong/webhook-secret. + +Logs each delivery (source IP / event / HMAC result / ref / action) to journald. +Added 2026-07-14 to diagnose the gitea->:9010 REAL-delivery path after the ufw fix +alone didn't restore auto-deploy (the prior `log_message = pass` made the listener +silent, so we couldn't see whether gitea was delivering / authenticating / ref-matching).""" +import hashlib, hmac, json, os, subprocess, sys, threading from http.server import BaseHTTPRequestHandler, HTTPServer + SECRET = open(os.path.expanduser("~/.config/soong/webhook-secret"), "rb").read().strip() DEPLOY = os.path.expanduser("~/soong-lab-deploy.sh") STATUS = os.path.expanduser("~/.config/soong/last-deploy.json") + + +def _log(msg): + print(f"[webhook] {msg}", file=sys.stderr, flush=True) + + class H(BaseHTTPRequestHandler): def do_POST(self): body = self.rfile.read(int(self.headers.get("Content-Length", 0))) + recv = self.headers.get("X-Gitea-Signature", "") mac = hmac.new(SECRET, body, hashlib.sha256).hexdigest() - if not hmac.compare_digest(mac, self.headers.get("X-Gitea-Signature", "")): + ok = hmac.compare_digest(mac, recv) + event = self.headers.get("X-Gitea-Event", "?") + _log(f"POST from {self.client_address[0]} event={event} clen={len(body)} " + f"sig_recv={recv[:12]!r} sig_exp={mac[:12]!r} hmac_ok={ok}") + if not ok: + _log("-> 401 bad signature (gitea-hook secret != listener secret)") self.send_response(401); self.end_headers(); self.wfile.write(b"bad signature\n"); return - try: ref = json.loads(body).get("ref", "") - except Exception: self.send_response(400); self.end_headers(); return + try: + ref = json.loads(body).get("ref", "") + except Exception: + _log("-> 400 bad json") + self.send_response(400); self.end_headers(); return + _log(f"ref={ref!r}") if ref != "refs/heads/main": + _log("-> 200 ignored (ref != refs/heads/main)") self.send_response(200); self.end_headers(); self.wfile.write(b"ignored " + ref.encode() + b"\n"); return + _log("-> 202 deploying") self.send_response(202); self.end_headers(); self.wfile.write(b"deploying\n") threading.Thread(target=lambda: subprocess.run(["bash", DEPLOY]), daemon=True).start() + def do_GET(self): self.send_response(200); self.end_headers() - try: st = open(STATUS).read().strip() - except Exception: st = "no deploy yet" + try: + st = open(STATUS).read().strip() + except Exception: + st = "no deploy yet" self.wfile.write(b"soong-webhook ok | last: " + st.encode()) - def log_message(self, *a): pass + + def log_message(self, *a): + pass + + HTTPServer(("0.0.0.0", 9010), H).serve_forever()