fix(soong-lab-ci): webhook auto-deploy real root cause = gitea ALLOWED_HOST_LIST + add listener logging

The ufw fix (prior commit) was necessary but insufficient. The DECISIVE blocker was gitea webhook.ALLOWED_HOST_LIST = 'external, 10.100.0.0/16' (NH3 only) — corviduo-dev is 10.250.50.152 (Anaheim), so gitea refused to deliver ('deny 10.250.50.152') and never opened the TCP connection. Fixed to 'external, 10.0.0.0/8' (whole fleet, matches the ufw choice) + gitea restart. Listener now logs every delivery (source-IP/hmac_ok/ref/action) — the old log_message=pass silence hid the whole failure. Proven end-to-end: real gitea delivery -> hmac_ok=True, ref=main, 202 deploying -> green deploy.
2026-07-14 09:19:38 -07:00
parent 4fc0c27485
commit 462d528bef
2 changed files with 64 additions and 24 deletions
@@ -56,22 +56,31 @@ ssh corviduo-dev 'bash ~/soong-lab-deploy.sh'
  it wasn't present initially).
 - **bifrost dep** resolves from the internal Gitea PyPI via `~/.netrc` (already
  present on corviduo-dev); no extra auth in the deploy script.
- **⚠️ ufw firewall — the auto-deploy silently never worked until 2026-07-14.**
-  The webhook LISTENER binds `0.0.0.0:9010` fine, but corviduo-dev's ufw is
-  `default-deny` (only 22 + 8080 were allowed), so gitea's deliveries from
-  ana-docker **silently timed out (DROP)** — every push landed as a no-op and
-  the studio drifted (v0.3.6 was a manual deploy; v0.3.7–v0.3.13 never
-  auto-deployed). The setup-time "test-delivery 204" was a RED HERRING: 204 is
-  gitea *queuing* the delivery, NOT the listener receiving it — it never proved
-  reachability. **Diagnosis signal**: `ssh ana-docker 'curl -m8 http://10.250.50.152:9010/'`
-  → HTTP 000 timeout while `ssh corviduo-dev 'curl localhost:9010'` → 200 = a
-  firewall/bind gap, not a listener bug. **Fix**: `sudo ufw allow from 10.0.0.0/8`
-  on corviduo (operator-directed 2026-07-14 — "that footgun happens a lot", so
-  accept the whole internal fleet rather than per-port). Confirmed end-to-end:
-  gitea→:9010 = 200, a signed `refs/heads/main` push → listener 202 → green
-  deploy; gitea hook secret force-synced to the listener's. The listener's HMAC
-  secret is the auth layer, so 10/8 exposure is fine. (NOT an `ALLOWED_HOST_LIST`
-  / SSRF issue — gitea's egress allowlist was never the blocker here.)
+- **⚠️ Auto-deploy silently never worked until 2026-07-14 — TWO compounding blockers.**
+  The LISTENER binds `0.0.0.0:9010` and works, but nothing gitea sent ever reached
+  it, so every push was a no-op (v0.3.6 was manual; v0.3.7–v0.3.15 never
+  auto-deployed until fixed). Two separate, both-real blockers:
+  1. **corviduo-dev ufw** — `default-deny`, only 22 + 8080 allowed, so a *direct*
+     TCP to :9010 from ana-docker DROP-timed-out. Fix: `ufw allow from 10.0.0.0/8`
+     (operator-directed — "that footgun happens a lot", accept the fleet).
+  2. **★ gitea `webhook.ALLOWED_HOST_LIST` (the DECISIVE one)** — was
+     `external, 10.100.0.0/16` (NH3 only); corviduo-dev is `10.250.50.152`
+     (Anaheim), so gitea **refused to deliver**: `webhook can only call allowed HTTP
+     servers ... deny '10.250.50.152'` — it never even opens the TCP connection, so
+     the ufw fix alone did nothing. Fix: `ALLOWED_HOST_LIST = external, 10.0.0.0/8`
+     in gitea `app.ini` (`/data/gitea/conf/app.ini`, `[webhook]`) + `docker restart
+     gitea` (~8s blip). The HMAC secret was already correct (once delivery arrives,
+     `hmac_ok=True`).
+  **RED HERRINGS that cost two diagnosis rounds:** (a) "test-delivery 204" is gitea
+  *queuing*, NOT delivering — never proves the round-trip; (b) a proxy test signing
+  with the *listener's own* secret (bypassing gitea) proves the listener but NOT
+  gitea's real delivery. **Diagnose from BOTH ends:** the SENDER (`docker logs gitea
+  --since 5m | grep webhook` → the `deny '<ip>'` line) AND an instrumented RECEIVER
+  — the listener now ships with delivery logging (`journalctl -u soong-webhook.service
+  | grep '\[webhook\]'` shows source-IP / hmac_ok / ref / action; the old
+  `log_message=pass` silence hid all of it). **Proof of fix:** a real gitea delivery
+  logs `POST from 10.250.50.70 ... hmac_ok=True`, `ref='refs/heads/main'`,
+  `-> 202 deploying` → green deploy of the latest main SHA.
 - **Red-run push-notify** via an **althing relay on nh3-dev** (`soong-ci-relay.timer`,
  2-min poll of corviduo's `last-deploy.json` → pings **soong-dev** via althing on a
  NEW red run; green runs stay silent = fire-and-forget). corviduo itself has no
@@ -1,27 +1,58 @@
 #!/usr/bin/env python3
 """Gitea push-webhook listener for soong-lab: on a verified push to main, run
-~/soong-lab-deploy.sh. HMAC-SHA256 (X-Gitea-Signature) vs ~/.config/soong/webhook-secret."""
-import hashlib, hmac, json, os, subprocess, threading
+~/soong-lab-deploy.sh. HMAC-SHA256 (X-Gitea-Signature) vs ~/.config/soong/webhook-secret.
+
+Logs each delivery (source IP / event / HMAC result / ref / action) to journald.
+Added 2026-07-14 to diagnose the gitea->:9010 REAL-delivery path after the ufw fix
+alone didn't restore auto-deploy (the prior `log_message = pass` made the listener
+silent, so we couldn't see whether gitea was delivering / authenticating / ref-matching)."""
+import hashlib, hmac, json, os, subprocess, sys, threading
 from http.server import BaseHTTPRequestHandler, HTTPServer
+
 SECRET = open(os.path.expanduser("~/.config/soong/webhook-secret"), "rb").read().strip()
 DEPLOY = os.path.expanduser("~/soong-lab-deploy.sh")
 STATUS = os.path.expanduser("~/.config/soong/last-deploy.json")
+
+
+def _log(msg):
+    print(f"[webhook] {msg}", file=sys.stderr, flush=True)
+
+
 class H(BaseHTTPRequestHandler):
    def do_POST(self):
        body = self.rfile.read(int(self.headers.get("Content-Length", 0)))
+        recv = self.headers.get("X-Gitea-Signature", "")
        mac = hmac.new(SECRET, body, hashlib.sha256).hexdigest()
-        if not hmac.compare_digest(mac, self.headers.get("X-Gitea-Signature", "")):
+        ok = hmac.compare_digest(mac, recv)
+        event = self.headers.get("X-Gitea-Event", "?")
+        _log(f"POST from {self.client_address[0]} event={event} clen={len(body)} "
+             f"sig_recv={recv[:12]!r} sig_exp={mac[:12]!r} hmac_ok={ok}")
+        if not ok:
+            _log("-> 401 bad signature (gitea-hook secret != listener secret)")
            self.send_response(401); self.end_headers(); self.wfile.write(b"bad signature\n"); return
-        try: ref = json.loads(body).get("ref", "")
-        except Exception: self.send_response(400); self.end_headers(); return
+        try:
+            ref = json.loads(body).get("ref", "")
+        except Exception:
+            _log("-> 400 bad json")
+            self.send_response(400); self.end_headers(); return
+        _log(f"ref={ref!r}")
        if ref != "refs/heads/main":
+            _log("-> 200 ignored (ref != refs/heads/main)")
            self.send_response(200); self.end_headers(); self.wfile.write(b"ignored " + ref.encode() + b"\n"); return
+        _log("-> 202 deploying")
        self.send_response(202); self.end_headers(); self.wfile.write(b"deploying\n")
        threading.Thread(target=lambda: subprocess.run(["bash", DEPLOY]), daemon=True).start()
+
    def do_GET(self):
        self.send_response(200); self.end_headers()
-        try: st = open(STATUS).read().strip()
-        except Exception: st = "no deploy yet"
+        try:
+            st = open(STATUS).read().strip()
+        except Exception:
+            st = "no deploy yet"
        self.wfile.write(b"soong-webhook ok | last: " + st.encode())
-    def log_message(self, *a): pass
+
+    def log_message(self, *a):
+        pass
+
+
 HTTPServer(("0.0.0.0", 9010), H).serve_forever()