fix(soong-lab-ci): webhook auto-deploy real root cause = gitea ALLOWED_HOST_LIST + add listener logging

The ufw fix (prior commit) was necessary but insufficient. The DECISIVE blocker
was gitea webhook.ALLOWED_HOST_LIST = 'external, 10.100.0.0/16' (NH3 only) —
corviduo-dev is 10.250.50.152 (Anaheim), so gitea refused to deliver ('deny
10.250.50.152') and never opened the TCP connection. Fixed to 'external,
10.0.0.0/8' (whole fleet, matches the ufw choice) + gitea restart.

Listener now logs every delivery (source-IP/hmac_ok/ref/action) — the old
log_message=pass silence hid the whole failure. Proven end-to-end: real gitea
delivery -> hmac_ok=True, ref=main, 202 deploying -> green deploy.

docs/runbooks/soong-lab-autodeploy.md

@@ -56,22 +56,31 @@ ssh corviduo-dev 'bash ~/soong-lab-deploy.sh'
   it wasn't present initially).
 - **bifrost dep** resolves from the internal Gitea PyPI via `~/.netrc` (already
   present on corviduo-dev); no extra auth in the deploy script.
-- **⚠️ ufw firewall — the auto-deploy silently never worked until 2026-07-14.**
+- **⚠️ Auto-deploy silently never worked until 2026-07-14 — TWO compounding blockers.**
-  The webhook LISTENER binds `0.0.0.0:9010` fine, but corviduo-dev's ufw is
+  The LISTENER binds `0.0.0.0:9010` and works, but nothing gitea sent ever reached
-  `default-deny` (only 22 + 8080 were allowed), so gitea's deliveries from
+  it, so every push was a no-op (v0.3.6 was manual; v0.3.7–v0.3.15 never
-  ana-docker **silently timed out (DROP)** — every push landed as a no-op and
+  auto-deployed until fixed). Two separate, both-real blockers:
-  the studio drifted (v0.3.6 was a manual deploy; v0.3.7–v0.3.13 never
+  1. **corviduo-dev ufw** — `default-deny`, only 22 + 8080 allowed, so a *direct*
-  auto-deployed). The setup-time "test-delivery 204" was a RED HERRING: 204 is
+     TCP to :9010 from ana-docker DROP-timed-out. Fix: `ufw allow from 10.0.0.0/8`
-  gitea *queuing* the delivery, NOT the listener receiving it — it never proved
+     (operator-directed — "that footgun happens a lot", accept the fleet).
-  reachability. **Diagnosis signal**: `ssh ana-docker 'curl -m8 http://10.250.50.152:9010/'`
+  2. **★ gitea `webhook.ALLOWED_HOST_LIST` (the DECISIVE one)** — was
-  → HTTP 000 timeout while `ssh corviduo-dev 'curl localhost:9010'` → 200 = a
+     `external, 10.100.0.0/16` (NH3 only); corviduo-dev is `10.250.50.152`
-  firewall/bind gap, not a listener bug. **Fix**: `sudo ufw allow from 10.0.0.0/8`
+     (Anaheim), so gitea **refused to deliver**: `webhook can only call allowed HTTP
-  on corviduo (operator-directed 2026-07-14 — "that footgun happens a lot", so
+     servers ... deny '10.250.50.152'` — it never even opens the TCP connection, so
-  accept the whole internal fleet rather than per-port). Confirmed end-to-end:
+     the ufw fix alone did nothing. Fix: `ALLOWED_HOST_LIST = external, 10.0.0.0/8`
-  gitea→:9010 = 200, a signed `refs/heads/main` push → listener 202 → green
+     in gitea `app.ini` (`/data/gitea/conf/app.ini`, `[webhook]`) + `docker restart
-  deploy; gitea hook secret force-synced to the listener's. The listener's HMAC
+     gitea` (~8s blip). The HMAC secret was already correct (once delivery arrives,
-  secret is the auth layer, so 10/8 exposure is fine. (NOT an `ALLOWED_HOST_LIST`
+     `hmac_ok=True`).
-  / SSRF issue — gitea's egress allowlist was never the blocker here.)
+  **RED HERRINGS that cost two diagnosis rounds:** (a) "test-delivery 204" is gitea
+  *queuing*, NOT delivering — never proves the round-trip; (b) a proxy test signing
+  with the *listener's own* secret (bypassing gitea) proves the listener but NOT
+  gitea's real delivery. **Diagnose from BOTH ends:** the SENDER (`docker logs gitea
+  --since 5m | grep webhook` → the `deny '<ip>'` line) AND an instrumented RECEIVER
+  — the listener now ships with delivery logging (`journalctl -u soong-webhook.service
+  | grep '\[webhook\]'` shows source-IP / hmac_ok / ref / action; the old
+  `log_message=pass` silence hid all of it). **Proof of fix:** a real gitea delivery
+  logs `POST from 10.250.50.70 ... hmac_ok=True`, `ref='refs/heads/main'`,
+  `-> 202 deploying` → green deploy of the latest main SHA.
 - **Red-run push-notify** via an **althing relay on nh3-dev** (`soong-ci-relay.timer`,
   2-min poll of corviduo's `last-deploy.json` → pings **soong-dev** via althing on a
   NEW red run; green runs stay silent = fire-and-forget). corviduo itself has no

services/soong-lab-ci/soong-webhook.py

@@ -1,27 +1,58 @@
 #!/usr/bin/env python3
 """Gitea push-webhook listener for soong-lab: on a verified push to main, run
-~/soong-lab-deploy.sh. HMAC-SHA256 (X-Gitea-Signature) vs ~/.config/soong/webhook-secret."""
+~/soong-lab-deploy.sh. HMAC-SHA256 (X-Gitea-Signature) vs ~/.config/soong/webhook-secret.
-import hashlib, hmac, json, os, subprocess, threading
+
+Logs each delivery (source IP / event / HMAC result / ref / action) to journald.
+Added 2026-07-14 to diagnose the gitea->:9010 REAL-delivery path after the ufw fix
+alone didn't restore auto-deploy (the prior `log_message = pass` made the listener
+silent, so we couldn't see whether gitea was delivering / authenticating / ref-matching)."""
+import hashlib, hmac, json, os, subprocess, sys, threading
 from http.server import BaseHTTPRequestHandler, HTTPServer
+
 SECRET = open(os.path.expanduser("~/.config/soong/webhook-secret"), "rb").read().strip()
 DEPLOY = os.path.expanduser("~/soong-lab-deploy.sh")
 STATUS = os.path.expanduser("~/.config/soong/last-deploy.json")
+
+
+def _log(msg):
+    print(f"[webhook] {msg}", file=sys.stderr, flush=True)
+
+
 class H(BaseHTTPRequestHandler):
     def do_POST(self):
         body = self.rfile.read(int(self.headers.get("Content-Length", 0)))
+        recv = self.headers.get("X-Gitea-Signature", "")
         mac = hmac.new(SECRET, body, hashlib.sha256).hexdigest()
-        if not hmac.compare_digest(mac, self.headers.get("X-Gitea-Signature", "")):
+        ok = hmac.compare_digest(mac, recv)
+        event = self.headers.get("X-Gitea-Event", "?")
+        _log(f"POST from {self.client_address[0]} event={event} clen={len(body)} "
+             f"sig_recv={recv[:12]!r} sig_exp={mac[:12]!r} hmac_ok={ok}")
+        if not ok:
+            _log("-> 401 bad signature (gitea-hook secret != listener secret)")
             self.send_response(401); self.end_headers(); self.wfile.write(b"bad signature\n"); return
-        try: ref = json.loads(body).get("ref", "")
+        try:
-        except Exception: self.send_response(400); self.end_headers(); return
+            ref = json.loads(body).get("ref", "")
+        except Exception:
+            _log("-> 400 bad json")
+            self.send_response(400); self.end_headers(); return
+        _log(f"ref={ref!r}")
         if ref != "refs/heads/main":
+            _log("-> 200 ignored (ref != refs/heads/main)")
             self.send_response(200); self.end_headers(); self.wfile.write(b"ignored " + ref.encode() + b"\n"); return
+        _log("-> 202 deploying")
         self.send_response(202); self.end_headers(); self.wfile.write(b"deploying\n")
         threading.Thread(target=lambda: subprocess.run(["bash", DEPLOY]), daemon=True).start()
+
     def do_GET(self):
         self.send_response(200); self.end_headers()
-        try: st = open(STATUS).read().strip()
+        try:
-        except Exception: st = "no deploy yet"
+            st = open(STATUS).read().strip()
+        except Exception:
+            st = "no deploy yet"
         self.wfile.write(b"soong-webhook ok | last: " + st.encode())
-    def log_message(self, *a): pass
+
+    def log_message(self, *a):
+        pass
+
+
 HTTPServer(("0.0.0.0", 9010), H).serve_forever()