fix(soong-lab-ci): webhook auto-deploy real root cause = gitea ALLOWED_HOST_LIST + add listener logging
The ufw fix (prior commit) was necessary but insufficient. The DECISIVE blocker
was gitea webhook.ALLOWED_HOST_LIST = 'external, 10.100.0.0/16' (NH3 only) —
corviduo-dev is 10.250.50.152 (Anaheim), so gitea refused to deliver ('deny
10.250.50.152') and never opened the TCP connection. Fixed to 'external,
10.0.0.0/8' (whole fleet, matches the ufw choice) + gitea restart.
Listener now logs every delivery (source-IP/hmac_ok/ref/action) — the old
log_message=pass silence hid the whole failure. Proven end-to-end: real gitea
delivery -> hmac_ok=True, ref=main, 202 deploying -> green deploy.
This commit is contained in:
@@ -56,22 +56,31 @@ ssh corviduo-dev 'bash ~/soong-lab-deploy.sh'
|
||||
it wasn't present initially).
|
||||
- **bifrost dep** resolves from the internal Gitea PyPI via `~/.netrc` (already
|
||||
present on corviduo-dev); no extra auth in the deploy script.
|
||||
- **⚠️ ufw firewall — the auto-deploy silently never worked until 2026-07-14.**
|
||||
The webhook LISTENER binds `0.0.0.0:9010` fine, but corviduo-dev's ufw is
|
||||
`default-deny` (only 22 + 8080 were allowed), so gitea's deliveries from
|
||||
ana-docker **silently timed out (DROP)** — every push landed as a no-op and
|
||||
the studio drifted (v0.3.6 was a manual deploy; v0.3.7–v0.3.13 never
|
||||
auto-deployed). The setup-time "test-delivery 204" was a RED HERRING: 204 is
|
||||
gitea *queuing* the delivery, NOT the listener receiving it — it never proved
|
||||
reachability. **Diagnosis signal**: `ssh ana-docker 'curl -m8 http://10.250.50.152:9010/'`
|
||||
→ HTTP 000 timeout while `ssh corviduo-dev 'curl localhost:9010'` → 200 = a
|
||||
firewall/bind gap, not a listener bug. **Fix**: `sudo ufw allow from 10.0.0.0/8`
|
||||
on corviduo (operator-directed 2026-07-14 — "that footgun happens a lot", so
|
||||
accept the whole internal fleet rather than per-port). Confirmed end-to-end:
|
||||
gitea→:9010 = 200, a signed `refs/heads/main` push → listener 202 → green
|
||||
deploy; gitea hook secret force-synced to the listener's. The listener's HMAC
|
||||
secret is the auth layer, so 10/8 exposure is fine. (NOT an `ALLOWED_HOST_LIST`
|
||||
/ SSRF issue — gitea's egress allowlist was never the blocker here.)
|
||||
- **⚠️ Auto-deploy silently never worked until 2026-07-14 — TWO compounding blockers.**
|
||||
The LISTENER binds `0.0.0.0:9010` and works, but nothing gitea sent ever reached
|
||||
it, so every push was a no-op (v0.3.6 was manual; v0.3.7–v0.3.15 never
|
||||
auto-deployed until fixed). Two separate, both-real blockers:
|
||||
1. **corviduo-dev ufw** — `default-deny`, only 22 + 8080 allowed, so a *direct*
|
||||
TCP to :9010 from ana-docker DROP-timed-out. Fix: `ufw allow from 10.0.0.0/8`
|
||||
(operator-directed — "that footgun happens a lot", accept the fleet).
|
||||
2. **★ gitea `webhook.ALLOWED_HOST_LIST` (the DECISIVE one)** — was
|
||||
`external, 10.100.0.0/16` (NH3 only); corviduo-dev is `10.250.50.152`
|
||||
(Anaheim), so gitea **refused to deliver**: `webhook can only call allowed HTTP
|
||||
servers ... deny '10.250.50.152'` — it never even opens the TCP connection, so
|
||||
the ufw fix alone did nothing. Fix: `ALLOWED_HOST_LIST = external, 10.0.0.0/8`
|
||||
in gitea `app.ini` (`/data/gitea/conf/app.ini`, `[webhook]`) + `docker restart
|
||||
gitea` (~8s blip). The HMAC secret was already correct (once delivery arrives,
|
||||
`hmac_ok=True`).
|
||||
**RED HERRINGS that cost two diagnosis rounds:** (a) "test-delivery 204" is gitea
|
||||
*queuing*, NOT delivering — never proves the round-trip; (b) a proxy test signing
|
||||
with the *listener's own* secret (bypassing gitea) proves the listener but NOT
|
||||
gitea's real delivery. **Diagnose from BOTH ends:** the SENDER (`docker logs gitea
|
||||
--since 5m | grep webhook` → the `deny '<ip>'` line) AND an instrumented RECEIVER
|
||||
— the listener now ships with delivery logging (`journalctl -u soong-webhook.service
|
||||
| grep '\[webhook\]'` shows source-IP / hmac_ok / ref / action; the old
|
||||
`log_message=pass` silence hid all of it). **Proof of fix:** a real gitea delivery
|
||||
logs `POST from 10.250.50.70 ... hmac_ok=True`, `ref='refs/heads/main'`,
|
||||
`-> 202 deploying` → green deploy of the latest main SHA.
|
||||
- **Red-run push-notify** via an **althing relay on nh3-dev** (`soong-ci-relay.timer`,
|
||||
2-min poll of corviduo's `last-deploy.json` → pings **soong-dev** via althing on a
|
||||
NEW red run; green runs stay silent = fire-and-forget). corviduo itself has no
|
||||
|
||||
Reference in New Issue
Block a user