feat(soong-lab-ci): red-run althing relay (nh3-dev poll -> ping soong-dev)

Per operator call (no gitea write token on the Worldtree-team VM): a 2-min systemd
--user timer on nh3-dev polls corviduo's last-deploy.json and pings soong-dev via
althing on a NEW red deploy (green stays silent). Delivers soong-dev's red-run
visibility without a credential on corviduo. Tested (red detect+format DRY, green quiet).
This commit is contained in:
vh
2026-07-13 14:28:39 -07:00
parent fb556586e3
commit cc0e3af87d
4 changed files with 43 additions and 3 deletions
@@ -0,0 +1,6 @@
[Unit]
Description=soong-lab CI red-run relay (poll corviduo -> ping soong-dev on red)
[Service]
Type=oneshot
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
ExecStart=%h/.local/bin/soong-ci-relay.sh
+23
View File
@@ -0,0 +1,23 @@
#!/usr/bin/env bash
# soong-lab CI red-run relay (runs on nh3-dev, which has althing): poll corviduo's
# last-deploy.json; on a NEW red run, ping soong-dev via althing. No cred on corviduo.
set -uo pipefail
export ALTHING_HANDLE=infra-ops
STATE="$HOME/.local/state/soong-ci-relay/last-at.txt"
mkdir -p "$(dirname "$STATE")"
CUR=$(ssh -o BatchMode=yes -o ConnectTimeout=10 corviduo-dev 'cat ~/.config/soong/last-deploy.json 2>/dev/null' 2>/dev/null) || exit 0
[ -z "$CUR" ] && exit 0
get(){ printf '%s' "$CUR" | python3 -c "import sys,json;print(json.load(sys.stdin).get('$1',''))" 2>/dev/null; }
AT=$(get at); RESULT=$(get result)
[ -z "$AT" ] && exit 0
LAST=$(cat "$STATE" 2>/dev/null || true)
[ "$AT" = "$LAST" ] && exit 0
printf '%s' "$AT" > "$STATE"
[ "$RESULT" != "red" ] && exit 0
SHA=$(get sha); STAGE=$(get stage); DETAIL=$(get detail)
MSG="soong-lab CI deploy FAILED (RED) — sha=$SHA, stage=$STAGE, detail: $DETAIL.
The studio was NOT touched (still on the previous good version — safe-abort).
A red runner-run despite green-local usually means a runner-vs-local env divergence.
Log: corviduo-dev:~/soong-lab-deploy.log | status: curl corviduo-dev:9010/"
if [ "${DRY_RUN:-0}" = "1" ]; then echo "[DRY] would post to soong-dev:"; echo "$MSG"; exit 0; fi
printf '%s' "$MSG" | althing-cli post --to soong-dev --subject "soong-lab CI: RED deploy ($SHA) — studio untouched"
@@ -0,0 +1,7 @@
[Unit]
Description=poll soong-lab CI status every 2 min
[Timer]
OnBootSec=90
OnUnitActiveSec=120
[Install]
WantedBy=timers.target