From fb7b5959f3219b5f2450810e4d53d18481c34bbc Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 18 Jul 2026 22:16:53 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20queue=20soong-?= =?UTF-8?q?lab=20auto-redeploy=20for=20next=20session?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deferred-work capture: Vuong approved wiring soong-lab auto-redeploy; operator deferred execution to a fresh context. Recorded in Recent decisions with the mechanism (WT-style CI-deploy step), the blocker (a vh-owned deploy SSH-key secret), and next-session steps; handoff updated to make it the primary task. The "5 AM clock" report was a hallucination in another window — clock verified correct (US/Pacific, NTP-synced), no change. --- .../2026-07-18-soong-lab-auto-redeploy.md | 37 +++++++++++++++++++ persistent-memory.md | 4 +- 2 files changed, 40 insertions(+), 1 deletion(-) create mode 100644 persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md diff --git a/persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md b/persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md new file mode 100644 index 0000000..a14db65 --- /dev/null +++ b/persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md @@ -0,0 +1,37 @@ +- `[2026-07-18]` **soong-lab auto-redeploy — APPROVED, queued for next session (DEFERRED, not started).** + +Vuong approved wiring auto-redeploy for soong-lab (relayed via soong-dev, thread +`01KXT3A6C3908TA4V9THV3AMH7`): new images should go live on corviduo-dev without +the manual `docker compose pull && up -d`. Host-side implementation is infra-ops's +lane; mechanism is infra-ops's call per fleet conventions. Operator deferred +execution — "we'll do soong on fresh context." + +**Chosen mechanism (recommended, agrees with soong-dev): Worldtree-style +CI-deploy step** — NOT watchtower polling. +- Add a deploy job/step to soong-lab's `.gitea/workflows/build-and-push.yml` that, + after the build+push job succeeds, **SSHes from the pfi-fleet runner to + corviduo-dev** and runs `cd /home/infra-ops/soong-lab-deploy && docker compose + pull && docker compose up -d`, then a **health-gate** (`curl -fsS + http://localhost:8443/api/version`). +- This is exactly how WT deploys the demo instance to the SAME host: see + `~/development/Worldtree/.gitea/workflows/deploy.yml` — the "Deploy to demo VM + + health-gate" step uses `secrets.DEMO_VM_SSH_KEY` / `DEMO_VM_HOST` / `DEMO_VM_USER`. + Explicit-over-implicit (visible in the run log, fires exactly on build success), + one less always-on service than watchtower. + +**Constraints (from soong-dev):** deploy on CI success only; keep the trigger +gated to `v*` tags + `workflow_dispatch` (as today); preserve the one-command +rollback posture (`docker compose down` / pin a previous tag). + +**BLOCKER — needs from vh (owner-only):** a **runner→corviduo-dev deploy SSH key** +as a repo secret (+ host/user), same class as WT's `DEMO_VM_SSH_KEY`. Likely +**reuse WT's existing demo-deploy key** (WT's runner already SSHes to 10.250.50.152 +as its deploy user). Repo secrets are vh-owner-only (see +[[2026-07-18-fleet-gitea-runner-build-recipe]]). + +**Next-session steps:** (1) confirm/obtain the deploy SSH-key secret from vh (reuse +WT's or mint fresh); (2) add the deploy job to build-and-push.yml (infra-ops has +push on vh/soong-lab); (3) dispatch a build to verify it deploys + health-gates; +(4) ping soong-dev so they sync DEPLOY.md's "open follow-up" note to the as-built +mechanism. Auto-pull (watchtower) explicitly NOT chosen. See +[[2026-07-18-soong-lab-containerize-cutover]]. diff --git a/persistent-memory.md b/persistent-memory.md index e6918c7..2967931 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -114,7 +114,7 @@ _As of 2026-07-18 — end of a long session. **Four major items ALL LANDED** (de **Open follow-ups (non-blocking — pick one up or not):** - **Zonos emotion:** sad axes/text pass on the 3 calibrated voices (only named-sad, untested); emotion-congruent-text pass (validates intensity, may rescue sad id); clone-char (Emmie/Penny/Natalie/Miranda) emotion rows use the mid-region fallback until measured. Presets are **provisional** (neutral-text ear-check was inconclusive). Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio,assemble_voice}.py` (run ON irv-ml1; dial-in studio = nohup :8898 on nh3-dev). dvalin thread at rest (`01KXT12FN0AS…`). → `persistent-memory.d/2026-07-18-zonos-gateway-0.2.1-emotion-presets.md` - **zonos-gateway CI-wire:** deployed irv-ml1 tree `/opt/docker/compose/zonos-gateway` is still NON-git (hand-updated build context) — git-connect + build-on-push like the other sisters. (Same pattern soong-lab now has.) -- **soong-lab:** cutover DONE; future redeploy = `cd /home/infra-ops/soong-lab-deploy && sudo docker compose pull && up -d`; optional auto-pull-on-`:latest` (watchtower/deploy-hook). → `persistent-memory.d/2026-07-18-soong-lab-containerize-cutover.md` +- **soong-lab:** cutover DONE + **production-proven** (soong-dev confirmed a real Soong turn saved a new design + served its portrait through the migrated volumes post-cutover). **NEXT-SESSION TASK (approved, queued): wire auto-redeploy** — WT-style CI-deploy step, blocked on a vh deploy-SSH-key secret. → `persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md` (+ cutover detail `persistent-memory.d/2026-07-18-soong-lab-containerize-cutover.md`; manual redeploy meanwhile = `cd /home/infra-ops/soong-lab-deploy && sudo docker compose pull && up -d`). **Zonos voice stack (LIVE, unchanged):** 8 voices in `zonos-gateway` (`:8890` irv-ml1) — defaults AmericanFemale/Male/BritishFemale/Cora + 4 clones Emmie/Penny/Natalie/Miranda; add a voice = drop `.wav` in `/opt/docker/compose/zonos-gateway/voices/` + `docker compose restart` (host-managed bind-mount, NO rebuild). Clone pipeline: `/mnt/smithy/voice_clones/.zip` → `assemble_voice.py` → drop. Dial-in studio http://10.100.10.50:8898/ (nohup on nh3-dev, relaunch `nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &`). @@ -126,6 +126,8 @@ _As of 2026-07-18 — end of a long session. **Four major items ALL LANDED** (de ## Recent decisions +- `[2026-07-18]` **soong-lab auto-redeploy APPROVED — QUEUED for next session (deferred, not started)** — Vuong approved (via soong-dev thread `01KXT3A6C3908TA4V9THV3AMH7`); mechanism = WT-style CI-deploy step (runner SSHes corviduo-dev → `compose pull && up -d` + health-gate); **blocked on a vh-owned runner→corviduo-dev deploy SSH-key secret** (reuse WT's demo-deploy key). Operator: "do soong on fresh context." → `persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md` + - `[2026-07-18]` **nh3-dev /tmp auto-clean enabled** — Debian ships /tmp with no tmpfiles age (`D /tmp 1777 root root -` → never cleans); this high-churn agent box had accreted **~190k stale temp dirs / 25G**. One-shot manual purge (194k→10k entries, 25G→1.7G; deleted top-level dirs/files >1d old, spared `/tmp/claude-*` by name + anything ≤1d). Then `/etc/tmpfiles.d/tmp.conf` = `D /tmp 1777 root root 3d` (daily `systemd-tmpfiles-clean.timer` removes >3d-untouched items; active files + socket dirs spared). Tunable via the age. Note the churn: ~10k /tmp entries/day here. - `[2026-07-18]` **soong-lab containerize cutover COMPLETE + LIVE** — systemd→container on corviduo-dev :8443 (image `vh/soong-lab:latest` v0.3.24), data migrated (Sindra + portraits) + backed up, old service+webhook retired, Homepage tile added, operator functional-confirmed. Deploy `/home/infra-ops/soong-lab-deploy/`; no proxy (co-located WT, plain-http callback). → `persistent-memory.d/2026-07-18-soong-lab-containerize-cutover.md`