memory: snapshot — infra-ops WT config-repo build queued; Booth shipped; #376 closed
Centered on the resume task: build the infra-ops-owned Worldtree per-deployment config repo + deploy tooling (operator-directed) with the one-time worldtree-dev boundary agreement. Also records: The Booth shipped (v0.1.3), the jackdaw-compose nh3-dev deploy, and the Worldtree #376 arc closed (per-instance config ruled by-design). Two detail files + the /tmp handoff.
This commit is contained in:
@@ -0,0 +1,18 @@
|
||||
`[2026-07-25]` **infra-ops to OWN a Worldtree per-deployment config repo + deploy tooling (operator-directed; IN-FLIGHT — build next session).**
|
||||
|
||||
**Decision.** Vuong directed (2026-07-25, this session) that Worldtree instance config should be a *tracked change*, **managed and deployed by infra-ops — not worldtree-dev**. Model: worldtree-dev owns the app/image (+ the baked baseline defaults); **infra-ops owns config-as-code for every deployment** and deploys it. This is the durable fix for the root cause behind the whole #376 arc — config was edited live on host bind-mounts (`/opt/<instance>/config/`) with zero version history, audit, or recovery.
|
||||
|
||||
**What "no worldtree-dev involvement" does and does NOT cover** (clarified with the operator this session):
|
||||
- **Build + deploy = infra-ops-only.** Deploying config = write the host bind-mount file + restart the container (the *exact* procedure already run this session — backup → replace → restart → health-gate → rollback-on-unhealthy). No worldtree-dev in the deploy loop. Their CI only swaps the IMAGE; it does NOT resync the host config bind-mount (confirmed #376 finding).
|
||||
- **ONE load-bearing exception — a one-time boundary agreement, NOT per-deploy involvement:** for the repo to *own* config it must be the **only writer**. worldtree-dev "live-bridges" (hand-edits mounted config directly on the box). If the repo deploys config *and* they keep live-editing → **two writers fighting the same files** = #376 all over again. So secure a one-time "yes" from worldtree-dev: *the config repo is now authoritative; stop hand-editing `/opt/<instance>/config`; route config changes through the repo.* (Five-minute agreement, not a design collab.)
|
||||
- **Standing coupling (not "involvement"):** the config *schema* is the app's, enforced by its boot validator (`core.config_validator`). infra-ops configs must stay schema-compatible with the deployed image; the boot gate is the loud backstop.
|
||||
|
||||
**Build shape (recommended):**
|
||||
- Gitea repo `worldtree-instance-configs` (infra-ops-owned), **dir per instance** (`demo/`, `personal/`, `pinned/` — the three on corviduo-dev 10.250.50.152: demo `worldtree-worldtree-api-1` :8080, personal `worldtree-personal-worldtree-api-1` :8081, pinned `worldtree-pinned-worldtree-api-1` :8082). Config dirs: demo `/opt/worldtree/config`, personal `/opt/worldtree-personal/config`, pinned `/opt/worldtree-pinned/config` (verify pinned's mount).
|
||||
- **SEED FROM CURRENT MOUNTED STATE, don't author fresh** — capture each instance's live config (incl. legitimate live-bridged deltas: personal carries `agent_architect` role [Soong/soong-lab] in model_roles.yaml + `ratatoskr-affect-full-allow` in policies.yaml that are NOT in the app repo — the operator ruled these are BY DESIGN, keep them). Losing them = breakage (the affect-render one gates mood rendering).
|
||||
- Deploy script (e.g. `scripts/deploy-wt-config <instance>`): git = source of truth → push to host bind-mount + `docker restart` (same pinned image, no pull) + health-gate + auto-rollback. This is the proven-this-session procedure, scripted.
|
||||
- Files per instance: `policies.yaml`, `model_roles.yaml` (+ whatever else is bind-mounted — `defaults.yaml`, `providers.yaml`, `matrix.yaml` all live in `/opt/<instance>/config`; decide scope — policies+model_roles are the authz/role layer, defaults/providers are heavier instance tunables).
|
||||
|
||||
**Tracking surface:** operator-directed 2026-07-25, carried by this snapshot + `/tmp/infra-ops-handoff.md`. No issue filed (infra-ops-internal build). Related fleet idiom to reuse: canonical-sync (`.corviduo-canonicals.toml` / `canonical_sync.py`). Later scale option (deferred, needs worldtree-dev): base+overlay with a merge step in their pipeline.
|
||||
|
||||
See [[2026-07-25-wt-376-per-instance-config-arc]] for the incident that produced this. Auto-memory: `reference_worldtree_perinstance_config`, `reference_corviduo_dev_emergency_ops`.
|
||||
@@ -0,0 +1,13 @@
|
||||
`[2026-07-23→25]` **Worldtree #376 config-divergence arc — wyrd grant fixed, drift guard demoted, per-instance config ruled BY DESIGN.**
|
||||
|
||||
**Trigger.** wyrd-dev needed `session.history.write` on the DEMO Worldtree (operator-approved) — add `wyrd-dev` to the `session-history-write-ratatoskr` policy rule. Attempting it surfaced that the demo runtime `/opt/worldtree/config/policies.yaml` (mtime Jul-17) had **silently diverged from the repo** — missing whole rules, not a faithful copy of any revision. Refused to hand-edit a divergent authz file on a managed box; worldtree-dev prescribed a **wholesale replace** with `main@55f3fde`. Executed (backup → replace → `docker restart worldtree-worldtree-api-1` same-image → health-gate → verify) — grant live, demo policies at parity. **This was the ONE genuine bug of the arc:** a demo-intended grant that wasn't ON demo.
|
||||
|
||||
**Drift guard (#376).** worldtree-dev shipped a startup guard (b131/`62b85e3`) that hashes mounted config vs the image's baked copy, logging `CONFIG DRIFT (#376)`. First reading found MORE drift: `model_roles.yaml` on both instances + personal `policies.yaml`. Captured the three runtime-vs-baked diffs (read-only) → all had **runtime-only content** (personal's `agent_architect` role + `ratatoskr-affect-full-allow` rule — live-bridged, ahead of repo). The guard's STOP-on-runtime-only rule earned its keep: a blind "sync to repo" would've deleted legitimate per-instance config.
|
||||
|
||||
**Operator ruling (2026-07-25 — the reframe).** Worldtree will run dozens-to-hundreds of instances at v1, each configured for its env. **Per-instance config deltas are the DESIGN, not rot; back-streaming to the canonical repo doesn't scale.** Everything stood down: demo model_roles normalize withdrawn, personal sync cancelled, post-mortem dissolved, `.bak` deleted. Guard demoted b132/`ad596b5` from ERROR alarm to INFO `CONFIG BASELINE (#376)` breadcrumb (WARNING only for a mounted file *entirely absent* = breakage-adjacent). Breaking-change protection stays in `core.config_validator`'s boot gate.
|
||||
|
||||
**infra-ops watcher — built then retired same day.** Wired an off-box `wt-drift-watch` (systemd --user timer on nh3-dev, alerts worldtree-dev on new `CONFIG DRIFT` startup lines) — then RETIRED it per the ruling (the error line is going away). Lesson banked in auto-memory `reference_worldtree_perinstance_config`.
|
||||
|
||||
**Governance notes worth keeping:**
|
||||
- The auto-mode guard **blocked** a peer-green-lit (worldtree-dev) config replace on the managed demo box because there was no *operator* consent for that specific change — correct: a config-mutation+restart on shared infra needs the operator's yes, not just a peer's. Surfaced it; the operator later stood the whole thing down. Good governance on both ends.
|
||||
- This arc is the direct evidence base for the [[2026-07-25-infra-ops-wt-config-repo]] decision (infra-ops owns config-as-code so live-edits stop being untracked).
|
||||
+18
-4
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-07-18_
|
||||
_Last updated: 2026-07-25_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -109,7 +109,7 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-07-18 (later session) — **soong-lab auto-redeploy WIRED + validated** (CI-deploy step live, deploys from /opt/soong-lab, health-gated; dispatch run #5 green — live container recreated onto a new image) and **worldtree-sdk 1.0.0 (Python) published to the vh Gitea PyPI** + acceptance-verified. Prior session's four landed items (Zonos 0.2.1 presets; soong-lab containerize cutover; Wyrd key; wtsdk npm@1.0.0) remain done. **No active in-flight task.**_
|
||||
_As of 2026-07-25 — **ACTIVE: build the infra-ops-owned Worldtree per-deployment config repo + deploy tooling** (operator-directed; the resume task for this snapshot). worldtree-dev owns the app/image; infra-ops owns config-as-code for every deployment and deploys it (write host bind-mount + `docker restart` same-image + health-gate — the proven-this-session procedure). Also secure the one-time boundary "yes" from worldtree-dev (repo is authoritative → they STOP live-editing `/opt/<instance>/config`, else two-writer drift returns). Seed from CURRENT mounted state (keep personal's live-bridged `agent_architect` + `ratatoskr-affect-full-allow` — ruled by-design). Full shape + caveats → `persistent-memory.d/2026-07-25-infra-ops-wt-config-repo.md` + the handoff. **Done this session:** The Booth shipped (v0.1.3, `services/booth/`, nh3-dev :8090, Homepage-linked, upload-pickup + image-viewer + copy-id); jackdaw-compose backend deployed (nh3-dev :8787) + its throwaway cloudflare tunnel torn down; ana-ml2 README refreshed to live GPU state; Worldtree #376 arc CLOSED (per-instance config ruled by-design; wyrd grant live; drift-watcher built+retired) → `persistent-memory.d/2026-07-25-wt-376-per-instance-config-arc.md`._
|
||||
|
||||
**Open follow-ups (non-blocking — pick one up or not):**
|
||||
- **Zonos emotion:** sad axes/text pass on the 3 calibrated voices (only named-sad, untested); emotion-congruent-text pass (validates intensity, may rescue sad id); clone-char (Emmie/Penny/Natalie/Miranda) emotion rows use the mid-region fallback until measured. Presets are **provisional** (neutral-text ear-check was inconclusive). Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio,assemble_voice}.py` (run ON irv-ml1; dial-in studio = nohup :8898 on nh3-dev). dvalin thread at rest (`01KXT12FN0AS…`). → `persistent-memory.d/2026-07-18-zonos-gateway-0.2.1-emotion-presets.md`
|
||||
@@ -118,14 +118,24 @@ _As of 2026-07-18 (later session) — **soong-lab auto-redeploy WIRED + validate
|
||||
|
||||
**Zonos voice stack (LIVE, unchanged):** 8 voices in `zonos-gateway` (`:8890` irv-ml1) — defaults AmericanFemale/Male/BritishFemale/Cora + 4 clones Emmie/Penny/Natalie/Miranda; add a voice = drop `<Name>.wav` in `/opt/docker/compose/zonos-gateway/voices/` + `docker compose restart` (host-managed bind-mount, NO rebuild). Clone pipeline: `/mnt/smithy/voice_clones/<name>.zip` → `assemble_voice.py` → drop. Dial-in studio http://10.100.10.50:8898/ (nohup on nh3-dev, relaunch `nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &`).
|
||||
|
||||
**althing monitor** ARMED (handle `infra-ops`, wake-listener; herald up). Re-arm after each fire (dies on /clear). Open watch: worldtree-dev (#368 forensics done; #363 research-wing ingest PARKED).
|
||||
**althing monitor** ARMED (handle `infra-ops`, wake-listener `bwraz1ai5`; herald up). ⚠️ Re-arm ONLY after an actual FIRE (`<task-notification> completed rc0`), never after a plain operator turn — re-arming while the prior listener is still live bounces rc3, and chaining the arm with `&`/`&&` orphans it (untracked → mail unwatched). Spawn `althing-wake-listener` as its OWN run_in_background task. Open watch: worldtree-dev (#376 closed; #363 research-wing ingest PARKED).
|
||||
|
||||
**eshpfi UNPUSHED** — this session's memory commits (`4bdf010`, `deae057`, + this snapshot) + the earlier `14a0004`..`438cd35` are committed but unpushed (push = operator's call). `stacks/heretic2-charrp-reasoning/` UNTRACKED; `graphify-out/GRAPH_REPORT.md` modified.
|
||||
**eshpfi UNPUSHED** — the Booth arc (`f4a5ba7`..`d813f15`), ana-ml2 refresh (`966324c`), nh3-dev inventory (`cd4d52e`) + this snapshot are committed but unpushed (push = operator's call). `stacks/heretic2-charrp-reasoning/` UNTRACKED; `graphify-out/GRAPH_REPORT.md` modified.
|
||||
|
||||
**PARKED (grok-code/Codex):** operator asked about fronting grok-code / Codex behind the LiteLLM gateway. Rec (given): raw models behind the gateway → **API keys** (native `xai/` + `openai/` providers, the GLM-passthrough pattern); fleet *consults* → the **Heid/Eitri peer-CLI** pattern (Codex already wired). Do NOT reverse-proxy the subscription CLIs (grok CLI / Codex CLI, OAuth-auth) into the gateway — ToS + account-ban risk + brittle. Untracked by operator choice; no decision made.
|
||||
|
||||
**Carried standing (non-blocking):** ana-ml2 GPU0 ~14 G reserve; irv-ml1 3090 oversubscription (kokoro :8193 + vibevoicefusion :9527 idle-pinned + zonos :1920 — operator declined to fix); rotate the 5 rest-server backup creds (operator, offline); Worldtree #363 research-wing ingest (parked, no deadline); T1 SFT LoRA dormant; Zonos2 engine still NATIVE (containerize deprioritized).
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-07-25]` **infra-ops to OWN a Worldtree per-deployment config repo + deploy tooling (operator-directed; IN-FLIGHT — build next session).** worldtree-dev owns app/image; infra-ops owns config-as-code per deployment + deploys it (write host bind-mount + `docker restart` same-image + health-gate). Build+deploy = infra-ops-only; the ONE exception is a one-time "repo is authoritative, STOP live-editing `/opt/<instance>/config`" agreement from worldtree-dev (else two-writer drift). Seed from current mounted state (keep by-design live-bridged deltas). → `persistent-memory.d/2026-07-25-infra-ops-wt-config-repo.md`
|
||||
|
||||
- `[2026-07-23→25]` **Worldtree #376 config-divergence arc CLOSED — per-instance config ruled BY DESIGN.** wyrd `session.history.write` demo grant was the one real bug (demo-intended grant not on demo; fixed via wholesale `policies.yaml` replace + restart). The b131 drift guard then surfaced broader divergence = legitimate live-bridged per-instance deltas; operator ruled deltas are the design not rot; guard demoted to INFO (b132); infra-ops drift-watcher built then retired same day. → `persistent-memory.d/2026-07-25-wt-376-per-instance-config-arc.md`, auto-memory `reference_worldtree_perinstance_config`
|
||||
|
||||
- `[2026-07-20→25]` **The Booth SHIPPED (v0.1.3) — ephemeral media drop board for CC sessions.** New fleet tool: user-systemd on nh3-dev :8090 (`services/booth/`, FastAPI+Jinja2, Corviduo "Australis" theme, 34 tests), Homepage-linked (Apps). Drop a folder in `~/booth-data/<name>` → browsable "booth" (auto-gallery of images/webm/audio, or a folder's own `index.html` verbatim), 24h TTL. Added across the session: browser/curl upload-for-pickup with human-readable ids (`4-wombat`), image viewer (Fit/1:1, conditional toggle), copy-id button (HTTP-LAN `execCommand` fallback). Registered in global CLAUDE.md tools. auto-memory `reference_booth_media_board`.
|
||||
|
||||
- `[2026-07-23]` **jackdaw-compose backend deployed as a persistent nh3-dev service (:8787).** Hosted for jackdaw-dev: thin stateless `bun server/index.ts` (from `~/development/jackdaw`) → LiteLLM `gen`, Origin-gated (INV-BK04/05), reached same-origin via their `:4500` bench's `/compose` proxy. `jackdaw-compose.service` (env/shared-key server-side, unit 0600, uncommitted). Also stood up + tore down a throwaway cloudflare quick-tunnel for their preview (`cloudflared` now installed at `~/bin`). In the nh3-dev README inventory (`cd4d52e`).
|
||||
|
||||
- `[2026-07-19]` **irv-ml1 ComfyUI — RTX VSR baked into canonical provisioning (comfy-dev ticket DONE).** RTXVideoSuperResolution node + `nvidia-vfx` dep were manual installs; documented both in the canonical `stacks/comfyui/README.md` runbook (this stack's provisioning IS the README — no automated provision script). Key durability insight: the **node** lives in `basedir/custom_nodes` (persistent, restic-included → durable) but the **`nvidia-vfx` wheel** lives in the venv under `run/` (disposable, restic-excluded → **dropped by any `rm -rf run/*` fresh-bootstrap**), so the pip step must re-run after every venv rebuild. Both steps run **as uid 1000** (root install → venv-ownership crash-loop, [[reference_irv_ml1_comfyui_mmartial]]); `--extra-index-url https://pypi.nvidia.com` kept **scoped to the nvidia-vfx install**, deliberately NOT a global compose `PIP_EXTRA_INDEX_URL` (would risk perturbing the pinned torch 2.12.1/SageAttention boot bootstrap). Node already live on the box; no host change, canonical runbook now replays it. comfy-dev informed.
|
||||
|
||||
- `[2026-07-19]` **vh private Gitea PyPI — consumer READ-access convention set + wyrd-dev provisioned.** Consuming agents read the internal vh PyPI (`https://gitea.phasefinal.com/api/packages/vh/pypi/simple/`) with a **shared read-only token** (operator call: shared, not per-consumer — read-only blast radius is small, per-agent Gitea identities aren't worth it). Minted a dedicated `read:package`-scoped PAT off **claude-bot** (`POST /users/claude-bot/tokens`, name `vh-pypi-read-consumers`; verified reads worldtree-sdk, write-probe 401), revocable/rotatable independently. uv auth = `UV_INDEX_GITEA_USERNAME=claude-bot` + `UV_INDEX_GITEA_PASSWORD=<token>` (or `~/.netrc`); pyproject uses `[[tool.uv.index]] name=gitea … explicit=true` + `[tool.uv.sources] <pkg> = { index = "gitea" }` (mirrors soong-lab's bifrost setup). Delivered to wyrd-dev (worldtree-sdk adoption) via mode-600 drop on nh3-dev, drop-and-shred. [[reference_claude_bot_gitea_creds]]
|
||||
@@ -232,6 +242,10 @@ _142 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-07-25]` **Chaining the althing wake-listener arm orphans it.** `reply && althing-wake-listener &` (or spawning `althing-wake-listener` with `&` *inside* a `run_in_background` task) → the `&`-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a *plain operator turn* (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn `althing-wake-listener` as its OWN `run_in_background` task, and re-arm ONLY after a real fire (`<task-notification> completed rc0`). Reclaim an orphan with `althing-cli stop-monitor` then re-arm.
|
||||
|
||||
- `[2026-07-25]` **Peer green-light ≠ operator consent for a managed-box mutation.** Auto-mode guard blocked a config-replace+restart on the Worldtree-team demo box that was authorized only by worldtree-dev's althing message — correctly: a persistent change to shared infra needs the *operator's* yes for that specific change, not a peer's. Surface it; don't route around the guard. (The operator then stood the whole change down — the guard's hold was the right call.)
|
||||
|
||||
- `[2026-07-18]` **Fleet Gitea CI foot-guns** (3 failed soong-lab builds): the pfi-fleet runner's `node:20-slim` job image has no docker/git so `actions/checkout` + `docker/*` marketplace actions all fail; `vh` is a USER so its packages are owner-write-only (claude-bot repo-admin-collab still 401s on push/publish, and can't set repo secrets — owner-only); `GITEA_`-prefixed secret names are reserved/illegal. Fixes in → `persistent-memory.d/2026-07-18-fleet-gitea-runner-build-recipe.md`
|
||||
|
||||
- `[2026-07-18]` **zonos-gateway local clone had NO git remote + a history unrelated to gitea's** — "committed to vh/zonos-gateway" was never pushed from that clone; two separate `git init` lineages, no merge-base. Reconcile = reset local→origin/main + overlay the changed files + push (NOT force — that erases gitea's voice-wav commits). Check `git remote -v` + `git merge-base` before assuming a clone is wired.
|
||||
|
||||
Reference in New Issue
Block a user