memory: snapshot for /clear — auto-archived 16 entries (Recent decisions 13, Tried and abandoned 3); index 303 -> 254
This commit is contained in:
@@ -1 +0,0 @@
|
||||
- `[2026-07-15]` **arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake.** Operator-directed full switch (comfy-dev executed, live in prod). Established: gen (`qwen3.6-35b-a3b-heretic`) is vision-enabled and was image-bench's predecessor as arbo's hero-judge; image-judge actually serves 4 roles (vision quality-scoring + identity-scoring + bbox grounding + an uncensored text tier), not just grounding. comfy-dev spot-check: gen faster on every task, grounding within ~3px, uncensoring preserved, and it FIXED a bug (image-judge's reasoning preamble broke json_object + stalled the router). Sequencing = short prod bake then evict (~30 GB GPU1 reclaim); revert = flip `ARBO_VISION_MODEL`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`.
|
||||
@@ -1,83 +0,0 @@
|
||||
- `[2026-07-18]` **soong-lab auto-redeploy — DONE + VALIDATED** (was approved/queued; executed same day on fresh context — see AS-BUILT at the bottom).
|
||||
|
||||
Vuong approved wiring auto-redeploy for soong-lab (relayed via soong-dev, thread
|
||||
`01KXT3A6C3908TA4V9THV3AMH7`): new images should go live on corviduo-dev without
|
||||
the manual `docker compose pull && up -d`. Host-side implementation is infra-ops's
|
||||
lane; mechanism is infra-ops's call per fleet conventions. Operator deferred
|
||||
execution — "we'll do soong on fresh context."
|
||||
|
||||
**Chosen mechanism (recommended, agrees with soong-dev): Worldtree-style
|
||||
CI-deploy step** — NOT watchtower polling.
|
||||
- Add a deploy job/step to soong-lab's `.gitea/workflows/build-and-push.yml` that,
|
||||
after the build+push job succeeds, **SSHes from the pfi-fleet runner to
|
||||
corviduo-dev** and runs `cd /home/infra-ops/soong-lab-deploy && docker compose
|
||||
pull && docker compose up -d`, then a **health-gate** (`curl -fsS
|
||||
http://localhost:8443/api/version`).
|
||||
- This is exactly how WT deploys the demo instance to the SAME host: see
|
||||
`~/development/Worldtree/.gitea/workflows/deploy.yml` — the "Deploy to demo VM +
|
||||
health-gate" step uses `secrets.DEMO_VM_SSH_KEY` / `DEMO_VM_HOST` / `DEMO_VM_USER`.
|
||||
Explicit-over-implicit (visible in the run log, fires exactly on build success),
|
||||
one less always-on service than watchtower.
|
||||
|
||||
**Constraints (from soong-dev):** deploy on CI success only; keep the trigger
|
||||
gated to `v*` tags + `workflow_dispatch` (as today); preserve the one-command
|
||||
rollback posture (`docker compose down` / pin a previous tag).
|
||||
|
||||
**BLOCKER — needs from vh (owner-only):** a **runner→corviduo-dev deploy SSH key**
|
||||
as a repo secret (+ host/user), same class as WT's `DEMO_VM_SSH_KEY`. Likely
|
||||
**reuse WT's existing demo-deploy key** (WT's runner already SSHes to 10.250.50.152
|
||||
as its deploy user). Repo secrets are vh-owner-only (see
|
||||
[[2026-07-18-fleet-gitea-runner-build-recipe]]).
|
||||
|
||||
**Next-session steps:** (1) confirm/obtain the deploy SSH-key secret from vh (reuse
|
||||
WT's or mint fresh); (2) add the deploy job to build-and-push.yml (infra-ops has
|
||||
push on vh/soong-lab); (3) dispatch a build to verify it deploys + health-gates;
|
||||
(4) ping soong-dev so they sync DEPLOY.md's "open follow-up" note to the as-built
|
||||
mechanism. Auto-pull (watchtower) explicitly NOT chosen. See
|
||||
[[2026-07-18-soong-lab-containerize-cutover]].
|
||||
|
||||
## AS-BUILT (2026-07-18, same-day execution)
|
||||
|
||||
**Mechanism landed** exactly as planned: `build-and-push.yml` gained a `Deploy to
|
||||
corviduo-dev + health-gate` step (after build+push) that SSHes the host as `deploy`
|
||||
and runs `docker compose pull && up -d` from `/opt/soong-lab`, then polls
|
||||
`http://localhost:8443/api/version` for 120s and fails the job loud if unhealthy. No
|
||||
compose is shipped from CI (the in-repo `docker-compose.yml` is a BUILD compose; the
|
||||
host pull-compose is infra-ops-managed). Kept the `v*`-tag/`workflow_dispatch` trigger.
|
||||
Skipped WT's disk-watermark gate + health-gated-`:latest`-advance (low cadence, easy
|
||||
rollback).
|
||||
|
||||
**Deploy identity = reuse WT's `deploy` account** (operator accepted the rec):
|
||||
- `deploy` (uid 1001, docker-group → no sudo) already owns `/opt/worldtree`; relocated
|
||||
soong-lab's deploy dir `/home/infra-ops/soong-lab-deploy` → **`/opt/soong-lab`**
|
||||
(deploy-owned), copied compose + `.env`. Named volumes (`soong-lab_soong-library`,
|
||||
`soong-lab_soong-portraits`) are project-scoped by compose `name: soong-lab` → followed
|
||||
the move untouched (dry-run `up -d` ADOPTED the running container, no recreate). Old dir
|
||||
**retired → `.retired-20260718`** (recoverable). Also lingering: `soong-lab-deploy.sh` /
|
||||
`.log` (dead pre-container webhook artifacts) — harmless, left in place.
|
||||
- **Dedicated soong-only ed25519 deploy key** minted (NOT literally WT's key — cleaner
|
||||
independent revocation), pubkey appended to `deploy`'s `authorized_keys`
|
||||
(fp `SHA256:MG7M3RiZJ176sLfblffb96V6W1qkRTgJ5dow1CpiY68`). Existing `deploy` key is
|
||||
plain/unrestricted, so parity held.
|
||||
|
||||
**The secret gate (the friction point):** repo Actions secrets are **vh-owner-only** —
|
||||
claude-bot's token is `write:package,read:repository` (403 on secret-write), and the vh
|
||||
package-scoped PAT also 403'd on `PUT …/actions/secrets/…`. So `DEPLOY_SSH_KEY` /
|
||||
`DEPLOY_HOST` (10.250.50.152) / `DEPLOY_USER` (deploy) HAD to be set by the operator.
|
||||
First operator attempt produced a **bad key paste** — the deploy step died with
|
||||
`Load key … error in libcrypto` + `Permission denied (publickey)` (build+push were green;
|
||||
live Soong never moved). Fix: operator re-set the secret; the minted key path was
|
||||
pre-validated from nh3-dev (`ssh -i … deploy@… 'cd /opt/soong-lab && docker compose config
|
||||
-q'` → OK, health 200) so the re-set was the only variable.
|
||||
|
||||
**Validation:** `workflow_dispatch` via claude-bot **basic auth** (its token lacks
|
||||
`write:repository` for the dispatch API; the account password works). Run #5 (task 1886)
|
||||
GREEN — live container recreated `sha256:…541f7730` → `…07526a08`, `StartedAt` fresh,
|
||||
health 200. `/api/version` now reports **0.3.25** (run #5 shipped soong-dev's 1c2f831
|
||||
STYLE_WORKFLOWS re-pin as validation cargo). soong-dev synced `docs/DEPLOY.md`
|
||||
(commit `00b67c3`). NB: tag **v0.3.25 exists only locally** — pushing it would re-trigger
|
||||
a redundant build+deploy of the same commit (operator's discretion).
|
||||
|
||||
**Ops now:** redeploy = tag `v*` or `workflow_dispatch` the CI (auto). Manual fallback =
|
||||
`sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`
|
||||
(the `.env` is `deploy`-owned 600, so infra-ops needs `sudo -u deploy`, not a bare `cd`).
|
||||
@@ -1,32 +0,0 @@
|
||||
`[2026-07-25]` **infra-ops Worldtree config-as-code repo — SHIPPED + boundary AGREED.**
|
||||
|
||||
**STATUS (2026-07-25, done this session):** `vh/worldtree-instance-configs` (private, gitea) built, pushed, validated; boundary agreement secured from worldtree-dev.
|
||||
|
||||
- **Repo:** dir-per-instance `demo/` + `personal/` (5 files each: `defaults.yaml`, `policies.yaml`, `model_roles.yaml`, `providers.yaml`, `matrix.yaml`), seeded byte-exact from live `/opt/<instance>/config`. `pinned/` = README stub only — **no `/app/config` bind-mount; config baked into frozen image `446e5807` (2026-05-13)**, so out-of-scope; deploy verb refuses it.
|
||||
- **Tool:** `scripts/deploy-wt-config <verb> <instance>` — `diff` (read-only repo-vs-host), `deploy` (in-run host backup → `install -o vh -g vh -m 644` → restart **api+matrix** → health-gate api `/health` → auto-rollback), `capture` (host→repo reconcile). Instance table in-script (demo→`/opt/worldtree/config`+`worldtree-worldtree-{api,matrix}-1`; personal→`/opt/worldtree-personal/config`+`worldtree-personal-worldtree-{api,matrix}-1`). Matrix sidecar shares the config mount but has no healthcheck → restart both, gate on api. Env `WT_CONFIG_HOST` (default `infra-ops@10.250.50.152`), `WT_HEALTH_WAIT` (90s). Local clone `~/development/worldtree-instance-configs`.
|
||||
- **Gitea plumbing (reusable):** nh3-dev **403s the gitea HTTP API** (public fail2ban + internal `:3000` both 403). Repo CREATE went via **ana-docker localhost API** (`ssh infra-ops@10.250.50.70` → `curl localhost:3000/api/v1/user/repos`, vh token from `~/.config/tea/config.yml`, operator-authorized one-time). PUSH went over **internal git-SSH `ssh://git@10.250.50.70:222`** (works from nh3-dev; auths as vh). `git init` defaulted to `master` → renamed `main` to match repo default_branch.
|
||||
- **Boundary AGREED (worldtree-dev, althing thread `01KYCAECRWVEF16EVKQAGT2N80`):** no hand-edits to `/opt/<instance>/config`; config changes route to infra-ops as deltas (worldtree-dev owns CONTENT + approval trail — the wyrd-grant shape — infra-ops lands+deploys). **Three-layer model:** image `config/` = baseline new instances seed from (theirs) → `vh/worldtree-instance-configs` = per-instance truth (ours) → host bind-mount = deploy target (written only by the tool). **Carve-out:** worldtree-dev's admin-API ops (`/admin/keys` mint, tier changes, session retirement, future runtime-grant surfaces) mutate instance **DATABASES not config files** → NOT config edits, stay in-band. If a future API writes config *files*, they flag at design time. b132 CONFIG BASELINE breadcrumb composes (INFO line = config-as-code diverges from image baseline, by design).
|
||||
- **No live deploy** done or needed — repo seeded == live (diff clean, capture round-trips zero-diff). Deploy path is dry-run-validated only; first real deploy needs operator per-change yes (managed box).
|
||||
|
||||
---
|
||||
|
||||
_Original plan (2026-07-25, pre-build):_
|
||||
|
||||
`[2026-07-25]` **infra-ops to OWN a Worldtree per-deployment config repo + deploy tooling (operator-directed).**
|
||||
|
||||
**Decision.** Vuong directed (2026-07-25, this session) that Worldtree instance config should be a *tracked change*, **managed and deployed by infra-ops — not worldtree-dev**. Model: worldtree-dev owns the app/image (+ the baked baseline defaults); **infra-ops owns config-as-code for every deployment** and deploys it. This is the durable fix for the root cause behind the whole #376 arc — config was edited live on host bind-mounts (`/opt/<instance>/config/`) with zero version history, audit, or recovery.
|
||||
|
||||
**What "no worldtree-dev involvement" does and does NOT cover** (clarified with the operator this session):
|
||||
- **Build + deploy = infra-ops-only.** Deploying config = write the host bind-mount file + restart the container (the *exact* procedure already run this session — backup → replace → restart → health-gate → rollback-on-unhealthy). No worldtree-dev in the deploy loop. Their CI only swaps the IMAGE; it does NOT resync the host config bind-mount (confirmed #376 finding).
|
||||
- **ONE load-bearing exception — a one-time boundary agreement, NOT per-deploy involvement:** for the repo to *own* config it must be the **only writer**. worldtree-dev "live-bridges" (hand-edits mounted config directly on the box). If the repo deploys config *and* they keep live-editing → **two writers fighting the same files** = #376 all over again. So secure a one-time "yes" from worldtree-dev: *the config repo is now authoritative; stop hand-editing `/opt/<instance>/config`; route config changes through the repo.* (Five-minute agreement, not a design collab.)
|
||||
- **Standing coupling (not "involvement"):** the config *schema* is the app's, enforced by its boot validator (`core.config_validator`). infra-ops configs must stay schema-compatible with the deployed image; the boot gate is the loud backstop.
|
||||
|
||||
**Build shape (recommended):**
|
||||
- Gitea repo `worldtree-instance-configs` (infra-ops-owned), **dir per instance** (`demo/`, `personal/`, `pinned/` — the three on corviduo-dev 10.250.50.152: demo `worldtree-worldtree-api-1` :8080, personal `worldtree-personal-worldtree-api-1` :8081, pinned `worldtree-pinned-worldtree-api-1` :8082). Config dirs: demo `/opt/worldtree/config`, personal `/opt/worldtree-personal/config`, pinned `/opt/worldtree-pinned/config` (verify pinned's mount).
|
||||
- **SEED FROM CURRENT MOUNTED STATE, don't author fresh** — capture each instance's live config (incl. legitimate live-bridged deltas: personal carries `agent_architect` role [Soong/soong-lab] in model_roles.yaml + `ratatoskr-affect-full-allow` in policies.yaml that are NOT in the app repo — the operator ruled these are BY DESIGN, keep them). Losing them = breakage (the affect-render one gates mood rendering).
|
||||
- Deploy script (e.g. `scripts/deploy-wt-config <instance>`): git = source of truth → push to host bind-mount + `docker restart` (same pinned image, no pull) + health-gate + auto-rollback. This is the proven-this-session procedure, scripted.
|
||||
- Files per instance: `policies.yaml`, `model_roles.yaml` (+ whatever else is bind-mounted — `defaults.yaml`, `providers.yaml`, `matrix.yaml` all live in `/opt/<instance>/config`; decide scope — policies+model_roles are the authz/role layer, defaults/providers are heavier instance tunables).
|
||||
|
||||
**Tracking surface:** operator-directed 2026-07-25, carried by this snapshot + `/tmp/infra-ops-handoff.md`. No issue filed (infra-ops-internal build). Related fleet idiom to reuse: canonical-sync (`.corviduo-canonicals.toml` / `canonical_sync.py`). Later scale option (deferred, needs worldtree-dev): base+overlay with a merge step in their pipeline.
|
||||
|
||||
See [[2026-07-25-wt-376-per-instance-config-arc]] for the incident that produced this. Auto-memory: `reference_worldtree_perinstance_config`, `reference_corviduo_dev_emergency_ops`.
|
||||
Reference in New Issue
Block a user