servers/corviduo-dev: add Worldtree-team dev VM to inventory

Captures the access path + emergency-ops runbook surfaced during the
2026-05-12 demo outage (Z_AI_API_KEY / runtime validator / :latest tag
drift cascade).

- servers/corviduo-dev/{README.md,ssh-target,system-details.txt}
- CLAUDE.md servers table + placement-rules note

The "docker-as-root sudo bypass" pattern (vh's docker-group membership
→ effective root for any bind-mount-able operation) is the canonical
fix path when /opt/worldtree*/.env edits are needed and the deploy
user's sudo is out of reach. The README documents the pattern with
the appropriate "docker-group ≈ sudo" security warning so future
emergency-ops sessions don't have to re-derive it.

Ownership stance matches SF client hosts: PFI hosts + provides
emergency-ops backstop, Worldtree-team owns OS config / deploys /
backup decisions. Coordinate data-affecting work with the architect.

CLAUDE.md

@@ -2,6 +2,14 @@
 
 This workspace is for managing PFI infrastructure — servers, Docker stacks, and related configs. Spawn a dedicated Claude Code session here when working on infra so it doesn't clutter AIPA-MCP development context.
 
+## Persistent memory
+
+`persistent-memory.md` at the repo root captures durable intent and
+supporting evidence (goals, decisions, foot-gun warnings, in-flight
+state) across context resets. Read it at session start; treat it as
+one input alongside this CLAUDE.md and the auto-memory system, not
+as the single source of truth.
+
 **New session starting here?** Read [`docs/orientation.md`](docs/orientation.md) first — fleet topology, backup architecture, governing principles, and all the NFS/DSM/naming gotchas that have cost past sessions time.
 
 **For SSH-driven work: use `scripts/elway`.** Write a playbook under
@@ -94,6 +102,7 @@ Observed and standardized across servers:
 | sfsrv-ana | 10.250.250.115 | Anaheim | **SureFire client (PFI-managed)** — Proxmox VE hypervisor | `servers/sfsrv-ana/README.md` |
 | sf-ana-container | 10.250.150.100 | Anaheim | **SureFire client (PFI-managed)** — container workload on sfsrv-ana | `servers/sf-ana-container/README.md` |
 | sf-r630 | iDRAC 10.250.250.110 | Anaheim | **SureFire client (PFI-managed)** — physical Dell R630, iDRAC-managed from PFI side | `servers/sf-r630/README.md` |
+| corviduo-dev | 10.250.50.152 | Anaheim | **Worldtree-team dev VM (PFI-hosted)** — runs the demo + personal + pinned Worldtree deployments vor/asset-engine talk to | `servers/corviduo-dev/README.md` |
 | nh3-docker | 10.100.50.40 | NH3 (`10.100.0.0/16`) | General-purpose Docker host (non-GPU VM on nh3-pve) | `servers/nh3-docker/README.md` |
 | nh3-pve | 10.100.250.60 | NH3 | Proxmox VE hypervisor | `servers/nh3-pve/README.md` |
 | nh3-nas | 10.100.50.50 | NH3 | Synology RS2418+ — NFS exports, rest-server-nh3, PBS-NH3 datastore backend | `servers/nh3-nas/README.md` |
@@ -113,6 +122,7 @@ Observed and standardized across servers:
 - ESH home-lab workloads (`esteban.net`) → `esh-docker-vm` (general) or `vm-esh-nas` (needs direct NFS mounts from 10.0.50.50). Not part of the PFI colo topology, but shares monitoring/backup tooling.
 - Cross-site services (e.g. Beszel hub, Dozzle hub) live on `ana-docker` and pull from agents on the other hosts.
 - **SureFire (SF) client hosts** (`sf-*`, `sfsrv-ana`) are PFI-managed under the hosting agreement — SSH, OS ops, backups are PFI's responsibility. Hardware and data belong to the client; coordinate anything that affects data with them.
+- **Worldtree-team dev VM** (`corviduo-dev`) is PFI-hosted (Anaheim subnet) but Worldtree-team-managed at the OS / application layer. PFI handles networking + emergency-ops backstop; OS configuration + deploy workflows + backup decisions live with the architect's team. Treat data-affecting work like SF hosts — coordinate before touching.
 - **Hypervisors** (`pfi-pve`, `nh3-pve`, `esh-pve`, `esh-pve-nas`) are tracked for inventory / capacity planning. Don't deploy Docker stacks directly on them; new workloads land as VMs. `server_inspect.sh` captures host-level detail only — VM/LXC/ZFS enumeration needs Proxmox-native tooling (`qm list`, `pvesh get …`, `zpool list`).
 
 ## How to refresh a server's state

servers/corviduo-dev/README.md

@@ -0,0 +1,153 @@
+# corviduo-dev
+
+Worldtree-team development VM. **PFI-hosted, Worldtree-team-managed
+at the OS / application layer.** Runs the demo + personal + pinned
+Worldtree deployments that vor, /vor-frame, and asset-engine talk
+to over the LAN.
+
+## Client context
+
+- **Client / team:** Worldtree (architect's team is the upstream owner)
+- **PFI role:** hosting + emergency-ops backstop. Ownership of OS
+  configuration, application data, and deploy workflows lives with
+  the Worldtree team. PFI provides networking, IP space, hypervisor
+  capacity, and emergency-access pathways. Treat data-affecting work
+  the same way we treat the SureFire `sf-*` hosts — coordinate via
+  the architect over [althing](../../scripts/elway) before touching
+  anything beyond a documented runbook.
+
+## Network
+
+- **LAN IP:** `10.250.50.152`
+- **Hostname (OS):** `corviduo-dev`
+- **FQDN:** not in DNS — reach by IP. Add to workstation `/etc/hosts`
+  if you'll be SSH'ing frequently:
+  ```
+  10.250.50.152  corviduo-dev
+  ```
+- **SSH:** `vh@10.250.50.152` (this dir's `ssh-target` carries that)
+- **Site:** Anaheim (PFI colo), `10.250.0.0/16` subnet
+
+## Infrastructure
+
+- **Hypervisor:** assumed `pfi-pve` (10.250.250.31). Not yet
+  enumerated — verify via `qm list` on pfi-pve when you next have a
+  reason to touch the hypervisor.
+- **OS:** Debian (verified via `hostname` + standard tooling; not yet
+  inspected with `server_inspect.sh`).
+
+## Deployments
+
+Three `docker compose` projects under `/opt/`:
+
+| Project dir | Container name | Role |
+|---|---|---|
+| `/opt/worldtree/` | `worldtree-worldtree-api-1` (port 8080) | Demo deployment — auto-deployed by Worldtree's CI on every push to `main`. The "shared" Worldtree instance vor + asset-engine talk to. |
+| `/opt/worldtree-personal/` | `worldtree-personal-worldtree-api-1` (port 8081) | Personal deployment — manually managed, separate compose project, persistent state independent of demo. |
+| `/opt/worldtree-pinned/` | `worldtree-pinned-worldtree-api-1` (port 8082) | Pinned deployment — intentionally lags behind `demo` for regression/comparison purposes. |
+
+Each project has:
+- `compose.yaml` (image, env wiring, healthcheck, port mapping)
+- `.env` (provider keys, ports, image tag — `deploy:deploy mode 600`)
+
+The image is pulled from `gitea.phasefinal.com/vh/worldtree`, SHA-pinned
+per deploy after Worldtree's CI workflow commit `8ef3801` (health-gated
+`:latest` advance — `:latest` only moves on successful `/health` probes).
+
+## Permissions model + the docker-as-root pattern
+
+`vh` is the SSH user for ops. **`vh` does NOT have passwordless sudo.**
+The `.env` files are `deploy:deploy mode 600` — vh cannot read or
+write them through the normal POSIX path.
+
+`vh` *is* in the `docker` group on this VM. That's effectively root
+for any operation that can be expressed as "run a container with a
+bind mount":
+
+```bash
+# Apply an edit to /opt/worldtree/.env (which vh can't read directly)
+# without sudo. Run as vh.
+ssh vh@10.250.50.152 'docker run --rm -i \
+  -v /opt/worldtree:/wt \
+  -v /var/run/docker.sock:/var/run/docker.sock \
+  docker:cli sh -s' <<'SH'
+sed -i '/^Z_AI_API_KEY=/d' /wt/.env  # idempotent strip
+echo "Z_AI_API_KEY=<value>" >> /wt/.env
+cd /wt && docker compose up -d --force-recreate --pull never
+SH
+```
+
+**Security warning:** docker-group membership is effectively
+unrestricted root via bind mounts. This is a well-known property of
+Linux docker installs, not a misconfiguration. Treat the existence
+of vh's docker-group membership as a privilege grant equivalent to
+sudo, and don't expand it casually.
+
+Notable gotchas:
+- `docker compose up -d` from inside the docker:cli sandbox does NOT
+  carry the host's registry credentials, so it will fail to pull
+  authenticated images. Workaround: `--pull never` if the image is
+  already cached on the host (Worldtree's deploy workflow always
+  pre-pulls the SHA-tagged image before the SSH `compose up` step).
+- The bare `compose.yaml` defaults `WORLDTREE_IMAGE` to
+  `gitea.phasefinal.com/vh/worldtree:latest` if the env var is
+  unset. Inside the docker:cli sandbox the env var doesn't carry,
+  so emergency-ops `compose up` may pin to `:latest` rather than the
+  intended SHA. Pass `-e WORLDTREE_IMAGE=...:<sha>` to the docker
+  run invocation when this matters.
+
+## Emergency runbook: provider-key restore
+
+When Worldtree's startup validator fails because provider keys are
+missing or empty (this happened 2026-05-12 to `Z_AI_API_KEY`):
+
+1. Get the key values from the architect (via althing) — both
+   `Z_AI_API_KEY` (LLM provider auth, `providers.zai.api_key`) and
+   `ZAI_API_KEY` (MCP service key for web reads/searches). These
+   are distinct keys.
+2. Apply via the docker-as-root pattern above. Idempotent: strip
+   any existing line for the same name before appending.
+3. Force-recreate: `docker compose up -d --force-recreate --pull never`
+   from the docker:cli sandbox.
+4. Verify: `GET http://10.250.50.152:8080/health` returns 200; boot
+   log shows "Application startup complete".
+5. Treat the keys as compromised-by-disclosure if they passed over
+   althing (multi-agent bus). Ask the architect to rotate; re-apply
+   the rotated values via the same flow.
+
+## Backup coverage
+
+**Currently none — known gap.** This VM is not in PBS, not in
+`backrest`, not in the fleet restic profiles. Failure modes:
+- VM hardware loss → app rebuild from Worldtree's git repo +
+  pipeline; data loss on `/data/state`, `/data/kb`, `/app/sessions`
+  volumes.
+- VM filesystem corruption → same.
+
+Worth coordinating with the architect on whether the data volumes
+(`worldtree-kb`, `worldtree-state`, `worldtree-sessions`) need
+external backup, and whether PFI hosts that via the standard restic
+pattern or whether the architect's team handles it via
+application-level snapshotting.
+
+## Refresh state
+
+```bash
+scripts/refresh-server-info.sh corviduo-dev
+```
+
+Reads via this dir's `ssh-target` (`vh@10.250.50.152`). The inspect
+script is read-only — safe to run any time.
+
+## History
+
+- **2026-05-12 outage:** Worldtree image `98e4ca0` introduced a
+  startup validator that fail-fasts on empty required provider keys
+  (`Z_AI_API_KEY` referenced by `providers.zai`). Demo `.env` had
+  the keys present-but-empty, container entered a restart loop.
+  Architect shipped a validator-scoping fix-forward (`70703b0`) and
+  the actual key values; infra applied the .env adds via the
+  docker-as-root pattern documented above. Same outage exposed a
+  `:latest` tag-drift bug that architect closed with commit
+  `8ef3801` (health-gated `:latest` advance in the deploy workflow).
+  Full thread: althing thread `01KRFZ349NSY8FEQDSM6TNEK7M`.

servers/corviduo-dev/ssh-target

@@ -0,0 +1 @@
+vh@10.250.50.152