esh-pfi-infrastructure/servers/corviduo-dev

corviduo-dev

Worldtree-team development VM. PFI-hosted, Worldtree-team-managed at the OS / application layer. Runs the demo + personal + pinned Worldtree deployments that vor, /vor-frame, and asset-engine talk to over the LAN.

Client context

• Client / team: Worldtree (architect's team is the upstream owner)
• PFI role: hosting + emergency-ops backstop. Ownership of OS configuration, application data, and deploy workflows lives with the Worldtree team. PFI provides networking, IP space, hypervisor capacity, and emergency-access pathways. Treat data-affecting work the same way we treat the SureFire sf-* hosts — coordinate via the architect over althing before touching anything beyond a documented runbook.

Network

• LAN IP: 10.250.50.152
• Hostname (OS): corviduo-dev
• FQDN: not in DNS — reach by IP. Add to workstation /etc/hosts if you'll be SSH'ing frequently:

  10.250.50.152  corviduo-dev
  

• SSH: vh@10.250.50.152 (this dir's ssh-target carries that)
• Site: Anaheim (PFI colo), 10.250.0.0/16 subnet

Infrastructure

• Hypervisor: assumed pfi-pve (10.250.250.31). Not yet enumerated — verify via qm list on pfi-pve when you next have a reason to touch the hypervisor.
• OS: Debian (verified via hostname + standard tooling; not yet inspected with server_inspect.sh).

Deployments

Three docker compose projects under /opt/:

Project dir	Container name	Role
/opt/worldtree/	worldtree-worldtree-api-1 (port 8080)	Demo deployment — auto-deployed by Worldtree's CI on every push to main. The "shared" Worldtree instance vor + asset-engine talk to.
/opt/worldtree-personal/	worldtree-personal-worldtree-api-1 (port 8081)	Personal deployment — manually managed, separate compose project, persistent state independent of demo.
/opt/worldtree-pinned/	worldtree-pinned-worldtree-api-1 (port 8082)	Pinned deployment — intentionally lags behind demo for regression/comparison purposes.

Each project has:

• compose.yaml (image, env wiring, healthcheck, port mapping)
• .env (provider keys, ports, image tag — deploy:deploy mode 600)

The image is pulled from gitea.phasefinal.com/vh/worldtree, SHA-pinned per deploy after Worldtree's CI workflow commit 8ef3801 (health-gated :latest advance — :latest only moves on successful /health probes).

Permissions model + the docker-as-root pattern

vh is the SSH user for ops. vh does NOT have passwordless sudo. The .env files are deploy:deploy mode 600 — vh cannot read or write them through the normal POSIX path.

vh is in the docker group on this VM. That's effectively root for any operation that can be expressed as "run a container with a bind mount":

# Apply an edit to /opt/worldtree/.env (which vh can't read directly)
# without sudo. Run as vh.
ssh vh@10.250.50.152 'docker run --rm -i \
  -v /opt/worldtree:/wt \
  -v /var/run/docker.sock:/var/run/docker.sock \
  docker:cli sh -s' <<'SH'
sed -i '/^Z_AI_API_KEY=/d' /wt/.env  # idempotent strip
echo "Z_AI_API_KEY=<value>" >> /wt/.env
cd /wt && docker compose up -d --force-recreate --pull never
SH

Security warning: docker-group membership is effectively unrestricted root via bind mounts. This is a well-known property of Linux docker installs, not a misconfiguration. Treat the existence of vh's docker-group membership as a privilege grant equivalent to sudo, and don't expand it casually.

Notable gotchas:

• docker compose up -d from inside the docker:cli sandbox does NOT carry the host's registry credentials, so it will fail to pull authenticated images. Workaround: --pull never if the image is already cached on the host (Worldtree's deploy workflow always pre-pulls the SHA-tagged image before the SSH compose up step).
• The bare compose.yaml defaults WORLDTREE_IMAGE to gitea.phasefinal.com/vh/worldtree:latest if the env var is unset. Inside the docker:cli sandbox the env var doesn't carry, so emergency-ops compose up may pin to :latest rather than the intended SHA. Pass -e WORLDTREE_IMAGE=...:<sha> to the docker run invocation when this matters.

Emergency runbook: provider-key restore

When Worldtree's startup validator fails because provider keys are missing or empty (this happened 2026-05-12 to Z_AI_API_KEY):

1. Get the key values from the architect (via althing) — both Z_AI_API_KEY (LLM provider auth, providers.zai.api_key) and ZAI_API_KEY (MCP service key for web reads/searches). These are distinct keys.
2. Apply via the docker-as-root pattern above. Idempotent: strip any existing line for the same name before appending.
3. Force-recreate: docker compose up -d --force-recreate --pull never from the docker:cli sandbox.
4. Verify: GET http://10.250.50.152:8080/health returns 200; boot log shows "Application startup complete".
5. Treat the keys as compromised-by-disclosure if they passed over althing (multi-agent bus). Ask the architect to rotate; re-apply the rotated values via the same flow.

Backup coverage

Currently none — known gap. This VM is not in PBS, not in backrest, not in the fleet restic profiles. Failure modes:

• VM hardware loss → app rebuild from Worldtree's git repo + pipeline; data loss on /data/state, /data/kb, /app/sessions volumes.
• VM filesystem corruption → same.

Worth coordinating with the architect on whether the data volumes (worldtree-kb, worldtree-state, worldtree-sessions) need external backup, and whether PFI hosts that via the standard restic pattern or whether the architect's team handles it via application-level snapshotting.

Refresh state

scripts/refresh-server-info.sh corviduo-dev

Reads via this dir's ssh-target (vh@10.250.50.152). The inspect script is read-only — safe to run any time.

History

• 2026-05-12 outage: Worldtree image 98e4ca0 introduced a startup validator that fail-fasts on empty required provider keys (Z_AI_API_KEY referenced by providers.zai). Demo .env had the keys present-but-empty, container entered a restart loop. Architect shipped a validator-scoping fix-forward (70703b0) and the actual key values; infra applied the .env adds via the docker-as-root pattern documented above. Same outage exposed a :latest tag-drift bug that architect closed with commit 8ef3801 (health-gated :latest advance in the deploy workflow). Full thread: althing thread 01KRFZ349NSY8FEQDSM6TNEK7M.