esh-pfi-infrastructure/servers/corviduo-dev

corviduo-dev

Worldtree-team development VM. PFI-hosted, Worldtree-team-managed at the OS / application layer. Runs the demo + personal + pinned Worldtree deployments that vor, /vor-frame, and asset-engine talk to over the LAN.

Client context

• Client / team: Worldtree (architect's team is the upstream owner)
• PFI role: hosting + emergency-ops backstop. Ownership of OS configuration, application data, and deploy workflows lives with the Worldtree team. PFI provides networking, IP space, hypervisor capacity, and emergency-access pathways. Treat data-affecting work the same way we treat the SureFire sf-* hosts — coordinate via the architect over althing before touching anything beyond a documented runbook.

Network

• LAN IP: 10.250.50.152
• Hostname (OS): corviduo-dev
• FQDN: not in DNS — reach by IP. Add to workstation /etc/hosts if you'll be SSH'ing frequently:

  10.250.50.152  corviduo-dev
  

• SSH: vh@10.250.50.152 (this dir's ssh-target carries that)
• Site: Anaheim (PFI colo), 10.250.0.0/16 subnet

Infrastructure

• Hypervisor: assumed pfi-pve (10.250.250.31). Not yet enumerated — verify via qm list on pfi-pve when you next have a reason to touch the hypervisor.
• OS: Debian (verified via hostname + standard tooling; not yet inspected with server_inspect.sh).

Deployments

Worldtree (Worldtree-team-managed, under /opt/)

Three docker compose projects under /opt/:

Project dir	Container name	Role
/opt/worldtree/	worldtree-worldtree-api-1 (port 8080)	Demo deployment — the "shared" Worldtree instance vor + asset-engine talk to.
/opt/worldtree-personal/	worldtree-personal-worldtree-api-1 (port 8081)	Personal deployment — separate compose project, persistent state independent of demo.
/opt/worldtree-pinned/	worldtree-pinned-worldtree-api-1 (port 8082)	Pinned deployment — intentionally lags behind demo for regression/comparison purposes.

Deploys are CI/CD-only — NOT an infra-ops job. All three deployments (demo, pinned, personal) are deployed exclusively by the Worldtree team's CI/CD pipeline. infra-ops does not deploy these manually. A peer "deploy Worldtree vX.Y.Z to " request sent to infra-ops is misrouted — point them back to their pipeline; do not run the docker-as-root pattern to do it. The docker-as-root pattern below is for emergency-ops only (provider-key restore), not routine deploys.

Each project has:

• compose.yaml (image, env wiring, healthcheck, port mapping)
• .env (provider keys, ports, image tag — deploy:deploy mode 600)

The image is pulled from gitea.phasefinal.com/vh/worldtree, SHA-pinned per deploy after Worldtree's CI workflow commit 8ef3801 (health-gated :latest advance — :latest only moves on successful /health probes).

PFI monitoring agents (infra-ops-managed, under /home/vh/)

Bootstrap 2026-05-24. Live at /home/vh/docker/compose/{beszel,dozzle-agent}/, not under /opt/docker/compose/ — corviduo-dev's /opt/ is owned by deploy:deploy (Worldtree team) and vh lacks passwordless sudo to create the fleet-standard path. Functionally identical; documented here so future infra-ops sessions find them.

Project dir	Container	Role
/home/vh/docker/compose/beszel/	beszel-agent (host port 45876, network_mode: host)	Agent for the Beszel hub at ana-docker:8090. Hub SSH-polls inbound using the seeded ed25519 pubkey.
/home/vh/docker/compose/dozzle-agent/	dozzle-agent (host port 7007)	Agent for the Dozzle hub at ana-docker:8088. Hub connects inbound over mTLS (auto-generated certs on first run).
/home/vh/docker/compose/dockge/	dockge (host port 5001)	Per-host compose UI. Scoped to PFI-managed stacks under /home/vh/docker/compose/ — does NOT see /opt/worldtree*/ (worldtree-team boundary). DOCKGE_HOST_STACKS_ROOT=/home/vh/docker overrides the fleet-default /opt/docker.

The empty traefik-net external docker network was created on this host as a side effect of dozzle-agent's compose (which declares it external). Future PFI services landing here can reuse it.

After bootstrap, the dozzle hub's .env on ana-docker had 10.250.50.152:7007 appended to DOZZLE_REMOTE_AGENT and was force-recreated. The beszel hub needs corviduo-dev added via the "Add System" UI action (host=10.250.50.152 port=45876) — pure-API equivalent uses pocketbase admin auth which we didn't script. One-time operator step.

Permissions model + the docker-as-root pattern

vh is the SSH user for ops. vh does NOT have passwordless sudo. The .env files are deploy:deploy mode 600 — vh cannot read or write them through the normal POSIX path.

vh is in the docker group on this VM. That's effectively root for any operation that can be expressed as "run a container with a bind mount":

# Apply an edit to /opt/worldtree/.env (which vh can't read directly)
# without sudo. Run as vh.
ssh vh@10.250.50.152 'docker run --rm -i \
  -v /opt/worldtree:/wt \
  -v /var/run/docker.sock:/var/run/docker.sock \
  docker:cli sh -s' <<'SH'
sed -i '/^Z_AI_API_KEY=/d' /wt/.env  # idempotent strip
echo "Z_AI_API_KEY=<value>" >> /wt/.env
cd /wt && docker compose up -d --force-recreate --pull never
SH

Security warning: docker-group membership is effectively unrestricted root via bind mounts. This is a well-known property of Linux docker installs, not a misconfiguration. Treat the existence of vh's docker-group membership as a privilege grant equivalent to sudo, and don't expand it casually.

Notable gotchas:

• docker compose up -d from inside the docker:cli sandbox does NOT carry the host's registry credentials, so it will fail to pull authenticated images. Workaround: --pull never if the image is already cached on the host (Worldtree's deploy workflow always pre-pulls the SHA-tagged image before the SSH compose up step).
• The bare compose.yaml defaults WORLDTREE_IMAGE to gitea.phasefinal.com/vh/worldtree:latest if the env var is unset. Inside the docker:cli sandbox the env var doesn't carry, so emergency-ops compose up may pin to :latest rather than the intended SHA. Pass -e WORLDTREE_IMAGE=...:<sha> to the docker run invocation when this matters.

Emergency runbook: provider-key restore

When Worldtree's startup validator fails because provider keys are missing or empty (this happened 2026-05-12 to Z_AI_API_KEY):

1. Get the key values from the architect (via althing) — both Z_AI_API_KEY (LLM provider auth, providers.zai.api_key) and ZAI_API_KEY (MCP service key for web reads/searches). These are distinct keys.
2. Apply via the docker-as-root pattern above. Idempotent: strip any existing line for the same name before appending.
3. Force-recreate: docker compose up -d --force-recreate --pull never from the docker:cli sandbox.
4. Verify: GET http://10.250.50.152:8080/health returns 200; boot log shows "Application startup complete".
5. Treat the keys as compromised-by-disclosure if they passed over althing (multi-agent bus). Ask the architect to rotate; re-apply the rotated values via the same flow.

Backup coverage

Currently none — known gap. This VM is not in PBS, not in backrest, not in the fleet restic profiles. Failure modes:

• VM hardware loss → app rebuild from Worldtree's git repo + pipeline; data loss on /data/state, /data/kb, /app/sessions volumes.
• VM filesystem corruption → same.

Worth coordinating with the architect on whether the data volumes (worldtree-kb, worldtree-state, worldtree-sessions) need external backup, and whether PFI hosts that via the standard restic pattern or whether the architect's team handles it via application-level snapshotting.

Refresh state

scripts/refresh-server-info.sh corviduo-dev

Reads via this dir's ssh-target (vh@10.250.50.152). The inspect script is read-only — safe to run any time.

History

• 2026-05-12 outage: Worldtree image 98e4ca0 introduced a startup validator that fail-fasts on empty required provider keys (Z_AI_API_KEY referenced by providers.zai). Demo .env had the keys present-but-empty, container entered a restart loop. Architect shipped a validator-scoping fix-forward (70703b0) and the actual key values; infra applied the .env adds via the docker-as-root pattern documented above. Same outage exposed a :latest tag-drift bug that architect closed with commit 8ef3801 (health-gated :latest advance in the deploy workflow). Full thread: althing thread 01KRFZ349NSY8FEQDSM6TNEK7M.

• 2026-05-25 config-as-bind-mount (Worldtree #205, v0.25.2): /app/config/ is now a bind-mounted host dir (${WORLDTREE_CONFIG_DIR:-./config}:/app/config in compose). Per-instance edits persist across image upgrades — no more in-container hacks for things like rate-limit defaults or per-agent profile overrides. Entrypoint shim runs as root, chowns + seeds the bind dir from /app/config-defaults/ on first boot when empty, then drops privileges via gosu to the worldtree user. Idempotent against pre-existing populated dirs. Default per_user_tokens_per_hour raised image-side from 1M to 1B at the same time. Shipped after a v0.25.1 attempt that wedged on a uid-mismatch in the cp+chown sequence (host bind dir was root-owned, container user uid 1000 couldn't write); v0.25.2 fixed via root-then-drop pattern. Full thread: 01KSEBNJBVPZ29M7PD9FJECYQP.