esh-pfi-infrastructure/servers/corviduo-dev/README.md

# corviduo-dev

Worldtree-team development VM. **PFI-hosted, Worldtree-team-managed
at the OS / application layer.** Runs the demo + personal + pinned
Worldtree deployments that vor, /vor-frame, and asset-engine talk
to over the LAN.

## Client context

- **Client / team:** Worldtree (architect's team is the upstream owner)
- **PFI role:** hosting + emergency-ops backstop. Ownership of OS
  configuration, application data, and deploy workflows lives with
  the Worldtree team. PFI provides networking, IP space, hypervisor
  capacity, and emergency-access pathways. Treat data-affecting work
  the same way we treat the SureFire `sf-*` hosts — coordinate via
  the architect over [althing](../../scripts/elway) before touching
  anything beyond a documented runbook.

## Network

- **LAN IP:** `10.250.50.152`
- **Hostname (OS):** `corviduo-dev`
- **FQDN:** not in DNS — reach by IP. Add to workstation `/etc/hosts`
  if you'll be SSH'ing frequently:
  ```
  10.250.50.152  corviduo-dev
  ```
- **SSH:** `vh@10.250.50.152` (this dir's `ssh-target` carries that)
- **Site:** Anaheim (PFI colo), `10.250.0.0/16` subnet

## Infrastructure

- **Hypervisor:** assumed `pfi-pve` (10.250.250.31). Not yet
  enumerated — verify via `qm list` on pfi-pve when you next have a
  reason to touch the hypervisor.
- **OS:** Debian (verified via `hostname` + standard tooling; not yet
  inspected with `server_inspect.sh`).

## Deployments

### Worldtree (Worldtree-team-managed, under `/opt/`)

Three `docker compose` projects under `/opt/`:

| Project dir | Container name | Role |
|---|---|---|
| `/opt/worldtree/` | `worldtree-worldtree-api-1` (port 8080) | Demo deployment — the "shared" Worldtree instance vor + asset-engine talk to. |
| `/opt/worldtree-personal/` | `worldtree-personal-worldtree-api-1` (port 8081) | Personal deployment — separate compose project, persistent state independent of demo. |
| `/opt/worldtree-pinned/` | `worldtree-pinned-worldtree-api-1` (port 8082) | Pinned deployment — intentionally lags behind `demo` for regression/comparison purposes. |

> **Deploys are CI/CD-only — NOT an infra-ops job.** All three
> deployments (demo, pinned, personal) are deployed exclusively by the
> Worldtree team's CI/CD pipeline. infra-ops does **not** deploy these
> manually. A peer "deploy Worldtree vX.Y.Z to <deployment>" request
> sent to infra-ops is **misrouted** — point them back to their
> pipeline; do not run the docker-as-root pattern to do it. The
> docker-as-root pattern below is for **emergency-ops only**
> (provider-key restore), not routine deploys.

Each project has:
- `compose.yaml` (image, env wiring, healthcheck, port mapping)
- `.env` (provider keys, ports, image tag — `deploy:deploy mode 600`)

The image is pulled from `gitea.phasefinal.com/vh/worldtree`, SHA-pinned
per deploy after Worldtree's CI workflow commit `8ef3801` (health-gated
`:latest` advance — `:latest` only moves on successful `/health` probes).

### PFI monitoring agents (infra-ops-managed, under `/home/vh/`)

Bootstrap 2026-05-24. Live at `/home/vh/docker/compose/{beszel,dozzle-agent}/`,
**not** under `/opt/docker/compose/` — corviduo-dev's `/opt/` is owned by
`deploy:deploy` (Worldtree team) and `vh` lacks passwordless sudo to
create the fleet-standard path. Functionally identical; documented here so
future infra-ops sessions find them.

| Project dir | Container | Role |
|---|---|---|
| `/home/vh/docker/compose/beszel/` | `beszel-agent` (host port 45876, `network_mode: host`) | Agent for the Beszel hub at ana-docker:8090. Hub SSH-polls inbound using the seeded ed25519 pubkey. |
| `/home/vh/docker/compose/dozzle-agent/` | `dozzle-agent` (host port 7007) | Agent for the Dozzle hub at ana-docker:8088. Hub connects inbound over mTLS (auto-generated certs on first run). |
| `/home/vh/docker/compose/dockge/` | `dockge` (host port 5001) | Per-host compose UI. Scoped to PFI-managed stacks under `/home/vh/docker/compose/` — does NOT see `/opt/worldtree*/` (worldtree-team boundary). `DOCKGE_HOST_STACKS_ROOT=/home/vh/docker` overrides the fleet-default `/opt/docker`. |

The empty `traefik-net` external docker network was created on this host as
a side effect of dozzle-agent's compose (which declares it external). Future
PFI services landing here can reuse it.

After bootstrap, the dozzle hub's `.env` on ana-docker had `10.250.50.152:7007`
appended to `DOZZLE_REMOTE_AGENT` and was force-recreated. The beszel hub
needs corviduo-dev added via the **"Add System"** UI action (host=10.250.50.152
port=45876) — pure-API equivalent uses pocketbase admin auth which we didn't
script. One-time operator step.

## Permissions model + the docker-as-root pattern

`vh` is the SSH user for ops. **`vh` does NOT have passwordless sudo.**
The `.env` files are `deploy:deploy mode 600` — vh cannot read or
write them through the normal POSIX path.

`vh` *is* in the `docker` group on this VM. That's effectively root
for any operation that can be expressed as "run a container with a
bind mount":

```bash
# Apply an edit to /opt/worldtree/.env (which vh can't read directly)
# without sudo. Run as vh.
ssh vh@10.250.50.152 'docker run --rm -i \
  -v /opt/worldtree:/wt \
  -v /var/run/docker.sock:/var/run/docker.sock \
  docker:cli sh -s' <<'SH'
sed -i '/^Z_AI_API_KEY=/d' /wt/.env  # idempotent strip
echo "Z_AI_API_KEY=<value>" >> /wt/.env
cd /wt && docker compose up -d --force-recreate --pull never
SH
```

**Security warning:** docker-group membership is effectively
unrestricted root via bind mounts. This is a well-known property of
Linux docker installs, not a misconfiguration. Treat the existence
of vh's docker-group membership as a privilege grant equivalent to
sudo, and don't expand it casually.

Notable gotchas:
- `docker compose up -d` from inside the docker:cli sandbox does NOT
  carry the host's registry credentials, so it will fail to pull
  authenticated images. Workaround: `--pull never` if the image is
  already cached on the host (Worldtree's deploy workflow always
  pre-pulls the SHA-tagged image before the SSH `compose up` step).
- The bare `compose.yaml` defaults `WORLDTREE_IMAGE` to
  `gitea.phasefinal.com/vh/worldtree:latest` if the env var is
  unset. Inside the docker:cli sandbox the env var doesn't carry,
  so emergency-ops `compose up` may pin to `:latest` rather than the
  intended SHA. Pass `-e WORLDTREE_IMAGE=...:<sha>` to the docker
  run invocation when this matters.

## Emergency runbook: provider-key restore

When Worldtree's startup validator fails because provider keys are
missing or empty (this happened 2026-05-12 to `Z_AI_API_KEY`):

1. Get the key values from the architect (via althing) — both
   `Z_AI_API_KEY` (LLM provider auth, `providers.zai.api_key`) and
   `ZAI_API_KEY` (MCP service key for web reads/searches). These
   are distinct keys.
2. Apply via the docker-as-root pattern above. Idempotent: strip
   any existing line for the same name before appending.
3. Force-recreate: `docker compose up -d --force-recreate --pull never`
   from the docker:cli sandbox.
4. Verify: `GET http://10.250.50.152:8080/health` returns 200; boot
   log shows "Application startup complete".
5. Treat the keys as compromised-by-disclosure if they passed over
   althing (multi-agent bus). Ask the architect to rotate; re-apply
   the rotated values via the same flow.

## Backup coverage

**Currently none — known gap.** This VM is not in PBS, not in
`backrest`, not in the fleet restic profiles. Failure modes:
- VM hardware loss → app rebuild from Worldtree's git repo +
  pipeline; data loss on `/data/state`, `/data/kb`, `/app/sessions`
  volumes.
- VM filesystem corruption → same.

Worth coordinating with the architect on whether the data volumes
(`worldtree-kb`, `worldtree-state`, `worldtree-sessions`) need
external backup, and whether PFI hosts that via the standard restic
pattern or whether the architect's team handles it via
application-level snapshotting.

## Refresh state

```bash
scripts/refresh-server-info.sh corviduo-dev
```

Reads via this dir's `ssh-target` (`vh@10.250.50.152`). The inspect
script is read-only — safe to run any time.

## History

- **2026-05-12 outage:** Worldtree image `98e4ca0` introduced a
  startup validator that fail-fasts on empty required provider keys
  (`Z_AI_API_KEY` referenced by `providers.zai`). Demo `.env` had
  the keys present-but-empty, container entered a restart loop.
  Architect shipped a validator-scoping fix-forward (`70703b0`) and
  the actual key values; infra applied the .env adds via the
  docker-as-root pattern documented above. Same outage exposed a
  `:latest` tag-drift bug that architect closed with commit
  `8ef3801` (health-gated `:latest` advance in the deploy workflow).
  Full thread: althing thread `01KRFZ349NSY8FEQDSM6TNEK7M`.

- **2026-05-25 config-as-bind-mount (Worldtree #205, v0.25.2):**
  `/app/config/` is now a bind-mounted host dir
  (`${WORLDTREE_CONFIG_DIR:-./config}:/app/config` in compose).
  Per-instance edits persist across image upgrades — no more
  in-container hacks for things like rate-limit defaults or
  per-agent profile overrides. Entrypoint shim runs as root,
  chowns + seeds the bind dir from `/app/config-defaults/` on
  first boot when empty, then drops privileges via `gosu` to the
  worldtree user. Idempotent against pre-existing populated dirs.
  Default `per_user_tokens_per_hour` raised image-side from 1M to
  1B at the same time. Shipped after a v0.25.1 attempt that wedged
  on a uid-mismatch in the cp+chown sequence (host bind dir was
  root-owned, container user uid 1000 couldn't write); v0.25.2
  fixed via root-then-drop pattern. Full thread:
  `01KSEBNJBVPZ29M7PD9FJECYQP`.