esh-pfi-infrastructure/services/secrets-broker/secrets-broker.contract.md

title, kind, status, owner, created, depends_on

title	kind	status	owner	created	depends_on
secrets-broker	module-contract	draft	infra-ops	2026-08-11	Vaultwarden (vaultwarden.phasefinal.com, on ana-docker; DB on pfi-postgres, in the pg_dump backup set) rbw (Rust Bitwarden CLI; unattended unlock daemon)

secrets-broker — fleet-wide credential registry over Vaultwarden

Purpose

A single place to store and look up any fleet secret — env files, API tokens, TLS certs/keys, SSH keys, DB creds, WireGuard keys — that should not live in a git repo and is today single-copy on a host. Vaultwarden is the durable central store; the broker is the thin programmatic surface (secret put|get|list|backfill) that agents, scripts, and the operator use. Solves two problems at once: durability (secrets currently backed up nowhere) and lookup (no canonical place to find a credential).

This is a central store, not (yet) a deploy-time source of truth: host .env files stay in place; the vault is the authoritative registry + backup. The pull-at-deploy evolution is explicitly out of scope for v1 (see Out of scope).

Architecture

• Store: a dedicated Vaultwarden service account (secrets-broker@…), its personal vault is the registry. (An Organization + per-host Collections is the textbook multi-user structure, but rbw's write path into org collections is weak — see the rbw capability gate — so v1 uses the service account's own vault with a folder/name taxonomy. Org migration is a future option if granular human ACLs are ever needed.)
• Client: bw (official Bitwarden CLI, installed to ~/.local on nh3-dev). Switched from rbw after rbw's register returned an undebuggable 400 against this Vaultwarden despite valid creds (a direct client_credentials grant + both prelogin paths return 200; rbw emits no HTTP logs). bw gives a clean unattended flow (login --apikey via BW_CLIENTID/BW_CLIENTSECRET env, unlock --passwordenv --raw → BW_SESSION) and full write support — org collections + attachments included, which reopens the org-vs-personal-vault choice rbw had foreclosed.
• Broker host + identity: runs as the session user (lkraven) on nh3-dev (this machine — where the live CC sessions + rbw live); rbw 1.15.0 is also present on nh3-extdev. Fleet ssh + the trust boundary for the bootstrap secret.
• Code home: eshpfi-management/services/secrets-broker/.

Data model / taxonomy (lookup-first)

• One item per secret. Item name encodes the path so lookup works even if folders are unavailable: <host>/<stack>/<file> (e.g. ana-docker/gitea/.env), or <domain>/<name> for non-host creds (gitea/claude-bot-token, wireguard/irv-ml1, certs/phasefinal-wildcard).
• Folders (if rbw supports add --folder — gate) group by class: hosts/, tokens/, certs/, ssh-keys/, db/, wireguard/, misc/.
• Item type by shape: a credential pair → Bitwarden login (username/password/ uri); an env file or freeform blob → secure note (whole file in the note body).
• Binary secrets (*.pem, *.key, *.pfx, acme.json, client_secrets.json) → base64 into a note field (rbw has no attachments). Cap ~64 KB/item; larger keys are flagged, not silently truncated.
• Metadata fields on every item: source_host, source_path, synced_at, sha256 (of the plaintext, for drift detection without decrypting to compare).

CLI surface (services/secrets-broker/secret)

• secret put <name> [--file PATH | --stdin] [--type note|login] [--field k=v]… — idempotent upsert (create if absent, else update in place, keyed by name). Never prints the secret value. Refuses .env.example / template inputs.
• secret get <name> [--field FIELD] [--file OUT] — fetch to stdout or a 0600 file; base64 fields decode with --file.
• secret list [--prefix P] — names + metadata only (never values).
• secret backfill --host H [--dry-run] — ssh to host H, enumerate real secret files (.env, env.sh, and the cert/key set), upsert each. --dry-run prints the plan (names, sizes, would-create/would-update) and writes nothing.

Bootstrap secret handling

• The service account's email + master password + personal API key (client_id/secret) live in a 0600 file at ~/.config/secrets-broker/bootstrap.env on nh3-dev (session-user-owned) — mirrors the ~/.config/<tool>/<token> pattern used across the fleet. The operator provisions the account and drops these creds (they never transit the chat; a 0600 template is pre-staged at bootstrap.env.template). This is the one secret that cannot live in the vault (secrets-zero); the crown jewel.
• bw unattended flow: bw login --apikey (reads BW_CLIENTID/BW_CLIENTSECRET from env, sourced from the bootstrap file) → bw unlock --passwordenv BW_PASSWORD --raw → a session token exported as BW_SESSION for subsequent calls. No pinentry/daemon (the abandoned rbw shim + .pinentry-key marker can be removed).

Invariants

1. Idempotent upserts — re-running put/backfill converges, never duplicates (name is the key).
2. Never log or echo secret values — not to stdout, stderr, logs, or the board. list and --dry-run show names/metadata only.
3. .env.example and templates are never stored — real secrets only.
4. Round-trip verified — backfill re-fetches and compares sha256 against the source before reporting a file "stored".
5. Bootstrap file is 0600, owned by the broker identity; the broker host is the trust boundary (the service account can read everything it can write — Bitwarden has no write-only).
6. Vault durability is a precondition — the Vaultwarden DB is in the pg_dump set (confirmed); v1 adds a periodic independent encrypted vault export → restic as belt-and-suspenders before it becomes load-bearing.

Phases

• Phase 0 — verify (read-only): DONE. Vault located + DB-backed-up; creds valid (direct grant 200); rbw abandoned (register 400); bw selected.
• Phase 1 — stand up: DONE (2026-08-11). Operator provisioned the service account (infra-ops@phasefinal.com); bw installed to ~/.local; bw config server + login --apikey + unlock --passwordenv all succeed; create→read→delete round-trip verified against the live vault. Remaining Phase-1 polish: decide the folder-vs-org taxonomy (bw reopened org) and lay it out.
• Phase 2 — backfill. secret backfill --host <h> --dry-run for every host, review, then real run; round-trip verify. Start with .env/env.sh, then the cert/key set.
• Phase 3 — CLI + keep-fresh. Harden the secret CLI; a periodic re-sync (host→vault) so rotated secrets don't drift; the encrypted export→restic job.

The rbw capability gate — RESOLVED (rbw 1.15.0, 2026-08-11)

Verified against the installed rbw 1.15.0 write surface on nh3-extdev:

• add/edit support --folder, --uri, username, and password + multi-line note. Folders work → taxonomy is folder + name-encoded path.
• No --organization/--collection on add → rbw cannot create items in an org collection. Resolution: v1 uses the service account's own vault + folders (pure-rbw, honors the rust-client choice; no bw write-path needed). Org + Collections stays deferred — it would force bw for writes for marginal multi-user-ACL benefit the fleet store doesn't need.
• add is editor-driven (first line = password, the rest = note). The secret put wrapper drives it non-interactively by setting $EDITOR/$VISUAL to a content-supplying shim (standard rbw scripting pattern) — no human editor in the automated path. Env-file blobs live in the note; single tokens as the password; metadata (sha256/source) as trailing note lines read back with get --field.

Consequence for human lookup: the registry is the service account's shared vault — the operator looks things up by logging in as secrets-broker (or pointing rbw at it), not via a shared collection in his personal account. Acceptable for a fleet infra store; the org path remains the escape hatch if per-user ACLs are ever needed.

Out of scope (v1)

• Deploy-time source-of-truth (deploys pulling secrets from the vault into .env). This is the powerful evolution; it changes every deploy path and makes a vault outage deploy-blocking. Revisit after the registry is proven.
• Organization + Collections with granular multi-user ACLs (rbw-write-limited; future).
• Bitwarden Secrets Manager — Vaultwarden does not implement it; not an option.

Failure modes / rollback

• Broker/vault down → lookups fail, but host .env are untouched and authoritative (backup semantics), so nothing stops running or deploying.
• Bad backfill write → idempotent upsert + round-trip verify catch it; items are versioned in Vaultwarden (restore prior).
• Bootstrap file compromise = full registry read → tight perms + broker-host trust are the control; rotate the service account's master password + API key on suspicion.