esh-pfi-infrastructure/services/secrets-broker/secrets-broker.contract.md

title, kind, status, owner, created, depends_on

title	kind	status	owner	created	depends_on
secrets-broker	module-contract	draft	infra-ops	2026-08-11	Vaultwarden (vaultwarden.phasefinal.com, on ana-docker; DB on pfi-postgres, in the pg_dump backup set) rbw (Rust Bitwarden CLI; unattended unlock daemon)

secrets-broker — fleet-wide credential registry over Vaultwarden

Purpose

A single place to store and look up any fleet secret — env files, API tokens, TLS certs/keys, SSH keys, DB creds, WireGuard keys — that should not live in a git repo and is today single-copy on a host. Vaultwarden is the durable central store; the broker is the thin programmatic surface (secret put|get|list|backfill) that agents, scripts, and the operator use. Solves two problems at once: durability (secrets currently backed up nowhere) and lookup (no canonical place to find a credential).

Scope (corrected 2026-08-11): this serves the CC sessions on THIS dev box (nh3-dev) — a per-box credential store + backup, NOT a fleet service. New dev boxes get this same stack copied and duplicated one at a time (each box: its own bw + a per-box bootstrap.env, backing up its own local secrets, hostname- namespaced). No daemon / no central broker — the subprocess CLI is the final shape. Local .env/env.sh files stay in place; the vault is the authoritative registry + backup. Deploy-time source-of-truth (pull-at-deploy) stays out of scope.

Architecture

• Store: the operator-provisioned service account (infra-ops@phasefinal.com) in the infra-ops org (id d30c6b58…), Default collection (id b829376a…). Items land in that org collection so the operator's primary account (shared into the org) sees them too; organised by folder (per box, named for the hostname) + a <host>/<path> name convention.
• Client: bw (official Bitwarden CLI, installed to ~/.local on nh3-dev). Switched from rbw after rbw's register returned an undebuggable 400 against this Vaultwarden despite valid creds (a direct client_credentials grant + both prelogin paths return 200; rbw emits no HTTP logs). bw gives a clean unattended flow (login --apikey via BW_CLIENTID/BW_CLIENTSECRET env, unlock --passwordenv --raw → BW_SESSION) and full write support — org collections + attachments included, which reopens the org-vs-personal-vault choice rbw had foreclosed.
• Runs as the session user (lkraven) on nh3-dev, invoked directly by CC sessions on this box. No ssh, no daemon; each dev box runs its own copy against its own local secrets.
• Code home: eshpfi-management/services/secrets-broker/ (the secret CLI is copied to each dev box; the per-box bootstrap.env lives in ~/.config/secrets-broker/, 0600, never committed).

Data model / taxonomy (lookup-first)

• One item per secret. Item name encodes the path so lookup works even if folders are unavailable: <host>/<stack>/<file> (e.g. ana-docker/gitea/.env), or <domain>/<name> for non-host creds (gitea/claude-bot-token, wireguard/irv-ml1, certs/phasefinal-wildcard).
• Folders group by class (one per box, named for the hostname); the item name carries the real path so lookup works regardless.
• Item type: everything is a secure note (type 2) — the file/secret content in the note body (arbitrary KEY=VALUE env content isn't a clean login shape).
• Binary secrets (*.pem, *.key, *.pfx) → base64 into a hidden field (content_b64); portable and rbw-agnostic. bw could use attachments, but base64 keeps a single fetch path. Cap ~64 KB/item.
• Metadata fields on every item: source_host, source_path, synced_at, sha256 (of the plaintext, for drift detection without decrypting to compare).

CLI surface (services/secrets-broker/secret)

• secret put <name> [--file PATH | --stdin] [--type note|login] [--field k=v]… — idempotent upsert (create if absent, else update in place, keyed by name). Never prints the secret value. Refuses .env.example / template inputs.
• secret get <name> [--field FIELD] [--file OUT] — fetch to stdout or a 0600 file; base64 fields decode with --file.
• secret list [--prefix P] — names + metadata only (never values).
• secret backfill --host H [--dry-run] — ssh to host H, enumerate real secret files (.env, env.sh, and the cert/key set), upsert each. --dry-run prints the plan (names, sizes, would-create/would-update) and writes nothing.

Bootstrap secret handling

• The service account's email + master password + personal API key (client_id/secret) live in a 0600 file at ~/.config/secrets-broker/bootstrap.env on nh3-dev (session-user-owned) — mirrors the ~/.config/<tool>/<token> pattern used across the fleet. The operator provisions the account and drops these creds (they never transit the chat; a 0600 template is pre-staged at bootstrap.env.template). This is the one secret that cannot live in the vault (secrets-zero); the crown jewel.
• bw unattended flow: bw login --apikey (reads BW_CLIENTID/BW_CLIENTSECRET from env, sourced from the bootstrap file) → bw unlock --passwordenv BW_PASSWORD --raw → a session token exported as BW_SESSION for subsequent calls. No pinentry/daemon (the abandoned rbw shim + .pinentry-key marker can be removed).

Invariants

1. Idempotent upserts — re-running put/backfill converges, never duplicates (name is the key).
2. Never log or echo secret values — not to stdout, stderr, logs, or the board. list and --dry-run show names/metadata only.
3. .env.example and templates are never stored — real secrets only.
4. Round-trip verified — backfill re-fetches and compares sha256 against the source before reporting a file "stored".
5. Bootstrap file is 0600, owned by the broker identity; the broker host is the trust boundary (the service account can read everything it can write — Bitwarden has no write-only).
6. Vault durability is a precondition — the Vaultwarden DB is in the pg_dump set (confirmed); v1 adds a periodic independent encrypted vault export → restic as belt-and-suspenders before it becomes load-bearing.

Phases

• Phase 0 — verify (read-only): DONE. Vault located + DB-backed-up; creds valid (direct grant 200); rbw abandoned (register 400); bw selected.
• Phase 1 — stand up: DONE (2026-08-11). Operator provisioned the service account (infra-ops@phasefinal.com); bw installed to ~/.local; bw config server + login --apikey + unlock --passwordenv all succeed; create→read→delete round-trip verified against the live vault. Remaining Phase-1 polish: decide the folder-vs-org taxonomy (bw reopened org) and lay it out.
• Phase 2 — backfill. secret backfill --host <h> --dry-run for every host, review, then real run; round-trip verify. Start with .env/env.sh, then the cert/key set.
• Phase 3 — CLI + keep-fresh. Harden the secret CLI; a periodic re-sync (host→vault) so rotated secrets don't drift; the encrypted export→restic job.

The rbw capability gate — RESOLVED (rbw 1.15.0, 2026-08-11)

Verified against the installed rbw 1.15.0 write surface on nh3-extdev:

• add/edit support --folder, --uri, username, and password + multi-line note. Folders work → taxonomy is folder + name-encoded path.
• No --organization/--collection on add → rbw cannot create items in an org collection. Resolution: v1 uses the service account's own vault + folders (pure-rbw, honors the rust-client choice; no bw write-path needed). Org + Collections stays deferred — it would force bw for writes for marginal multi-user-ACL benefit the fleet store doesn't need.
• add is editor-driven (first line = password, the rest = note). The secret put wrapper drives it non-interactively by setting $EDITOR/$VISUAL to a content-supplying shim (standard rbw scripting pattern) — no human editor in the automated path. Env-file blobs live in the note; single tokens as the password; metadata (sha256/source) as trailing note lines read back with get --field.

Consequence for human lookup: the registry is the service account's shared vault — the operator looks things up by logging in as secrets-broker (or pointing rbw at it), not via a shared collection in his personal account. Acceptable for a fleet infra store; the org path remains the escape hatch if per-user ACLs are ever needed.

Out of scope (v1)

• Deploy-time source-of-truth (deploys pulling secrets from the vault into .env). This is the powerful evolution; it changes every deploy path and makes a vault outage deploy-blocking. Revisit after the registry is proven.
• Organization + Collections with granular multi-user ACLs (rbw-write-limited; future).
• Bitwarden Secrets Manager — Vaultwarden does not implement it; not an option.

Known limitations (deferred)

• secret put namespace warning is non-blocking + stderr — an interactive caller sees the new-top-level-namespace heads-up, but a scripted caller that suppresses stderr can still store an item under a typo'd/missing prefix silently (jackdaw-dev, 2026-08-12). Deliberately not blocking (some names are legitimately domain-scoped, e.g. gitea/…, certs/…, so auto-prefixing would misfire). Revisit with an opt-in --strict flag (nonzero exit on a new namespace) only if scripted callers appear.

Failure modes / rollback

• Broker/vault down → lookups fail, but host .env are untouched and authoritative (backup semantics), so nothing stops running or deploying.
• Bad backfill write → idempotent upsert + round-trip verify catch it; items are versioned in Vaultwarden (restore prior).
• Bootstrap file compromise = full registry read → tight perms + broker-host trust are the control; rotate the service account's master password + API key on suspicion.