esh-pfi-infrastructure/services/secrets-broker/secrets-broker.contract.md

---
title: secrets-broker
kind: module-contract
status: draft
owner: infra-ops
created: 2026-08-11
depends_on:
  - Vaultwarden (vaultwarden.phasefinal.com, on ana-docker; DB on pfi-postgres, in the pg_dump backup set)
  - rbw (Rust Bitwarden CLI; unattended unlock daemon)
---

# secrets-broker — fleet-wide credential registry over Vaultwarden

## Purpose

A single place to **store and look up any fleet secret** — env files, API tokens,
TLS certs/keys, SSH keys, DB creds, WireGuard keys — that should not live in a git
repo and is today single-copy on a host. Vaultwarden is the durable central store;
the broker is the thin programmatic surface (`secret put|get|list|backfill`) that
agents, scripts, and the operator use. Solves two problems at once: **durability**
(secrets currently backed up nowhere) and **lookup** (no canonical place to find a
credential).

This is a **central store**, not (yet) a deploy-time source of truth: host `.env`
files stay in place; the vault is the authoritative *registry + backup*. The
pull-at-deploy evolution is explicitly out of scope for v1 (see Out of scope).

## Architecture

- **Store:** a **dedicated Vaultwarden service account** (`secrets-broker@…`), its
  personal vault *is* the registry. (An Organization + per-host Collections is the
  textbook multi-user structure, but **rbw's write path into org collections is
  weak** — see the rbw capability gate — so v1 uses the service account's own vault
  with a folder/name taxonomy. Org migration is a future option if granular
  human ACLs are ever needed.)
- **Client:** **`bw`** (official Bitwarden CLI, installed to `~/.local` on nh3-dev).
  Switched from `rbw` after rbw's `register` returned an undebuggable 400 against this
  Vaultwarden despite valid creds (a direct `client_credentials` grant + both prelogin
  paths return 200; rbw emits no HTTP logs). bw gives a clean unattended flow
  (`login --apikey` via `BW_CLIENTID`/`BW_CLIENTSECRET` env, `unlock --passwordenv
  --raw` → `BW_SESSION`) and full write support — org collections + attachments
  included, which reopens the org-vs-personal-vault choice rbw had foreclosed.
- **Broker host + identity:** runs as the session user (`lkraven`) on **nh3-dev**
  (this machine — where the live CC sessions + rbw live); rbw 1.15.0 is also present
  on nh3-extdev. Fleet ssh + the trust boundary for the bootstrap secret.
- **Code home:** `eshpfi-management/services/secrets-broker/`.

## Data model / taxonomy (lookup-first)

- One **item per secret**. Item **name encodes the path** so lookup works even if
  folders are unavailable: `<host>/<stack>/<file>` (e.g. `ana-docker/gitea/.env`),
  or `<domain>/<name>` for non-host creds (`gitea/claude-bot-token`,
  `wireguard/irv-ml1`, `certs/phasefinal-wildcard`).
- **Folders** (if rbw supports `add --folder` — gate) group by class: `hosts/`,
  `tokens/`, `certs/`, `ssh-keys/`, `db/`, `wireguard/`, `misc/`.
- **Item type by shape:** a credential pair → Bitwarden *login* (username/password/
  uri); an env file or freeform blob → *secure note* (whole file in the note body).
- **Binary secrets** (`*.pem`, `*.key`, `*.pfx`, `acme.json`, `client_secrets.json`)
  → **base64 into a note field** (rbw has no attachments). Cap ~64 KB/item; larger
  keys are flagged, not silently truncated.
- Metadata fields on every item: `source_host`, `source_path`, `synced_at`,
  `sha256` (of the plaintext, for drift detection without decrypting to compare).

## CLI surface (`services/secrets-broker/secret`)

- `secret put <name> [--file PATH | --stdin] [--type note|login] [--field k=v]…`
  — idempotent **upsert** (create if absent, else update in place, keyed by name).
  Never prints the secret value. Refuses `.env.example` / template inputs.
- `secret get <name> [--field FIELD] [--file OUT]` — fetch to stdout or a `0600`
  file; base64 fields decode with `--file`.
- `secret list [--prefix P]` — names + metadata only (never values).
- `secret backfill --host H [--dry-run]` — ssh to host H, enumerate real secret
  files (`.env`, `env.sh`, and the cert/key set), upsert each. `--dry-run` prints
  the plan (names, sizes, would-create/would-update) and writes nothing.

## Bootstrap secret handling

- The service account's **email + master password + personal API key
  (client_id/secret)** live in a **`0600` file** at
  `~/.config/secrets-broker/bootstrap.env` on nh3-dev (session-user-owned) — mirrors
  the `~/.config/<tool>/<token>` pattern used across the fleet. **The operator
  provisions the account and drops these creds** (they never transit the chat; a
  `0600` template is pre-staged at `bootstrap.env.template`). This is the one secret
  that cannot live in the vault (secrets-zero); the crown jewel.
- bw unattended flow: `bw login --apikey` (reads `BW_CLIENTID`/`BW_CLIENTSECRET`
  from env, sourced from the bootstrap file) → `bw unlock --passwordenv BW_PASSWORD
  --raw` → a session token exported as `BW_SESSION` for subsequent calls. No
  pinentry/daemon (the abandoned rbw shim + `.pinentry-key` marker can be removed).

## Invariants

1. **Idempotent upserts** — re-running `put`/`backfill` converges, never duplicates
   (name is the key).
2. **Never log or echo secret values** — not to stdout, stderr, logs, or the board.
   `list` and `--dry-run` show names/metadata only.
3. **`.env.example` and templates are never stored** — real secrets only.
4. **Round-trip verified** — backfill re-fetches and compares `sha256` against the
   source before reporting a file "stored".
5. **Bootstrap file is `0600`**, owned by the broker identity; the broker host is
   the trust boundary (the service account can *read* everything it can write —
   Bitwarden has no write-only).
6. **Vault durability is a precondition** — the Vaultwarden DB is in the pg_dump set
   (confirmed); v1 adds a periodic independent **encrypted vault export → restic** as
   belt-and-suspenders before it becomes load-bearing.

## Phases

- **Phase 0 — verify (read-only): DONE.** Vault located + DB-backed-up; creds valid
  (direct grant 200); rbw abandoned (register 400); bw selected.
- **Phase 1 — stand up: DONE (2026-08-11).** Operator provisioned the service account
  (`infra-ops@phasefinal.com`); bw installed to `~/.local`; `bw config server` +
  `login --apikey` + `unlock --passwordenv` all succeed; **create→read→delete
  round-trip verified** against the live vault. Remaining Phase-1 polish: decide the
  folder-vs-org taxonomy (bw reopened org) and lay it out.
- **Phase 2 — backfill.** `secret backfill --host <h> --dry-run` for every host,
  review, then real run; round-trip verify. Start with `.env`/`env.sh`, then the
  cert/key set.
- **Phase 3 — CLI + keep-fresh.** Harden the `secret` CLI; a periodic re-sync
  (host→vault) so rotated secrets don't drift; the encrypted export→restic job.

## The rbw capability gate — RESOLVED (rbw 1.15.0, 2026-08-11)

Verified against the installed `rbw 1.15.0` write surface on nh3-extdev:
- `add`/`edit` support **`--folder`**, `--uri`, username, and password + multi-line
  note. **Folders work** → taxonomy is folder + name-encoded path.
- **No `--organization`/`--collection` on `add`** → rbw *cannot* create items in an
  org collection. **Resolution: v1 uses the service account's own vault + folders**
  (pure-rbw, honors the rust-client choice; no `bw` write-path needed). Org +
  Collections stays deferred — it would force `bw` for writes for marginal
  multi-user-ACL benefit the fleet store doesn't need.
- `add` is **editor-driven** (first line = password, the rest = note). The `secret
  put` wrapper drives it non-interactively by setting `$EDITOR`/`$VISUAL` to a
  content-supplying shim (standard rbw scripting pattern) — no human editor in the
  automated path. Env-file blobs live in the note; single tokens as the password;
  metadata (sha256/source) as trailing note lines read back with `get --field`.

Consequence for human lookup: the registry is the **service account's shared vault**
— the operator looks things up by logging in *as* `secrets-broker` (or pointing rbw
at it), not via a shared collection in his personal account. Acceptable for a fleet
infra store; the org path remains the escape hatch if per-user ACLs are ever needed.

## Out of scope (v1)

- **Deploy-time source-of-truth** (deploys *pulling* secrets from the vault into
  `.env`). This is the powerful evolution; it changes every deploy path and makes a
  vault outage deploy-blocking. Revisit after the registry is proven.
- **Organization + Collections** with granular multi-user ACLs (rbw-write-limited;
  future).
- **Bitwarden Secrets Manager** — Vaultwarden does not implement it; not an option.

## Failure modes / rollback

- Broker/vault down → lookups fail, but host `.env` are untouched and authoritative
  (backup semantics), so nothing stops running or deploying.
- Bad backfill write → idempotent upsert + round-trip verify catch it; items are
  versioned in Vaultwarden (restore prior).
- Bootstrap file compromise = full registry read → tight perms + broker-host trust
  are the control; rotate the service account's master password + API key on
  suspicion.