esh-pfi-infrastructure/services/secrets-broker/secrets-broker.contract.md

---
title: secrets-broker
kind: module-contract
status: draft
owner: infra-ops
created: 2026-08-11
depends_on:
  - Vaultwarden (vaultwarden.phasefinal.com, on ana-docker; DB on pfi-postgres, in the pg_dump backup set)
  - rbw (Rust Bitwarden CLI; unattended unlock daemon)
---

# secrets-broker — fleet-wide credential registry over Vaultwarden

## Purpose

A single place to **store and look up any fleet secret** — env files, API tokens,
TLS certs/keys, SSH keys, DB creds, WireGuard keys — that should not live in a git
repo and is today single-copy on a host. Vaultwarden is the durable central store;
the broker is the thin programmatic surface (`secret put|get|list|backfill`) that
agents, scripts, and the operator use. Solves two problems at once: **durability**
(secrets currently backed up nowhere) and **lookup** (no canonical place to find a
credential).

**Scope (corrected 2026-08-11):** this serves the **CC sessions on THIS dev box
(nh3-dev)** — a per-box credential store + backup, NOT a fleet service. New dev
boxes get this same stack **copied and duplicated one at a time** (each box: its own
`bw` + a per-box `bootstrap.env`, backing up its own local secrets, hostname-
namespaced). **No daemon / no central broker** — the subprocess CLI is the final
shape. Local `.env`/`env.sh` files stay in place; the vault is the authoritative
*registry + backup*. Deploy-time source-of-truth (pull-at-deploy) stays out of scope.

## Architecture

- **Store:** the operator-provisioned service account (`infra-ops@phasefinal.com`)
  in the **`infra-ops` org** (id `d30c6b58…`), **Default collection** (id
  `b829376a…`). Items land in that org collection so the operator's **primary
  account** (shared into the org) sees them too; organised by **folder** (per box,
  named for the hostname) + a `<host>/<path>` name convention.
- **Client:** **`bw`** (official Bitwarden CLI, installed to `~/.local` on nh3-dev).
  Switched from `rbw` after rbw's `register` returned an undebuggable 400 against this
  Vaultwarden despite valid creds (a direct `client_credentials` grant + both prelogin
  paths return 200; rbw emits no HTTP logs). bw gives a clean unattended flow
  (`login --apikey` via `BW_CLIENTID`/`BW_CLIENTSECRET` env, `unlock --passwordenv
  --raw` → `BW_SESSION`) and full write support — org collections + attachments
  included, which reopens the org-vs-personal-vault choice rbw had foreclosed.
- **Runs as** the session user (`lkraven`) on **nh3-dev**, invoked directly by CC
  sessions on this box. No ssh, no daemon; each dev box runs its own copy against its
  own local secrets.
- **Code home:** `eshpfi-management/services/secrets-broker/` (the `secret` CLI is
  copied to each dev box; the per-box `bootstrap.env` lives in
  `~/.config/secrets-broker/`, `0600`, never committed).

## Data model / taxonomy (lookup-first)

- One **item per secret**. Item **name encodes the path** so lookup works even if
  folders are unavailable: `<host>/<stack>/<file>` (e.g. `ana-docker/gitea/.env`),
  or `<domain>/<name>` for non-host creds (`gitea/claude-bot-token`,
  `wireguard/irv-ml1`, `certs/phasefinal-wildcard`).
- **Folders** group by class (one per box, named for the hostname); the item name
  carries the real path so lookup works regardless.
- **Item type:** everything is a *secure note* (type 2) — the file/secret content in
  the note body (arbitrary `KEY=VALUE` env content isn't a clean login shape).
- **Binary secrets** (`*.pem`, `*.key`, `*.pfx`) → **base64 into a hidden field**
  (`content_b64`); portable and rbw-agnostic. bw could use attachments, but base64
  keeps a single fetch path. Cap ~64 KB/item.
- Metadata fields on every item: `source_host`, `source_path`, `synced_at`,
  `sha256` (of the plaintext, for drift detection without decrypting to compare).

## CLI surface (`services/secrets-broker/secret`)

- `secret put <name> [--file PATH | --stdin] [--type note|login] [--field k=v]…`
  — idempotent **upsert** (create if absent, else update in place, keyed by name).
  Never prints the secret value. Refuses `.env.example` / template inputs.
- `secret get <name> [--field FIELD] [--file OUT]` — fetch to stdout or a `0600`
  file; base64 fields decode with `--file`.
- `secret list [--prefix P]` — names + metadata only (never values).
- `secret backfill --host H [--dry-run]` — ssh to host H, enumerate real secret
  files (`.env`, `env.sh`, and the cert/key set), upsert each. `--dry-run` prints
  the plan (names, sizes, would-create/would-update) and writes nothing.

## Bootstrap secret handling

- The service account's **email + master password + personal API key
  (client_id/secret)** live in a **`0600` file** at
  `~/.config/secrets-broker/bootstrap.env` on nh3-dev (session-user-owned) — mirrors
  the `~/.config/<tool>/<token>` pattern used across the fleet. **The operator
  provisions the account and drops these creds** (they never transit the chat; a
  `0600` template is pre-staged at `bootstrap.env.template`). This is the one secret
  that cannot live in the vault (secrets-zero); the crown jewel.
- bw unattended flow: `bw login --apikey` (reads `BW_CLIENTID`/`BW_CLIENTSECRET`
  from env, sourced from the bootstrap file) → `bw unlock --passwordenv BW_PASSWORD
  --raw` → a session token exported as `BW_SESSION` for subsequent calls. No
  pinentry/daemon (the abandoned rbw shim + `.pinentry-key` marker can be removed).

## Invariants

1. **Idempotent upserts** — re-running `put`/`backfill` converges, never duplicates
   (name is the key).
2. **Never log or echo secret values** — not to stdout, stderr, logs, or the board.
   `list` and `--dry-run` show names/metadata only.
3. **`.env.example` and templates are never stored** — real secrets only.
4. **Round-trip verified** — backfill re-fetches and compares `sha256` against the
   source before reporting a file "stored".
5. **Bootstrap file is `0600`**, owned by the broker identity; the broker host is
   the trust boundary (the service account can *read* everything it can write —
   Bitwarden has no write-only).
6. **Vault durability is a precondition** — the Vaultwarden DB is in the pg_dump set
   (confirmed); v1 adds a periodic independent **encrypted vault export → restic** as
   belt-and-suspenders before it becomes load-bearing.

## Phases

- **Phase 0 — verify (read-only): DONE.** Vault located + DB-backed-up; creds valid
  (direct grant 200); rbw abandoned (register 400); bw selected.
- **Phase 1 — stand up: DONE (2026-08-11).** Operator provisioned the service account
  (`infra-ops@phasefinal.com`); bw installed to `~/.local`; `bw config server` +
  `login --apikey` + `unlock --passwordenv` all succeed; **create→read→delete
  round-trip verified** against the live vault. Remaining Phase-1 polish: decide the
  folder-vs-org taxonomy (bw reopened org) and lay it out.
- **Phase 2 — backfill.** `secret backfill --host <h> --dry-run` for every host,
  review, then real run; round-trip verify. Start with `.env`/`env.sh`, then the
  cert/key set.
- **Phase 3 — CLI + keep-fresh.** Harden the `secret` CLI; a periodic re-sync
  (host→vault) so rotated secrets don't drift; the encrypted export→restic job.

## The rbw capability gate — RESOLVED (rbw 1.15.0, 2026-08-11)

Verified against the installed `rbw 1.15.0` write surface on nh3-extdev:
- `add`/`edit` support **`--folder`**, `--uri`, username, and password + multi-line
  note. **Folders work** → taxonomy is folder + name-encoded path.
- **No `--organization`/`--collection` on `add`** → rbw *cannot* create items in an
  org collection. **Resolution: v1 uses the service account's own vault + folders**
  (pure-rbw, honors the rust-client choice; no `bw` write-path needed). Org +
  Collections stays deferred — it would force `bw` for writes for marginal
  multi-user-ACL benefit the fleet store doesn't need.
- `add` is **editor-driven** (first line = password, the rest = note). The `secret
  put` wrapper drives it non-interactively by setting `$EDITOR`/`$VISUAL` to a
  content-supplying shim (standard rbw scripting pattern) — no human editor in the
  automated path. Env-file blobs live in the note; single tokens as the password;
  metadata (sha256/source) as trailing note lines read back with `get --field`.

Consequence for human lookup: the registry is the **service account's shared vault**
— the operator looks things up by logging in *as* `secrets-broker` (or pointing rbw
at it), not via a shared collection in his personal account. Acceptable for a fleet
infra store; the org path remains the escape hatch if per-user ACLs are ever needed.

## Out of scope (v1)

- **Deploy-time source-of-truth** (deploys *pulling* secrets from the vault into
  `.env`). This is the powerful evolution; it changes every deploy path and makes a
  vault outage deploy-blocking. Revisit after the registry is proven.
- **Organization + Collections** with granular multi-user ACLs (rbw-write-limited;
  future).
- **Bitwarden Secrets Manager** — Vaultwarden does not implement it; not an option.

## Known limitations (deferred)
- **`secret put` namespace warning is non-blocking + stderr** — an interactive caller
  sees the new-top-level-namespace heads-up, but a *scripted* caller that suppresses
  stderr can still store an item under a typo'd/missing prefix silently (jackdaw-dev,
  2026-08-12). Deliberately not blocking (some names are legitimately domain-scoped,
  e.g. `gitea/…`, `certs/…`, so auto-prefixing would misfire). Revisit with an opt-in
  `--strict` flag (nonzero exit on a new namespace) only if scripted callers appear.

## Failure modes / rollback

- Broker/vault down → lookups fail, but host `.env` are untouched and authoritative
  (backup semantics), so nothing stops running or deploying.
- Bad backfill write → idempotent upsert + round-trip verify catch it; items are
  versioned in Vaultwarden (restore prior).
- Bootstrap file compromise = full registry read → tight perms + broker-host trust
  are the control; rotate the service account's master password + API key on
  suspicion.