feat(secrets-broker): nh3-dev backfill complete (25/25) + attachment + resilient run

Scope corrected to per-dev-box (CC sessions on this box), not a fleet service; each
box duplicates the stack and backs up its own local secrets, hostname-namespaced.

CLI:
- backfill is local-only (scan this box's ~/development/*/{env.sh,.env} + ~/.config
  credentials; exclude bootstrap.env/examples/AIPA-Data archives).
- large files (>6000 B) route to a bw ATTACHMENT instead of the note field
  (Vaultwarden caps notes at ~10000 encrypted chars); get/verify read it back.
- backfill catches per-item failures and continues (bw errors raise BwError,
  main converts to a clean exit); idempotent upsert makes re-runs safe.

Backfilled all 25 nh3-dev secret files into the infra-ops org's Default collection
(folder = hostname), every one round-trip verified (2 large via attachment, 23 via
note). README added for duplicating the stack to new dev boxes. Contract scope +
data-model sections updated (bw, org/collection, per-box).

services/secrets-broker/README.md

@@ -0,0 +1,65 @@
+# secrets-broker
+
+Per-dev-box credential **store + backup** over the PFI Vaultwarden
+(`vaultwarden.phasefinal.com`). Lets the CC sessions on a dev box stash and look up
+secrets — API keys, tokens, `env.sh` / `.env` files, TLS keys — that shouldn't live
+in git and are otherwise single-copy on the box.
+
+**Not a fleet service.** Each dev box runs its own copy of this stack against its own
+local secrets; items are hostname-namespaced (`<hostname>/…`) in the shared
+`infra-ops` org so they don't collide. No daemon — the `secret` CLI shells out to
+`bw` per call (~a few seconds; fine for occasional session use).
+
+## Files
+
+| File | What |
+|---|---|
+| `secret` | the CLI (`put` / `get` / `list` / `backfill`) — copy this to each box |
+| `secrets-broker.contract.md` | the spec |
+| `~/.config/secrets-broker/bootstrap.env` | **per-box** service-account creds, `0600`, **never committed** |
+
+## Set up on a new dev box
+
+1. **Install `bw`** (user-prefix, no sudo):
+   ```bash
+   npm install -g @bitwarden/cli --prefix "$HOME/.local"   # -> ~/.local/bin/bw
+   ```
+2. **Provision creds** (operator): the service account already exists
+   (`infra-ops@phasefinal.com`); drop its creds into the bootstrap file:
+   ```bash
+   mkdir -p ~/.config/secrets-broker && chmod 700 ~/.config/secrets-broker
+   $EDITOR ~/.config/secrets-broker/bootstrap.env   # RBW_EMAIL, RBW_MASTER_PW, BW_CLIENTID, BW_CLIENTSECRET
+   chmod 600 ~/.config/secrets-broker/bootstrap.env
+   ```
+   (`RBW_*` names are historical — bw reads them the same.)
+3. **Copy the CLI** and point at the vault:
+   ```bash
+   cp secret ~/.local/bin/secret          # or run in place
+   bw config server https://vaultwarden.phasefinal.com
+   ```
+4. **Back up this box's secrets:**
+   ```bash
+   secret backfill --dry-run   # review what it would store
+   secret backfill             # write + round-trip-verify each
+   ```
+
+## Usage
+
+```bash
+secret put myproj/.env --file ./env.sh --folder $(hostname)   # upsert a secret
+secret put api/some-token --stdin                             # from stdin
+secret get myproj/.env                                        # -> note body (stdout)
+secret get certs/foo.pem --field content_b64 --file foo.pem   # binary -> 0600 file
+secret list --prefix $(hostname)/                             # names + metadata only
+secret backfill [--dry-run]                                   # this box's local secrets
+```
+
+## Notes
+
+- **`bootstrap.env` is the one secret that can't be vaulted** (secrets-zero) — it's
+  excluded from backfill. Keep it `0600`; it's this box's crown jewel.
+- Values are only ever printed by an explicit `get`; `list` / `--dry-run` show
+  names + metadata (`sha256`, `synced_at`, `source_path`) only.
+- Upsert is idempotent (keyed by name) — re-running `backfill` refreshes, never
+  duplicates. Safe to re-run if a run is interrupted.
+- The Vaultwarden DB is in the pg_dump backup set; the store itself is durable.

services/secrets-broker/secret

@@ -1,29 +1,34 @@
 #!/usr/bin/env python3
-"""secret — fleet credential store & retrieve over Vaultwarden (bw-backed).
+"""secret — per-box credential store & retrieve over Vaultwarden (bw-backed).
 
-Stores each secret as an item in the `infra-ops` org's Default collection (so the
+Serves the CC sessions on this dev box. Stores each secret as an item in the
-operator's primary account, an org member, sees it too), organised by folder +
+`infra-ops` org's Default collection (so the operator's primary account, an org
-a `<host>/<stack>/<file>` name convention. Text secrets (env files, tokens) live
+member, sees it too), organised by folder (named for the hostname) + a
-in the item note; binary secrets (keys/certs) are base64'd into a hidden field.
+`<host>/<path>` name convention. Small text -> item note; small binary -> base64
+hidden field; anything over the note cap -> a bw attachment. Metadata fields carry
+sha256 + source for drift detection.
 
-Auth bootstraps from ~/.config/secrets-broker/bootstrap.env (0600): apikey login
+Auth bootstraps from ~/.config/secrets-broker/bootstrap.env (0600): apikey login +
-+ master-password unlock -> per-invocation session. Secret VALUES are never printed
+master-password unlock -> per-invocation session. Secret VALUES are never printed
 except by an explicit `get`.
 
 Commands:
   secret put  <name> (--file P | --stdin) [--folder C] [--field k=v]...
   secret get  <name> [--field F] [--file OUT]
   secret list [--prefix P]
-  secret backfill --host H [--dry-run]     # enumerate a host's .env/env.sh, upsert
+  secret backfill [--dry-run]              # scan THIS box's local secrets, upsert each
 """
 import argparse
 import base64
+import glob
 import hashlib
 import json
 import os
-import shlex
+import shutil
+import socket
 import subprocess
 import sys
+import tempfile
 import time
 from pathlib import Path
 
@@ -33,6 +38,11 @@ BOOTSTRAP = CFG / "bootstrap.env"
 SERVER = "https://vaultwarden.phasefinal.com"
 ORG_ID = "d30c6b58-773c-4e39-916e-8e33ca7f8b81"
 COLLECTION_ID = "b829376a-9db1-4091-a4ae-9a36132ad4c2"
+NOTE_MAX = 6000  # plaintext bytes; larger -> attachment (Vaultwarden note cap ~10000 encrypted chars)
+
+
+class BwError(Exception):
+    pass
 
 
 def die(msg):
@@ -61,7 +71,7 @@ def bw(args, env, session=None, stdin=None, check=True):
     cmd = [BW] + args + (["--session", session] if session else [])
     r = subprocess.run(cmd, env=env, input=stdin, capture_output=True)
     if check and r.returncode != 0:
-        die(f"bw {args[0]} failed: {r.stderr.decode(errors='replace').strip()}")
+        raise BwError(f"bw {args[0]}: {r.stderr.decode(errors='replace').strip()}")
     return r
 
 
@@ -86,8 +96,7 @@ def encode(obj, env):
 
 
 def find(env, s, name):
-    r = bw(["list", "items", "--search", name], env, s)
+    for it in json.loads(bw(["list", "items", "--search", name], env, s).stdout.decode() or "[]"):
-    for it in json.loads(r.stdout.decode() or "[]"):
         if it.get("name") == name:
             return it
     return None
@@ -103,6 +112,67 @@ def folder_id(env, s, cls):
     return json.loads(r.stdout.decode())["id"]
 
 
+def store_secret(env, s, name, data, folder_name, extra_fields):
+    """Upsert a secret item; route by size: note / base64-field / attachment.
+    Returns (item_id, sha256, mode)."""
+    sha = hashlib.sha256(data).hexdigest()
+    try:
+        text, is_text = data.decode("utf-8"), True
+    except UnicodeDecodeError:
+        text, is_text = None, False
+    large = len(data) > NOTE_MAX
+    fields = [{"name": "sha256", "value": sha, "type": 0},
+              {"name": "synced_at", "value": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "type": 0}]
+    fields += extra_fields
+    attach_name = None
+    if large:
+        attach_name = name.split("/")[-1] or "secret"
+        notes = f"<{len(data)} bytes — stored as attachment '{attach_name}'>"
+        fields.append({"name": "storage", "value": "attachment", "type": 0})
+    elif is_text:
+        notes = text
+    else:
+        notes = "<binary — value in content_b64 field>"
+        fields.append({"name": "content_b64", "value": base64.b64encode(data).decode(), "type": 1})
+    item = {"type": 2, "name": name, "notes": notes, "organizationId": ORG_ID,
+            "collectionIds": [COLLECTION_ID], "folderId": folder_id(env, s, folder_name),
+            "fields": fields, "secureNote": {"type": 0}}
+    existing = find(env, s, name)
+    if existing:
+        item["id"] = existing["id"]
+        bw(["edit", "item", existing["id"]], env, s, stdin=encode(item, env))
+        item_id = existing["id"]
+        full = json.loads(bw(["get", "item", item_id], env, s).stdout.decode())
+        for att in full.get("attachments") or []:   # clear stale attachments (no dupes on re-store)
+            bw(["delete", "attachment", att["id"], "--itemid", item_id], env, s)
+    else:
+        item_id = json.loads(bw(["create", "item"], env, s, stdin=encode(item, env)).stdout.decode())["id"]
+    if large:
+        d = tempfile.mkdtemp()
+        try:
+            fp = os.path.join(d, attach_name)
+            Path(fp).write_bytes(data)
+            bw(["create", "attachment", "--itemid", item_id, "--file", fp], env, s)
+        finally:
+            shutil.rmtree(d, ignore_errors=True)
+    return item_id, sha, ("attachment" if large else ("field" if not is_text else "note"))
+
+
+def read_secret(env, s, name):
+    """Return the stored secret bytes for `name` (attachment / base64-field / note)."""
+    it = find(env, s, name)
+    if not it:
+        return None
+    full = json.loads(bw(["get", "item", it["id"]], env, s).stdout.decode())
+    atts = full.get("attachments") or []
+    if atts:
+        return bw(["get", "attachment", atts[0]["fileName"], "--itemid", full["id"], "--raw"], env, s).stdout
+    for f in full.get("fields") or []:
+        if f["name"] == "content_b64":
+            return base64.b64decode(f["value"])
+    return (full.get("notes") or "").encode()
+
+
 def cmd_put(a):
     env = load_env()
     s = session(env)
@@ -112,35 +182,12 @@ def cmd_put(a):
         data = sys.stdin.buffer.read()
     else:
         die("put needs --file or --stdin")
-    sha = hashlib.sha256(data).hexdigest()
+    extra = []
-    try:
-        notes, binary = data.decode("utf-8"), False
-    except UnicodeDecodeError:
-        notes, binary = "<binary — value in the content_b64 field>", True
-    fields = [
-        {"name": "sha256", "value": sha, "type": 0},
-        {"name": "synced_at", "value": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "type": 0},
-    ]
     for kv in (a.field or []):
         k, v = kv.split("=", 1)
-        fields.append({"name": k, "value": v, "type": 0})
+        extra.append({"name": k, "value": v, "type": 0})
-    if binary:
+    _, sha, mode = store_secret(env, s, a.name, data, a.folder, extra)
-        fields.append({"name": "content_b64", "value": base64.b64encode(data).decode(), "type": 1})
+    print(f"stored: {a.name}  (sha256 {sha[:12]}, {len(data)} bytes, {mode})")
-    item = {
-        "type": 2, "name": a.name, "notes": notes,
-        "organizationId": ORG_ID, "collectionIds": [COLLECTION_ID],
-        "folderId": folder_id(env, s, a.folder), "fields": fields,
-        "secureNote": {"type": 0},
-    }
-    existing = find(env, s, a.name)
-    if existing:
-        item["id"] = existing["id"]
-        bw(["edit", "item", existing["id"]], env, s, stdin=encode(item, env))
-        action = "updated"
-    else:
-        bw(["create", "item"], env, s, stdin=encode(item, env))
-        action = "created"
-    print(f"{action}: {a.name}  (sha256 {sha[:12]}, {len(data)} bytes{', binary' if binary else ''})")
 
 
 def cmd_get(a):
@@ -151,19 +198,15 @@ def cmd_get(a):
     if a.field:
         for f in full.get("fields") or []:
             if f["name"] == a.field:
-                if a.field == "content_b64" and a.file:
-                    Path(a.file).write_bytes(base64.b64decode(f["value"]))
-                    os.chmod(a.file, 0o600)
-                    return print(f"wrote {a.file} (0600)")
                 return print(f["value"])
         die(f"no field '{a.field}' on {a.name}")
-    content = full.get("notes") or ""
+    data = read_secret(env, s, a.name)
     if a.file:
-        Path(a.file).write_text(content)
+        Path(a.file).write_bytes(data)
         os.chmod(a.file, 0o600)
         print(f"wrote {a.file} (0600)")
     else:
-        sys.stdout.write(content if content.endswith("\n") else content + "\n")
+        sys.stdout.buffer.write(data if data.endswith(b"\n") else data + b"\n")
 
 
 def cmd_list(a):
@@ -181,63 +224,68 @@ def cmd_list(a):
     print(f"# {n} item(s)", file=sys.stderr)
 
 
-def _host_secret_files(host):
+LOCAL_GLOBS = [
-    find_cmd = (r"find /opt/docker -maxdepth 4 \( -name .env -o -name env.sh \) "
+    "development/*/env.sh", "development/*/*/env.sh",
-                r"! -name '*.example' -type f 2>/dev/null")
+    "development/*/.env", "development/*/*/.env",
-    r = subprocess.run(["ssh", host, find_cmd], capture_output=True, text=True)
+    ".config/secrets/env.sh",
-    return [p for p in r.stdout.split("\n") if p.strip()]
+    ".config/*/*.env", ".config/*/*token*", ".config/*/*-key*",
+    ".config/*/*.pem", ".config/*/*.key", ".config/*/secrets.env", ".config/*/cred*",
+]
+LOCAL_EXCLUDE = ("secrets-broker/bootstrap.env", ".example", "/node_modules/",
+                 "/.cargo/", "/.cache/", "/AIPA-Data/")
 
 
-def _name_for(host, path):
+def _local_secret_files():
-    parts = Path(path).parts  # e.g. /opt/docker/compose/gitea/.env
+    home = str(Path.home())
-    stack = parts[-2] if len(parts) >= 2 else "root"
+    hits = set()
-    return f"{host}/{stack}/{Path(path).name}"
+    for g in LOCAL_GLOBS:
+        hits.update(glob.glob(f"{home}/{g}"))
+    return [p for p in sorted(hits)
+            if not any(e in p for e in LOCAL_EXCLUDE) and Path(p).is_file()]
+
+
+def _local_name(path):
+    home = str(Path.home())
+    rel = path[len(home) + 1:] if path.startswith(home + "/") else path.lstrip("/")
+    return f"{socket.gethostname()}/{rel}"
 
 
 def cmd_backfill(a):
-    host = a.host
+    files = _local_secret_files()
-    files = _host_secret_files(host)
+    host = socket.gethostname()
     if not files:
-        return print(f"{host}: no .env/env.sh found under /opt/docker")
+        return print(f"{host}: no local secret files matched")
-    env = load_env()
-    s = None if a.dry_run else session(env)
     print(f"{host}: {len(files)} secret file(s)")
+    if a.dry_run:
+        for path in files:
+            data = Path(path).read_bytes()
+            tag = "attachment" if len(data) > NOTE_MAX else "note"
+            print(f"  WOULD store  {_local_name(path)}\t({len(data)} bytes, "
+                  f"sha {hashlib.sha256(data).hexdigest()[:12]}, {tag})")
+        print("  (dry-run — nothing written; bootstrap.env / *.example / AIPA-Data excluded)")
+        return
+    env = load_env()
+    s = session(env)
+    ok, failed = 0, []
     for path in files:
-        name = _name_for(host, path)
+        name = _local_name(path)
-        data = subprocess.run(["ssh", host, f"cat {shlex.quote(path)}"], capture_output=True).stdout
+        data = Path(path).read_bytes()
-        sha = hashlib.sha256(data).hexdigest()
-        if a.dry_run:
-            print(f"  WOULD store  {name}\t({len(data)} bytes, sha {sha[:12]})  <- {path}")
-            continue
         try:
-            notes = data.decode("utf-8")
+            _, sha, mode = store_secret(env, s, name, data, host,
-        except UnicodeDecodeError:
+                                        [{"name": "source_host", "value": host, "type": 0},
-            notes = "<binary>"
+                                         {"name": "source_path", "value": path, "type": 0}])
-        fields = [
+            back = read_secret(env, s, name)
-            {"name": "sha256", "value": sha, "type": 0},
+            v = back is not None and hashlib.sha256(back).hexdigest() == sha
-            {"name": "source_host", "value": host, "type": 0},
+            print(f"  stored {name}\t({len(data)}b, {mode}) verify={'OK' if v else 'MISMATCH'}")
-            {"name": "source_path", "value": path, "type": 0},
+            ok += 1
-            {"name": "synced_at", "value": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "type": 0},
+        except BwError as e:
-        ]
+            print(f"  FAILED {name}: {e}")
-        item = {"type": 2, "name": name, "notes": notes, "organizationId": ORG_ID,
+            failed.append(name)
-                "collectionIds": [COLLECTION_ID], "folderId": folder_id(env, s, "hosts"),
+    print(f"{host}: {ok}/{len(files)} stored" + (f"; {len(failed)} FAILED: {', '.join(failed)}" if failed else ""))
-                "fields": fields, "secureNote": {"type": 0}}
-        existing = find(env, s, name)
-        if existing:
-            item["id"] = existing["id"]
-            bw(["edit", "item", existing["id"]], env, s, stdin=encode(item, env))
-            verb = "updated"
-        else:
-            bw(["create", "item"], env, s, stdin=encode(item, env))
-            verb = "stored "
-        # round-trip verify
-        back = json.loads(bw(["get", "item", find(env, s, name)["id"]], env, s).stdout.decode())
-        ok = hashlib.sha256((back.get("notes") or "").encode()).hexdigest() == sha
-        print(f"  {verb} {name}\t({len(data)}b) verify={'OK' if ok else 'MISMATCH'}")
 
 
 def main():
-    p = argparse.ArgumentParser(prog="secret", description="fleet credential store over Vaultwarden")
+    p = argparse.ArgumentParser(prog="secret", description="per-box credential store over Vaultwarden")
     sub = p.add_subparsers(dest="cmd", required=True)
     pp = sub.add_parser("put"); pp.add_argument("name")
     pp.add_argument("--file"); pp.add_argument("--stdin", action="store_true")
@@ -246,10 +294,13 @@ def main():
     pg = sub.add_parser("get"); pg.add_argument("name")
     pg.add_argument("--field"); pg.add_argument("--file"); pg.set_defaults(fn=cmd_get)
     pl = sub.add_parser("list"); pl.add_argument("--prefix"); pl.set_defaults(fn=cmd_list)
-    pb = sub.add_parser("backfill"); pb.add_argument("--host", required=True)
+    pb = sub.add_parser("backfill", help="scan THIS box's local secret files and upsert each")
     pb.add_argument("--dry-run", action="store_true"); pb.set_defaults(fn=cmd_backfill)
     a = p.parse_args()
-    a.fn(a)
+    try:
+        a.fn(a)
+    except BwError as e:
+        die(str(e))
 
 
 if __name__ == "__main__":

services/secrets-broker/secrets-broker.contract.md

@@ -21,18 +21,21 @@ agents, scripts, and the operator use. Solves two problems at once: **durability
 (secrets currently backed up nowhere) and **lookup** (no canonical place to find a
 credential).
 
-This is a **central store**, not (yet) a deploy-time source of truth: host `.env`
+**Scope (corrected 2026-08-11):** this serves the **CC sessions on THIS dev box
-files stay in place; the vault is the authoritative *registry + backup*. The
+(nh3-dev)** — a per-box credential store + backup, NOT a fleet service. New dev
-pull-at-deploy evolution is explicitly out of scope for v1 (see Out of scope).
+boxes get this same stack **copied and duplicated one at a time** (each box: its own
+`bw` + a per-box `bootstrap.env`, backing up its own local secrets, hostname-
+namespaced). **No daemon / no central broker** — the subprocess CLI is the final
+shape. Local `.env`/`env.sh` files stay in place; the vault is the authoritative
+*registry + backup*. Deploy-time source-of-truth (pull-at-deploy) stays out of scope.
 
 ## Architecture
 
-- **Store:** a **dedicated Vaultwarden service account** (`secrets-broker@…`), its
+- **Store:** the operator-provisioned service account (`infra-ops@phasefinal.com`)
-  personal vault *is* the registry. (An Organization + per-host Collections is the
+  in the **`infra-ops` org** (id `d30c6b58…`), **Default collection** (id
-  textbook multi-user structure, but **rbw's write path into org collections is
+  `b829376a…`). Items land in that org collection so the operator's **primary
-  weak** — see the rbw capability gate — so v1 uses the service account's own vault
+  account** (shared into the org) sees them too; organised by **folder** (per box,
-  with a folder/name taxonomy. Org migration is a future option if granular
+  named for the hostname) + a `<host>/<path>` name convention.
-  human ACLs are ever needed.)
 - **Client:** **`bw`** (official Bitwarden CLI, installed to `~/.local` on nh3-dev).
   Switched from `rbw` after rbw's `register` returned an undebuggable 400 against this
   Vaultwarden despite valid creds (a direct `client_credentials` grant + both prelogin
@@ -40,10 +43,12 @@ pull-at-deploy evolution is explicitly out of scope for v1 (see Out of scope).
   (`login --apikey` via `BW_CLIENTID`/`BW_CLIENTSECRET` env, `unlock --passwordenv
   --raw` → `BW_SESSION`) and full write support — org collections + attachments
   included, which reopens the org-vs-personal-vault choice rbw had foreclosed.
-- **Broker host + identity:** runs as the session user (`lkraven`) on **nh3-dev**
+- **Runs as** the session user (`lkraven`) on **nh3-dev**, invoked directly by CC
-  (this machine — where the live CC sessions + rbw live); rbw 1.15.0 is also present
+  sessions on this box. No ssh, no daemon; each dev box runs its own copy against its
-  on nh3-extdev. Fleet ssh + the trust boundary for the bootstrap secret.
+  own local secrets.
-- **Code home:** `eshpfi-management/services/secrets-broker/`.
+- **Code home:** `eshpfi-management/services/secrets-broker/` (the `secret` CLI is
+  copied to each dev box; the per-box `bootstrap.env` lives in
+  `~/.config/secrets-broker/`, `0600`, never committed).
 
 ## Data model / taxonomy (lookup-first)
 
@@ -51,13 +56,13 @@ pull-at-deploy evolution is explicitly out of scope for v1 (see Out of scope).
   folders are unavailable: `<host>/<stack>/<file>` (e.g. `ana-docker/gitea/.env`),
   or `<domain>/<name>` for non-host creds (`gitea/claude-bot-token`,
   `wireguard/irv-ml1`, `certs/phasefinal-wildcard`).
-- **Folders** (if rbw supports `add --folder` — gate) group by class: `hosts/`,
+- **Folders** group by class (one per box, named for the hostname); the item name
-  `tokens/`, `certs/`, `ssh-keys/`, `db/`, `wireguard/`, `misc/`.
+  carries the real path so lookup works regardless.
-- **Item type by shape:** a credential pair → Bitwarden *login* (username/password/
+- **Item type:** everything is a *secure note* (type 2) — the file/secret content in
-  uri); an env file or freeform blob → *secure note* (whole file in the note body).
+  the note body (arbitrary `KEY=VALUE` env content isn't a clean login shape).
-- **Binary secrets** (`*.pem`, `*.key`, `*.pfx`, `acme.json`, `client_secrets.json`)
+- **Binary secrets** (`*.pem`, `*.key`, `*.pfx`) → **base64 into a hidden field**
-  → **base64 into a note field** (rbw has no attachments). Cap ~64 KB/item; larger
+  (`content_b64`); portable and rbw-agnostic. bw could use attachments, but base64
-  keys are flagged, not silently truncated.
+  keeps a single fetch path. Cap ~64 KB/item.
 - Metadata fields on every item: `source_host`, `source_path`, `synced_at`,
   `sha256` (of the plaintext, for drift detection without decrypting to compare).