From 6ca455a15fd65940bcfcec96c912bbb2a890b771 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 2 Sep 2026 17:49:42 -0700 Subject: [PATCH] =?UTF-8?q?feat(scripts):=20provision-mac-dsh.sh=20?= =?UTF-8?q?=E2=80=94=20one=20script=20for=20a=20Mac,=20end=20to=20end?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three Macs and six accounts were done by hand, and the fourth would have repeated every mistake the first three taught. This script carries them. Each guard is something a hand-run got wrong first: - an account may not own its own home. A `sudo mkdir` before sysadminctl leaves /Users/ root-owned; the account then authenticates, gets a shell, has a correct $HOME and cannot write to it. Surfaced on the Studio as a bare "Permission denied" hours after the account looked fine. - `sudo -u` keeps the CALLER's $HOME. Without -H the install's rm -rf aims at the wrong account — it did, at a working install, and only permissions stopped it. The remote half refuses to run unless $HOME matches the target. - the provider ships a hard-coded model catalog that the web GUI reads INDEPENDENTLY of agent-default-model, so a correct default still showed DeepSeek models in the picker. `models:` replaces it. - reasoningEffort / maxTokens / defaultContextWindow are all measured against the seat; the harness defaults fail on every one. - the key is scoped per machine and the scope is VERIFIED (200 on gen-reasoning, 403 on gen), not trusted from the mint. The first run found two more: it named the vault item after the IP (`mac-10-0-10-10/`, unreadable beside esh-mac-studio) and its config check used grep -A3 where the block needs -A4, so it printed an empty model and passed anyway. Both fixed, and verification now asserts the model rather than only the answer token — a check that cannot fail is not a check. Run twice against the same account to confirm idempotence, then against vhpfi. docs/runbooks/mac-provisioning.md carries the operator-run stage and the traps that are not the script's to solve. --- docs/runbooks/mac-provisioning.md | 90 ++++++++++ scripts/provision-mac-dsh.sh | 284 ++++++++++++++++++++++++++++++ 2 files changed, 374 insertions(+) create mode 100644 docs/runbooks/mac-provisioning.md create mode 100755 scripts/provision-mac-dsh.sh diff --git a/docs/runbooks/mac-provisioning.md b/docs/runbooks/mac-provisioning.md new file mode 100644 index 0000000..d71d288 --- /dev/null +++ b/docs/runbooks/mac-provisioning.md @@ -0,0 +1,90 @@ +# Provisioning a Mac for the fleet + +Three Macs are provisioned this way as of 2026-09-02: `vuongs-mac-mini` +(10.100.79.2), `esh-macbook-air` (10.0.10.83), `esh-mac-studio` (10.0.10.10). +None is in `servers/` or `dns/internal.yaml` — they are the operator's personal +machines, not PFI-managed fleet hosts, and registering them there would imply +otherwise. + +## Two stages + +**Stage 1 — an account I can reach.** Operator-run, because it needs a password +I do not have. See "Operator steps" below. + +**Stage 2 — the harness.** `scripts/provision-mac-dsh.sh [name]`, +idempotent, run once per account. + + scripts/provision-mac-dsh.sh 10.0.10.10 vhpfi esh-mac-studio + scripts/provision-mac-dsh.sh --check 10.0.10.10 vhpfi + +## Operator steps (stage 1) + +```zsh +sudo sysadminctl -addUser infra-ops -fullName "PFI infra-ops" \ + -shell /bin/zsh -home /Users/infra-ops -password - +sudo dseditgroup -o edit -a infra-ops -t user admin +sudo mkdir -p /Users/infra-ops/.ssh +sudo tee /Users/infra-ops/.ssh/authorized_keys >/dev/null <<'KEY' +ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN+1HBwfXrkfTYWdcnWCjLJ6VLAGC87gxH5h5vKaaA3c infra-ops@pfi-fleet +KEY +sudo chown -R infra-ops:staff /Users/infra-ops/.ssh +sudo chmod 700 /Users/infra-ops/.ssh; sudo chmod 600 /Users/infra-ops/.ssh/authorized_keys +``` + +Then hand over a throwaway password; infra-ops rotates it, vaults it as +`/infra-ops-password`, and installs a `visudo`-validated NOPASSWD drop-in. + +⚠ **Install the key and prove key-auth on a FRESH connection BEFORE touching the +password.** These machines have no out-of-band access — a failed rotation means +the operator walks to the machine. + +## ⚠ Traps, all of them paid for + +**An account may not own its own home.** If `/Users/` was created by a +`sudo mkdir` before `sysadminctl` ran, sysadminctl adopts the existing directory +and leaves it **root-owned**. The account then authenticates, gets a shell, has a +correct `$HOME`, and cannot write to it — surfacing as a bare `Permission denied` +from `mkdir` long after the account looked healthy. The script detects and fixes +this; a hand-run will not. + +**`sudo -u ` keeps the CALLER's `$HOME`.** Without `-H` and an explicit +`HOME=`, `"$HOME/.local"` resolves to the caller's home. On 2026-09-02 that +pointed an `rm -rf` at a working install in another account; only filesystem +permissions stopped it. The remote half of the script refuses to run unless +`$HOME` matches the target account. + +**A wrong username looks exactly like a wrong password.** sshd answers +`Permission denied (publickey,password,keyboard-interactive)` for a bad user, a +bad password, AND a user outside `com.apple.access_ssh`. Two of the three Macs +produced a false diagnosis this way. **Check `dscl . -list /Users` first** — the +Studio's operator account is `vhpfi`, not `lkraven`. + +**`com.apple.access_ssh` gates SSH when it exists, but admins usually pass +anyway** through a nested group. Do not *create* the group if absent: doing so +flips SSH from open-to-all to members-only and can lock out the operator. + +**Password rotation: use `dscl . -passwd`, not `sysadminctl`.** With FileVault on +and no Secure Token on the account, `sysadminctl -resetPasswordFor` refuses with +*"Operation is not permitted without secure token unlock"*. `dscl` works +precisely because there is no token to desync. True on all three Macs. + +⚠ **FileVault kills remote access across reboots.** The machine sits at the +pre-boot unlock screen with no network until someone unlocks it physically. +Nothing unattended should depend on a Mac being reachable after a restart. + +**macOS has no `adduser`, `useradd`, or `timeout`.** Use `sysadminctl`, and wrap +the ssh call locally rather than reaching for a remote `timeout`. + +## Harness specifics + +Node is installed **private to the account** under `~/.local/node`, +checksum-verified — deliberately not Homebrew, which owns `/opt/homebrew` and +edits PATH. One **device-scoped** gateway key per machine (`-dsh`, scoped +to `gen-reasoning`), shared by that machine's accounts and vaulted at +`/litellm-dsh-key`; the script verifies the scope (200 on gen-reasoning, +403 on gen) rather than trusting the mint. + +See `stacks/litellm/README.md` for why `reasoningEffort: high` works at all, and +`scripts/provision-mac-dsh.sh` for the three measured limits (`reasoningEffort`, +`maxTokens`, `defaultContextWindow`) and the hard-coded model catalog the web +GUI reads independently of the default model. diff --git a/scripts/provision-mac-dsh.sh b/scripts/provision-mac-dsh.sh new file mode 100755 index 0000000..646b29b --- /dev/null +++ b/scripts/provision-mac-dsh.sh @@ -0,0 +1,284 @@ +#!/usr/bin/env bash +# Provision the DeepSeek Harness (`dsh`) on a Mac, for one account, pointed at +# the fleet's LiteLLM gen-reasoning seat. +# +# scripts/provision-mac-dsh.sh [name] +# scripts/provision-mac-dsh.sh 10.0.10.10 vhpfi esh-mac-studio +# scripts/provision-mac-dsh.sh --check 10.0.10.10 vhpfi +# +# `name` is the vault/key-alias namespace and defaults to the machine's own +# hostname. Give it explicitly to match what is already in the vault -- the +# first run derived it from the IP and produced `mac-10-0-10-10/`, which is +# unreadable next to esh-mac-studio / esh-macbook-air / vuongs-mac-mini. +# +# Written after doing this by hand on three Macs and six accounts. Every value +# and every guard below is something a hand-run got wrong first. +# +# ───────────────────────────────────────────────────────────────────────────── +# WHAT IT DOES, AND WHY EACH STEP EXISTS +# +# 1. Node under ~/.local/node, checksum-verified against nodejs.org. +# NOT Homebrew: a package manager owning /opt/homebrew and editing PATH is +# a bigger footprint than this task earns on someone's daily driver. +# +# 2. npm -g @deepseek-ai/dsh with npm_config_prefix=~/.local, so the install +# is contained in the account and nothing lands system-wide. +# +# 3. A DEVICE-SCOPED LiteLLM key (models: [gen-reasoning]), minted per host, +# never the shared all-agents key. A laptop travels; losing one should be +# one revocation, not a fleet-wide rotation. The scope is VERIFIED after +# minting, not assumed -- see `feedback_retiring_a_model_orphans_scoped_keys`. +# +# 4. ~/.dsh/.credentials.yaml (0600) + a cordis.patch.yml in BOTH profiles. +# +# 5. Verification: a real headless task must return the expected token, and +# the composed config must show gen-reasoning. A green install that cannot +# answer a prompt is the failure this script exists to stop shipping. +# +# ───────────────────────────────────────────────────────────────────────────── +# ⚠ THE FIVE THINGS THAT BIT DURING THE HAND-RUNS +# +# ⚠ `sudo -u ` KEEPS THE CALLER'S $HOME. Without -H (and an explicit +# HOME=), "$HOME/.local" resolves to the CALLER's home and the install's +# `rm -rf` aims at the wrong account. On 2026-09-02 this pointed a wipe at +# a working install; only filesystem permissions stopped it. The remote +# script below refuses to run unless $HOME matches the target account. +# +# ⚠ macOS HAS NO `timeout`. Wrap the ssh call locally instead. +# +# ⚠ THE PROVIDER SHIPS A HARD-CODED MODEL CATALOG. `dsh-llm-deepseek` returns +# deepseek-v4-flash/-pro/-flash-vision-exp to "discovery consumers" -- i.e. +# the web GUI's model picker -- INDEPENDENTLY of agent-default-model. Set +# `models:` or the GUI offers three models our gateway does not serve while +# headless runs work fine. This one shipped to the operator before it was +# caught. +# +# ⚠ reasoning_effort IS NOT A UNIVERSAL VOCABULARY. The seat takes only +# xhigh/medium/low; the harness emits off/low/high/max. `high` is mapped to +# `xhigh` by the gateway hook conf/reasoning_effort_map.py. Without that +# hook the seat 400s and `low` -- its WEAKEST tier -- is the only value +# that works. +# +# ⚠ THE HARNESS DEFAULT maxTokens IS 256000 against a 262144-token seat, +# leaving 6144 for input. A two-word prompt overflowed it. +# +# ⚠ ALPHA SOFTWARE. dsh is 0.1.x and its README promises compatibility-breaking +# changes. Re-run this after an upgrade rather than assuming config survives. +set -euo pipefail + +GATEWAY="http://10.250.50.70:4000/v1" +MODEL="gen-reasoning" +NODE_VER="v24.9.0" +ADMIN_KEY_FILE="$HOME/.config/litellm/infra-ops-key" +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +SECRET="$REPO_ROOT/services/secrets-broker/secret" + +CHECK=0 +[[ "${1:-}" == "--check" ]] && { CHECK=1; shift; } +HOST="${1:-}"; ACCOUNT="${2:-}"; NAME="${3:-}" + +if [[ -z "$HOST" || -z "$ACCOUNT" ]]; then + sed -n '2,12p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + exit 2 +fi + +say() { printf '%s\n' "$*"; } +step() { printf '\n── %s\n' "$*"; } + +# How do we reach this account? Direct if its own key auth works, else via +# infra-ops + sudo. Probed, not assumed -- the accounts differ per machine +# (the Studio has vhpfi where the others have lkraven). +step "reaching $ACCOUNT@$HOST" +SSH_DIRECT=(ssh -o BatchMode=yes -o ConnectTimeout=8 -i "$HOME/.ssh/infra-ops_ed25519") +if timeout 15 "${SSH_DIRECT[@]}" "$ACCOUNT@$HOST" true 2>/dev/null; then + MODE=direct + say " direct key auth as $ACCOUNT" +elif timeout 15 "${SSH_DIRECT[@]}" "infra-ops@$HOST" "sudo -n -H -u $ACCOUNT env HOME=/Users/$ACCOUNT true" 2>/dev/null; then + MODE=viasudo + say " via infra-ops + NOPASSWD sudo -> $ACCOUNT" +else + say " ✗ cannot reach $ACCOUNT@$HOST directly or through infra-ops sudo." + say " Provision infra-ops on this host first (see docs/runbooks/mac-provisioning.md)." + exit 1 +fi + +# Run a script in the TARGET account, with HOME correct in both modes. +remote() { + if [[ "$MODE" == direct ]]; then + timeout "${1:-300}" "${SSH_DIRECT[@]}" "$ACCOUNT@$HOST" "bash -s -- ${2:-}" + else + timeout "${1:-300}" "${SSH_DIRECT[@]}" "infra-ops@$HOST" \ + "sudo -n -H -u $ACCOUNT env HOME=/Users/$ACCOUNT bash -s -- ${2:-}" + fi +} + +if (( CHECK )); then + step "check only — nothing will be changed" + remote 60 <<'EOF' +export PATH="$HOME/.local/node/bin:$HOME/.local/bin:$PATH" +printf ' HOME %s\n' "$HOME" +printf ' node %s\n' "$(node --version 2>/dev/null || echo ABSENT)" +printf ' dsh %s\n' "$(dsh --version 2>/dev/null || echo ABSENT)" +printf ' credentials %s\n' "$(test -f ~/.dsh/.credentials.yaml && echo present || echo ABSENT)" +printf ' model %s\n' "$(dsh --profile headless --dump-config 2>/dev/null | grep -A4 'id: agent-default-model' | sed -n 's/^ *model: //p' || echo '?')" +EOF + exit 0 +fi + +# ── device-scoped gateway key ──────────────────────────────────────────────── +step "gateway key" +# Name the key and vault item after the MACHINE, not its address: addresses +# change, and `mac-10-0-10-10/` is unreadable in a vault listing. +if [[ -z "$NAME" ]]; then + NAME=$(timeout 15 "${SSH_DIRECT[@]}" "${MODE:+infra-ops}@$HOST" hostname -s 2>/dev/null \ + | tr '[:upper:]' '[:lower:]' | tr -cd 'a-z0-9-') + NAME="${NAME:-mac-$(printf '%s' "$HOST" | tr '.' '-')}" +fi +KEY_ALIAS="${NAME}-dsh" +VAULT_ITEM="${NAME}/litellm-dsh-key" +PWTMP="$(mktemp)"; chmod 600 "$PWTMP" +trap 'shred -u "$PWTMP" 2>/dev/null || rm -f "$PWTMP"' EXIT + +# One key per MACHINE, shared by its accounts: the blast radius is the device, +# so a second key per account would be extra state with no extra containment. +if "$SECRET" get "$VAULT_ITEM" >"$PWTMP" 2>/dev/null && [[ -s "$PWTMP" ]]; then + say " reusing this machine's vaulted key ($VAULT_ITEM)" +else + [[ -r "$ADMIN_KEY_FILE" ]] || { say " ✗ no LiteLLM admin key at $ADMIN_KEY_FILE"; exit 1; } + curl -s -m 20 -X POST "${GATEWAY%/v1}/key/generate" \ + -H "Authorization: Bearer $(cat "$ADMIN_KEY_FILE")" -H "Content-Type: application/json" \ + -d "{\"key_alias\":\"$KEY_ALIAS\",\"models\":[\"$MODEL\"], + \"metadata\":{\"host\":\"$HOST\",\"account\":\"$ACCOUNT\",\"purpose\":\"DeepSeek Harness\",\"minted_by\":\"infra-ops\"}}" \ + | python3 -c "import sys,json;d=json.load(sys.stdin);k=d.get('key'); +open('$PWTMP','w').write(k or '');print(' minted',d.get('key_alias'),d.get('models'))" + [[ -s "$PWTMP" ]] || { say " ✗ key mint failed"; exit 1; } + "$SECRET" put "$VAULT_ITEM" --file "$PWTMP" \ + --field host="$HOST" --field alias="$KEY_ALIAS" --field scope="$MODEL" >/dev/null + say " vaulted at $VAULT_ITEM" +fi + +# ⚠ VERIFY THE SCOPE. A key that silently reaches more than intended is worse +# than no scoping, because it looks contained. +KEY="$(cat "$PWTMP")" +allowed=$(curl -s -m 60 -o /dev/null -w '%{http_code}' "$GATEWAY/chat/completions" \ + -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \ + -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":4}") +denied=$(curl -s -m 60 -o /dev/null -w '%{http_code}' "$GATEWAY/chat/completions" \ + -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \ + -d '{"model":"gen","messages":[{"role":"user","content":"hi"}],"max_tokens":4}') +say " scope: $MODEL -> $allowed gen -> $denied (want 200 / 403)" +[[ "$allowed" == 200 && "$denied" == 403 ]] || { say " ✗ key scope is not what was requested"; exit 1; } + +# ── install + configure ────────────────────────────────────────────────────── +# ⚠ AN ACCOUNT MAY NOT OWN ITS OWN HOME. If /Users/ was created by a +# `sudo mkdir` before sysadminctl ran, sysadminctl adopts the existing +# directory and leaves it root-owned. The account then authenticates, gets a +# shell, has a correct $HOME -- and cannot write to it. Presents as a bare +# "Permission denied" from mkdir, hours after the account looked fine. +# Found on esh-mac-studio, 2026-09-02. +step "home ownership" +timeout 60 "${SSH_DIRECT[@]}" "infra-ops@$HOST" " + owner=\$(stat -f '%Su' /Users/$ACCOUNT) + if [ \"\$owner\" != '$ACCOUNT' ]; then + echo \" /Users/$ACCOUNT was owned by \$owner — chowning to $ACCOUNT:staff\" + sudo -n chown $ACCOUNT:staff /Users/$ACCOUNT + else + echo ' ok: owned by $ACCOUNT' + fi + stat -f ' %N %Su:%Sg %Sp' /Users/$ACCOUNT" + +step "installing node $NODE_VER + dsh in $ACCOUNT" +remote 600 "$ACCOUNT $NODE_VER $KEY $GATEWAY $MODEL" <<'EOF' +set -euo pipefail +ACCOUNT="$1"; NODE_VER="$2"; KEY="$3"; GATEWAY="$4"; MODEL="$5" +# ⚠ Guard: refuse if HOME is not the target account's. `sudo -u` without -H +# keeps the caller's HOME and the rm -rf below would hit the wrong account. +[ "$HOME" = "/Users/$ACCOUNT" ] || { echo "REFUSING: HOME=$HOME, expected /Users/$ACCOUNT"; exit 1; } +PREFIX="$HOME/.local"; mkdir -p "$PREFIX/bin" +TARBALL="node-${NODE_VER}-darwin-arm64" +if [ "$("$PREFIX/node/bin/node" --version 2>/dev/null)" != "$NODE_VER" ]; then + cd "$(mktemp -d)" + curl -fsSLO "https://nodejs.org/dist/${NODE_VER}/${TARBALL}.tar.gz" + curl -fsSLO "https://nodejs.org/dist/${NODE_VER}/SHASUMS256.txt" + grep " ${TARBALL}.tar.gz$" SHASUMS256.txt | shasum -a 256 -c - + rm -rf "$PREFIX/node" + tar -xzf "${TARBALL}.tar.gz"; mv "${TARBALL}" "$PREFIX/node" +fi +export PATH="$PREFIX/node/bin:$PREFIX/bin:$PATH"; export npm_config_prefix="$PREFIX" +npm install -g @deepseek-ai/dsh 2>&1 | tail -2 +echo " dsh $(dsh --version)" + +mkdir -p ~/.dsh +dsh --profile headless --dump-default-config >/dev/null 2>&1 || true # materialise profiles +printf 'LITELLM_API_KEY: %s\n' "$KEY" > ~/.dsh/.credentials.yaml +chmod 700 ~/.dsh; chmod 600 ~/.dsh/.credentials.yaml + +for p in headless web; do + mkdir -p ~/.dsh/profiles/$p + cat > ~/.dsh/profiles/$p/cordis.patch.yml </dev/null || cat >> ~/.zprofile <<'ZP' + +# DeepSeek Harness + its private Node runtime (contained under ~/.local) +export PATH="$HOME/.local/node/bin:$HOME/.local/bin:$PATH" +ZP +EOF + +# ── verification ───────────────────────────────────────────────────────────── +step "verify" +out=$(remote 300 <<'EOF' +export PATH="$HOME/.local/node/bin:$HOME/.local/bin:$PATH" +# ⚠ -A4, not -A3: the block is id/name/config/provider/model, so -A3 stops one +# line short and the check silently reports nothing. A check that cannot fail +# is not a check -- the assertion below is what makes this one load-bearing. +printf ' composed model: %s\n' "$(dsh --profile web --dump-config 2>/dev/null | grep -A4 'id: agent-default-model' | sed -n 's/^ *model: //p')" +printf ' login shell : %s\n' "$(zsh -lc 'command -v dsh' 2>/dev/null || echo 'NOT on PATH')" +dsh --profile headless "Reply with exactly PROVISION-OK and nothing else." 2>&1 | tail -3 +EOF +) +say "$out" +if printf '%s' "$out" | grep -q "composed model: $MODEL" && printf '%s' "$out" | grep -q 'PROVISION-OK'; then + say "" + say " ✓ $ACCOUNT@$HOST provisioned and answering through $MODEL" +else + say "" + say " ✗ install completed but verification failed (wrong model, or no answer)." + say " Do not call this done." + exit 1 +fi