Vuong Hoang
968ec71cce
The adapter is STAGED, NOT SHIPPED: adapter_config.json and adapter_model.safetensors are in /tank/aimodels/voice-adapters/lv-mccarthy-4b-v1 on fv-ml1, verified byte-identical to gx10's checkpoint-300 at the source, after the local hop and at the destination. The seat CANNOT load it -- nothing in --lora-modules references it -- and /v1/models was checked after staging and still serves exactly the three registered voices plus base. The adapter carries a README recording the full gate result AND its cost, so it cannot be read as clean by anyone who finds the directory without the gate record. A copy is committed at scripts/mccarthy-corpus/gate-results/ADAPTER-README.md for review. It leads with "STAGED, NOT SHIPPED" and says that if lv-mccarthy is absent from the seat's model list, the decision has not been made. The compose edit that would actually register it is prepared in the working tree and deliberately LEFT UNCOMMITTED. stacks/ is canonical intent that deploy-stack.sh reads; committing it would assert an intent that is precisely the open question. Ship-script fix: <run-dir> must be an absolute REMOTE path. A leading ~ is expanded by the local shell, so ~/r49-runs/... became /home/lkraven/... and gx10 answered "Permission denied" -- a confusing error for a path problem. It now refuses a non-absolute path outright, with the reason. Verified: the guard fires on a ~ path and the real staging succeeded on the absolute one.
88 lines
3.9 KiB
Bash
Executable File
88 lines
3.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Ship an r49 voice LoRA from the training box to the voices seat's adapter store.
|
|
#
|
|
# scripts/r49-corpus/ship-voice-adapter.sh <run-dir> <checkpoint> <adapter-name>
|
|
# e.g. ... /home/infra-ops/r49-runs/mccarthy-4b-pairs-3ep checkpoint-300 lv-mccarthy-4b-v1
|
|
#
|
|
# ⚠ <run-dir> MUST BE AN ABSOLUTE REMOTE PATH. A leading `~` is expanded by the LOCAL
|
|
# shell before the script sees it, so `~/r49-runs/...` becomes /home/lkraven/... and the
|
|
# source read fails with "Permission denied" on gx10 -- a confusing error for a path
|
|
# problem. The guard below refuses it outright rather than letting ssh report it.
|
|
#
|
|
# Run from nh3-dev. gx10 and fv-ml1 both hold only authorized_keys, so neither can
|
|
# INITIATE to the other -- the relay through this box is the path, and it costs nothing
|
|
# because the WAN hop happens once either way.
|
|
#
|
|
# ⚠ THE SHA VERIFICATION IS THE POINT, not decoration. "copied the adapter" is a claim
|
|
# with no honest form but a read-back, and an adapter that is silently truncated or
|
|
# half-written loads without complaint and serves a subtly different voice. The digest is
|
|
# taken at the SOURCE, after the local hop, and at the DESTINATION, and any mismatch
|
|
# aborts before the seat is ever told about the file.
|
|
#
|
|
# Deliberately does NOT touch the compose file or restart the seat. Registering the
|
|
# adapter is a reviewed edit to stacks/voices-seat/compose.yaml plus deploy-stack.sh,
|
|
# because that file is where the gate verdict and its caveats are written down for
|
|
# whoever reads it next.
|
|
set -euo pipefail
|
|
|
|
RUN=${1:?usage: ship-voice-adapter.sh <run-dir> <checkpoint> <adapter-name>}
|
|
CKPT=${2:?missing checkpoint}
|
|
NAME=${3:?missing adapter name}
|
|
SRC_HOST=${SRC_HOST:-infra-ops@10.100.50.60} # pfi-gx10
|
|
DST_HOST=${DST_HOST:-infra-ops@10.251.50.54} # fv-ml1
|
|
DST_ROOT=${DST_ROOT:-/tank/aimodels/voice-adapters}
|
|
case "$RUN" in
|
|
/*) ;;
|
|
*) echo "== REFUSING: <run-dir> must be an ABSOLUTE remote path, got '$RUN'."
|
|
echo " A leading ~ is expanded by the local shell and will point at the wrong user."
|
|
exit 2 ;;
|
|
esac
|
|
STAGE=$(mktemp -d)
|
|
trap 'rm -rf "$STAGE"' EXIT
|
|
|
|
say(){ echo "[ship $(date +%H:%M:%S)] $*"; }
|
|
|
|
say "source $SRC_HOST:$RUN/checkpoints/$CKPT"
|
|
say "dest $DST_HOST:$DST_ROOT/$NAME"
|
|
|
|
# The two files vLLM actually loads. A checkpoint dir also holds optimizer state and RNG
|
|
# that must NOT be shipped -- they are large, useless to the seat, and would make the
|
|
# destination digest disagree with any future re-ship.
|
|
FILES="adapter_config.json adapter_model.safetensors"
|
|
|
|
say "digest at SOURCE"
|
|
SRC_SUMS=$(ssh -o BatchMode=yes "$SRC_HOST" "cd '$RUN/checkpoints/$CKPT' && sha256sum $FILES")
|
|
echo "$SRC_SUMS" | sed 's/^/ /'
|
|
|
|
say "pull to local stage"
|
|
for f in $FILES; do
|
|
rsync -a "$SRC_HOST:$RUN/checkpoints/$CKPT/$f" "$STAGE/$f"
|
|
done
|
|
STAGE_SUMS=$(cd "$STAGE" && sha256sum $FILES)
|
|
if [ "$SRC_SUMS" != "$STAGE_SUMS" ]; then
|
|
say "ABORT: digest changed on the source->local hop"; echo "$STAGE_SUMS"; exit 1
|
|
fi
|
|
say " hop 1 verified"
|
|
|
|
say "push to dest"
|
|
ssh -o BatchMode=yes "$DST_HOST" "mkdir -p '$DST_ROOT/$NAME'"
|
|
for f in $FILES; do
|
|
rsync -a "$STAGE/$f" "$DST_HOST:$DST_ROOT/$NAME/$f"
|
|
done
|
|
DST_SUMS=$(ssh -o BatchMode=yes "$DST_HOST" "cd '$DST_ROOT/$NAME' && sha256sum $FILES")
|
|
if [ "$SRC_SUMS" != "$DST_SUMS" ]; then
|
|
say "ABORT: digest differs at the destination"; echo "$DST_SUMS"; exit 1
|
|
fi
|
|
say " hop 2 verified -- byte-identical to the checkpoint on gx10"
|
|
|
|
say "destination listing"
|
|
ssh -o BatchMode=yes "$DST_HOST" "ls -la '$DST_ROOT/$NAME'" | sed 's/^/ /'
|
|
cat <<NEXT
|
|
|
|
NEXT, and neither is done for you:
|
|
1. Write $DST_ROOT/$NAME/README.md carrying the gate verdict AND its caveats, so the
|
|
adapter cannot be read as clean by anyone who finds it without the gate record.
|
|
2. Add - $NAME=/adapters/$NAME to stacks/voices-seat/compose.yaml --lora-modules,
|
|
with the verdict in a comment, then: scripts/deploy-stack.sh fv-ml1 voices-seat
|
|
NEXT
|