vh
a2b5b58eee
Replaces a bare nohup on irv-ml1:8410 that would not have survived a reboot, handed over by waterland-dev. Tracks vh/waterland @ main (PR #4 merged; main HEAD is exactly the pinned 8025366). Build context is a checkout at /opt/waterland-studio/src, deliberately OUTSIDE the compose dir — deploy-stack.sh rsyncs stacks/<stack>/ with --delete and would otherwise eat it. The Dockerfile is passed out-of-context. Three landmines, all measured: 1. Both uv extras are load-bearing at build AND run. jobs.py shells the renderer out as a literal with no --extra flags, so uv would re-sync at runtime and prune cupy — silently dropping to the numpy path at ~21x wall time. UV_NO_SYNC pins it; UV_OFFLINE makes any failure loud instead of quietly slow. 2. cupy needs CUDA HEADERS for its NVRTC compile, not just the driver and the wheel's runtime libs. The host has a system CUDA toolkit so the nohup process found them by accident; a slim image does not, and every render died 1.7s in with 'Failed to find CUDA headers' printed through argparse's usage banner — which reads like a CLI bug, not a missing toolkit. Fixed with cupy-cuda12x[ctk] (hundreds of MB, vs ~6 GB for a -devel base). 3. The A6000 is host device 1 but container device 0, since compose exposes exactly one GPU. CUDA_VISIBLE_DEVICES_TARGET=0 inside; copying the host's value selects a device that does not exist. /root/.cupy is a volume because the NVRTC compile costs ~17s: verified at 23.3s cold vs 6.1s warm, and re-verified across a restart (23.2s on a fresh cache volume, 6.0s once populated). Warm 256^2+anim beats the 7.4s recorded against bare metal, so containerising cost nothing. Job store seeded with the 4 jobs from the displaced instance. Serial by design (one replica, one card) and unauthenticated, so it stays LAN/WireGuard-only.
47 lines
1.4 KiB
Bash
Executable File
47 lines
1.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Refresh the waterland checkout and rebuild the studio image.
|
|
#
|
|
# The checkout deliberately lives OUTSIDE the compose directory:
|
|
# deploy-stack.sh rsyncs stacks/<stack>/ with --delete, so a checkout kept
|
|
# beside compose.yaml would be deleted by the next deploy of this stack.
|
|
#
|
|
# Run ON irv-ml1:
|
|
# /opt/docker/compose/waterland-studio/update.sh
|
|
set -euo pipefail
|
|
|
|
SRC=/opt/waterland-studio/src
|
|
COMPOSE_DIR=/opt/docker/compose/waterland-studio
|
|
REPO_URL=${WATERLAND_REPO_URL:-https://gitea.phasefinal.com/vh/waterland.git}
|
|
REF=${WATERLAND_REF:-main}
|
|
|
|
if [ ! -d "$SRC/.git" ]; then
|
|
echo "no checkout at $SRC — clone it first:"
|
|
echo " sudo mkdir -p $(dirname "$SRC")"
|
|
echo " sudo git clone $REPO_URL $SRC"
|
|
exit 1
|
|
fi
|
|
|
|
echo "==> fetching $REF"
|
|
git -C "$SRC" fetch --prune origin
|
|
git -C "$SRC" checkout -q "$REF"
|
|
git -C "$SRC" reset --hard "origin/$REF"
|
|
echo "==> now at $(git -C "$SRC" rev-parse --short HEAD): $(git -C "$SRC" log -1 --format=%s)"
|
|
|
|
echo "==> rebuilding"
|
|
cd "$COMPOSE_DIR"
|
|
docker compose build --pull
|
|
|
|
echo "==> restarting"
|
|
docker compose up -d
|
|
|
|
echo "==> waiting for health"
|
|
for _ in $(seq 1 30); do
|
|
if curl -fsS -m 5 http://127.0.0.1:8410/api/health >/dev/null 2>&1; then
|
|
echo "healthy: $(curl -fsS -m 5 http://127.0.0.1:8410/api/health)"
|
|
exit 0
|
|
fi
|
|
sleep 3
|
|
done
|
|
echo "did not come healthy within 90s — check: docker compose logs --tail 50" >&2
|
|
exit 1
|