scripts/upgrade-irv-ml1-cuda: drop pipe-to-head (SIGPIPE under pipefail)
Phase 1 silently aborted right after the 'cuda-drivers candidate' step — before snapshotting, stopping containers, installing the new driver, or rebooting. Symptom: phase2 showed driver still at 570.124.06 and containers with multi-hour uptimes (proof they were never stopped). Cause: `apt-cache policy cuda-drivers | head -10`. head reads its 10 lines and closes the pipe early; that delivers SIGPIPE to apt-cache, which under `set -o pipefail` propagates as a non-zero exit, and `set -e` immediately terminates the remote shell. The same hazard existed in phase2's `docker ps ... | head -15`. Fix is one-line per call site: drop the head. apt-cache policy output for cuda-drivers is short (~10 lines), and docker ps's container list on irv-ml1 is small enough to print fully.
This commit is contained in:
@@ -89,7 +89,11 @@ remote_phase1() {
|
||||
|
||||
echo
|
||||
echo "=== cuda-drivers candidate ==="
|
||||
apt-cache policy cuda-drivers | head -10
|
||||
# Don't `| head` here — apt-cache policy emits enough that head
|
||||
# closes the pipe early and SIGPIPEs apt-cache, which under
|
||||
# set -o pipefail aborts the whole script before we get to the
|
||||
# actual install + reboot. Output is short anyway.
|
||||
apt-cache policy cuda-drivers
|
||||
|
||||
echo
|
||||
echo "=== snapshotting current nvidia packages (rollback reference) ==="
|
||||
@@ -133,7 +137,8 @@ remote_phase2() {
|
||||
sleep 8
|
||||
echo
|
||||
echo "=== container status ==="
|
||||
docker ps --format "table {{.Names}}\t{{.Status}}" | head -15
|
||||
# `| head -15` would SIGPIPE docker ps under pipefail; just print all.
|
||||
docker ps --format "table {{.Names}}\t{{.Status}}"
|
||||
}
|
||||
|
||||
remote_rollback() {
|
||||
|
||||
Reference in New Issue
Block a user