scripts/upgrade-irv-ml1-cuda: drop pipe-to-head (SIGPIPE under pipefail)
Phase 1 silently aborted right after the 'cuda-drivers candidate' step — before snapshotting, stopping containers, installing the new driver, or rebooting. Symptom: phase2 showed driver still at 570.124.06 and containers with multi-hour uptimes (proof they were never stopped). Cause: `apt-cache policy cuda-drivers | head -10`. head reads its 10 lines and closes the pipe early; that delivers SIGPIPE to apt-cache, which under `set -o pipefail` propagates as a non-zero exit, and `set -e` immediately terminates the remote shell. The same hazard existed in phase2's `docker ps ... | head -15`. Fix is one-line per call site: drop the head. apt-cache policy output for cuda-drivers is short (~10 lines), and docker ps's container list on irv-ml1 is small enough to print fully.
This commit is contained in:
@@ -89,7 +89,11 @@ remote_phase1() {
|
|||||||
|
|
||||||
echo
|
echo
|
||||||
echo "=== cuda-drivers candidate ==="
|
echo "=== cuda-drivers candidate ==="
|
||||||
apt-cache policy cuda-drivers | head -10
|
# Don't `| head` here — apt-cache policy emits enough that head
|
||||||
|
# closes the pipe early and SIGPIPEs apt-cache, which under
|
||||||
|
# set -o pipefail aborts the whole script before we get to the
|
||||||
|
# actual install + reboot. Output is short anyway.
|
||||||
|
apt-cache policy cuda-drivers
|
||||||
|
|
||||||
echo
|
echo
|
||||||
echo "=== snapshotting current nvidia packages (rollback reference) ==="
|
echo "=== snapshotting current nvidia packages (rollback reference) ==="
|
||||||
@@ -133,7 +137,8 @@ remote_phase2() {
|
|||||||
sleep 8
|
sleep 8
|
||||||
echo
|
echo
|
||||||
echo "=== container status ==="
|
echo "=== container status ==="
|
||||||
docker ps --format "table {{.Names}}\t{{.Status}}" | head -15
|
# `| head -15` would SIGPIPE docker ps under pipefail; just print all.
|
||||||
|
docker ps --format "table {{.Names}}\t{{.Status}}"
|
||||||
}
|
}
|
||||||
|
|
||||||
remote_rollback() {
|
remote_rollback() {
|
||||||
|
|||||||
Reference in New Issue
Block a user