fix(pfi-gx10): install python3-dev — Triton JIT-compiles C at first use
Triton builds its CUDA-utils shim with gcc the first time a kernel runs, and needs Python.h to do it. Without python3-dev the box looks entirely healthy: torch imports, the 49 GB base loads, LoRA attaches with the right parameter count, and then the first training step dies with a bare CalledProcessError naming a gcc invocation and an exit code. The real message -- "fatal error: Python.h: No such file or directory" -- is discarded, because Triton sends the compiler's stdout to DEVNULL. It cost a probe run and a model load to find something a one-line manual compile answered immediately. Same shape as the dots-tts container needing a C compiler at runtime: a JIT dependency invisible at install time that only surfaces under load. The verify step runs the compile rather than checking the package is present, because `dpkg -l python3-dev` would pass while the compile still failed on a missing library path or header directory.
This commit is contained in:
@@ -73,6 +73,18 @@ steps:
|
||||
shell: DEBIAN_FRONTEND=noninteractive apt-get install -y -qq cuda-toolkit
|
||||
when: "! test -x /usr/local/cuda/bin/nvcc"
|
||||
|
||||
- name: Install python3-dev + build-essential (Triton JITs C at RUNTIME)
|
||||
# Triton compiles its CUDA-utils shim with gcc on FIRST USE and needs
|
||||
# Python.h to do it. Without python3-dev, torch imports fine, the model
|
||||
# loads fine, LoRA attaches fine -- and then the first training step dies
|
||||
# with a CalledProcessError from gcc whose real message ("fatal error:
|
||||
# Python.h: No such file or directory") is swallowed, because Triton
|
||||
# sends the compiler's stdout to DEVNULL. Cost a probe run to find.
|
||||
# Same shape as the dots-tts container needing a C compiler at runtime.
|
||||
sudo: true
|
||||
shell: DEBIAN_FRONTEND=noninteractive apt-get install -y -qq python3-dev build-essential
|
||||
when: "! test -f /usr/include/python3.12/Python.h"
|
||||
|
||||
- name: Install the NVIDIA Container Toolkit
|
||||
sudo: true
|
||||
shell: DEBIAN_FRONTEND=noninteractive apt-get install -y -qq nvidia-container-toolkit
|
||||
@@ -101,6 +113,18 @@ verify:
|
||||
shell: sg docker -c 'docker run --rm --gpus all nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi -L' | grep -q GB10
|
||||
changed_when: "false"
|
||||
|
||||
- name: Triton can actually compile its runtime shim
|
||||
# Asserting the effective behaviour. `dpkg -l python3-dev` would pass while
|
||||
# the compile still failed for a missing header or library path.
|
||||
shell: |
|
||||
V=/home/infra-ops/ml/.venv/lib/python3.12/site-packages/triton/backends/nvidia
|
||||
test -d "$V" || exit 0
|
||||
gcc "$V/driver.c" -O3 -shared -fPIC -Wno-psabi -o /tmp/.tritoncheck.so \
|
||||
-l:libcuda.so.1 -L"$V/lib" -L/lib/aarch64-linux-gnu \
|
||||
-I"$V/include" -I/usr/include/python3.12
|
||||
rm -f /tmp/.tritoncheck.so
|
||||
changed_when: "false"
|
||||
|
||||
- name: The userspace torch stack still sees the GPU
|
||||
# Guards against a toolkit install disturbing the working driver.
|
||||
shell: /home/infra-ops/ml/.venv/bin/python -c 'import torch;assert torch.cuda.is_available()'
|
||||
|
||||
Reference in New Issue
Block a user