Triton builds its CUDA-utils shim with gcc the first time a kernel runs,
and needs Python.h to do it. Without python3-dev the box looks entirely
healthy: torch imports, the 49 GB base loads, LoRA attaches with the
right parameter count, and then the first training step dies with a bare
CalledProcessError naming a gcc invocation and an exit code.
The real message -- "fatal error: Python.h: No such file or directory" --
is discarded, because Triton sends the compiler's stdout to DEVNULL. It
cost a probe run and a model load to find something a one-line manual
compile answered immediately.
Same shape as the dots-tts container needing a C compiler at runtime: a
JIT dependency invisible at install time that only surfaces under load.
The verify step runs the compile rather than checking the package is
present, because `dpkg -l python3-dev` would pass while the compile still
failed on a missing library path or header directory.
The userspace half is done and needed no root. This covers what does:
NOPASSWD sudo for infra-ops (ending the gx10 fleet exception), the docker
group, the CUDA toolkit for nvcc, and the NVIDIA Container Toolkit wired
into dockerd.
Unrun -- it needs one interactive invocation to supply the sudo password,
because no gx10 credential is vaulted and elway prompts via getpass. After
its first step lands, the box stops being an exception and later runs are
unattended.
Guards worth noting: the sudoers drop-in is validated with visudo -cf
before install, since a malformed one locks every sudo user out of a box
sitting on a desk with no out-of-band access; and cuda-toolkit is
installed rather than the cuda metapackage, so the working 580.173.02
driver is not replaced.
vh