vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(elway): probe NOPASSWD with sudo -n true, never sudo -n -v

Browse Source 
`sudo -v` refreshes the auth timestamp, and a NOPASSWD-only rule creates
no timestamp to refresh, so on sudo >= 1.9.15 `sudo -n -v` returns
non-zero while every real command runs passwordless. Measured:

  pfi-gx10    sudo 1.9.15p5   sudo -n -v rc=1   sudo -n true rc=0
  nh3-docker  sudo 1.9.13p3   sudo -n -v rc=0   sudo -n true rc=0
  ana-docker  sudo 1.9.13p3   sudo -n -v rc=0   sudo -n true rc=0
  irv-ml1     sudo 1.9.13p3   sudo -n -v rc=0   sudo -n true rc=0

Only pfi-gx10 is new enough to hit it today, but every host does as it
moves past 1.9.13, and the failure mode is bad: elway prompts for a
password on a host with working NOPASSWD sudo, which in a
non-interactive run is an EOFError partway through a playbook.

The same probe in my own notes cost this session directly. gx10 looked
like a fleet exception with no NOPASSWD sudo when it had it from account
creation, and the operator was asked for a password that was never
needed. Corrected in auto-memory too.

Also lands the gx10 privileged outfit playbook, now green at 5/5:
NOPASSWD sudo, nvcc, docker group, a CUDA container seeing the GB10, and
the userspace torch stack still working afterward.
This commit is contained in:
vh
2026-09-01 23:37:42 -07:00
parent 0b48517909
commit c0e352a47b
3 changed files with 84 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+10 -2
View File
@@ -133,8 +133,16 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
**first use of every kernel pays a JIT compile** and cubin-only libraries will fail.
⚠⚠ **ALWAYS WARM UP BEFORE TIMING** — an unwarmed benchmark read 27 TFLOP/s (it was timing the
JIT) and nearly became a phantom "tensor cores are broken" report.
🔴 **BLOCKED ON SUDO** for nvcc / container toolkit / docker group: **infra-ops has NO NOPASSWD
sudo on gx10** (fleet exception) and **no credential is vaulted**.
✅ **PRIVILEGED HALF DONE 2026-09-01** — nvcc, NVIDIA Container Toolkit, docker group; playbook
`gx10-outfit-privileged.yaml` green 5/5 (a CUDA container sees the GB10). **Bare metal, fully
outfitted.** ⚠⚠ **I was NEVER blocked** — infra-ops had NOPASSWD sudo all along; I probed with
**`sudo -n -v`, which is WRONG on sudo >= 1.9.15** (it refreshes a timestamp NOPASSWD never
creates) and asked the operator for a password that did not exist to need. **Always
`sudo -n true`.** Fixed in `scripts/elway` (same probe, would have broken fleet-wide as hosts
pass sudo 1.9.13) and in auto-memory. Password rotated + vaulted at
`pfi-gx10/infra-ops-password` (kept not locked: **no iDRAC on this box**, so it is the only
console break-glass). ⚠ Docker's buildkit DB was **already corrupt** — the restart surfaced it,
did not cause it; cleared `/var/lib/docker/buildkit` (cache, 0 images/containers).
→ `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000
Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores