Conditions ran unprivileged no matter what the step declared, and that fails
in the dangerous direction. A root-requiring `when:` -- `pvesh get ...` exits
255 for a non-root user -- returns non-zero, elway reports the step `skipped`,
and a playbook that never performed its change reports overall OK. "Skipped"
is indistinguishable from working idempotency, so the run looks correct.
Found the hard way on esh-pve: three consecutive runs of an exclusion playbook
reported success while changing nothing. Only the verify phase caught it, by
continuing to report the thing the steps claimed to have handled -- which is
exactly why verify runs unconditionally.
`creates:` had the same fault from the other side: a path under /root is
unreadable to the login user, so `test -e` said absent and the step re-ran
every time. It now correctly reports the file as already present.
Sudo-less steps are unaffected: their conditions still evaluate as the login
user, which is what they mean. Only a step that declares sudo: true gets
privileged condition evaluation, so no existing playbook changes meaning
unless it was already silently broken.
infra-ops and infra-hermes act as the same OS identity and dockerd does not
log exec per caller, so host-side changes carry no fingerprint. Git cannot
close the gap either: every commit here is attributed to Vuong Hoang by
convention, which is correct for authorship and useless for attribution.
On 2026-09-18 a second session edited the searxng stack mid-deploy, crash-
looping fleet search for ~4 minutes, and the author was unidentifiable.
scripts/ops-log records one line per host-changing action and holds a
lightweight claim so two agents do not deploy the same stack at once.
Four design questions, settled:
* Central on nh3-dev, not per-host and not the post office. Both agents
run as the same unix user there, so one file is shared with zero
provisioning. Per-host needs a writable path on ~25 heterogeneous boxes
and stores "we changed host Y" on host Y. journald looked free but shows
an unprivileged reader only their own _UID, which would have split the
log silently between the infra-ops and lkraven halves of the fleet.
* The claim is advisory and enforced in the tooling. deploy-stack.sh
refuses a foreign claim across the diff, the prompt and the apply -- the
whole review window, which is where the collision happened. Acquire is
mkdir, so it is atomic rather than probably-fine. Stale claims auto-break
and the break is recorded.
* Writers are automatic. deploy-stack.sh and elway record themselves; a log
that depends on remembering is the same class of instrument as a health
check that passes in both states.
* There is a detector. `ops-log audit` asks each host what changed on disk
and compares it to the newest log line for that stack, covering the
manual ssh-and-edit path the automatic writers structurally cannot.
ops-log being absent or broken never blocks a deploy; only a live foreign
claim does. `ops-log baseline` marks the 136 stacks that predate the
instrument so the detector starts from today rather than reporting the whole
fleet forever and training us to ignore it.
An unreachable host reports INCOMPLETE and exit 5, never clean.
`sudo -v` refreshes the auth timestamp, and a NOPASSWD-only rule creates
no timestamp to refresh, so on sudo >= 1.9.15 `sudo -n -v` returns
non-zero while every real command runs passwordless. Measured:
pfi-gx10 sudo 1.9.15p5 sudo -n -v rc=1 sudo -n true rc=0
nh3-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
ana-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
irv-ml1 sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
Only pfi-gx10 is new enough to hit it today, but every host does as it
moves past 1.9.13, and the failure mode is bad: elway prompts for a
password on a host with working NOPASSWD sudo, which in a
non-interactive run is an EOFError partway through a playbook.
The same probe in my own notes cost this session directly. gx10 looked
like a fleet exception with no NOPASSWD sudo when it had it from account
creation, and the operator was asked for a password that was never
needed. Corrected in auto-memory too.
Also lands the gx10 privileged outfit playbook, now green at 5/5:
NOPASSWD sudo, nvcc, docker group, a CUDA container seeing the GB10, and
the userspace torch stack still working afterward.
Avoids the "paste the full output" friction. Every elway run now
writes its full streamed output to two files in addition to the
terminal:
/tmp/elway-last.log
always overwritten — the easy "what just happened" target
~/.cache/elway/runs/<ts>-<host>-<playbook>.log
timestamped permanent record; accumulates across runs
Implementation: small _Tee class wraps sys.stdout for the duration
of main() so all `print(...)` calls fan out to the real terminal +
both file handles. Subprocess output already goes through print()
via _stream_process, so the build/healthz/etc. text is captured.
ANSI color codes are kept in the file so colors are preserved in
log viewers that handle them (less -R, modern tail). Strip with
`sed 's/\x1b\[[0-9;]*m//g'` for paste-elsewhere.
New flags:
--log <path> override path; replaces both default destinations
--no-log terminal-only, skip both files
Path of the permanent log is printed at the top of every run so
you know where it landed without remembering the timestamp pattern.
Previously the startup logic ran `probe_sudo()` if any step in the
playbook declared `sudo: true`, regardless of whether that step's
creates:/when: gates would actually let it fire. The result on the
task-board deploy rerun was a spurious password prompt followed by
six SKIPPED lines — the prompt served no purpose.
New flow:
- Remove the upfront probe in main().
- SSHContext.sudo_password defaults to None; new sudo_probed flag
tracks whether we've already prompted this session.
- run_shell_step + run_upload_step call ensure_sudo(ctx) only at
the point a sudo step is actually executing — i.e. after its
skip conditions have been evaluated and passed. Idempotent:
probes at most once per playbook run.
Tradeoff accepted: if the user fat-fingers the password, they see it
mid-run on the first sudo step rather than upfront. `stop_on_fail`
(default true) halts cleanly; they rerun. Lower friction for the
common idempotent-rerun case, same recoverability.
Verified against playbooks/deploy-task-board.yaml — prior run
prompted + completed in 1.7s; new run completes in 1.7s with no
prompt because every sudo step skip-gated.
Tier 1 — pre-step skip conditions:
when: <remote shell expr> skip unless expr exits 0
creates: <remote path> skip if path already exists
removes: <remote path> skip if path is already absent
Any of the three saying "skip" marks the step `skipped` and moves on.
Evaluated under bash -c on the remote so `!`, `[[`, pipes etc. behave
consistently regardless of the default remote shell.
Tier 2 — post-step change detection:
changed_when: <remote shell expr>
Evaluated after a successful step. Exit 0 → step counts as `changed`
(default). Exit != 0 → `ok` (ran, nothing actually different).
Without this field, successful steps default to `changed`, matching
Ansible's shell/command defaults. Useful on verify steps:
`changed_when: "false"` reports them as `ok` since they only attest.
Status model moved from pass/fail to four states:
ok / changed / failed / skipped
Summary reports each count; overall outcome is CHANGED if any step
changed, OK if none did, FAILED on any non-skipped failure.
Rerunnable smoke: playbooks/elway-smoke.yaml now proves it. On a
clean target the cold run reports 4 changed, 3 ok. Rerunning with
the same vars reports 2 skipped / 2 changed (upload + log-record
have no idempotency hooks and are always `changed`). Overriding
--var greeting=... re-runs the gated step exactly as intended.
Doc block at the top of the script updated with the new schema
fields and state machine.
`scripts/elway` is a ~600-line Python tool (stdlib + python3-yaml) for
driving one-off ssh commands, ad-hoc file uploads, and YAML playbooks
against a single host. Fills the gap between "single ssh one-liner"
and "reach for Ansible."
Highlights:
- Three invocation modes: --shell, --upload (LOCAL:REMOTE[:MODE]),
and --playbook <path>
- Playbook schema: inline vars, list of steps, optional verify block.
Template via {{ var }}; CLI --var overrides inline defaults
- stop_on_fail global (default on), per-step override. Verify phase
always runs, even after a halt — you see end-state regardless
- Sudo handled once: probes NOPASSWD; if not, prompts locally via
getpass, validates up-front, then feeds via `sudo -S` per step.
Password never written to disk/logs. Upload-with-sudo stages to
/tmp then sudo-mv + sudo-chmod
- SSH connection reuse via ControlMaster (60s persist) keeps
multi-step playbooks responsive (~30ms/step reuse vs ~550ms cold)
- Live interleaved stdout/stderr with per-step prefix and colored
pass/fail summary. --dry-run prints the plan without executing
- Shebang pinned to /usr/bin/python3 to bypass venv-shadowing
when python3-yaml lives in the system site-packages
Smoke test (playbooks/elway-smoke.yaml) covers vars + upload + verify;
drove out a YAML-scalar-coercion bug before first commit (`shell: false`
parsed to Python bool, crashed the templater — now coerced to string
at load time with a clear error on nulls).
vh