From ea0642f9420f21d535f6870192af5bb00a471d3a Mon Sep 17 00:00:00 2001 From: ScottW514 Date: Fri, 14 Aug 2026 21:14:57 -0400 Subject: [PATCH] Record live-fire drill results: emission witness, HV, lid-IR, X-3, G-10 Phase 5 A-1 emission witness and A-5 HV telemetry pass on live burns; A-2 lid-IR characterized as a weak signal at 40% (gate left watch-only); pgood confirmed unusable as a witness on this PSU. Phase 4 X-3 (0.1s job-based disarm on M2) and G-10 (disarm counts down in Hold) pass. Adds live_fire_drills.py (arm-lifecycle sampler over TCP + HTTP). --- docs/BRINGUP.md | 80 ++++++- scripts/bench/live_fire_drills.py | 362 ++++++++++++++++++++++++++++++ 2 files changed, 435 insertions(+), 7 deletions(-) create mode 100644 scripts/bench/live_fire_drills.py diff --git a/docs/BRINGUP.md b/docs/BRINGUP.md index 0b2454b..d3ef961 100644 --- a/docs/BRINGUP.md +++ b/docs/BRINGUP.md @@ -112,13 +112,79 @@ MOTION and re-probed rather than false-passed, and the second probe returned MOTION OK (p2p x=3919, y=1636) — the DRV8825s are not wedged after the drill session's rail cycles. -**Still pending — operator-present:** the connection-flood -`machine_is_idle` fail-closed check under motion (X-2); the stale-gate -/ motion-integrity armed drills across Phases 4 and 6 (physical -button); and the Phase 5 live-fire work — emission-witness live check -and lid-IR fire characterization to set `cool_fire_ir_delta` from the -ambient baseline below. Live fire only with the operator armed: eye -protection, fire watch, exhaust running. +**X-2 connection-flood robustness exercised (dry):** a 500-connection +slow-drip flood from a second LAN host drove forgectrl from 7 to a peak +of 379 open fds, where it plateaued — MHD's own connection handling +caps concurrency far below the raised 4096 `RLIMIT_NOFILE`, so the +flood could not manufacture the EMFILE that the X-2 fix guards against. +The daemon never crashed, the kernel `cnc/state` stayed readable +throughout (two local `/status` probes timed out at the peak and +recovered within a second), and it returned to 7 fds with `/status` +`200` after the flood drained. The fail-closed branch itself +(`machine_is_idle()` returns busy on any `rd_attr` failure) is present +in `status.c` and covered by the `-Werror` CI build; distinctly +exercising the EMFILE→busy path wants a host unit test that injects the +read failure (the clean X-2 closure — the runtime flood cannot reach it +while MHD caps connections below the fd limit). Note for F-15/X-6: the +absence of an explicit `MHD_OPTION_CONNECTION_LIMIT` + per-IP cap is +still the deferred half; the default ceiling held here but a per-IP cap +remains the right hardening. + +**Live-fire drills PASS (operator armed, S400/40% vector marks on +scrap, `scripts/bench/live_fire_drills.py`):** + +- **Phase 5 A-1 emission witness — PASS.** On a commanded fire window + `cnc/laser_on_sampled` (surfaced as `/status` `laser.emission_samples`) + goes to its full 255 count and returns to 0 at Idle, across two + separate burns. This is the reliable live-emission witness. +- **Phase 5 A-5 HV telemetry — PASS.** `pic/hv_current` (`hv_current_raw`) + tracks the cut: 0 at idle, 0→1023/661/482 raw during the three burns + (the tube draws real current). The only HV witness on this PSU — + `hv_voltage` is grounded, as the audit noted. +- **Phase 5 A-2 lid IR — characterized, gate left watch-only.** A 40 % + vector cut lifts the four `pic/lid_ir` channels only ~+3 counts over + the ambient baseline (37/36/40/40 → peaks ~40/39/42/43) — barely above + the ±3-count ambient noise, i.e. a weak fire signal at this power. + `cool_fire_ir_delta` therefore stays 0 (watch-only) until a + representative high-power job is characterized; a real ignition flare + is far brighter than a cut, so the eventual threshold sits well above + both the cut delta and the noise (a floor near 15 counts is the + working target, not yet committed). forgectrl's per-job telemetry line + logs baseline/peak for all four channels. +- **pgood is not a usable witness on this PSU.** `cnc/laser_pgood_sampled` + stayed 0 (forgectrl reads <128 as "not good") through every burn even + while `hv_current` rail'd and the tube cut — so A-1's "surface + laser_pgood loss" warning is a false alarm on this hardware and must + be gated/suppressed here (or documented as expected); the emission and + HV witnesses are the trustworthy ones. Recorded for the A-1 follow-up. +- **Phase 4 X-3 job-based disarm — PASS.** A job ending in `M2` + (program end, as LightBurn sends) disarms in **0.1 s** at Idle; a job + with no program end falls back to the ~60 s `laser_disarm_s` idle + grace (measured 56.8 s). The window is job-based, not 60-s-idle-based. +- **Phase 4 G-10 disarm-in-Hold — PASS.** Armed, fired a +X move, + feed-held mid-move (`Hold:1`); the disarm grace counts down while held + and closes the window at 61.3 s (the bug left a job abandoned in Hold + armed for hours). + +**Live defect caught and fixed by the campaign:** the liveness probe's +enclosure guard read the combined-doors EV_SW bit with inverted sense +(bit 3 set = *closed*, per the switch map; the guard treated set as +*open*), so the wedge probe skipped on every spawn with the lid closed +and would have moved the gantry with it open. Fixed in forgectrl +`424f185`, hot-deployed; the probe then ran for real and behaved as +designed — a gray-zone first read (head-accel p2p x=455, below the ≥500 +threshold) was retried rather than false-passed, and the retry returned +MOTION OK (p2p x=3919, y=1636). + +**Deferred (fiddly or config-dependent, not blocking):** G-4 (arm +re-checks the coolant fire gate after the button wait) wants the pump +killed in the instant after the press — cleanest as a host unit test; +the code re-check is in place. Phase 6's "armed job refuses at the stale +origin after an underrun" is config-dependent (GRBL mode permits +unhomed cutting), and the core underrun behavior — `pulse data +underrun; position no longer trusted` with the homing anchor unlinked — +is already logged in the dry dead-man drills above. Live fire only with +the operator armed: eye protection, fire watch, exhaust running. The lid-IR **ambient baseline** for the fire-watch characterization is captured on this image (600 samples over 5.6 min at 2 Hz, lid closed, diff --git a/scripts/bench/live_fire_drills.py b/scripts/bench/live_fire_drills.py new file mode 100644 index 0000000..bec54dd --- /dev/null +++ b/scripts/bench/live_fire_drills.py @@ -0,0 +1,362 @@ +#!/usr/bin/env python3 +"""Live-fire bench drills - Phases 4, 5, 6. Runs from a LAN host against +grblHAL over TCP (argv[1] or GF_HOST, port 23) and forgectrl over HTTP +(:8080). LIVE LASER: the operator must be armed with eye protection, a +fire watch, an extinguisher, and the exhaust running. Every drill waits +for the operator to press the physical arm button before the machine +fires; nothing here defeats that gate. + +Drills (pass a name): + witness Phase 5 A-1/A-2/A-5: a short vector mark at S400. Samples + forgectrl /status (emission_samples, lid_ir[], hv_current) + and /cool/status (armed, fire_watch) at ~8 Hz through the + job. PASS: emission_samples goes nonzero during the fire + window and returns to 0; lid_ir peak recorded per channel + vs the ambient baseline; hv_current range logged. Also + asserts X-3: armed drops at Idle with the job close, not at + +60 s. + hold Phase 4 G-10: arm + start a longer job, feed-hold mid-run, + then hold. PASS: the disarm grace counts down in Hold and + the window closes (armed -> false) without the job resuming. + faultpos Phase 6 G-2/G-3: after a run that was stopped by an + underrun (position no longer trusted), a subsequent armed + job must refuse to cut at the stale origin - the sender + alarms and re-home is required. Reads homed via /status. + +The G-4 arm-refuses-when-a-fire-gate-is-active drill is operator-manual +(kill the pump during the button wait); this harness prints the cue. +""" +import json +import os +import socket +import sys +import time +import urllib.request + +HOST = sys.argv[2] if len(sys.argv) > 2 else os.environ.get('GF_HOST') +if not HOST: + raise SystemExit('usage: live_fire_drills.py [host] (or set GF_HOST)') +PORT = 23 +BASE = 'http://%s:8080' % HOST + +# Ambient lid-IR baseline (2026-08-14, lid closed, idle): per-channel means. +IR_BASELINE = [37.3, 36.3, 39.5, 40.0] + + +def get_json(path): + with urllib.request.urlopen(BASE + path, timeout=4) as r: + return json.load(r) + + +class Grbl: + def __init__(self, host, port): + self.s = socket.create_connection((host, port), timeout=5) + self.s.settimeout(0.2) + self.buf = b'' + time.sleep(0.5) + self.drain() + + def drain(self): + try: + while True: + d = self.s.recv(4096) + if not d: + break + self.buf += d + except socket.timeout: + pass + out, self.buf = self.buf, b'' + return out.decode('ascii', 'replace') + + def cmd(self, line, timeout=5.0): + self.s.sendall(line.encode() + b'\n') + deadline = time.time() + timeout + text = '' + while time.time() < deadline: + text += self.drain() + if 'ok' in text or 'error' in text or 'ALARM' in text: + return text.strip() + time.sleep(0.02) + return '(timeout) ' + text.strip() + + def status(self): + self.s.sendall(b'?') + t = time.time() + 1.0 + text = '' + while time.time() < t: + text += self.drain() + if '>' in text: + break + time.sleep(0.02) + if '<' in text and '>' in text: + return text[text.rfind('<'):text.rfind('>') + 1] + return '' + + def state(self): + st = self.status() + return st[1:].split('|')[0] if st else '' + + def rt(self, ch): + self.s.sendall(ch) + + def wait_state(self, want, timeout=60.0, poll=0.1): + deadline = time.time() + timeout + while time.time() < deadline: + s = self.state() + if s.startswith(want): + return s + time.sleep(poll) + return self.state() + + +def sample_forgectrl(): + """One combined /status + /cool/status sample, or None on error.""" + try: + st = get_json('/status') + cs = get_json('/cool/status') + except Exception: + return None + return { + 't': time.time(), + 'kstate': st.get('state'), + 'emission': st.get('laser', {}).get('emission_samples'), + 'pgood': st.get('laser', {}).get('pgood_samples'), + 'faults': st.get('faults'), + 'hv': st.get('hv_current_raw'), + 'ir': st.get('lid_ir'), + 'armed': cs.get('armed'), + 'fire_watch': cs.get('fire_watch'), + 'verdict': cs.get('verdict'), + 'phase': cs.get('phase'), + 'reason': cs.get('reason'), + } + + +def arm_cue(): + print('\n>>> OPERATOR: eye protection on, exhaust running, fire watch,') + print('>>> extinguisher in reach, scrap under the head with room to move.') + print('>>> The job is starting. The white button will light and the') + print('>>> stream will BLOCK until you press the physical arm button.') + print('>>> The machine fires only after your press.\n') + + +def run_and_sample(g, gcode_lines, sample_hz=8, overall_timeout=200): + """Stream gcode; sample forgectrl through the whole arm -> fire -> + disarm lifecycle. The arm phase (fan run + flow interrogation) plus + the operator button wait present grblHAL as Idle, so completion must + NOT trigger on an early Idle. Complete on one of: + - real fire captured: emission was seen > 0, then grbl Idle > 3 s; + - no-fire disarm: armed went True then False, grbl Idle, > 15 s in; + - overall timeout. + """ + samples = [] + period = 1.0 / sample_hz + s0 = sample_forgectrl() + if s0: + samples.append(s0) + for ln in gcode_lines: + g.s.sendall(ln.encode() + b'\n') + t_start = time.time() + next_t = t_start + seen_emission = False + seen_armed = False + disarmed_now = False + idle_since = None + while time.time() - t_start < overall_timeout: + now = time.time() + if now >= next_t: + smp = sample_forgectrl() + if smp: + samples.append(smp) + if smp['emission'] and smp['emission'] > 0: + seen_emission = True + if smp['armed']: + seen_armed = True + disarmed_now = seen_armed and not smp['armed'] + next_t = now + period + st = g.state() + if st.startswith('Idle'): + if idle_since is None: + idle_since = now + idle_for = now - idle_since + if seen_emission and idle_for > 3.0: + break # captured the burn + if disarmed_now and (now - t_start) > 15 and idle_for > 3.0: + break # armed then disarmed, no fire + else: + idle_since = None + time.sleep(0.05) + return samples + + +def prepare(g): + """Guarantee a clean Idle start: clear a latched Door hold (lid was + opened to inspect) or an Alarm before the run.""" + st = g.status() + if 'Door' in st or 'Hold' in st: + g.rt(b'\x18') # soft reset clears the hold + time.sleep(2) + g.drain() + st = g.status() + if 'Alarm' in st: + print('unlock: %s' % g.cmd('$X')) + st = g.status() + return st + + +def drill_witness(g): + print('=== Phase 5 witness drill: S400 vector mark ===') + print('connect: %s' % prepare(g)) + base = sample_forgectrl() + print('pre-fire: %s' % base) + arm_cue() + # A small square outline at S400, motion-only feed so the fire window + # is unambiguous. Absolute-relative: use G91 so no homing is needed. + job = [ + 'G91', 'G21', # relative, mm + 'M4', # dynamic laser mode, spindle enable + 'S400', + 'G1 X20 F600', + 'G1 Y20 F600', + 'G1 X-20 F600', + 'G1 Y-20 F600', + 'M5', # laser off + 'G90', + 'M2', # program end: X-3 job-based disarm trigger + ] + samples = run_and_sample(g, job) + # Analysis. + emis = [s['emission'] for s in samples if s['emission'] is not None] + peak_emis = max(emis) if emis else 0 + end_emis = emis[-1] if emis else None + ir_peak = [0, 0, 0, 0] + hv_vals = [] + for s in samples: + if s['ir'] and len(s['ir']) == 4: + for i in range(4): + ir_peak[i] = max(ir_peak[i], s['ir'][i]) + if s['hv'] is not None: + hv_vals.append(s['hv']) + armed_seen = any(s['armed'] for s in samples) + pgood_vals = [s['pgood'] for s in samples if s['pgood'] is not None] + print('\n--- results ---') + print('emission_samples: peak=%s end=%s (PASS if peak>0 and end==0)' + % (peak_emis, end_emis)) + print('pgood_samples during job: peak=%s (>=128 = power-good)' + % (max(pgood_vals) if pgood_vals else '-')) + print('lid_ir peak=%s vs baseline=%s delta=%s' + % (ir_peak, IR_BASELINE, + [round(ir_peak[i] - IR_BASELINE[i], 1) for i in range(4)])) + print('hv_current range: %s..%s' % (min(hv_vals) if hv_vals else '-', + max(hv_vals) if hv_vals else '-')) + print('armed observed during job: %s' % armed_seen) + # X-3: measure time-to-disarm after the job completes at Idle. The + # job-based window (Phase 4) should disarm promptly at Idle entry, + # not wait out the ~60 s laser_disarm_s grace. + t0 = time.time() + disarm_dt = None + while time.time() - t0 < 75: + s = sample_forgectrl() + if s and not s['armed']: + disarm_dt = time.time() - t0 + break + time.sleep(1) + print('X-3 time-to-disarm after Idle: %s s (job-based PASS if prompt, ' + 'not ~60 s grace)' + % (round(disarm_dt, 1) if disarm_dt is not None else '>75 (REVIEW)')) + ok = peak_emis > 0 and end_emis == 0 + print('WITNESS emission %s' % ('PASS' if ok else 'REVIEW - see values above')) + # Emit the recommended cool_fire_ir_delta floor. + worst = max(ir_peak[i] - IR_BASELINE[i] for i in range(4)) + print('suggest cool_fire_ir_delta >= max(15, %.0f) once several jobs ' + 'confirm the peak delta' % (2 * worst if worst > 0 else 15)) + return samples + + +def drill_hold(g): + print('=== Phase 4 G-10 drill: disarm grace counts down in Hold ===') + print('connect: %s' % prepare(g)) + arm_cue() + # +X move at F300 (~5 mm/s). Held after ~2 s of motion (~10 mm), + # so ~30 mm of +X clearance is plenty. + job = ['G91', 'G21', 'M4', 'S400', 'G1 X40 F300'] + for ln in job: + g.s.sendall(ln.encode() + b'\n') + print('armed; waiting for motion to start (arm + your button press)...') + st = g.wait_state('Run', 180) # arming + button wait, then motion + if not st.startswith('Run'): + print('FAIL: motion never started (state=%s) - arm refused or no press' + % st) + g.cmd('M5', timeout=1) + g.rt(b'\x18') + return + print('moving under laser: %s; feed-hold in 2 s' % st) + time.sleep(2) + g.rt(b'!') # feed hold mid-move + st = g.wait_state('Hold', 5) + print('feed-held mid-move: %s' % st) + print('watching the disarm grace count down IN HOLD (not resuming)...') + t0 = time.time() + disarmed_at = None + while time.time() - t0 < 120: + s = sample_forgectrl() + held = g.state().startswith('Hold') + if s and not s['armed']: + disarmed_at = time.time() - t0 + break + if not held: + print('note: left Hold (state=%s) before disarm' % g.state()) + time.sleep(1) + # Recover: laser off, abort out of hold. + g.cmd('M5', timeout=1) + g.rt(b'\x18') # soft reset / abort out of hold + time.sleep(1) + print('G-10 disarmed in Hold after %s s (PASS if it disarms while held; ' + 'the bug left it armed for hours)' + % (round(disarmed_at, 1) if disarmed_at else 'NOT WITHIN 120 - REVIEW')) + if 'Alarm' in g.status(): + g.cmd('$X') + + +def drill_faultpos(g): + print('=== Phase 6 G-2/G-3 drill: stale origin refused after underrun ===') + s = get_json('/status') + print('homed=%s (an underrun should have cleared this)' % s.get('homed')) + if s.get('homed'): + print('NOTE: homed is still true - run the SIGSTOP/underrun drill ' + 'first, then re-run this to confirm the refusal.') + return + print('attempting an armed cut at the stale origin - it must refuse/alarm') + prepare(g) + arm_cue() + r = g.cmd('M4 S400', timeout=2) + r2 = g.cmd('G1 X10 F300', timeout=3) + st = g.state() + print('controller response: %s / %s state=%s' % (r, r2, st)) + print('FAULTPOS %s' % ('PASS (refused/alarm at stale origin)' + if ('error' in (r + r2).lower() or 'Alarm' in st) else + 'REVIEW - cut was accepted; check G-3 anchor invalidation')) + g.cmd('M5', timeout=1) + + +def main(): + drill = sys.argv[1] if len(sys.argv) > 1 else '' + drills = {'witness': drill_witness, 'hold': drill_hold, + 'faultpos': drill_faultpos} + if drill not in drills: + print(__doc__) + return 2 + g = Grbl(HOST, PORT) + try: + drills[drill](g) + finally: + # Always leave the laser commanded off. + try: + g.cmd('M5', timeout=1) + except Exception: + pass + return 0 + + +if __name__ == '__main__': + sys.exit(main())