forgefirm/scripts/bench

ForgeFIRM bench tools

Hardware-verification tools for the ForgeFIRM bench. All run ON the target board (dev image, python3 present) unless noted. The dev image installs them under /usr/share/forgetest/bench/, and the acceptance tool's Bench diagnostics tab (http://<machine>:8090/#bench, the site, Developers, "Acceptance") runs them with their arguments and the output on the page - takeover and scope tools get forgectrl and the controller stopped and started around the run, live tools need the operator acknowledgment; the acceptance catalog itself is built from ports of these drills. The tools that also run from a LAN host use gfbench.py: GF_HOST names the machine (sysfs through ssh - the ssh on PATH, or the client named by GF_SSH, for example GF_SSH='wsl -d <distro> -- ssh' from Windows - Grbl and forgectrl over the LAN); with GF_HOST unset they run on the board itself (sysfs directly, everything on 127.0.0.1), which is how the bench page runs them, with their data files under /data/forgetest/bench/ (FORGETEST_BENCH_DATA; next to the tool otherwise) and the panel token in GF_TOKEN. Tools that drive the thermal hardware directly (the flow characterization family) run with forgectrl - the thermal-hardware owner - and the controller stopped: the page's takeover does that; from a host, stop them first.

Tool	Purpose
feeder.c	Underrun proof: streams NOP pulse bytes to /dev/glowforge with wall-clock pacing, bounded queue depth, deadman flock, SCHED_FIFO. Usage: feeder <hz> <seconds> <depth_ms>. Proven envelope: 100 kHz × 120 s under full load, 0.2 ms worst write latency. Cross-compile with build-feeder.sh (WSL).
bench_phase2.py	End-of-data protocol bench: underrun detection/ack, parked no-replay guard, resume(0), continuous-feed stability, 20× run/underrun cycles. Motion-safe (motors locked, laser latched).
check_pwm.py	Laser PWM register check: reads PWM2 PWMCR/PWMPR via /dev/mem, expects divider 13 × ~127 counts ≈ 40 kHz.
pwm_sweep.py	LASER_PWM scope test (runs on the board): check = read-only safety readbacks + PWM2 dump; sweep = steps PWMSAR through 50/25/75/6/100 % duty with 4 s holds, then restores. Locked state only (controller and forgectrl stopped, the pulse device closed - the page's takeover): the sweep relocks the latch itself and refuses to write if FIRE reads driven or LASER_ON reads active.
pwm_hold.py	Holds one PWMSAR value for a scope-measurement window (pwm_hold.py <sar> <seconds>), then restores. Same locked-state rule and guard.
fire_test.py	FIRE drop-timing scope test (runs on the board): A = latch locked (expects nothing on FIRE/LASER_ON), B = latch unlocked / normal end-of-data, U = true underrun. Duty 0 throughout; refuses to unlock if HV reports good.
pwm_stream_test.py	LASER_PWM stream-path scope test (runs on the board, controller and forgectrl stopped): streams power bytes only (no step bytes, no FIRE bits, motor_lock=15, latch locked) through /dev/glowforge so the scope verifies the real power path, including the run-start duty reset and the consecutive-power-byte drop; position counters compared before/after. Exit 0 = counters unmoved, idle at the end, no FIRE/emission read back.
gate_a_kernel_drills.py	Kernel laser-safety drills (run on the board with forgectrl stopped so the pulse device is free): K1 controlled-stop deceleration floor, K2 resume waypoint honors the locked latch, K3 a mid-ramp latch unlock never re-arms the FIRE drive. Software witnesses (cnc/state, laser_enable, laser_on, laser_on_sampled, interlock bit 3) plus the PSU-connector LASER_ON scope point; K3 refuses to run if HV reports good.
laser_stream_test.py	Host-side laser pulse-stream emission harness: runs the native null-sink controller with GFSINK_DUMP, drives small laser jobs over TCP, and checks the dumped bytes against the kernel feeder contract (leading power byte, no back-to-back power bytes, FIRE only inside cutting moves, every stream ends FIRE-clear, no FIRE on a stepless gap, no FIRE leak across cycle churn, the rapids after an M5 executed at idle ship dark, and the next job in the same process fires at the level the previous one ended at). Runs in the grblHAL repo's CI.
laser_lifecycle_test.py	Host-side operator-armed-window lifecycle harness (null-sink controller): arm once per job with M5/M3 persistence, the M2 close, sender-change re-consent, the disarm grace counting down in Hold, and arm refusal under a blocking cooling verdict. Runs in the grblHAL repo's CI.
z_envelope_test.py	Host-side Z envelope harness (null-sink controller): the Z soft limit belongs to the driver, not to $20, so the driver re-applies sys.work_envelope, sys.homed and sys.soft_limits for Z from the settings-changed chain. Checks that an unreferenced Z is collapsed to where the lens stands and blocks a move each way, that X and Y stay free, and that neither a $20 write (the core clears the soft-limit mask in the setter) nor a $132 write (which un-homes the axis as well) frees Z. Runs in the grblHAL repo's CI.
live_fire_drills.py	LIVE LASER drills, on the board (the bench page) or from a LAN host (GF_HOST): live_fire_drills.py <drill> [S] [F] - witness (emission witness, lid-IR peaks vs the ambient baseline, HV current, job-based disarm on M2), hold (disarm grace in Hold), faultpos (armed job refuses a stale origin after an underrun), ircut (lid-IR characterization cut at S/F), pthresh (laser power-threshold ladder: 13 constant-power rungs from 2 % to 30 % of full on scrap; the lowest rung that marks is the tube's striking threshold and reads directly as the $35 value - requires $35 = 0 for the run), dladder (density ladder at a chosen base period), pcurve (laser performance-curve ladder: one 100 mm line per level at 10 mm/s under M3, the laser off between rungs and a mid-ladder rung repeated at the end; reads pic/hv_current and the head thermopile head/beam_detect_analog (a scatter detector in the beam path upstream of the final mirror, so it sees the beam, not the material) from sysfs at ~25 Hz on the board, brackets each rung on the controller's Run/Idle states, and reports per rung the current with a clipped-at-1023 flag, the thermopile delta over its laser-off baseline and in-line drift, then the normalized curve, monotonicity, a line fit with its threshold intercept and the repeat-rung drift; JSON record with the raw trace in the bench data directory; rungs follow laser_power_model, a comma list overrides; a curve measurement wants $35 = 0), dpatch [F] [pitch] [length] (depth witness for the density dose curve: two rows of small serpentine-filled patches, row A CW at feeds giving relative doses 1.0 to 0.25 of the reference feed, row B at the reference feed at 100/80/60/45/30 % density; the operator matches each row-B patch to the row-A patch of equal depth, which reads the density's light fraction off the material beside the thermopile's prediction; JSON record), m4feeds [S] [F1] [F2] (the density time base across feeds: one out-and-back line pair per feed at the same S under M4 density, one armed run; the operator reads within-line evenness and reversal darkness at both feeds - M4's velocity scaling is what holds dose per mm through the accel), m4corner [S] [F] (M4 velocity-scaled power into corners: a corner-heavy vector pattern at 30 % under M4 density, one armed run; the operator confirms every commanded segment marks - the floor makes a dropout unreachable - and the drill asserts the arm report, one discharge window and dark after), m5dark (the rapids after an M5 ship dark: one 20 mm line at M3 S400, M5, dwell, rapid back, dwell, rapid forward; PASS when the 25 Hz current trace shows one discharge segment and reads dark after the M5 and laser_on_sampled never re-lights; the catalog's laser.m5-rapid-dark is its port), flowload (cooling under laser load, one armed run per invocation, the conf keys it writes put back at the end, the pump never commanded off: t1 reproduces the flow-check trip with the check on at its defaults and two 30 x 4 mm CW fills at F1500 starting on the press with no dark dwell, and reports the engine's rise/dT verdict beside the 25 Hz trace of both coolant sensors, the current, the digital witness and the heater output in 5 s bins across the window, with the shape at fire start; t2 <secs> [pct] runs with the check off and one fill of about secs lit seconds at CW or at pct density, and reports the lag to each sensor, the rise per raw-second of hv_current and what a full 50 s window would add against the 1.6 C margin; fit fits rise against dose over every t2 record; JSON records), expstop (armed kill on the expected-stop path; needs the panel token - GF_TOKEN, or the board's token file) and ctrlstart (the separate controller restart after it). Every drill waits for the operator's physical arm press; eye protection, fire watch, extinguisher, and exhaust are mandatory.
pacing_test.py	Protocol-loop pacing check (runs on the board, dry motion): idle and parked-in-Hold states are coarse-paced, active motion is tight-paced, and a feed-hold/resume mid-move preserves position with no feeder starve.
planner_blocks_test.py	Host-side planner buffer depth harness (null-sink controller): $398 written to 400 and to 1000, the controller restarted on the same settings store each time, and it must answer on the port, report the depth in its status report and run a move. A byte-wide ring index once made 255 blocks or more spin the start. Runs in the grblHAL repo's CI.
xy_mode_test.py	Host-side XY microstep mode harness (null-sink controller): xy_microsteps in the shared config sets $100/$101 and the machine tick, a typed $100 is overwritten on the spot, a value that is not a mode falls back to x8 with a warning, and $110/$111 are held under a tick lowered with GFSINK_RATE. Runs in the grblHAL repo's CI.
raster_dry.py	A dry raster at top speed at each XY microstep mode (runs on the board; the controller in GRBL mode, idle, no other Grbl client): per mode given (default 8, 16, 32) it stores xy_microsteps through forgectrl, waits for the restarted controller, streams 60 passes of 150 mm at F12000 with the laser off, and reports the peak feed, the controller CPU, the kernel counters against the start, cnc/underruns and any clamped-event line. Needs 150 mm of free +X and 12 mm of free +Y travel. Ends at x8.
xy_pattern_accel.py	The XY microstep modes by the head accelerometer with the machine silent (runs on the board; the machine homed and at home, the lid closed, no other Grbl client): per mode it takes the cooling engine's quiet hold with the pump (POST /cool/quiet?on=1&pump=1: every fan, the pump and the TEC off), waits the fixed 10 s, samples the LIS2HH12 over /dev/i2c-3 in one SMBus block transaction per reading at about 600 Hz, and runs the pattern from home at F12000: to (18, 9) in, to (9, 9) in, a 9 in circle from its mid-bottom back to (9, 9), to (9, 0), home. Reports the cruise-window RMS and peak-to-peak per leg and overall, the leg times, the kernel counters against home and cnc/underruns; JSON with the trace in the bench data directory. Ends at x8.
arc_tolerance_sweep.py	How fine an arc the controller can plan (runs on the board; the machine homed and at home, the lid closed, no other Grbl client, the machine silent through the quiet hold): a $12 ladder on the 9 in circle at F12000 at the mode given (--mode, default 16). Per rung: the chords and the chord boundaries a second, the circle time against the ideal, the lowest feed mid-circle and the fewest free planner blocks, the controller CPU, clamped events, underruns and the accelerometer's cruise RMS. $12 goes back to what it was on every exit path; the head returns home.
gfbench.py	Not a tool: the helper the board/host tools share - HOST/LOCAL from GF_HOST, board(cmd) (local sh -c or ssh), the factory coolant conversion degc(), data_path() (FORGETEST_BENCH_DATA or next to the tool), forgectrl's HTTP API with the panel token, setting(key) (from forgectrl, or from /data/forgefirm/forgefirm.conf on the board while forgectrl is stopped).
lens_travel.py	Lens travel drill (runs on the board; stops the controller through forgectrl and restarts it; lens motion only): steps the lens over sysfs the way the focus card's lens home does and counts, per condition and round, the hall's hysteresis band (down off the rising edge until the hall leaves home, back up until it reads home), the rising edge's height above the bottom stop after a drive onto it, and the top stop's height above the edge (a drive up, down until the hall leaves home, back up; the difference). Conditions: the card's half-step drive onto the stop, a short drive, a long one, the factory's full-step home mode; --ladder drives a list of half-step descents below the leave-home point and counts each back (exact until the stop, short by an even number after a stall), --current picks the run current, the hold current, or hold down / run up, --settle and --cadence vary the timing; `--park bottom
lens_stop_accel.py	Lens stop detection by the head accelerometer (runs on the board; stops the controller through forgectrl and restarts it; lens motion only; stall drills on the bench reference machine only): reads the LIS2HH12 straight over /dev/i2c-3 in six-byte bursts at 800 Hz (the iio path waits a sample period per read), steps the lens one half-step at a time from the hall's rising edge toward each stop and past it, and prints per step the peak-to-peak on each axis. A free step rings strongly on every second half-step; at a stop the ring dies, and a rotor slip a few steps later is a burst three times any free ring. --find N runs the contact rule N times per stop (a strong-parity step ringing under --thresh, or a burst over four times it, calls contact; the lens backs off --back and the count home proves no slip). --save keeps the per-step sample traces as JSON.
fan_test.py	Fan/coolant bench (board or host; controller running): snapshots fan PWMs/tachs/temps, drives M8 → cut fans, M9 → cooldown → idle, verifying via tach readbacks.
fan_floor_measure.py	The numbers the airflow gates ship with (board or host): spinup opens a run session with M8 from idle and samples the four tachs and the purge current at 1 Hz, reporting per fan the steady speed, the time to 90 percent and the spread over the steady window, plus the purge current at idle and at run duty (the pump is always on; a dead one reads about 1), and candidate floors at 55 percent; cut samples only, during a real cut, for the spread under load. Results as JSON in the bench data directory.
flow_characterize.py	Coolant flow characterization using the factory temperature curve (board or host; forgectrl and controller stopped): baseline → flow → no-flow → recovery, printing the ΔT bands and their separation. Takes the heater duty as an argument (flow_characterize.py 30); aborts if downstream passes 45 °C.
flow_matrix.py	The flow-detection design matrix (board or host; forgectrl and controller stopped; with flow_sampler.py from /usr/share/forgetest/bench/): duty × duration × flow/no-flow, every run from a common cooled baseline, interleaved repeats. One heating trace yields the metric at every candidate duration, so cost and precision come from the same 60 runs. Prints a cost table, a precision table (mean±sd, worst-case margin, d′) and a ranked shortlist. flow_matrix.py [duties] [repeats] (or env FM_DUTIES, FM_REPEATS, FM_RESULTS); results/log in the bench data directory, resumable.
flow_sustained.py	Long-run test of the real re-check cadence via M8 (board or host; controller running): counts verdicts/false faults against the configured cool_flow_rise and tracks whether the loop accumulates heat. flow_sustained.py [minutes].
temp_calibrate.py (supply-* modes)	The power supply's sensor (pic/pwr_temp, raw) against a thermometer on its heatsink: supply-watch, supply-point <C>, supply-fit; the fit is printed beside the documented unverified guess (sensors). Three points during a long cut settle it.
critical_tier_drill.py	The coolant critical tier on a rising temperature (board or host): sets the ceiling, the resume gate and the critical line a few tenths above the live upstream reading and lets the engine's own flow-check heater warm the loop through them inside one M8 session, expecting OVERTEMP at the ceiling and then CRITICAL (fire blocked, hold, no resume) with the fault ending at M9; restores the settings and cycles a session so the engine re-reads them. Results as JSON in the bench data directory.
aa_offset_check.py	Coolant offset correction under the run airflow (board; controller running; dark, no press): M8 brings the fans to the run profile while the raw coolant counts, /status and the engine's readings are averaged before, during and after; with cool_aa_offset_counts at the machine's value the readings hold still while the raw counts step, at zero they drop by about a degree. aa_offset_check.py [dwell_s].
offset_probe.py	Coolant-sensor offset probe (board; forgectrl idle; dark, no press): switches one actuator at a time (exhaust at 100/50/25 %, intakes, air assist, purge, heater, pump, TEC, lid lamp, then all run fans) with both thermistors sampled at 25 Hz and scores the common-mode step at every edge and the level toggling inside every dwell; offset_probe.py ladder runs the air-assist duty ladder alone. Every value is restored on exit. JSON record in the bench data directory.
flow_warm_validate.py	Runs the real check from a heater-warmed baseline (board or host; forgectrl and controller stopped; flow_warm_validate.py [cycles_per_case]; results/log in the bench data directory; exit 1 if any run is misclassified). Note the ceiling: 100 % duty pushes the downstream sensor past 50 °C in 30 s while the bulk barely moves, so warm-loop validation above ~23 °C needs the laser, not the heater.
flow_recheck_char.py	Characterizes short in-run re-checks and the differential metric (board or host; forgectrl and controller stopped; flow_recheck_char.py [heater_pct] [window_s]); shows why over-temp cannot see a stopped pump and why passive warming trends are ambiguous.
flow_confirm_drill.py	Coolant flow suspicion/confirmation drill (runs on the board): one continuous M8 session walks the verdict state machine through real pump-off transients - verified → SUSPECT (+ immediate re-check) → cleared → SUSPECT → FAULT (consecutive) → recovered - printing PASS/FAIL per transition. Leaves the machine idle (M9, pump on, heater off).
flow_escalate_drill.py	Coolant starved-re-check escalation drill (runs on the board, controller running): sets the engine's confirmation budget cool_confirm_max_s to a short value through forgectrl's settings (flow_escalate_drill.py [budget_s], default 60, the setting's minimum) and restores it after; with the pump off the job-start check reads SUSPECT, the stagnant loop cannot pass the settle gate inside the budget, and the engine must escalate to FAULT. PASS/FAIL (exit status), leaves the machine idle.
flow_sampler.py	Board-side coolant sampler used by the flow tools (flow_sampler.py <duration_s> <interval_s>, prints elapsed,raw_down,raw_up); run on the board (dev image: /usr/share/forgetest/bench/) so cadence does not depend on ssh latency.
temp_calibrate.py	Coolant temperature spot-check helper (board or host): watch [seconds] / point <measured_C> [note] / fit - pairs a measured temperature with averaged raw ADC readings and fits a per-machine line to sanity-check the factory curve against a thermometer. Points accumulate in temp_calibration.json in the bench data directory.
build-glowforge.sh	Cross-compiles grblHAL-glowforge (the canonical driver repo) in the Yocto build environment, borrowing the recipe toolchain. Run: bash <path>/build-glowforge.sh (from Windows, launch it through the WSL distro from PowerShell; Git Bash mangles /mnt/c paths). Env: FF_SRC_TOP, FF_BUILD_TOP. This is the production controller build.
build-forgectrl.sh	Cross-compiles forgectrl (the canonical control-daemon repo) the same way, borrowing the toolchain from the forgectrl recipe workdir (regenerate with bitbake forgectrl after a clean).
accel_fast.py	Direct-I2C sampler for the two head-bus LIS2HH12s (runs on the board; unbinds/rebinds st-accel around the capture, 800 Hz ODR, ~270 Hz per device polled): optional mid-capture jogs via local grblHAL TCP. CSV to /tmp/accel.csv. The head accel is i2c-3 0x1e.
bump_seek.py	Accelerometer bump-seek homing prototype (runs on the board): creeps toward a rail in bounded jog segments via grblHAL TCP, learns the moving-noise baseline per segment, detects the contact jolt (~530 Hz sampling, 2-sample confirm), jog-cancels (0x85) and backs off. CSV to /tmp/bump.csv.
accel_crash_probe.py	Head-accel crash-detector de-risk drill (runs on the board; the head crash detector): arms the head LIS2HH12's on-chip interrupt generator (IG_CFG1/IG_THS/IG_DUR1) and polls the latched IG_SRC1 for a strike, reporting the axes and raw magnitude. coexist (default) reaches the IG registers with I2C_SLAVE_FORCE while st_accel stays bound, so it proves whether the detector can be forgectrl-only with the liveness path untouched; unbind frees the device (stop the controller and forgectrl first). Touches only the IG registers, no full-scale change, no emission, no commanded motion by default (provoke a trip by hand or with one --jog). CSV to /tmp/accel_crash.csv.
build-feeder.sh	Cross-compiles feeder.c the same way.
puls_profile.py	Decodes factory .puls streams (raw or GF1-headered) into velocity/accel profiles: peak speeds, ramp-slope fits, per-move segments, Z cadence. Runs anywhere (stdlib only). Source of the factory-true grblHAL defaults: 700/590 mm/s² accel, 200 mm/s max rate, 28160 Hz travel tick.
cp_watchdog_timing.py	HV charge-pump watchdog one-shot timing (runs on the board): latches every CHG_PUMP feed pulse in GPIO3's edge detector (pin 24 only, IMR untouched, ICR2 restored on exit) and polls the !Q (charge_pump_alive) and !HV_ENABLE (hv_enable) pads through /dev/mem in a tight loop while it commands short local jogs; prints per-run t_w (last pulse → Q fall), Q → HV_ENABLE delay, priming latency and the feed period, with the loop's worst gap as the resolution. Motion only, laser locked, no other Grbl client attached.
resume_dark_lead.py	Pause/resume safety-chain timing (runs on the board, as root): samples LASER_ON, FIRE, HV_ENABLE, the charge-pump watchdog, the button and the doors straight off the SoC pads through /dev/mem at ~2 kHz, with motion dated from the kernel step counters, across a pause and a resume driven by the operator's button presses. Reports how long HV_ENABLE survives the stream stopping, how fast the chain re-arms on the resume, and - on --run live - the dark lead between FIRE going back on and LASER_ON following it, in milliseconds and in millimeters at the job's feed. --run dry (default) commands no laser at all and --auto P,R drives the pause and resume with !/~ for an unattended rehearsal; --run live needs the arm press, eye protection, fire watch, extinguisher and exhaust. GRBL mode, no other Grbl client attached.
pgood_probe.py	The supply's power-good line (J1_14, cnc/laser_pgood) against the laser chain, without a scope (runs on the board, as root): polls the kernel readbacks (laser_pgood reported as the raw pin level, laser_on, laser_enable, charge_pump_alive) and the switch device's HV_ENABLE and doors bits at a few hundred hertz, hv_current at 20 Hz, and prints every transition with a timestamp plus a per-line summary. Drive the machine meanwhile (a dry jog, an armed cut, a pause, a lid open); the probe only watches. It needs no file on the board: run it as python3 - --secs 90 over an ssh session with the script on standard input.
bench_m2.py	Motion-quality bench, runs against the board over TCP:23: bounded round-trip jogs (sanity, max-rate, diagonal) + feed-hold/resume mid-move, reporting peak feed, state transitions, and position drift.

Data files kept beside the tools: flow_matrix_results.json / flow_matrix_log.txt (the 60-run flow-detection matrix), flow_warm_results.json / flow_warm_log.txt (warm-baseline validation), temp_calibration.json (coolant sensor spot-checks), and lid_ir_ambient_baseline.csv (lid-IR ambient, lid closed, idle).

The build scripts borrow the Yocto cross toolchain + sysroot from a target recipe work directory in the build tree (FF_BUILD_TOP, default ../forgefirm/build); if that path ages out after a bitbake -c clean, rebuild the named recipe or point TC at any current target recipe workdir (or build a proper SDK with bitbake meta-toolchain).

How the coolant-flow fire-gate threshold was derived

forgectrl's cooling engine refuses to let the laser fire unless a heater-based flow check passes: with the pump commanded on, the loop heater runs at COOL_FLOW_HEATER_PCT (40 %) for COOL_FLOW_CHECK_S (50 s) and the downstream sensor's rise over its settled baseline must stay below COOL_FLOW_RISE_C (14.4 °C) - a stopped pump lets the heater's output pool at the downstream sensor instead of being carried away. The three constants live in forgectrl/src/cool.h and are the compiled defaults behind the cool_flow_* settings.

The 14.4 °C threshold and the 40 %/50 s operating point come from flow_matrix.py: 6 duties × 2 flow states × 5 interleaved repeats = 60 heating runs, every run started from a common cooled baseline, both sensors sampled at 1 Hz by flow_sampler.py, and one heating trace scored at every candidate check duration. The committed flow_matrix_results.json and flow_matrix_log.txt are that data set. The selection rule: the cheapest duty at which every observed no-flow rise exceeded every observed flow rise with a comfortable d′ (40 % / 50 s: flow ≤ 12.75 °C over 17 observations, no-flow ≥ 16.04 °C over 8, d′ 8.4, ~0.8 °C of loop heating per check), with the threshold set at the balanced midpoint of the two bands (14.4 °C). flow_warm_validate.py then re-ran the real check from heater-warmed baselines (flow_warm_results.json).

To reproduce on another machine: boot the dev image and run the flow-matrix tool from the bench page (#bench; a takeover, ~1.6 h for the full matrix; the results land under /data/forgetest/bench/), or from a host with GF_HOST (and GF_SSH if ssh needs a wrapper), forgectrl and the controller stopped, flow_matrix.py [duties] [repeats]. The same derivation is also built into forgectrl as the panel's Diagnostics → flow-calibrate tool (3 trials per case at the operating point, reports both bands and a recommended threshold; the bench value it recommends lands within a few tenths of a degree of 14.4). Apply a per-machine value through the cool_flow_rise setting rather than editing the constant.