diff --git a/docs/BRINGUP.md b/docs/BRINGUP.md new file mode 100644 index 0000000..8e08dde --- /dev/null +++ b/docs/BRINGUP.md @@ -0,0 +1,123 @@ +# ForgeFIRM bring-up status & cold-start runbook + +Last updated: **2026-07-26** — the day the machine first moved under grblHAL. +Read together with `AUDIT_ACTION_PLAN.md` in the project root (sibling of +this repo; per-finding status of the 2026-07-03 audit) and +`kernel-module-glowforge/UAPI.md` (the pulse-stream feeder contract). + +## Where the project stands + +**Audit phases 0–5: complete and hardware-verified.** Both motion blockers +fixed (cnc probe / 40v-supply; SDMA script relocated to `<26 0xF00>` with a +pre-run integrity guard); the end-of-data protocol reworked and bench-proven +(underrun is a first-class `underrun` state behind the `streaming` attr; +16/16 protocol bench); laser PWM verified at 39.98 kHz (register level); +`CONFIG_PREEMPT=y`; uEnv/u-boot/ulfius build integrity restored; legacy +cloud mode repaired (nvmem identity → hostname XXX-XXX verified on fuses; +deadman/safety loop; camera error paths). + +**Phase 6 spike: achieved.** +- grblHAL (unmodified core) runs on the board, speaking Grbl + 1.1f over **TCP port 23** (LightBurn-confirmed). +- Underrun proof: 100 kHz × 120 s under full load, 150 ms queue, 0.2 ms + worst write latency, zero underruns. Measured SDMA script ceiling: + **~165 kHz effective** (~6 µs/byte). +- **The step backend works**: the driver resamples grblHAL's step + events into pulse bytes and live-feeds `/dev/glowforge`. X and Y jogs + from TCP G-code move the real gantry; grblHAL and kernel position + counters agree step-for-step. Motion-only: the laser latch is forced + locked, byte bit 4 is never emitted. + +## The bench + +- **Board**: SSH `root@172.16.1.130`, empty password + (`ssh -o PreferredAuthentications=none` logs straight in). Dev image + (`forgefirm-image-dev`) on SD; BusyBox userland + python3 + gdb/strace. + Serial console on ttymxc0 available at the bench. +- **Deploying kernels**: re-burn the SD with the freshly built + `forgefirm-image-dev-glowforge.rootfs.wic.gz` (deploy dir below). Where + the boot flow loads the kernel from was never fully traced (the wic has + no boot partition; the eMMC env area reads empty) — re-burning works and + is the procedure. **Module-only changes hot-swap**: scp `glowforge.ko` + over `/lib/modules//extras/`, then `rmmod glowforge && modprobe + glowforge`. NOTE: a module reload turns off the lid LED (relight via + `/sys/class/leds/lid_led*/target`) and resets analog config (below). +- **Build host**: WSL2 distro `forge-yocto`, tree at + `~/dev/openglow-forgefirm`. `~/src-sync.sh` rsyncs the Windows repos in + (now includes `python3-gfhardware`). Build: + `cd ~/dev/openglow-forgefirm/forgefirm && kas shell + kas/forgefirm-glowforge.yml -c 'bitbake forgefirm-image + forgefirm-image-dev'`. Artifacts: + `forgefirm/build/tmp/deploy/images/glowforge/`. +- **Shell gotchas** (cost real time): PowerShell mangles embedded double + quotes in git-commit here-strings (avoid `"` in messages); `wsl -- bash + -c '...'` eats `$VAR` expansions (use script files run via PowerShell, + not Git Bash, which MSYS-mangles `/mnt/c` paths). + +## Running the step backend (grblHAL on the board) + +Source: the ForgeFIRM grblHAL step backend (grblHAL core + the +glowforge pulse-stream sink). + +1. Build: cross-compile the backend in the forge-yocto WSL distro (from + PowerShell). Produces `build-arm/grblHAL_glowforge` in the WSL tree + (`-O1 -g`, `GLOWFORGE_DEFAULTS=ON` → machine scaling baked in: + 53.333 µsteps/mm XY @ ×8, 2.832 half-steps/mm Z, 0.417" Z travel). +2. Deploy to `/usr/bin/grblHAL_glowforge` on the board. +3. **Analog machine config — required after every module reload/boot; the + kernel does NOT do this** (the cloud stack normally did): + ```sh + echo 150 > /sys/glowforge/pic/x_step_current # run current; 33 = weak hold. + echo 150 > /sys/glowforge/pic/y_step_current # Factory-true values TBD + echo 8 > /sys/glowforge/cnc/x_mode # ×8 microstepping to match + echo 8 > /sys/glowforge/cnc/y_mode # the baked steps/mm + echo 8 > /sys/glowforge/cnc/motor_lock # Z locked; X/Y free + echo 1 > /sys/glowforge/cnc/laser_latch # laser locked out + ``` +4. Start: `cd /data && GFSINK=/dev/glowforge grblHAL_glowforge -p 23 -n -t 1.0` + (`-t 1.0` is REQUIRED: the real-time throttle is what bounds the + queue). Env knobs: `GFSINK_RATE` (machine tick, default 10000 Hz), + `GFSINK_DEPTH_MS` (queue depth = feed-hold latency, default 200). +5. Connect LightBurn/UGS to `172.16.1.130:23`, or jog raw: + `$J=G91X40F1200`. + +## Hardware facts bank (measured) + +- SDMA pulse engine: ring free = 128 MiB − 32 KiB gap; script effective + ceiling ~165 kHz; position counters (`sdma_context` sc0/1/2 = X/Y/Z + steps, sc3 = bytes) match grblHAL exactly. +- Byte layout & rules: see the UAPI.md feeder contract (authoritative). +- Z: bit 6 SET = lens UP = +Z (hardware-verified; pulsedata.py was the + inverted party, fixed). Home = hall trigger at TOP; usable travel ≈ 30 + half-steps ≈ 10.6 mm ≈ 0.417"; 0.3534 mm/half-step. Never blind-drive Z + — hall-supervised only. +- XY: 0.15 mm per full step; DIR bit set = −X / +Y (Y1/Y2 complementary). +- Laser PWM: 39.98 kHz register-verified (divider 13 × 127 counts). +- Switches: truthy = closed/OK; SW_INTERLOCK reads False on units without + the rear plug — must NOT gate motion (beam is hardware-gated). +- Machine identity from OCOTP nvmem: serial 00000000 → hostname XXX-XXX + (matches the factory label). + +## Next work (in rough order) + +1. **Backend milestone 2 — motion quality**: motion is loud/jerky at the + 10 kHz tick. Raise `GFSINK_RATE` to 20–50 kHz (finer step-timing + quantization), find factory-true run currents (puls-file headers carry + them — see gfutilities settings map XSrc/YSrc) and sane accel/max-rate; + verify smooth diagonal moves and feed-hold mid-move. +2. **Laser mapping** (gated on the scope session): spindle → power bytes + (bit 7) + bit 4 laser-enable, M3/M4/$32 semantics, PWM-reset rule per + the contract. **No live fire before the standing scope gates**: LASER_PWM + waveform vs factory capture + ≤1-tick laser drop at underrun. +3. **Homing**: X/Y home switch GPIOs exist in the cnc pin map (unused so + far); wire as grblHAL limits or keep StallGuard-less factory scheme; + Z homes against the hall sensor (top). +4. **6.5 safety mapping**: door/estop evdev → feed-hold/halt in the + backend; underrun → grblHAL alarm; interlock-trip recovery check. +5. **6.6 camera service**: persistent MJPEG (ulfius, forgectrl) — also the + natural time for the deferred 5.6 emulator smoke (homing images). +6. **Housekeeping**: upstream the settings-write crash fix (grblHAL + crashes on every runtime $-settings write — NULL chained + `grbl.on_settings_changed` in gcode.c's gc_init); Phase 7 doc sweep + (CLAUDE.md charter refresh, README roadmap); kas flip + first GitHub + release per kas/README.md once ready to publish. diff --git a/scripts/bench/README.md b/scripts/bench/README.md new file mode 100644 index 0000000..af03381 --- /dev/null +++ b/scripts/bench/README.md @@ -0,0 +1,16 @@ +# ForgeFIRM bench tools + +Hardware-verification tools for the ForgeFIRM bench. All run ON the +target board (dev image, python3 present) unless noted. + +| Tool | Purpose | +|---|---| +| `feeder.c` | Spike-step-3 underrun proof: streams NOP pulse bytes to `/dev/glowforge` with wall-clock pacing, bounded queue depth, deadman flock, SCHED_FIFO. Usage: `feeder `. Passed 100 kHz × 120 s under full load with 0.2 ms worst write latency. Cross-compile with `build-feeder.sh` (WSL). | +| `bench_phase2.py` | End-of-data protocol bench (audit M2–M5): underrun detection/ack, parked no-replay guard, resume(0), continuous-feed stability, 20× run/underrun cycles. Motion-safe (motors locked, laser latched). 16/16 PASS on 2026-07-26. | +| `check_pwm.py` | Laser PWM register check (audit M8): reads PWM2 PWMCR/PWMPR via /dev/mem, expects divider 13 × ~127 counts ≈ 40 kHz. The scope on LASER_PWM remains the final pre-live-fire gate. | +| `build-feeder.sh` | Cross-compiles `feeder.c` the same way. | + +The build scripts borrow the Yocto cross toolchain + sysroot from the ulfius +2.7.15 work directory in the WSL build tree; if that path ages out after a +`bitbake -c clean`, point `TC` at any current target recipe workdir (or build +a proper SDK with `bitbake meta-toolchain`). diff --git a/scripts/bench/bench_phase2.py b/scripts/bench/bench_phase2.py new file mode 100644 index 0000000..77d8e29 --- /dev/null +++ b/scripts/bench/bench_phase2.py @@ -0,0 +1,166 @@ +#!/usr/bin/env python3 +"""Phase 2.8 bench test: the reworked end-of-data protocol (audit M2-M5). + +Motion-safe by construction: all four motors are locked via motor_lock (the +SDMA script masks the STEP bits), the laser latch is locked (LASER_ON pin is +Hi-Z and the hardware safety chain is open anyway), and only NOP (0x00) and +laser-bit (0x10) bytes are streamed. + +Exercises: normal completion, underrun detection/ack (M5), parked no-replay +guard (M4), alldone GPIO clear (M2, via GPIO2_DR readback - the data register +retains the last value the script wrote), resume(0) (M3), continuous-feed +stability, and 20x run/underrun cycling (M3 wedge check). +""" +import os, re, sys, time, mmap, struct + +C = "/sys/glowforge/cnc" +DEV = "/dev/glowforge" +GPIO2_BASE = 0x020A0000 # GPIO2 DR at offset 0 + +passed = failed = 0 +def chk(name, cond, detail=""): + global passed, failed + tag = "PASS" if cond else "FAIL" + if cond: passed += 1 + else: failed += 1 + print(f"{tag}: {name} [{detail}]", flush=True) + +def rd(attr): + with open(f"{C}/{attr}") as f: return f.read().strip() +def wr(attr, val): + with open(f"{C}/{attr}", "w") as f: f.write(str(val)) +def state(): return rd("state") + +def gpio2_dr(): + with open("/dev/mem", "rb") as f: + m = mmap.mmap(f.fileno(), 4096, mmap.MAP_SHARED, mmap.PROT_READ, + offset=GPIO2_BASE) + v = struct.unpack(" idle ---- +wr("streaming", 0) +clear_data() +feed(0x00, 2 * SEC) +wr("run", 1) +chk("T1 run starts", state() == "running", state()) +s = wait_state("idle", 10) +chk("T1 normal completion -> idle", s == "idle", s) + +# ---- T2: underrun (streaming=1): state, counter, run refused ---- +u0 = int(rd("underruns")) +wr("streaming", 1) +clear_data() +feed(0x00, 1 * SEC) +wr("run", 1) +s = wait_state("underrun", 10) +chk("T2 underrun state entered", s == "underrun", s) +chk("T2 underrun counter incremented", int(rd("underruns")) == u0 + 1, rd("underruns")) +try: + wr("run", 1) + chk("T2 run refused while unacked", False, "run was accepted!") +except OSError as e: + chk("T2 run refused while unacked", True, str(e)) + +# ---- T3: parked no-replay guard (M4): append while parked, counters frozen ---- +ctx_a = rd("sdma_context") +feed(0x01, 1000) # X-step bytes appended while parked (motors locked anyway) +time.sleep(1.0) +ctx_b = rd("sdma_context") +chk("T3 byte counter frozen while parked", sc(ctx_a, 3) == sc(ctx_b, 3), + f"sc3 {sc(ctx_a,3)} -> {sc(ctx_b,3)}") +chk("T3 head frozen while parked", sc(ctx_a, 4) == sc(ctx_b, 4), + f"sc4 {sc(ctx_a,4)} -> {sc(ctx_b,4)}") +wr("stop", 1) +chk("T3 stop acks underrun -> idle", state() == "idle", state()) +clear_data() + +# ---- T4: alldone clears laser+step GPIO bits (M2) ---- +wr("streaming", 1) +feed(0x10, 1 * SEC) # laser bit set, no steps +wr("run", 1) +s = wait_state("underrun", 10) +chk("T4 underrun after laser-bit stream", s == "underrun", s) +time.sleep(0.2) +dr = gpio2_dr() +chk("T4 LASER_ENABLE/LASER_ON_HEAD bits low in GPIO2_DR", (dr >> 30) == 0, + f"DR=0x{dr:08x}") +chk("T4 STEP bits low in GPIO2_DR", + (dr & ((1 << 20) | (1 << 21) | (1 << 22) | (1 << 29))) == 0, f"DR=0x{dr:08x}") +wr("stop", 1) +clear_data() + +# ---- T5: resume(0) completes instead of wedging (M3) ---- +wr("streaming", 0) +feed(0x00, SEC // 2) +wr("resume", 0) +chk("T5 resume(0) starts", state() == "running", state()) +s = wait_state("idle", 10) +chk("T5 resume(0) completes -> idle (no wedge)", s == "idle", s) +clear_data() + +# ---- T6: continuous feed never underruns; stopping the feed does ---- +wr("streaming", 1) +feed(0x00, SEC) +wr("run", 1) +t0 = time.time(); ok = True +while time.time() - t0 < 5: + feed(0x00, SEC // 4) + if state() != "running": + ok = False + break + time.sleep(0.2) +chk("T6 no underrun while feeding (5 s)", ok, state()) +s = wait_state("underrun", 10) +chk("T6 underrun after feed stops", s == "underrun", s) +wr("stop", 1) +clear_data() + +# ---- T7: 20x run/underrun cycles, no wedge (M3) ---- +anomalies = 0 +for i in range(20): + feed(0x00, SEC // 10) + wr("run", 1) + s = wait_state("underrun", 5) + if s != "underrun": + anomalies += 1 + print(f" cycle {i}: state={s}", flush=True) + wr("stop", 1) +chk("T7 20 run/underrun cycles clean", anomalies == 0, f"{anomalies} anomalies") + +# ---- wrap up ---- +wr("streaming", 0) +clear_data() +os.close(fd) +wr("motor_lock", 0) +print(f"\nRESULT: {passed} passed, {failed} failed; underruns total={rd('underruns')}", + flush=True) +sys.exit(1 if failed else 0) diff --git a/scripts/bench/build-feeder.sh b/scripts/bench/build-feeder.sh new file mode 100644 index 0000000..2b661c2 --- /dev/null +++ b/scripts/bench/build-feeder.sh @@ -0,0 +1,9 @@ +#!/bin/bash +set -e +TC=/home/builder/dev/openglow-forgefirm/forgefirm/build/tmp/work/cortexa9t2hf-neon-fslc-linux-gnueabi/ulfius/2.7.15 +SP="$(cd "$(dirname "$0")" && pwd)" +"$TC/recipe-sysroot-native/usr/bin/arm-fslc-linux-gnueabi/arm-fslc-linux-gnueabi-gcc" \ + --sysroot="$TC/recipe-sysroot" \ + -mthumb -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a9 \ + -O2 -Wall -Wextra -o "$SP/feeder" "$SP/feeder.c" +echo FEEDER-OK diff --git a/scripts/bench/check_pwm.py b/scripts/bench/check_pwm.py new file mode 100644 index 0000000..af28cb8 --- /dev/null +++ b/scripts/bench/check_pwm.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 +"""Phase 3.2 software check: laser PWM carrier frequency from PWM2 registers. + +Expected with the fsl,extra-prescale=<13> fix: PWMCR prescaler divider = 13, +PWMPR ~125 (127 counts - 2), effective carrier = perclk / (13 * 127) ~= 40 kHz. +Without the fix: divider 1 -> ~520 kHz (audit M8). Safe: read-only register +inspection; the laser PWM output feeds the PSU power input, firing stays gated +by the hardware chain. The definitive gate remains the scope on LASER_PWM. +""" +import mmap, struct + +PWM2_BASE = 0x02084000 +PERCLK_HZ = 66_000_000 # ipg_high; cross-check against the EPIT rate in dmesg + +with open("/dev/mem", "rb") as f: + m = mmap.mmap(f.fileno(), 4096, mmap.MAP_SHARED, mmap.PROT_READ, + offset=PWM2_BASE) + cr, sr, ir, sar, pr = struct.unpack("<5I", m[:20]) + m.close() + +prescaler_field = (cr >> 4) & 0xFFF +divider = prescaler_field + 1 +period_counts = pr + 2 +freq = PERCLK_HZ / (divider * period_counts) if period_counts else 0 +enabled = cr & 1 + +print(f"PWMCR=0x{cr:08x} PWMSAR={sar} PWMPR={pr} enabled={bool(enabled)}") +print(f"prescaler divider = {divider} (field {prescaler_field})") +print(f"period counts = {period_counts}") +print(f"carrier frequency = {freq/1000:.2f} kHz") +ok = divider == 13 and 120 <= period_counts <= 135 and 38_000 <= freq <= 42_000 +print(f"\n{'PASS' if ok else 'FAIL'}: expected divider 13, ~127 counts, ~40 kHz") diff --git a/scripts/bench/feeder.c b/scripts/bench/feeder.c new file mode 100644 index 0000000..a35f7be --- /dev/null +++ b/scripts/bench/feeder.c @@ -0,0 +1,164 @@ +/* + * feeder.c - ForgeFIRM spike step 3: prove no-underrun continuous live + * feeding of the glowforge.ko SDMA pulse ring under load (audit 6.3). + * + * Streams NOP (0x00) pulse bytes to /dev/glowforge, pacing by wall clock to + * hold a bounded queue depth (like a real grblHAL backend would - audit M6), + * with the deadman flock held. Motors must be locked and the laser latch + * locked by the caller (bench_phase6.sh does this). + * + * Reports: feed statistics, worst scheduling stall, ENOMEM count, and the + * final driver state (expect "running" throughout, "underrun" only after + * the deliberate starve at the end). + * + * Usage: feeder + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define DEV "/dev/glowforge" +#define CNC "/sys/glowforge/cnc/" + +static double now_s(void) +{ + struct timespec ts; + clock_gettime(CLOCK_MONOTONIC, &ts); + return ts.tv_sec + ts.tv_nsec / 1e9; +} + +static int wr_attr(const char *attr, const char *val) +{ + char path[128]; + int fd, ret; + snprintf(path, sizeof path, CNC "%s", attr); + fd = open(path, O_WRONLY); + if (fd < 0) return -1; + ret = (int)write(fd, val, strlen(val)); + close(fd); + return ret < 0 ? -1 : 0; +} + +static int rd_attr(const char *attr, char *buf, size_t len) +{ + char path[128]; + int fd; + ssize_t n; + snprintf(path, sizeof path, CNC "%s", attr); + fd = open(path, O_RDONLY); + if (fd < 0) return -1; + n = read(fd, buf, len - 1); + close(fd); + if (n < 0) return -1; + while (n > 0 && (buf[n-1] == '\n')) n--; + buf[n] = 0; + return 0; +} + +int main(int argc, char **argv) +{ + long freq = argc > 1 ? atol(argv[1]) : 10000; + long duration = argc > 2 ? atol(argv[2]) : 60; + long depth_ms = argc > 3 ? atol(argv[3]) : 150; + static unsigned char chunk[8192]; /* NOP bytes: no step, no laser */ + char state[32], u0[16], u1[16]; + long long target, enqueued = 0, enomem = 0, writes = 0; + double t0, t_end, last = 0, max_stall = 0, max_wr = 0; + long depth_bytes = (long)((double)freq * depth_ms / 1000.0); + int fd; + + memset(chunk, 0, sizeof chunk); + + /* SCHED_FIFO like a real feeder; fall back silently if not permitted */ + struct sched_param sp = { .sched_priority = 10 }; + sched_setscheduler(0, SCHED_FIFO, &sp); + + fd = open(DEV, O_WRONLY); + if (fd < 0) { perror("open " DEV); return 1; } + if (flock(fd, LOCK_EX) != 0) { perror("flock"); return 1; } + + char fbuf[16]; + snprintf(fbuf, sizeof fbuf, "%ld", freq); + if (wr_attr("step_freq", fbuf)) { perror("step_freq"); return 1; } + wr_attr("streaming", "1"); + rd_attr("underruns", u0, sizeof u0); + + lseek(fd, 0, SEEK_SET); /* clear data + position */ + + /* Prefill one queue depth, then start the run */ + while (enqueued < depth_bytes) { + long n = depth_bytes - enqueued; + if (n > (long)sizeof chunk) n = sizeof chunk; + if (write(fd, chunk, n) < 0) { perror("prefill"); return 1; } + enqueued += n; + } + if (wr_attr("run", "1")) { perror("run"); return 1; } + + t0 = now_s(); + t_end = t0 + duration; + last = t0; + printf("feeding: %ld Hz for %ld s, queue depth %ld ms (%ld bytes)\n", + freq, duration, depth_ms, depth_bytes); + + while (1) { + double t = now_s(); + if (t >= t_end) break; + if (t - last > max_stall) max_stall = t - last; + last = t; + + /* wall-clock pacing: keep enqueued = consumed-so-far + depth */ + target = (long long)((t - t0) * freq) + depth_bytes; + while (enqueued < target) { + long n = (long)(target - enqueued); + if (n > (long)sizeof chunk) n = sizeof chunk; + double w0 = now_s(), w1; + if (write(fd, chunk, n) < 0) { + if (errno == ENOMEM) { enomem++; break; } + perror("write"); return 1; + } + w1 = now_s(); + if (w1 - w0 > max_wr) max_wr = w1 - w0; + enqueued += n; + writes++; + } + + struct timespec ts = { 0, 20 * 1000 * 1000 }; /* 20 ms */ + nanosleep(&ts, NULL); + } + + rd_attr("state", state, sizeof state); + printf("after %ld s: state=%s enqueued=%lld writes=%lld enomem=%lld\n", + duration, state, enqueued, writes, enomem); + printf("max loop stall: %.1f ms, max write latency: %.1f ms\n", + max_stall * 1e3, max_wr * 1e3); + int fed_ok = (strcmp(state, "running") == 0) && enomem == 0; + + /* Deliberate starve: stop feeding, expect a clean underrun */ + double ts0 = now_s(); + do { + struct timespec ts = { 0, 50 * 1000 * 1000 }; + nanosleep(&ts, NULL); + rd_attr("state", state, sizeof state); + } while (strcmp(state, "running") == 0 && now_s() - ts0 < 10 + depth_ms / 1000.0); + rd_attr("underruns", u1, sizeof u1); + printf("after starve: state=%s underruns %s -> %s\n", state, u0, u1); + int starve_ok = strcmp(state, "underrun") == 0; + + wr_attr("stop", "1"); /* acknowledge */ + lseek(fd, 0, SEEK_SET); + wr_attr("streaming", "0"); + flock(fd, LOCK_UN); + close(fd); + + printf("%s: feed %s, starve->underrun %s\n", + (fed_ok && starve_ok) ? "PASS" : "FAIL", + fed_ok ? "clean" : "FAILED", starve_ok ? "detected" : "MISSED"); + return (fed_ok && starve_ok) ? 0 : 1; +}