forgetest: motion.job and laser.recorder-dark

forgectrl's job runner (POST /job, and the dose-curve recorder and the
sheet wizards moved onto it) needs a test of a posted program, and nothing
in the catalog ran the recorder's start: only its refusal and its status.

motion.job posts programs the way a client does, as a multipart form. A
program with a $ line and a job with no name are refused with 400, and a
job beside the suite's own Grbl client with 409, each with nothing moved. A
dark program (out 20 mm, a dwell, back) then plays: /status names
job:forgetest as the lease holder of kind sender, GET /job reports it
running, and a second job, a port jog, and a settings write are each
refused in the job's name. At its end the record says done with every line
and the runner's own M2 acknowledged, no discharge and no LASER_ON sample,
the kernel's counters saw the 20 mm and are back where they began, and the
lease is free. POST /job/abort stops a 40 mm move short, into the
controller's alarm state, with the lease free. A job sent into the alarm
fails at its first line with nothing moved; the same job with unlock=1
clears the alarm and returns the head.

laser.recorder-dark starts the recorder and never presses. While the
controller waits at its arm: the recorder holds the machine as a sender,
GET /job shows its run (not a posted program's), the floor and the curve
read 0 and off, POST /job/abort refuses to stop a run it did not start, a
posted job is refused in the recorder's name, and two seconds of witnesses
stay dark. POST /curve/stop ends it: stopped before the ladder fired, the
lease free, both laser keys as they were. The ladder begins at the
controller's X0 Y0, so the head is returned by a posted job.

covers: setup.sheet and forgectrl.lease name src/jobrun.*, since the sheet
wizards stream through the runner and the recorder's refusal comes from it;
setup.sheet names src/lease.* for the hold its jobs take inside the
wizard's.

Proven on the bench reference, the job runner's daemon hot-deployed over
image 20260920130136. motion.job: PASS; the kernel saw 20.000 mm out and
0.000 back, 6 lines with 7 sent and acknowledged, every witness zero, the
abort at 9.83 mm of 40 into Alarm, the unlocking job with its $X and its M2
the two lines more. Its first run failed, and found a defect in the daemon
and not in the test: the record said done with the head 0.19 mm short of
its end, because the controller acknowledges M2 before the pulse engine has
played the last of its ring. The runner now waits for the kernel to be
idle, and the test passes unchanged. laser.recorder-dark: PASS; the
controller reached its arm wait, 16 samples dark, the stop left the
controller Idle and the head where it was. setup.sheet: PASS with the
operator at the machine. forgectrl.lease: PASS. The unit suite passes (420)
with no undefined name.

Acceptance. These two tests are the gate for forgectrl's job runner, with
setup.sheet for the sheet wizards' side of it.

forgetest/forgetest/suite/forgectrl.py

@@ -584,7 +584,7 @@ def _text(body):
       subsystem="forgectrl", kind="auto", est_min=2,
       covers=[("forgectrl", "src/lease.*"), ("forgectrl", "src/main.c"), ("forgectrl", "src/status.*"),
               ("forgectrl", "src/wizdark.*"), ("forgectrl", "src/diag.*"), ("forgectrl", "src/curverec.*"),
-              ("forgectrl", "src/update.*"), ("forgectrl", "src/logs.*"), ("forgectrl", "src/super.*"),
+              ("forgectrl", "src/jobrun.*"), ("forgectrl", "src/update.*"), ("forgectrl", "src/logs.*"), ("forgectrl", "src/super.*"),
               ("forgectrl", "src/events.*")],
       description="The switches check is started and left waiting at its first prompt: it moves "
                   "nothing, and it holds the machine lease for as long as it waits. /status must "

forgetest/forgetest/suite/laser.py

@@ -30,7 +30,8 @@ from ..catalog import test
 from .. import hw
 from ..runner import Failed
 from .motion import (kernel_xy_mm, kernel_start, check_kernel_returned, wait_state,
-                     wait_state_text, wait_left_state, wait_idle, drain_text, Watch)
+                     wait_state_text, wait_left_state, wait_idle, drain_text, Watch, machine_idle,
+                     _job, _job_post, _job_wait, _lease_holder, _port_state, _words)
 
 _LASER_COVERS = [("grblhal-glowforge", "src/**"), ("kernel-module-glowforge", "**"),
                  ("forgectrl", "src/super.c"), ("forgectrl", "src/cool.c"),
@@ -1339,3 +1340,119 @@ def port_dark(ctx):
               "the head's beam detector rose %s during a port jog", beam_rise)
     ctx.log("PASS: lit on the G1 (peak %s), and three port jogs under the same open window with M3 S400 "
             "modal shipped dark by all three witnesses", peak)
+
+
+# ---------------------------------------------------------------- the recorder, as a job of the runner's
+
+def _laser_keys(fc):
+    s = fc.settings()
+    return {k: s.get(k) for k in ("laser_floor_density", "laser_dose_curve")}
+
+
+def _arming(fc):
+    rep = ((fc.status().get("grbl") or {}).get("report") or {})
+    return bool((rep.get("laser") or {}).get("arming"))
+
+
+@test("laser.recorder-dark", title="The dose-curve recorder takes the machine as a job, arms, and is stopped dark",
+      subsystem="laser", kind="auto", mode="grbl", est_min=2,
+      covers=_LASER_COVERS + [("forgectrl", "src/curverec.*"), ("forgectrl", "src/jobrun.*"),
+                              ("forgectrl", "src/jobstream.*"), ("forgectrl", "src/lease.*"),
+                              ("forgectrl", "src/settings.*")],
+      requires=["motion.job"],
+      steps=["Lid closed, bed clear. Nobody presses the button: the recorder is stopped before its press.",
+             "The ladder begins at the controller's X0 Y0: the head goes there, and is brought back."],
+      description="POST /curve/record starts the dose ladder as a job of the job runner's. While it "
+                  "waits for a press that never comes: /status names the recorder as the lease "
+                  "holder of kind sender, GET /job reports the run (owner recorder, not a posted "
+                  "program), the floor and the curve are overridden (0 and off), the controller "
+                  "reaches its arm wait, POST /job/abort refuses to stop a run it did not start, a "
+                  "posted job is refused in the recorder's name, and the kernel's LASER_ON sample "
+                  "count and the HV current stay dark. POST /curve/stop then ends it: the recorder "
+                  "says it was stopped before the ladder fired, the lease is free, and both laser "
+                  "keys read as they did before. The head is returned by a posted job.")
+def recorder_dark(ctx):
+    ev = ctx.evidence
+    fc = hw.Forgectrl()
+    machine_idle(ctx)
+    x0, y0 = kernel_start(ctx)
+    keys0 = _laser_keys(fc)
+    base = sample(ctx)
+    ctx.check(base and not base["emission"], "emission_samples nonzero before the recording (%s)", base)
+    ctx.check(not _lease_holder(fc), "the machine is not free at the start: %s", _lease_holder(fc))
+    started = False
+    samples = []
+    try:
+        st, body = fc.post("/curve/record")
+        ctx.check(st == 200 and isinstance(body, dict) and body.get("state") == "waiting",
+                  "POST /curve/record -> %s %s", st, _words(body)[:200])
+        started = True
+        ok = ctx.wait_for(lambda: _lease_holder(fc).get("owner") == "recorder", 5, poll=0.1)
+        held = _lease_holder(fc)
+        ev["holder"] = held
+        ctx.check(ok is not None and held.get("kind") == "sender", "the lease does not name the recorder: %s", held)
+        rec = _job(fc)
+        ev["job_record"] = rec
+        ctx.check(rec["state"] == "running" and rec["owner"] == "recorder" and rec["program"] is False,
+                  "GET /job while the recorder runs: %s", rec)
+        over = _laser_keys(fc)
+        ev["override"] = over
+        ctx.check(str(over["laser_dose_curve"]) == "off" and float(over["laser_floor_density"] or 0) == 0.0,
+                  "the floor and the curve are not overridden for the ladder: %s", over)
+
+        ok = ctx.wait_for(lambda: _arming(fc), 40, poll=0.25)
+        ctx.check(ok is not None, "the controller never reached its arm wait: %s",
+                  (fc.status().get("grbl") or {}).get("report"))
+        ctx.log("the recorder holds the machine as %s, and the controller waits for the press", held)
+        st, body = fc.post("/job/abort")
+        ev["abort_route"] = [st, _words(body)[:160]]
+        ctx.check(st == 409 and "stop it where it was started" in _words(body),
+                  "POST /job/abort on the recorder's run -> %s %s", st, _words(body)[:160])
+        st, body = _job_post(fc, "G21\n")
+        ev["job_beside"] = [st, _words(body)[:160]]
+        ctx.check(st == 409 and "the dose-curve recorder holds the machine" in _words(body),
+                  "a posted job beside the recorder -> %s %s", st, _words(body)[:160])
+        for _ in range(16):                 # two seconds in the arm wait, by the witnesses
+            samples.append(sample(ctx))
+            ctx.sleep(0.125)
+        ctx.check(_lease_holder(fc).get("owner") == "recorder", "a refused request took the lease away")
+    finally:
+        if started:
+            st, body = fc.post("/curve/stop")
+            ev["stopped"] = body if isinstance(body, dict) else _words(body)[:160]
+    ctx.check(st == 200 and isinstance(body, dict) and body.get("state") == "failed"
+              and "before the ladder fired" in body.get("reason", ""), "POST /curve/stop -> %s %s",
+              st, _words(body)[:200])
+    ok = ctx.wait_for(lambda: not _lease_holder(fc), 10, poll=0.2)
+    ctx.check(ok is not None, "the lease was not given back: %s", _lease_holder(fc))
+    keys1 = _laser_keys(fc)
+    ev["keys"] = {"before": keys0, "after": keys1}
+    ctx.check(keys1 == keys0, "the laser keys were not restored: %s, were %s", keys1, keys0)
+
+    lit = [s for s in samples if s and s["emission"]]
+    hv_max = max((s["hv"] for s in samples if s and s["hv"] is not None), default=0)
+    ev["witnesses"] = {"samples": len(samples), "with_emission": len(lit), "hv_max": hv_max,
+                       "job": _job(fc)["emission"]}
+    ctx.check(not lit and hv_max <= HV_DARK_MAX and _job(fc)["emission"]["laser_on_samples"] == 0,
+              "the recorder's arm wait was not dark: %s", ev["witnesses"])
+
+    # The ladder's first move took the head to the controller's X0 Y0.
+    ctx.sleep(1.0)
+    dx, dy = (a - b for a, b in zip(kernel_xy_mm(ctx), (x0, y0)))
+    ev["moved_mm"] = [round(dx, 3), round(dy, 3)]
+    ev["controller_after_stop"] = _port_state(fc).get("state")
+    ctx.log("after the stop the controller is in %s; the head is %.3f, %.3f mm from where it was found",
+            ev["controller_after_stop"], dx, dy)
+    if abs(dx) > 0.05 or abs(dy) > 0.05:
+        st, body = _job_post(fc, "G21\nG91\nG1 X%.3f Y%.3f F2400\nG90\n" % (-dx, -dy), unlock="1")
+        ctx.check(st == 200, "the returning job -> %s %s", st, _words(body)[:200])
+        rec, _far = _job_wait(ctx, fc, 60)
+        ctx.check(rec["state"] == "done", "the returning job: %s", rec)
+    elif str(ev["controller_after_stop"]).startswith("Alarm"):
+        st, body = _job_post(fc, "G21\n", unlock="1")
+        rec, _far = _job_wait(ctx, fc, 20)
+        ctx.check(st == 200 and rec["state"] == "done", "the unlocking job: %s %s", st, rec)
+    check_kernel_returned(ctx, ev, (x0, y0), tag="recorder")
+    machine_idle(ctx)
+    ctx.log("PASS: the recorder ran as a job of the runner's, held the machine as a sender through its arm "
+            "wait with every witness dark, and its stop freed the machine and restored the laser keys")

forgetest/forgetest/suite/motion.py

@@ -2236,3 +2236,191 @@ def port_jog(ctx):
     machine_idle(ctx)
     ctx.log("PASS: the port jogged beside the client, the cancel and the client's own line each stopped "
             "a jog short, the bound held, and the release and the energize went through their routes")
+
+
+# ---------------------------------------------------------------- the job runner
+
+def _job(fc):
+    st, body = fc.get("/job")
+    if st != 200 or not isinstance(body, dict):
+        raise hw.HwError("forgectrl /job -> %s %s" % (st, body))
+    return body
+
+
+def _job_post(fc, program, name="forgetest", **fields):
+    """POST /job the way a client does: a multipart form, the program as
+    the file part."""
+    mark = "forgetestJobBoundary7d1"
+    form = ([("name", name)] if name is not None else []) + sorted(fields.items())
+    body = "".join('--%s\r\nContent-Disposition: form-data; name="%s"\r\n\r\n%s\r\n' % (mark, k, v)
+                   for k, v in form)
+    body += ('--%s\r\nContent-Disposition: form-data; name="program"; filename="job.gcode"\r\n'
+             'Content-Type: text/plain\r\n\r\n%s\r\n--%s--\r\n' % (mark, program, mark))
+    return fc.post("/job", data=body.encode("ascii"),
+                   headers={"Content-Type": "multipart/form-data; boundary=%s" % mark})
+
+
+def _job_wait(ctx, fc, timeout):
+    """Until the job is over; the record, and the farthest the kernel saw X go."""
+    x0 = kernel_xy_mm(ctx)[0]
+    far = 0.0
+    deadline = time.time() + timeout
+    while time.time() < deadline:
+        far = max(far, abs(kernel_xy_mm(ctx)[0] - x0))
+        if _job(fc)["state"] != "running":
+            break
+        ctx.sleep(0.1)
+    return _job(fc), far
+
+
+def _lease_holder(fc):
+    return ((fc.status().get("lease") or {}).get("holder") or {})
+
+
+@test("motion.job", title="The job runner: a posted program runs with the daemon as its sender",
+      subsystem="motion", kind="auto", mode="grbl", est_min=3,
+      covers=_MOTION_COVERS + [("forgectrl", "src/jobrun.*"), ("forgectrl", "src/jobpost.*"),
+                               ("forgectrl", "src/jobstream.*"), ("forgectrl", "src/lease.*"),
+                               ("forgectrl", "src/main.c"), ("forgectrl", "src/status.*")],
+      requires=["motion.jog-roundtrip"],
+      steps=["Bed clear; the head needs 60 mm of free travel toward +X. Nobody touches the gantry."],
+      description="POST /job takes a G-code program as a multipart form and runs it with the daemon "
+                  "as the machine's one sender. A program with a $ line, and one with no name, are "
+                  "refused with 400 and nothing moved. With the suite connected as a Grbl client the "
+                  "job is refused with 409 (a sender is connected) and nothing moved. A dark program "
+                  "(out 20 mm, a dwell, back) then plays: while it does, /status names job:forgetest "
+                  "as the lease holder of kind sender, GET /job reports it running, and a second job, "
+                  "a port jog, and a settings write are each refused in the holder's name. At its "
+                  "end the record says done with every line and the runner's own M2 acknowledged, no "
+                  "discharge and no LASER_ON sample, the kernel's counters saw the 20 mm and are back "
+                  "where they began, and the lease is free. POST /job/abort then stops a long move "
+                  "short: the record says failed (aborted), the controller is in its alarm state, and "
+                  "the lease is free. A job sent into the alarm without unlock fails at its first "
+                  "line with nothing moved; the same job with unlock=1 clears the alarm with the "
+                  "runner's own $X and returns the head to where it was found.")
+def job_runner(ctx):
+    ev = ctx.evidence
+    fc = hw.Forgectrl()
+    machine_idle(ctx)
+    x0, y0 = kernel_start(ctx)
+    rest = _job(fc)
+    ev["record_keys"] = sorted(rest)
+    ctx.check({"state", "owner", "program", "lines", "sent", "acked", "elapsed_s", "lit", "emission",
+               "reason"} <= set(rest), "GET /job lacks keys: %s", sorted(rest))
+    ctx.check(rest["state"] != "running", "a job is already running: %s", rest)
+    ctx.check(not _lease_holder(fc), "the machine is not free at the start: %s", _lease_holder(fc))
+
+    # Refused at the door, nothing moved.
+    refused = {}
+    st, body = _job_post(fc, "G21\n$X\nG91\nG1 X5 F1200\n")
+    refused["a $ line"] = [st, _words(body)[:160]]
+    ctx.check(st == 400 and "line 2" in _words(body) and "$ command" in _words(body),
+              "a program with a $ line -> %s %s", st, _words(body)[:160])
+    st, body = _job_post(fc, "G21\nG91\nG1 X5 F1200\n", name=None)
+    refused["no name"] = [st, _words(body)[:160]]
+    ctx.check(st == 400 and "name" in _words(body), "a job with no name -> %s %s", st, _words(body)[:160])
+    with ctx.grbl() as g:
+        clean_slate(ctx, g)
+        # The lease asks the controller's published state, which follows the connection.
+        ok = ctx.wait_for(lambda: ((fc.status().get("lease") or {}).get("observed") or {}).get("sender"),
+                          10, poll=0.2)
+        ctx.check(ok is not None, "the controller never reported the suite's client as connected")
+        st, body = _job_post(fc, "G21\nG91\nG1 X5 F1200\n")
+        refused["a connected client"] = [st, _words(body)[:160]]
+        ctx.check(st == 409 and "sender is connected" in _words(body),
+                  "a job beside a connected Grbl client -> %s %s", st, _words(body)[:160])
+    ev["refused_at_the_door"] = refused
+    ctx.check(abs(kernel_xy_mm(ctx)[0] - x0) < 0.01, "a refused job moved the head")
+    ok = ctx.wait_for(lambda: not ((fc.status().get("lease") or {}).get("observed") or {}).get("sender"),
+                      10, poll=0.2)
+    ctx.check(ok is not None, "the controller still reports the suite's client as connected")
+
+    moved = 0.0
+    try:
+        # A dark program plays, and holds the machine while it does.
+        st, body = _job_post(fc, "(forgetest motion.job)\nG21\nG91\nG1 X20 F1200 ; out\nG4 P2\n"
+                                 "G1 X-20 F1200\nG90\n")
+        ctx.check(st == 200 and isinstance(body, dict) and body.get("state") == "running",
+                  "POST /job -> %s %s", st, _words(body)[:200])
+        ok = ctx.wait_for(lambda: _lease_holder(fc).get("owner") == "job:forgetest", 5, poll=0.1)
+        held = _lease_holder(fc)
+        ev["holder"] = held
+        ctx.log("the lease while the job plays: %s", held)
+        ctx.check(ok is not None and held.get("kind") == "sender" and "forgetest" in (held.get("words") or ""),
+                  "the lease does not name the job: %s", held)
+        rec = _job(fc)
+        ctx.check(rec["state"] == "running" and rec["owner"] == "job:forgetest" and rec["program"] is True
+                  and rec["lines"] == 6, "the record while it plays: %s", rec)
+        beside = {}
+        for name, call in (("a second job", lambda: _job_post(fc, "G21\n", name="second")),
+                           ("a port jog", lambda: fc.post("/motion/jog", params={"x": "1"})),
+                           ("a settings write", lambda: fc.post("/settings", params={
+                               "ui_units": fc.settings().get("ui_units") or "metric"}))):
+            st, body = call()
+            beside[name] = [st, _words(body)[:160]]
+            ctx.check(st == 409 and "a job (forgetest) holds the machine" in _words(body),
+                      "%s beside the job -> %s %s", name, st, _words(body)[:160])
+        ev["refused_beside_the_job"] = beside
+        rec, far = _job_wait(ctx, fc, 40)
+        x1 = kernel_xy_mm(ctx)[0]
+        ev["dark_job"] = {"record": rec, "farthest_mm": round(far, 3), "kernel": [x0, x1]}
+        ctx.log("the dark job: %s; the kernel saw %.3f mm out and ended %.3f from the start",
+                rec, far, x1 - x0)
+        ctx.check(rec["state"] == "done" and rec["reason"] == "", "the job did not end well: %s", rec)
+        ctx.check(rec["sent"] == rec["acked"] == rec["lines"] + 1,
+                  "lines %s, sent %s, acked %s (the runner's M2 is the one more)",
+                  rec["lines"], rec["sent"], rec["acked"])
+        ctx.check(rec["lit"] is False and rec["emission"]["laser_on_samples"] == 0,
+                  "a dark job's witnesses: %s", rec["emission"])
+        ctx.check(19.0 < far < 21.0, "the kernel saw %.3f mm of the 20", far)
+        ctx.check(abs(x1 - x0) < 0.1, "the job did not end where it began: %.3f mm off", x1 - x0)
+        ctx.check(not _lease_holder(fc), "the lease was not given back: %s", _lease_holder(fc))
+
+        # The abort stops a long move short.
+        st, body = _job_post(fc, "G21\nG91\nG1 X40 F300\nG90\n")
+        ctx.check(st == 200, "the long job -> %s %s", st, _words(body)[:200])
+        ctx.sleep(2.0)
+        st, body = fc.post("/job/abort")
+        ctx.check(st == 200 and isinstance(body, dict) and body.get("state") == "failed"
+                  and "aborted" in body.get("reason", ""), "POST /job/abort -> %s %s", st, _words(body)[:200])
+        ctx.sleep(1.0)
+        moved = kernel_xy_mm(ctx)[0] - x0
+        after = _port_state(fc)
+        ev["abort"] = {"record": body, "moved_mm": round(moved, 3), "controller": after.get("state")}
+        ctx.log("aborted a 40 mm move at %.3f mm; the controller is in %s", moved, after.get("state"))
+        ctx.check(1.0 < moved < 35.0, "the abort did not stop the move short: moved %.3f of 40", moved)
+        ctx.check(str(after.get("state", "")).startswith("Alarm"),
+                  "the controller is in %s after the abort, not its alarm state", after.get("state"))
+        ctx.check(not _lease_holder(fc), "the lease was not given back after the abort: %s", _lease_holder(fc))
+
+        # Into the alarm: refused by the controller without the unlock, and
+        # with it the head goes back where it was found.
+        back = "G21\nG91\nG1 X%.3f F1200\nG90\n" % -moved
+        st, body = _job_post(fc, back)
+        ctx.check(st == 200, "the job into the alarm -> %s %s", st, _words(body)[:200])
+        rec, far = _job_wait(ctx, fc, 20)
+        ev["into_the_alarm"] = rec
+        ctx.check(rec["state"] == "failed" and "line 1" in rec["reason"],
+                  "a job into the alarm without unlock: %s", rec)
+        ctx.check(abs((kernel_xy_mm(ctx)[0] - x0) - moved) < 0.05, "a refused job moved the head")
+        st, body = _job_post(fc, back, unlock="1")
+        ctx.check(st == 200, "the unlocking job -> %s %s", st, _words(body)[:200])
+        rec, far = _job_wait(ctx, fc, 40)
+        ev["unlocked"] = rec
+        ctx.log("the unlocking job: %s", rec)
+        ctx.check(rec["state"] == "done" and rec["sent"] == rec["acked"] == rec["lines"] + 2,
+                  "the unlocking job (its $X and its M2 are the two more): %s", rec)
+        moved = kernel_xy_mm(ctx)[0] - x0
+    finally:
+        if _job(fc)["state"] == "running":
+            fc.post("/job/abort")
+            ctx.sleep(1.0)
+        left = kernel_xy_mm(ctx)[0] - x0
+        if abs(left) > 0.05:
+            ctx.log("returning the head %.3f mm through a Grbl client", -left)
+            _return_x(ctx, left)
+    check_kernel_returned(ctx, ev, (x0, y0), tag="job")
+    machine_idle(ctx)
+    ctx.log("PASS: a posted program ran as job:forgetest with the lease held as a sender, everything "
+            "beside it was refused in its name, the abort stopped a move short and freed the machine, "
+            "and the unlocking job cleared the alarm and returned the head")

forgetest/forgetest/suite/setup_sheet.py

@@ -28,6 +28,7 @@ from .setup_dark import run_check, Restore
 SHEET_COVERS = [("forgectrl", "src/wizlive.*"), ("forgectrl", "src/wizrun.h"),
                 ("forgectrl", "src/lens.*"), ("forgectrl", "src/sheet.*"),
                 ("forgectrl", "src/font_hershey.*"), ("forgectrl", "src/jobstream.*"),
+                ("forgectrl", "src/jobrun.*"), ("forgectrl", "src/lease.*"),
                 ("forgectrl", "src/curverec.*"), ("forgectrl", "src/wizdark.*"),
                 ("forgectrl", "src/wiz.*"), ("forgectrl", "src/setup.*"),
                 ("forgectrl", "src/main.c"), ("forgectrl", "src/ui/wizard.*"),