Catalog: the cloud client's latch at the run, and a hold that never clears

The cloud client changes (python3-gfhardware: the latch unlocks at the
run and nowhere earlier, the warm-up is supervised, a live feed must
land and finish, fire needs a power byte first, the homing runner
always stops). This commit carries the catalog tests that hold the
part the bench can see.

forgetest/forgetest/suite/cloud.py:
- cloud.dark-print (new, kind operator, one press, dark by
  construction) replaces cloud.verdict-hold. The print arms on the
  press, waits for the engine's acknowledgment and runs: the laser
  latch is locked at the button and through the wait, unlocked only
  for the run (immediately before it starts), locked again when the
  job ends, and the print completes. The engine's own warm-up release
  stays proven by cooling.floor-and-warm-up.
- cloud.verdict-refuse (new, kind operator, one press, dark): the
  start gate far above the coolant and cloud_hold_max_s at its minimum
  keep the armed print under the warm-up past the bound. The client
  waits with the latch locked (sampled every two seconds), cancels at
  the bound with its own log line, never runs, closes the armed window,
  and the print ends ':cancelled'. The settings are restored.
- cloud.verdict-hold is retired: its release rode the loop heater at
  the flow-check duty against a gate one degree above the coolant,
  inside the upstream reading's noise band, and its run hovered for
  minutes with the engine's "warm-up stalled" line in the log. The two
  tests above prove the client's contract without the thermal race.
- cloud.oversize-stream already reads cnc/streaming back at both ends
  of the run, which is the readback the client now insists on. A
  forced streaming write failure has no seam on the board (the write
  goes to sysfs as root) and stays a host test.

tests/test_cloud_suite.py follows: excerpts and failure cases for the
two new tests in place of the retired one's.

Proof. Host: the forgetest unit tests. Bench reference:
cloud.dark-print passed (locked at the button, unlocked in the run,
locked after, ':completed'), cloud.verdict-refuse passed (held 60 s,
31 latch samples all locked, ':cancelled', settings restored);
motion.deadman's kill during $H ended the runner on SIGTERM alone; and
the retired cloud.verdict-hold passed once more on the new client
before it went (the latch locked through an 8 minute warm-up hold, the
release ran the print to completion).
This commit is contained in:
ScottW514
2026-09-14 19:45:20 -04:00
parent d465625a98
commit f080d5d9cb
2 changed files with 188 additions and 79 deletions
+106 -45
View File
@@ -1300,42 +1300,96 @@ def lid_during_button_wait_body(ctx, ev, off, job, offset):
# ---------------------------------------------------------------- the cooling verdict # ---------------------------------------------------------------- the cooling verdict
HOLD_KEYS = ("cool_temp_min", "cool_temp_start") HOLD_KEYS = ("cool_temp_min", "cool_temp_start")
WARMUP_ABOVE_C = 1.0 # the start gate this far above the loop: a few heater minutes REFUSE_KEYS = HOLD_KEYS + ("cloud_hold_max_s",)
WARMUP_RELEASE_S = 720 # the flow heater warms the bulk 0.4 to 0.8 C a minute HOLD_REFUSE_S = 60 # the shortest bound the client takes
HOLD_REFUSE_LINE = "cooling hold held 60 s - relocking the laser"
WARMUP_WAIT_LINE = "waiting on the cooling engine: WARMUP" WARMUP_WAIT_LINE = "waiting on the cooling engine: WARMUP"
WARMUP_RELEASE_LINE = "cooling verdict clean after WARMUP; starting the run"
@test("cloud.verdict-hold", title="A cloud print armed under the warm-up gate waits for the engine", @test("cloud.dark-print", title="A dark cloud print: the latch locked until the run, unlocked "
subsystem="cloud", kind="operator", est_min=14, "for it, locked after it",
subsystem="cloud", kind="operator", est_min=3,
covers=_MACHINE_RUN + [("python3-gfhardware", "gfhardware/coolsvc.py"), covers=_MACHINE_RUN + [("python3-gfhardware", "gfhardware/coolsvc.py"),
("forgectrl", "src/cool.*"), ("forgectrl", "src/main.c")], ("forgectrl", "src/cool.*"), ("forgectrl", "src/main.c")],
requires=["cloud.lid-during-button-wait", "cooling.floor-and-warm-up"], actions=["button"], requires=["cloud.lid-during-button-wait", "cooling.floor-and-warm-up"], actions=["button"],
steps=[OFFLINE_STEP, steps=[OFFLINE_STEP,
"Coolant at room temperature (10 to 30 C). The test sets the warm-up gate just above " "Bed clear (the job is dark: a 30 s square at S0, nothing fires). Press the button "
"the coolant and restores it. Press the button when it lights: the print then waits " "when it lights: the print waits for the engine, runs, and finishes."],
"for the warm-up (a few minutes with the loop heater on), runs dark, and finishes."], description="The cooling-engine contract in cloud mode at its ordinary end, on the engine "
description="The cooling-engine contract in cloud mode, on the engine itself: the start gate " "itself: the print arms on the press, waits for the engine's acknowledgment (the "
"set just above the coolant makes the armed session open under the warm-up " "fans at their floors), and runs. The laser latch is locked at the button and "
"(WARMUP: fire blocked, hold), and the client waits it out after the button " "through the wait, unlocks only for the run (immediately before it starts), and "
"instead of canceling - nothing runs, the latch stays unlocked for the armed " "is locked again when the job ends; the print completes. A hold the engine never "
"window - until the engine releases into run, and the release starts the print, " "clears is cloud.verdict-refuse; the engine's own warm-up release is "
"which completes. The hold mid-run (the same laser-off pause as the button's, " "cooling.floor-and-warm-up.")
"resumed on the engine's resume_ok) and the bound on a hold are host-tested: the " def dark_print(ctx):
"gates apply at session open, so no setting can produce a hold mid-run here.") ev = ctx.evidence
def verdict_hold(ctx): fc = ctx.forgectrl
offset = enter_offline(ctx)
job = offline_job(ctx, "dark.puls", seconds=30)
off = Offline().__enter__()
try:
dark_print_body(ctx, ev, off, job, offset, fc)
finally:
off.__exit__(None, None, None)
offline_cleanup(ctx)
def dark_print_body(ctx, ev, off, job, offset, fc):
off.print_ready(9005, job)
got = wait_log(ctx, offset, ["waiting for button"], 120)
ctx.check(got["waiting for button"], "the print never reached the button wait")
ev["latch_locked_at_button"] = latch_locked()
ctx.check(ev["latch_locked_at_button"], "the latch is unlocked at the button wait")
ctx.act("button", "press", text="The button is lit: press it. The print runs dark and finishes.",
until=lambda: wait_print_running(ctx, offset, 0.1) is not None, timeout=PRESS_TIMEOUT_S)
run = wait_print_running(ctx, offset, 60)
ctx.check(run is not None, "the print did not start after the press")
ctx.sleep(1.0)
ev["latch_locked_in_run"] = latch_locked()
ctx.check(ev["latch_locked_in_run"] is False, "the latch is locked during the run")
fin = wait_action_finished(ctx, offset, "print", 180)
ev["print finished"] = message(fin)
ctx.check(fin and COMPLETED in fin, "the print did not complete: %s", message(fin) or "no finish line")
st, cs = fc.get("/cool/status")
ev["armed_after"] = cs.get("armed") if isinstance(cs, dict) else None
ctx.check(not ev["armed_after"], "armed window still open after the print")
ev["latch_locked_after"] = latch_locked()
ctx.check(ev["latch_locked_after"], "the latch is unlocked after the print")
ev["events"] = offline_events(off)
ctx.log("PASS: locked at the button, unlocked for the run, locked after it; the print completed")
@test("cloud.verdict-refuse", title="A cloud print held by the engine past the bound is canceled with "
"the latch never unlocked",
subsystem="cloud", kind="operator", est_min=4,
covers=_MACHINE_RUN + [("python3-gfhardware", "gfhardware/coolsvc.py"),
("forgectrl", "src/cool.*")],
requires=["cloud.dark-print"], actions=["button"],
steps=[OFFLINE_STEP,
"Coolant at room temperature (10 to 30 C). The test sets the warm-up gate well above "
"the coolant and the hold bound to its minimum, and restores both. Press the button "
"when it lights: nothing runs; a minute later the print cancels by itself."],
description="The other end of the cooling-engine contract before a run: a verdict that "
"refuses fire and never clears. The start gate far above the coolant keeps the "
"armed session under the warm-up for longer than the hold bound (cloud_hold_max_s "
"at its minimum, 60 s). The client waits with the laser latch locked the whole "
"time - it is never unlocked before the verdict passes, so there is nothing to "
"relock - then cancels the print at the bound: no run, the armed window closed, "
"the print ':cancelled'. Dark by construction.")
def verdict_refuse(ctx):
ev = ctx.evidence ev = ctx.evidence
fc = ctx.forgectrl fc = ctx.forgectrl
before = fc.settings() before = fc.settings()
orig = {k: before.get(k, "") for k in HOLD_KEYS} orig = {k: before.get(k, "") for k in REFUSE_KEYS}
ev["orig"] = orig ev["orig"] = orig
up = (fc.status().get("coolant") or {}).get("up_c") up = (fc.status().get("coolant") or {}).get("up_c")
ctx.check(up is not None and 10.0 <= up <= 30.0, "coolant outside the test's range (up_c %s)", up) ctx.check(up is not None and 10.0 <= up <= 30.0, "coolant outside the test's range (up_c %s)", up)
offset = enter_offline(ctx) offset = enter_offline(ctx)
job = offline_job(ctx, "hold.puls", seconds=30) job = offline_job(ctx, "refuse.puls", seconds=30)
off = Offline().__enter__() off = Offline().__enter__()
try: try:
verdict_hold_body(ctx, ev, off, job, offset, fc, up, orig) verdict_refuse_body(ctx, ev, off, job, offset, fc, up, orig)
finally: finally:
st, body = fc.post("/settings", params=orig) st, body = fc.post("/settings", params=orig)
ctx.log("restore: POST /settings %s -> %s", orig, st) ctx.log("restore: POST /settings %s -> %s", orig, st)
@@ -1343,46 +1397,53 @@ def verdict_hold(ctx):
offline_cleanup(ctx) offline_cleanup(ctx)
def verdict_hold_body(ctx, ev, off, job, offset, fc, up, orig): def verdict_refuse_body(ctx, ev, off, job, offset, fc, up, orig):
gate = round(up + WARMUP_ABOVE_C, 1) gate = round(up + 8.0, 1) # eight degrees: the heater cannot get there in a minute
st, body = fc.post("/settings", params={"cool_temp_start": str(gate), st, body = fc.post("/settings", params={"cool_temp_start": str(gate),
"cool_temp_min": orig["cool_temp_min"]}) "cool_temp_min": orig["cool_temp_min"],
ctx.check(st == 200, "POST /settings cool_temp_start=%s -> %s %s", gate, st, body) "cloud_hold_max_s": str(HOLD_REFUSE_S)})
ctx.check(st == 200, "POST /settings cool_temp_start=%s cloud_hold_max_s=%s -> %s %s",
gate, HOLD_REFUSE_S, st, body)
ev["gate"] = gate ev["gate"] = gate
off.print_ready(9005, job) off.print_ready(9006, job)
got = wait_log(ctx, offset, ["waiting for button"], 120) got = wait_log(ctx, offset, ["waiting for button"], 120)
ctx.check(got["waiting for button"], "the print never reached the button wait") ctx.check(got["waiting for button"], "the print never reached the button wait")
ctx.act("button", "press", text="The button is lit: press it. The print then waits for the warm-up.", ev["latch_locked_at_button"] = latch_locked()
ctx.check(ev["latch_locked_at_button"], "the latch is unlocked at the button wait")
ctx.act("button", "press", text="The button is lit: press it. Nothing runs; the print cancels "
"itself a minute later.",
until=lambda: log_has(offset, WARMUP_WAIT_LINE), timeout=PRESS_TIMEOUT_S) until=lambda: log_has(offset, WARMUP_WAIT_LINE), timeout=PRESS_TIMEOUT_S)
got = wait_log(ctx, offset, [WARMUP_WAIT_LINE], 30) got = wait_log(ctx, offset, [WARMUP_WAIT_LINE], 30)
ctx.check(got[WARMUP_WAIT_LINE], "the armed print did not wait on the warm-up") ctx.check(got[WARMUP_WAIT_LINE], "the armed print did not wait on the warm-up")
st, c = fc.get("/cool/status") t0 = time.time()
ev["warmup"] = c samples = []
ctx.check(isinstance(c, dict) and c.get("verdict") == "WARMUP" and c.get("hold") is True ctx.notice("The engine holds the print; the client gives up at the bound. Do nothing.")
and c.get("fire_ok") is False,
"the engine is not holding the armed session on the warm-up: %s", c)
ctx.check(wait_print_running(ctx, offset, 2) is None, "the run started under the warm-up hold")
ctx.notice("Warm-up: the loop heater brings the coolant to the gate; a few minutes. Do nothing.")
try: try:
t0 = time.time() while time.time() - t0 < HOLD_REFUSE_S + 30:
run = wait_print_running(ctx, offset, WARMUP_RELEASE_S) ctx.checkpoint()
samples.append(latch_locked())
if log_has(offset, HOLD_REFUSE_LINE):
break
ctx.sleep(2)
finally: finally:
ctx.clear_notice() ctx.clear_notice()
ev["release_s"] = round(time.time() - t0) ev["hold_s"] = round(time.time() - t0)
got = wait_log(ctx, offset, [WARMUP_RELEASE_LINE], 5) ev["latch_samples"] = {"n": len(samples), "unlocked": samples.count(False)}
ev["log"] = {WARMUP_WAIT_LINE: True, WARMUP_RELEASE_LINE: bool(got[WARMUP_RELEASE_LINE])} ctx.check(log_has(offset, HOLD_REFUSE_LINE), "the client did not cancel at the hold bound (%d s)",
ctx.check(got[WARMUP_RELEASE_LINE], "the warm-up did not release into the run within %d s", ev["hold_s"])
WARMUP_RELEASE_S) ctx.check(all(samples), "the latch unlocked during the hold (%d of %d samples)",
ctx.check(run is not None, "the print did not start after the warm-up") samples.count(False), len(samples))
fin = wait_action_finished(ctx, offset, "print", 180) ctx.check(wait_print_running(ctx, offset, 1) is None, "the run started under the refused verdict")
fin = wait_action_finished(ctx, offset, "print", 120)
ev["print finished"] = message(fin) ev["print finished"] = message(fin)
ctx.check(fin and COMPLETED in fin, "the print did not complete after the warm-up: %s", ctx.check(fin and CANCELLED in fin, "the held print did not end ':cancelled': %s",
message(fin) or "no finish line") message(fin) or "no finish line")
st, cs = fc.get("/cool/status") st, cs = fc.get("/cool/status")
ev["armed_after"] = cs.get("armed") if isinstance(cs, dict) else None ev["armed_after"] = cs.get("armed") if isinstance(cs, dict) else None
ctx.check(not ev["armed_after"], "armed window still open after the print") ctx.check(not ev["armed_after"], "armed window still open after the cancel")
ctx.check(latch_locked(), "the latch is unlocked after the cancel")
ev["events"] = offline_events(off) ev["events"] = offline_events(off)
ctx.log("PASS: the armed print waited %s s on the warm-up, then ran and completed", ev["release_s"]) ctx.log("PASS: held %s s with the latch locked throughout, then canceled at the bound", ev["hold_s"])
@test("cloud.pause-resume", title="Button pauses and resumes a cloud print (factory backtrack + lead)", @test("cloud.pause-resume", title="Button pauses and resumes a cloud print (factory backtrack + lead)",
+82 -34
View File
@@ -914,22 +914,63 @@ class CloudSuiteTests(unittest.TestCase):
self.assertEqual(self.fc.posts, []) self.assertEqual(self.fc.posts, [])
self.assertTrue(any("PASS: lid open at the button prompt" in l for l in run.lines)) self.assertTrue(any("PASS: lid open at the button prompt" in l for l in run.lines))
# -- the cooling verdict: the armed print waits on the warm-up ------------ # -- the cooling verdict: locked until the run, a hold that never clears --
def test_verdict_hold_on_the_bench_excerpt(self): def dark_parts(self):
self.in_offline() """(the print prologue through the button prompt, the run and the
self.fc.state["status"]["coolant"] = {"up_c": 21.5, "down_c": 21.4} print's end with no pause in it)."""
self.fc.state["settings"].update({"cool_temp_min": "5", "cool_temp_start": "16"})
self.fc.state["cool"].update({"verdict": "OK", "hold": False, "fire_ok": True, "armed": False})
lines = fixture("pause") lines = fixture("pause")
pre, rest = cut(lines, "waiting for button") pre, rest = cut(lines, "waiting for button")
pre = pre + [rest[0]] pre = pre + [rest[0]]
# the run and the print's end, without the pause in the middle
_, run = cut(rest, "machine:_run_loop starting run") _, run = cut(rest, "machine:_run_loop starting run")
run = [l for l in run if "paus" not in l and "resum" not in l] run = [l for l in run if "paus" not in l and "resum" not in l]
stamp = "2026-08-17T09:44:14.%06d+00:00 gfcloud[1927] INFO machine:_verdict_wait " return pre, run
wait = [stamp % 100000 + "waiting on the cooling engine: WARMUP (WARM-UP: coolant 21.5 C under "
"the 22.5 C start gate - heater on, hold)"] def test_dark_print_on_the_bench_excerpt(self):
release = [stamp % 200000 + "cooling verdict clean after WARMUP; starting the run"] self.in_offline()
self.fc.state["cool"].update({"verdict": "OK", "hold": False, "fire_ok": True, "armed": False})
pre, run = self.dark_parts()
self.offline_print_hooks(pre)
def press():
# the run: the client unlocks the latch right before it, and
# relocks when the job ends
self._attr("cnc/interlock_circuit", "37")
self.append(run[:1], delay=0.05)
def ended():
self._attr("cnc/interlock_circuit", "45")
self.append(run[1:], delay=0.0)
threading.Timer(2.0, ended).start()
hooks = {"press it. The print runs dark": press}
run_ = self.run_test(cloud.dark_print, hooks=hooks, test_id="cloud.dark-print")
ev = run_.evidence
self.assertTrue(ev["latch_locked_at_button"])
self.assertFalse(ev["latch_locked_in_run"])
self.assertTrue(ev["latch_locked_after"])
self.assertIn(":completed", ev["print finished"])
self.assertTrue(any("PASS: locked at the button" in l for l in run_.lines))
def test_dark_print_fails_when_the_latch_is_open_at_the_button(self):
self.in_offline()
pre, run = self.dark_parts()
self._attr("cnc/interlock_circuit", "37") # unlocked before the press
self.offline_print_hooks(pre)
hooks = {"press it. The print runs dark": lambda: self.append(run, delay=0.05)}
self.assertFails(cloud.dark_print, "the latch is unlocked at the button wait", hooks=hooks)
def test_verdict_refuse_on_the_bench_excerpt(self):
self.in_offline()
self.fc.state["status"]["coolant"] = {"up_c": 21.5, "down_c": 21.4}
self.fc.state["settings"].update({"cool_temp_min": "5", "cool_temp_start": "16",
"cloud_hold_max_s": ""})
self.fc.state["cool"].update({"verdict": "OK", "hold": False, "fire_ok": True, "armed": False})
pre, run = self.dark_parts()
stamp = "2026-08-17T09:44:14.%06d+00:00 gfcloud[1927] %s machine:_verdict_wait "
wait = [stamp % (100000, "INFO") + "waiting on the cooling engine: WARMUP (WARM-UP: coolant "
"21.5 C under the 29.5 C start gate - heater on, hold)"]
refuse = [stamp % (200000, "WARNING") + "cooling hold held 60 s - relocking the laser",
"2026-08-17T09:44:15.000000+00:00 gfcloud[1927] INFO basemachine:_finish_action print "
"[9006]: finished with event \":cancelled\""]
def on_post(path, form): def on_post(path, form):
if path == "/settings" and form.get("cool_temp_start") not in (None, "0", "16"): if path == "/settings" and form.get("cool_temp_start") not in (None, "0", "16"):
@@ -942,37 +983,44 @@ class CloudSuiteTests(unittest.TestCase):
def press(): def press():
self.append(wait, delay=0.05) self.append(wait, delay=0.05)
def released(): def refused():
self.fc.state["cool"].update({"verdict": "OK", "hold": False, "fire_ok": True, self.fc.state["cool"].update({"verdict": "OK", "hold": False, "fire_ok": True,
"armed": False, "phase": "run"}) "armed": False, "phase": "idle"})
self.append(release + run, delay=0.0) self.append(refuse, delay=0.0)
threading.Timer(3.0, released).start() threading.Timer(2.5, refused).start()
hooks = {"press it. The print then waits": press} hooks = {"press it. Nothing runs": press}
run_ = self.run_test(cloud.verdict_hold, hooks=hooks, test_id="cloud.verdict-hold") run_ = self.run_test(cloud.verdict_refuse, hooks=hooks, test_id="cloud.verdict-refuse")
ev = run_.evidence ev = run_.evidence
self.assertEqual(ev["gate"], 22.5) self.assertEqual(ev["gate"], 29.5)
self.assertEqual(ev["warmup"]["verdict"], "WARMUP") self.assertTrue(ev["latch_locked_at_button"])
self.assertTrue(ev["log"]["cooling verdict clean after WARMUP; starting the run"]) self.assertEqual(ev["latch_samples"]["unlocked"], 0)
self.assertIn(":completed", ev["print finished"]) self.assertGreaterEqual(ev["latch_samples"]["n"], 1)
posted = [f for p, f in self.fc.posts if p == "/settings"] self.assertIn(":cancelled", ev["print finished"])
self.assertEqual(posted[0]["cool_temp_start"], "22.5") posted = [f for p_, f in self.fc.posts if p_ == "/settings"]
self.assertEqual(posted[-1], {"cool_temp_min": "5", "cool_temp_start": "16"}) self.assertEqual(posted[0]["cool_temp_start"], "29.5")
self.assertTrue(any("PASS: the armed print waited" in l for l in run_.lines)) self.assertEqual(posted[0]["cloud_hold_max_s"], "60")
self.assertEqual(posted[-1], {"cool_temp_min": "5", "cool_temp_start": "16", "cloud_hold_max_s": ""})
self.assertTrue(any("PASS: held" in l for l in run_.lines))
def test_verdict_hold_fails_when_the_print_runs_under_the_hold(self): def test_verdict_refuse_fails_when_the_latch_unlocks_during_the_hold(self):
self.in_offline() self.in_offline()
self.fc.state["status"]["coolant"] = {"up_c": 21.5, "down_c": 21.4} self.fc.state["status"]["coolant"] = {"up_c": 21.5, "down_c": 21.4}
self.fc.state["settings"].update({"cool_temp_min": "5", "cool_temp_start": "16"}) self.fc.state["settings"].update({"cool_temp_min": "5", "cool_temp_start": "16"})
self.fc.state["cool"].update({"verdict": "WARMUP", "hold": True, "fire_ok": False, "armed": True}) self.fc.state["cool"].update({"verdict": "WARMUP", "hold": True, "fire_ok": False, "armed": True})
lines = fixture("pause") pre, run = self.dark_parts()
pre, rest = cut(lines, "waiting for button") stamp = "2026-08-17T09:44:14.%06d+00:00 gfcloud[1927] %s machine:_verdict_wait "
pre = pre + [rest[0]] wait = [stamp % (100000, "INFO") + "waiting on the cooling engine: WARMUP (no reason given)"]
_, run = cut(rest, "machine:_run_loop starting run") refuse = [stamp % (200000, "WARNING") + "cooling hold held 60 s - relocking the laser",
stamp = "2026-08-17T09:44:14.%06d+00:00 gfcloud[1927] INFO machine:_verdict_wait " "2026-08-17T09:44:15.000000+00:00 gfcloud[1927] INFO basemachine:_finish_action print "
wait = [stamp % 100000 + "waiting on the cooling engine: WARMUP (no reason given)"] "[9006]: finished with event \":cancelled\""]
self.offline_print_hooks(pre) self.offline_print_hooks(pre)
hooks = {"press it. The print then waits": lambda: self.append(wait + run[:1], delay=0.05)}
self.assertFails(cloud.verdict_hold, "the run started under the warm-up hold", hooks=hooks) def press():
self.append(wait, delay=0.05)
self._attr("cnc/interlock_circuit", "37") # unlocked under the hold
threading.Timer(2.5, lambda: self.append(refuse, delay=0.0)).start()
hooks = {"press it. Nothing runs": press}
self.assertFails(cloud.verdict_refuse, "the latch unlocked during the hold", hooks=hooks)
# -- a paused print canceled by the lid, a running one by the app -------- # -- a paused print canceled by the lid, a running one by the app --------
def cancel_parts(self): def cancel_parts(self):