Let the page run a campaign's remaining tests as a queue

A campaign is mostly waiting for the next Start. The page now offers two
queues, and each takes every test of its kinds the campaign does not
already count as satisfied: Unattended for the auto tests, which need
nobody in the room, and Operator and live for the ones that need somebody
at the machine, since they prompt and they fire the laser. The buttons
say how many they would run and ask before starting; the live queue names
the tests that fire and takes the acknowledgment once, for all of them.

A queue runs one test at a time through the runner's single slot, in
prerequisite order. Registration order otherwise, so a run reads down the
page, but a prerequisite inside the queue always goes first. It stops on
the first result that is not a PASS: a FAIL closes the campaign, and
carrying on would only open a second one behind the operator's back. A
test the runner refuses to start is skipped with the reason on the page
and the rest carry on, which is what happens to an auto test waiting on an
operator one: run the attended queue, then the unattended one again.

The queue lives in the runner, not in the tab, so reloading the page or
closing it leaves the run alone. While one is up it holds the machine
between its tests as well as during them, so a single Start and the bench
tools are refused rather than cutting in. Stop the queue cancels what is
still waiting and lets the run in progress finish; Abort ends that one
too, and lands as the non-PASS that stops the queue. Every run a queue
starts records which one put it there.

Also moves the /state ETag test to the end of its class. It invalidates,
timestamps are whole seconds, and a PASS stamped in the same second as an
invalidate is deliberately not inheritable, so on a fast run it decided
the inheritance an earlier test was checking.

forgetest is a dev-only component and can never appear in a coverage map,
so this has no acceptance catalog consequence.

forgetest/forgetest/catalog.py

@@ -120,6 +120,35 @@ def get(id, registry=None):
     return (registry if registry is not None else REGISTRY).get(id)
 
 
+def order_by_requires(tests, selected):
+    """`selected` ids in an order that runs a prerequisite before the test
+    that names it.
+
+    Registration order otherwise, so a run reads down the page. A
+    prerequisite outside the selection places nothing: either it is
+    already satisfied, or the run that needs it will be refused and
+    recorded as skipped. validate() has ruled out cycles; the stack check
+    only keeps a malformed registry from recursing forever.
+    """
+    want = set(selected)
+    by_id = {t.id: t for t in tests}
+    out, placed = [], set()
+
+    def visit(tid, stack):
+        if tid in placed or tid not in want or tid not in by_id or tid in stack:
+            return
+        stack.add(tid)
+        for r in by_id[tid].requires:
+            visit(r, stack)
+        stack.discard(tid)
+        placed.add(tid)
+        out.append(tid)
+
+    for t in tests:
+        visit(t.id, set())
+    return out
+
+
 def validate(registry=None):
     """Every `requires` names a known test and there are no cycles."""
     reg = registry if registry is not None else REGISTRY

forgetest/forgetest/page.py

@@ -76,6 +76,16 @@ pre#log{background:#1d1e26;color:#d7dae0;font-family:ui-monospace,Consolas,monos
 .switch .on{color:var(--warn);font-weight:600}
 .req.over{color:var(--dim)}
 .grp{margin-top:6px}
+.queue{background:#f7f8fa;margin:10px 0 0;padding:10px 12px}
+.queue h2{margin-bottom:8px}
+.queue .hint{margin-top:8px}
+.qline{font-size:12.5px;color:var(--dim);line-height:1.7;margin-top:8px}
+.qline b{color:var(--txt)}
+.qbar{display:flex;height:6px;border-radius:3px;overflow:hidden;background:#e2e4e9;margin:6px 0}
+.qbar i{display:block}
+.qbar i.ok{background:var(--ok)}.qbar i.bad{background:var(--red)}
+.qbar i.skip{background:var(--warn)}.qbar i.now{background:var(--blue)}
+.badge.queued{background:#dde7f5;color:#24507f}
 .tool .argrow{display:flex;gap:8px;flex-wrap:wrap;margin:6px 0}
 .tool .argrow label{font-size:12px;color:var(--dim)}
 </style></head><body>
@@ -88,6 +98,19 @@ pre#log{background:#1d1e26;color:#d7dae0;font-family:ui-monospace,Consolas,monos
    <div class='banner'><div class='auth' id='auth'>?</div>
     <div class='kv' id='banner'></div></div>
    <div id='invnote'></div><div id='msgs'></div>
+   <div class='card queue'><h2>Run what is left</h2>
+    <div class='actions'>
+     <button class='pri' id='q-unattended' onclick='startBatch("unattended")'>Unattended</button>
+     <button class='pri' id='q-attended' onclick='startBatch("attended")'>Operator and live</button>
+     <button class='danger' id='q-stop' onclick='stopBatch()' disabled>Stop the queue</button>
+    </div>
+    <div id='qstate'></div><div id='qmsg'></div>
+    <p class='hint'>Each queue takes every test of its kind that the campaign does not
+     already count as satisfied, runs them one at a time in prerequisite order, and stops
+     on the first result that is not a PASS. The unattended queue needs nobody in the room.
+     The other one does: it prompts, and it fires the laser. Stop-the-queue cancels what is
+     still waiting and lets the run in progress finish; Abort ends that one too.</p>
+   </div>
    <div class='actions' style='margin-top:10px'>
     <button class='pri' id='exportbtn' onclick='doExport()'>Export release artifact</button>
     <a id='dljson' href='/export/acceptance.json' style='display:none'><button>acceptance.json</button></a>
@@ -159,7 +182,7 @@ function api(method,path,body,cb,hdrs,timeoutMs){var x=new XMLHttpRequest();x.op
 /* The state poll. It carries the last ETag, so an unchanged state costs
    a 304 and no re-render; a run gets a faster tick, and every action
    pulls the next poll forward instead of waiting out the interval. */
-function pollDelay(){return (state&&state.running)?1000:2000}
+function pollDelay(){return (state&&(state.running||batchActive()))?1000:2000}
 function schedule(ms){if(pollTimer)window.clearTimeout(pollTimer);pollTimer=window.setTimeout(poll,ms)}
 function kick(){schedule(120)}
 function poll(){if(polling){schedule(150);return}polling=true;
@@ -181,12 +204,14 @@ function setIgnoreReq(on){ignoreReq=!!on;try{window.localStorage.setItem('forget
  if(state&&catalog)renderGroups()}
 /* A start already sent but not yet seen in the state counts as busy, so
    the buttons grey out on the click rather than on the next poll. The
-   window is capped in case the answer never arrives. */
-function isBusy(){if(state&&state.running)return true;
+   window is capped in case the answer never arrives. A queue holds the
+   machine between its tests as well as during them. */
+function batchActive(){return !!(state&&state.batch&&!state.batch.finished)}
+function isBusy(){if(state&&(state.running||batchActive()))return true;
  return !!(pending&&(nowMs()-pending)<PENDING_MS)}
 function fmtTs(t){return t?t.replace('T',' ').replace('Z',' UTC'):'-'}
 function render(){if(!state||!catalog)return;
- if(state.running){pending=0;pendingId=null}
+ if(state.running||batchActive()){pending=0;pendingId=null}
  var m=state.manifest||{};setText($('hdrver'),(m.version||'?'));setText($('hdrsub'),m.image||'');
  var a=$('auth');setText(a,'Release authorized: '+(state.authorized?'YES':'NO'));setProp(a,'className','auth'+(state.authorized?' yes':''));
  var c=state.campaign,cn=state.counts||{};var b='';
@@ -197,7 +222,34 @@ function render(){if(!state||!catalog)return;
  setHtml($('banner'),b);
  var inv=state.invalidate;setHtml($('invnote'),inv?"<div class='note'>Full campaign required since "+esc(fmtTs(inv.ts))+": "+esc(inv.reason)+"</div>":'');
  var ms='';(state.messages||[]).forEach(function(x){ms+="<div class='note'>"+esc(x)+"</div>"});setHtml($('msgs'),ms);
- renderGroups();renderRun();if(bench)renderBench()}
+ renderQueue();renderGroups();renderRun();if(bench)renderBench()}
+/* The two queues: what each would run now, and how the running one is
+   getting on. Built from the state, so a reload picks the queue back up
+   exactly where it is - the queue lives in the runner, not in this tab. */
+var QUEUES=[['unattended','Unattended'],['attended','Operator and live']];
+function renderQueue(){var av=state.batch_available||{},b=state.batch,busy=isBusy();
+ QUEUES.forEach(function(p){var e=$('q-'+p[0]);if(!e)return;
+  var ids=av[p[0]]||[];
+  setText(e,ids.length?(p[1]+' ('+ids.length+')'):(p[1]+' (none left)'));
+  setDis(e,busy||!ids.length);
+  setProp(e,'title',!ids.length?('nothing left: every '+p[0]+' test is satisfied')
+   :(busy?'a run is in progress':('in order: '+ids.join(', '))))});
+ setDis($('q-stop'),!batchActive()||!!(b&&b.stopping));
+ if(!b){setHtml($('qstate'),'');return}
+ var total=b.order.length||1,ok=0,bad=0;
+ b.done.forEach(function(x){if(x.result==='PASS')ok++;else bad++});
+ function seg(cls,n){return n?("<i class='"+cls+"' style='width:"+(100*n/total)+"%'></i>"):''}
+ var h="<div class='qbar'>"+seg('ok',ok)+seg('bad',bad)+seg('skip',b.skipped.length)+
+   (b.current?seg('now',1):'')+"</div><div class='qline'><b>"+esc(b.group)+"</b> queue, opened "+
+   esc(fmtTs(b.ts))+" &middot; ";
+ h+=b.finished?('finished '+esc(fmtTs(b.finished))):(b.current?('running <b>'+esc(b.current)+'</b>'):
+   (b.stopping?'stopping':'starting'));
+ h+=" &middot; <b>"+ok+"</b> passed, <b>"+bad+"</b> not, <b>"+b.skipped.length+
+   "</b> skipped, <b>"+b.pending.length+"</b> waiting";
+ if(b.stopped)h+="<br><span class='req'>stopped: "+esc(b.stopped)+"</span>";
+ if(b.skipped.length)h+="<br>skipped: "+esc(b.skipped.map(function(x){return x.test+' ('+x.reason+')'}).join('; '));
+ if(b.pending.length)h+="<br>waiting: <span class='tid'>"+esc(b.pending.join(', '))+"</span>";
+ setHtml($('qstate'),h+"</div>")}
 /* The rows are built once for a given catalog and then only updated in
    place: a poll never rewrites the table, so a Start button survives the
    press that is landing on it. */
@@ -225,11 +277,13 @@ function buildGroups(){var groups={},order=[];
  catalog.forEach(function(t){rowEls[t.id]={st:$('st-'+t.id),last:$('last-'+t.id),btn:$('btn-'+t.id),
   unmet:$('unmet-'+t.id),note:$('note-'+t.id),detdyn:$('detdyn-'+t.id)}})}
 function updateGroups(){if(!rowEls||!state)return;var busy=isBusy();
+ var queued={};if(batchActive()){(state.batch.pending||[]).forEach(function(x){queued[x]=1})}
  catalog.forEach(function(t){var e=rowEls[t.id];if(!e)return;var s=state.tests[t.id]||{};
   var st;
   if(pendingId===t.id&&isBusy()&&!state.running){st="<span class='st running'>starting&hellip;</span>"}
   else{st="<span class='st "+esc(s.status)+"'>"+esc(s.status||'none')+"</span>";
    if(s.required&&s.status!=='running')st+="<br><span class='req'>required: "+esc(s.reason)+"</span>"}
+  if(queued[t.id])st+="<br><span class='badge queued'>queued</span>";
   setHtml(e.st,st);
   var last=s.last?(esc(s.last.result)+' '+esc(fmtTs(s.last.ts))):'-';
   if(s.status==='inherited'&&s.origin)last+="<br><span class='tid'>from "+esc(s.origin.campaign)+" on "+esc(s.origin.image)+"</span>";
@@ -275,7 +329,28 @@ function startTest(id){if(isBusy())return;var t=findTest(id);var body={test:id};
  api('POST','/start',body,function(s,d){
   if(s!==200){pending=0;pendingId=null;rowMsg[id]=d.message||d.error;updateGroups()}
   setMsg('actmsg',d.message||d.error,s!==200);kick()})}
-function confirmLive(){return window.confirm('LIVE LASER TEST.\n\nConfirm before starting:\n - eye protection on, everyone in the room\n - fire watch present, extinguisher at hand\n - exhaust running, lid closed, scrap in place\n - you will press the physical button to arm when prompted\n\nStart the test?')}
+function confirmLive(live){return window.confirm(
+ (live?('LIVE LASER QUEUE.\n\nThese fire the laser:\n - '+live.join('\n - ')+'\n'):'LIVE LASER TEST.\n')+
+ '\nConfirm before starting:\n - eye protection on, everyone in the room\n - fire watch present, extinguisher at hand\n - exhaust running, lid closed, scrap in place\n - you will press the physical button to arm when prompted\n\n'+
+ (live?'Start the queue?':'Start the test?'))}
+/* A queue takes the machine for a long stretch, so both what it will run
+   and the acknowledgment it needs are put in front of the operator once,
+   before anything starts. */
+function startBatch(group){if(isBusy())return;
+ var ids=(state&&state.batch_available&&state.batch_available[group])||[];
+ if(!ids.length)return;
+ var body={group:group};if(ignoreReq)body.ignore_requires=true;
+ var live=[];catalog.forEach(function(t){if(t.kind==='live'&&ids.indexOf(t.id)>=0)live.push(t.id)});
+ if(live.length){if(!confirmLive(live))return;body.ack_live=true}
+ else if(!window.confirm('Run these '+ids.length+' test(s), in this order?\n\n - '+ids.join('\n - ')))return;
+ pending=nowMs();pendingId=null;rowMsg={};updateGroups();renderQueue();
+ setMsg('qmsg','starting the '+group+' queue...');
+ api('POST','/batch',body,function(s,d){
+  if(s!==200){pending=0;updateGroups();renderQueue()}
+  setMsg('qmsg',d.message||d.error,s!==200);kick()})}
+function stopBatch(){var e=$('q-stop');if(e.disabled)return;setDis(e,true);
+ setMsg('qmsg','stopping the queue...');
+ api('POST','/batch/stop',{},function(s,d){setMsg('qmsg',d.message||d.error,s!==200);kick()})}
 function promptBusy(on){var b=$('promptb').getElementsByTagName('button');for(var i=0;i<b.length;i++)setDis(b[i],on)}
 function answerIdx(i){if(!curPrompt)return;var p=curPrompt,v=p.options[i];
  promptBusy(true);setMsg('actmsg','answer sent: '+v);

forgetest/forgetest/runner.py

@@ -10,6 +10,13 @@ the campaign rules (campaign.py) do the rest.
 Bench tools are subprocesses (bench.py registry): same single slot, same
 log pane, no campaign effect.
 
+A queue (BATCH_GROUPS) runs what a campaign still owes, one test at a
+time through that same single slot: the unattended tests for an empty
+room, the attended ones for an operator at the machine. It runs them in
+prerequisite order and stops on the first result that is not a PASS,
+because a FAIL closes the campaign and going on would quietly open a
+second one.
+
 Safety, in code rather than convention: a live test starts only with the
 operator's acknowledgment in the request; the runner never touches the
 laser latch; a takeover always ends with forgectrl started again, and a
@@ -31,6 +38,16 @@ from .log import now_ts, data_dir
 
 MAX_LINES = 4000
 
+# The two queues the page offers. A campaign is mostly waiting: the
+# unattended tests need nobody in the room, the attended ones need the
+# operator at the machine, and sorting them that way lets one set run
+# while the operator is elsewhere. Each queue takes everything of its
+# kinds the campaign does not already count as satisfied.
+BATCH_GROUPS = {
+    "unattended": ("auto",),
+    "attended": ("operator", "live"),
+}
+
 
 class Aborted(Exception):
     pass
@@ -308,6 +325,7 @@ class Runner:
         self._lock = threading.Lock()
         self.current = None
         self.last = None
+        self.batch = None
         self.messages = []
         self.boot_ref = None
         self.recover()
@@ -363,6 +381,10 @@ class Runner:
         st["running"] = r.snapshot() if r and not r.finished else None
         last = r if (r and r.finished) else self.last
         st["last_run"] = last.snapshot() if last else None
+        st["batch"] = self.batch_snapshot()
+        # what each queue would run if started now, so the page can label
+        # its buttons with the work rather than a bare verb
+        st["batch_available"] = {g: self.batch_selection(g, st) for g in BATCH_GROUPS}
         return st, records
 
     def busy(self):
@@ -407,19 +429,27 @@ class Runner:
         evidence records which prerequisites were unmet (the release gate
         needs every test satisfied anyway, so nothing is hidden - the
         record just says the order was the operator's)."""
+        if self.batch_active():
+            return False, "a queue is running"
+        ok, msg, _ = self._start_test(test_id, ack_live, ignore_requires)
+        return ok, msg
+
+    def _start_test(self, test_id, ack_live=False, ignore_requires=False, batch=None):
+        """As start_test, and hands back the Run so the queue driver can
+        follow it without racing another start for `current`."""
         t = _catalog.get(test_id, self.registry)
         if t is None:
-            return False, "unknown test"
+            return False, "unknown test", None
         with self._lock:
             if self.busy():
-                return False, "a run is in progress"
+                return False, "a run is in progress", None
             state, _ = self.state()
             ts = state["tests"][t.id]
             missing = list(ts["missing_requires"])
             if missing and not ignore_requires:
-                return False, "prerequisites not satisfied: %s" % ", ".join(missing)
+                return False, "prerequisites not satisfied: %s" % ", ".join(missing), None
             if t.kind == "live" and not ack_live:
-                return False, "live test: acknowledge eye protection, fire watch, and exhaust first"
+                return False, "live test: acknowledge eye protection, fire watch, and exhaust first", None
             campaign = self._open_campaign_if_needed(state)
             run = Run("test", t.id, t.title)
             self.last = self.current
@@ -430,10 +460,117 @@ class Runner:
             run.evidence["prerequisites"] = {"overridden": True, "missing": missing, "ts": now_ts()}
         if t.kind == "live":
             run.evidence["operator"] = {"ack_live": True, "ts": now_ts()}
+        if batch:
+            run.log("queued by the %s queue opened %s" % (batch["group"], batch["ts"]))
+            run.evidence["batch"] = {"group": batch["group"], "ts": batch["ts"]}
         th = threading.Thread(target=self._exec_test, args=(t, run, campaign), daemon=True,
                               name="forgetest-run")
         th.start()
-        return True, "started"
+        return True, "started", run
+
+    # -- the queue ----------------------------------------------------------
+    def batch_active(self):
+        b = self.batch
+        return bool(b and not b["finished"])
+
+    def batch_selection(self, group, state=None):
+        """The ids the given queue would run, in prerequisite order: every
+        test of those kinds that the campaign does not already count as
+        satisfied, which is exactly what is left to do."""
+        kinds = BATCH_GROUPS.get(group)
+        if kinds is None:
+            return None
+        if state is None:
+            state, _ = self.state()
+        tests = self.tests()
+        want = [t.id for t in tests
+                if t.kind in kinds and not state["tests"][t.id]["satisfied"]]
+        return _catalog.order_by_requires(tests, want)
+
+    def start_batch(self, group, ack_live=False, ignore_requires=False):
+        if group not in BATCH_GROUPS:
+            return False, "unknown queue", None
+        if self.batch_active():
+            return False, "a queue is already running", None
+        if self.busy():
+            return False, "a run is in progress", None
+        order = self.batch_selection(group)
+        if not order:
+            return False, "nothing to run: every %s test is already satisfied" % group, None
+        live = [tid for tid in order
+                if _catalog.get(tid, self.registry).kind == "live"]
+        if live and not ack_live:
+            return False, ("this queue fires the laser (%s): acknowledge eye protection, "
+                           "fire watch, and exhaust first" % ", ".join(live)), None
+        with self._lock:
+            if self.batch_active() or self.busy():
+                return False, "a run is in progress", None
+            self.batch = {"group": group, "ts": now_ts(), "order": list(order), "done": [],
+                          "skipped": [], "current": None, "stop": False, "stopped": None,
+                          "finished": None, "ack_live": bool(ack_live),
+                          "ignore_requires": bool(ignore_requires)}
+            b = self.batch
+        self._note("queue %s: %d test(s) to run: %s" % (group, len(order), ", ".join(order)))
+        threading.Thread(target=self._drive_batch, args=(b,), daemon=True,
+                         name="forgetest-queue").start()
+        return True, "queue started: %d test(s)" % len(order), list(order)
+
+    def stop_batch(self):
+        """Cancel what is still queued. The run in progress finishes and is
+        recorded; Abort is the lever that stops that one."""
+        b = self.batch
+        if not self.batch_active():
+            return False, "no queue is running"
+        left = len(self.batch_snapshot()["pending"])
+        b["stop"] = True
+        b["stopped"] = b["stopped"] or "stopped by the operator"
+        self._note("queue %s: stop requested, %d test(s) will not run" % (b["group"], left))
+        return True, "queue stopped; the run in progress finishes"
+
+    def _drive_batch(self, b):
+        """One test at a time, in order, until the queue empties or a run
+        comes back anything other than PASS. A FAIL closes the campaign, so
+        carrying on would only open a second one behind the operator's
+        back; an ABORTED means they asked it to stop."""
+        try:
+            for tid in b["order"]:
+                if b["stop"]:
+                    break
+                b["current"] = tid
+                ok, msg, run = self._start_test(tid, ack_live=b["ack_live"],
+                                                ignore_requires=b["ignore_requires"], batch=b)
+                if not ok:
+                    b["skipped"].append({"test": tid, "reason": msg})
+                    self._note("queue %s: skipped %s: %s" % (b["group"], tid, msg))
+                    continue
+                # stop_batch lets the run in progress finish; Abort is what
+                # ends this one, and lands here as a non-PASS result.
+                while run.finished is None:
+                    time.sleep(0.2)
+                result = run.finished["result"]
+                b["done"].append({"test": tid, "result": result})
+                if result != _campaign.PASS:
+                    b["stop"] = True
+                    b["stopped"] = "%s on %s" % (result, tid)
+                    self._note("queue %s: stopped on %s (%s)" % (b["group"], tid, result))
+                    break
+        except Exception as e:  # noqa: BLE001 - a broken queue must not wedge the runner
+            b["stopped"] = "%s: %s" % (type(e).__name__, e)
+            self._note("queue %s: driver errored: %s" % (b["group"], b["stopped"]))
+        finally:
+            b["current"] = None
+            b["finished"] = now_ts()
+
+    def batch_snapshot(self):
+        b = self.batch
+        if b is None:
+            return None
+        done_ids = set(x["test"] for x in b["done"]) | set(x["test"] for x in b["skipped"])
+        return {"group": b["group"], "ts": b["ts"], "order": list(b["order"]),
+                "done": list(b["done"]), "skipped": list(b["skipped"]),
+                "pending": [t for t in b["order"] if t not in done_ids and t != b["current"]],
+                "current": b["current"], "stopping": bool(b["stop"]) and not b["finished"],
+                "stopped": b["stopped"], "finished": b["finished"]}
 
     # -- baseline around every run -----------------------------------------
     def _baseline_pre(self, run):
@@ -490,6 +627,8 @@ class Runner:
 
     # -- bench tools ---------------------------------------------------------
     def start_bench(self, tool_id, args=None, ack_live=False):
+        if self.batch_active():
+            return False, "a queue is running"
         if self.bench is None:
             return False, "no bench registry"
         tool = self.bench.get(tool_id)

forgetest/forgetest/server.py

@@ -19,6 +19,9 @@ Routes
   GET  /log                 the raw JSONL
   GET  /export/acceptance.json | .md   the last export
   POST /start {test, ack_live, ignore_requires}   start an acceptance test
+  POST /batch {group, ack_live, ignore_requires}  run everything a queue
+                            still owes, in prerequisite order
+  POST /batch/stop          cancel what is still queued
   POST /bench/start {tool, args, ack_live}
   POST /answer {prompt_id, value}
   POST /abort
@@ -261,6 +264,14 @@ class Handler(BaseHTTPRequestHandler):
                 ok, msg = r.start_test(str(body.get("test", "")), ack_live=bool(body.get("ack_live")),
                                        ignore_requires=bool(body.get("ignore_requires")))
                 self._send(200 if ok else 409, {"ok": ok, "message": msg})
+            elif path == "/batch":
+                ok, msg, order = r.start_batch(str(body.get("group", "")),
+                                               ack_live=bool(body.get("ack_live")),
+                                               ignore_requires=bool(body.get("ignore_requires")))
+                self._send(200 if ok else 409, {"ok": ok, "message": msg, "order": order})
+            elif path == "/batch/stop":
+                ok, msg = r.stop_batch()
+                self._send(200 if ok else 409, {"ok": ok, "message": msg})
             elif path == "/bench/start":
                 args = body.get("args") or {}
                 if not isinstance(args, dict):

forgetest/tests/test_queue.py

@@ -0,0 +1,342 @@
+"""The two queues: what they select, the order they run in, and where
+they stop.
+
+A campaign is mostly waiting, so the page offers to run what is still
+owed in one go: the unattended tests for an empty room, the attended ones
+for an operator at the machine. The rules that matter here are that a
+queue takes exactly the unsatisfied tests of its kinds, runs a
+prerequisite before the test that names it, stops on the first result
+that is not a PASS (a FAIL closes the campaign, so going on would open a
+second one behind the operator), and never fires the laser without the
+acknowledgment.
+"""
+import json
+import os
+import shutil
+import tempfile
+import threading
+import time
+import unittest
+import urllib.error
+import urllib.request
+
+import helpers
+from forgetest import catalog, server
+from forgetest.log import Log
+from forgetest.runner import BATCH_GROUPS, Failed, Runner
+
+
+def t_pass(ctx):
+    ctx.log("ok")
+
+
+def t_fail(ctx):
+    ctx.check(False, "deliberate")
+
+
+def t_prompt(ctx):
+    ctx.prompt("Ready?", ("Continue",))
+
+
+def t_slow(ctx):
+    ctx.sleep(30)
+
+
+class OrderTests(unittest.TestCase):
+    """order_by_requires is a pure function over the requires graph."""
+
+    def order(self, tests, selected):
+        return catalog.order_by_requires(tests, selected)
+
+    def test_prerequisite_first(self):
+        a = helpers.make_test("s.a", [], fn=t_pass)
+        b = helpers.make_test("s.b", [], requires=("s.a",), fn=t_pass)
+        # selection order must not matter; registration order breaks ties
+        self.assertEqual(self.order([b, a], ["s.b", "s.a"]), ["s.a", "s.b"])
+        self.assertEqual(self.order([a, b], ["s.b", "s.a"]), ["s.a", "s.b"])
+
+    def test_chain(self):
+        c = helpers.make_test("s.c", [], requires=("s.b",), fn=t_pass)
+        b = helpers.make_test("s.b", [], requires=("s.a",), fn=t_pass)
+        a = helpers.make_test("s.a", [], fn=t_pass)
+        self.assertEqual(self.order([c, b, a], ["s.c", "s.b", "s.a"]), ["s.a", "s.b", "s.c"])
+
+    def test_registration_order_otherwise(self):
+        ts = [helpers.make_test("s.%d" % i, [], fn=t_pass) for i in range(4)]
+        self.assertEqual(self.order(ts, [t.id for t in ts]), ["s.0", "s.1", "s.2", "s.3"])
+
+    def test_prerequisite_outside_the_selection_places_nothing(self):
+        a = helpers.make_test("s.a", [], fn=t_pass)
+        b = helpers.make_test("s.b", [], requires=("s.a",), fn=t_pass)
+        self.assertEqual(self.order([a, b], ["s.b"]), ["s.b"])
+
+    def test_unknown_and_empty(self):
+        a = helpers.make_test("s.a", [], fn=t_pass)
+        self.assertEqual(self.order([a], []), [])
+        self.assertEqual(self.order([a], ["s.nope"]), [])
+
+    def test_a_cycle_does_not_recurse_forever(self):
+        """validate() rejects cycles; a malformed registry must still not
+        hang the runner."""
+        a = helpers.make_test("s.a", [], requires=("s.b",), fn=t_pass)
+        b = helpers.make_test("s.b", [], requires=("s.a",), fn=t_pass)
+        out = self.order([a, b], ["s.a", "s.b"])
+        self.assertEqual(sorted(out), ["s.a", "s.b"])
+
+
+class QueueTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.tmp = tempfile.mkdtemp(prefix="forgetest-queue-")
+        os.environ["FORGETEST_DATA"] = cls.tmp
+        os.environ["FORGETEST_MARKER"] = os.path.join(cls.tmp, "marker")
+        cls.man = helpers.make_manifest()
+        cls.reg = helpers.registry(
+            # auto: b requires a, so a queue must run a first even though
+            # b registers earlier
+            helpers.make_test("q.b", [("forgectrl", "src/ui.c")], requires=("q.a",), fn=t_pass),
+            helpers.make_test("q.a", [("forgectrl", "src/auth.c")], fn=t_pass),
+            helpers.make_test("q.c", [("forgectrl", "src/cool.c")], fn=t_pass),
+            # attended
+            helpers.make_test("q.op", [("forgectrl", "src/main.c")], kind="operator", fn=t_prompt),
+            helpers.make_test("q.live", [("grblhal-glowforge", "src/**")], kind="live", fn=t_pass),
+        )
+        cls.log = Log(os.path.join(cls.tmp, "results.jsonl"))
+        cls.runner = Runner(cls.log, cls.man, cls.reg)
+        cls.token = server.load_token(os.path.join(cls.tmp, "token"))
+        cls.app = server.App(cls.runner, cls.token, export_dir=os.path.join(cls.tmp, "export"))
+        cls.srv = server.make_server(cls.app, "127.0.0.1", 0)
+        cls.port = cls.srv.server_address[1]
+        threading.Thread(target=cls.srv.serve_forever, daemon=True).start()
+
+    @classmethod
+    def tearDownClass(cls):
+        cls.srv.shutdown()
+        cls.srv.server_close()
+        shutil.rmtree(cls.tmp, ignore_errors=True)
+
+    def setUp(self):
+        # A clean history per test: a PASS left by an earlier one would be
+        # inherited and drop that test out of the selection. Truncating is
+        # also the shrunk-file path Log.read has to notice.
+        open(self.log.path, "w").close()
+        self.assertEqual(self.log.read(), [])
+        self.runner.batch = None
+
+    def call(self, method, path, body=None):
+        url = "http://127.0.0.1:%d%s" % (self.port, path)
+        hdrs = {"Host": "127.0.0.1:%d" % self.port, "X-ForgeFIRM-Token": self.token}
+        data = None
+        if body is not None:
+            data = json.dumps(body).encode()
+            hdrs["Content-Type"] = "application/json"
+        req = urllib.request.Request(url, data=data, method=method, headers=hdrs)
+        try:
+            with urllib.request.urlopen(req, timeout=10) as r:
+                return r.status, json.loads(r.read().decode())
+        except urllib.error.HTTPError as e:
+            return e.code, json.loads(e.read().decode())
+
+    def wait_queue(self, timeout=30):
+        deadline = time.time() + timeout
+        while time.time() < deadline:
+            st, d = self.call("GET", "/state")
+            if d["batch"] and d["batch"]["finished"]:
+                return d
+            time.sleep(0.05)
+        self.fail("the queue did not finish")
+
+    # -- selection ---------------------------------------------------------
+    def test_groups_split_the_catalog_by_who_has_to_be_there(self):
+        self.assertEqual(BATCH_GROUPS["unattended"], ("auto",))
+        self.assertEqual(sorted(BATCH_GROUPS["attended"]), ["live", "operator"])
+        st, d = self.call("GET", "/state")
+        self.assertEqual(d["batch_available"]["unattended"], ["q.a", "q.b", "q.c"])
+        self.assertEqual(d["batch_available"]["attended"], ["q.op", "q.live"])
+
+    def test_selection_runs_prerequisites_first(self):
+        """q.b registers before q.a but requires it."""
+        st, d = self.call("GET", "/state")
+        order = d["batch_available"]["unattended"]
+        self.assertLess(order.index("q.a"), order.index("q.b"))
+
+    def test_satisfied_tests_drop_out(self):
+        st, d = self.call("POST", "/start", {"test": "q.c"})
+        self.assertEqual(st, 200)
+        self.wait_idle()
+        st, d = self.call("GET", "/state")
+        self.assertEqual(d["tests"]["q.c"]["status"], "pass")
+        self.assertNotIn("q.c", d["batch_available"]["unattended"])
+        self.assertEqual(d["batch_available"]["unattended"], ["q.a", "q.b"])
+
+    def wait_idle(self, timeout=20):
+        deadline = time.time() + timeout
+        while time.time() < deadline:
+            st, d = self.call("GET", "/state")
+            if not d["running"]:
+                return d
+            time.sleep(0.05)
+        self.fail("run did not finish")
+
+    # -- running -----------------------------------------------------------
+    def test_unattended_queue_runs_everything_in_order(self):
+        st, d = self.call("POST", "/batch", {"group": "unattended"})
+        self.assertEqual(st, 200)
+        self.assertEqual(d["order"], ["q.a", "q.b", "q.c"])
+        d = self.wait_queue()
+        b = d["batch"]
+        self.assertEqual([x["test"] for x in b["done"]], ["q.a", "q.b", "q.c"])
+        self.assertEqual(set(x["result"] for x in b["done"]), {"PASS"})
+        self.assertEqual(b["skipped"], [])
+        self.assertIsNone(b["stopped"])
+        for tid in ("q.a", "q.b", "q.c"):
+            self.assertEqual(d["tests"][tid]["status"], "pass", tid)
+        # and the run records say which queue put them there
+        st, rec = self.call("GET", "/result?test=q.b")
+        self.assertEqual(rec["evidence"]["batch"]["group"], "unattended")
+
+    def test_an_empty_queue_is_refused_not_started(self):
+        self.call("POST", "/batch", {"group": "unattended"})
+        self.wait_queue()
+        st, d = self.call("POST", "/batch", {"group": "unattended"})
+        self.assertEqual(st, 409)
+        self.assertIn("already satisfied", d["message"])
+
+    def test_a_failure_stops_the_queue(self):
+        reg = helpers.registry(
+            helpers.make_test("f.a", [("forgectrl", "src/ui.c")], fn=t_pass),
+            helpers.make_test("f.b", [("forgectrl", "src/auth.c")], fn=t_fail),
+            helpers.make_test("f.c", [("forgectrl", "src/cool.c")], fn=t_pass),
+        )
+        tmp = tempfile.mkdtemp(prefix="forgetest-fail-")
+        try:
+            r = Runner(Log(os.path.join(tmp, "r.jsonl")), self.man, reg)
+            ok, msg, order = r.start_batch("unattended")
+            self.assertTrue(ok, msg)
+            self.assertEqual(order, ["f.a", "f.b", "f.c"])
+            deadline = time.time() + 20
+            while not r.batch["finished"] and time.time() < deadline:
+                time.sleep(0.05)
+            b = r.batch_snapshot()
+            self.assertEqual([x["test"] for x in b["done"]], ["f.a", "f.b"])
+            self.assertEqual(b["done"][1]["result"], "FAIL")
+            self.assertEqual(b["stopped"], "FAIL on f.b")
+            self.assertEqual(b["pending"], ["f.c"], "f.c ran after a FAIL closed the campaign")
+        finally:
+            shutil.rmtree(tmp, ignore_errors=True)
+
+    def test_a_test_that_cannot_start_is_skipped_not_silently_dropped(self):
+        """An unmet prerequisite outside the queue refuses the start; the
+        queue records why and carries on."""
+        reg = helpers.registry(
+            helpers.make_test("k.op", [("forgectrl", "src/main.c")], kind="operator", fn=t_pass),
+            helpers.make_test("k.a", [("forgectrl", "src/ui.c")], requires=("k.op",), fn=t_pass),
+            helpers.make_test("k.b", [("forgectrl", "src/auth.c")], fn=t_pass),
+        )
+        tmp = tempfile.mkdtemp(prefix="forgetest-skip-")
+        try:
+            r = Runner(Log(os.path.join(tmp, "r.jsonl")), self.man, reg)
+            ok, msg, order = r.start_batch("unattended")
+            self.assertEqual(order, ["k.a", "k.b"])
+            deadline = time.time() + 20
+            while not r.batch["finished"] and time.time() < deadline:
+                time.sleep(0.05)
+            b = r.batch_snapshot()
+            self.assertEqual([x["test"] for x in b["skipped"]], ["k.a"])
+            self.assertIn("prerequisites not satisfied", b["skipped"][0]["reason"])
+            self.assertEqual([x["test"] for x in b["done"]], ["k.b"])
+            self.assertIsNone(b["stopped"])
+        finally:
+            shutil.rmtree(tmp, ignore_errors=True)
+
+    # -- safety and exclusion ----------------------------------------------
+    def test_an_attended_queue_with_a_live_test_needs_the_acknowledgment(self):
+        st, d = self.call("POST", "/batch", {"group": "attended"})
+        self.assertEqual(st, 409)
+        self.assertIn("fires the laser", d["message"])
+        self.assertIn("q.live", d["message"])
+        st, d = self.call("GET", "/state")
+        self.assertIsNone(d["batch"], "a refused queue must not exist")
+
+    def test_the_acknowledgment_reaches_every_live_run(self):
+        st, d = self.call("POST", "/batch", {"group": "attended", "ack_live": True})
+        self.assertEqual(st, 200)
+        # the operator test prompts; answer it so the queue can reach the live one
+        deadline = time.time() + 20
+        while time.time() < deadline:
+            st, s = self.call("GET", "/state")
+            if s["running"] and s["running"]["prompt"]:
+                self.call("POST", "/answer", {"prompt_id": s["running"]["prompt"]["id"],
+                                              "value": "Continue"})
+                break
+            if s["batch"] and s["batch"]["finished"]:
+                break
+            time.sleep(0.05)
+        d = self.wait_queue()
+        self.assertEqual([x["test"] for x in d["batch"]["done"]], ["q.op", "q.live"])
+        st, rec = self.call("GET", "/result?test=q.live")
+        self.assertTrue(rec["evidence"]["operator"]["ack_live"])
+
+    def test_a_single_start_is_refused_while_a_queue_runs(self):
+        reg = helpers.registry(
+            helpers.make_test("s.slow", [("forgectrl", "src/ui.c")], fn=t_slow),
+            helpers.make_test("s.other", [("forgectrl", "src/auth.c")], fn=t_pass),
+        )
+        tmp = tempfile.mkdtemp(prefix="forgetest-excl-")
+        try:
+            r = Runner(Log(os.path.join(tmp, "r.jsonl")), self.man, reg)
+            r.start_batch("unattended")
+            deadline = time.time() + 10
+            while r.current is None and time.time() < deadline:
+                time.sleep(0.02)
+            ok, msg = r.start_test("s.other")
+            self.assertFalse(ok)
+            self.assertEqual(msg, "a queue is running")
+            ok, msg = r.start_bench("anything")
+            self.assertFalse(ok)
+            self.assertEqual(msg, "a queue is running")
+            ok, msg, _ = r.start_batch("unattended")
+            self.assertFalse(ok)
+            self.assertIn("already running", msg)
+            r.stop_batch()
+            r.abort()
+            deadline = time.time() + 20
+            while not r.batch["finished"] and time.time() < deadline:
+                time.sleep(0.05)
+        finally:
+            shutil.rmtree(tmp, ignore_errors=True)
+
+    def test_stop_cancels_what_is_waiting(self):
+        reg = helpers.registry(
+            helpers.make_test("p.slow", [("forgectrl", "src/ui.c")], fn=t_slow),
+            helpers.make_test("p.next", [("forgectrl", "src/auth.c")], fn=t_pass),
+        )
+        tmp = tempfile.mkdtemp(prefix="forgetest-stop-")
+        try:
+            r = Runner(Log(os.path.join(tmp, "r.jsonl")), self.man, reg)
+            r.start_batch("unattended")
+            deadline = time.time() + 10
+            while r.current is None and time.time() < deadline:
+                time.sleep(0.02)
+            ok, msg = r.stop_batch()
+            self.assertTrue(ok, msg)
+            self.assertTrue(r.batch_snapshot()["stopping"])
+            r.abort()          # the run in progress needs its own lever
+            deadline = time.time() + 20
+            while not r.batch["finished"] and time.time() < deadline:
+                time.sleep(0.05)
+            b = r.batch_snapshot()
+            self.assertNotIn("p.next", [x["test"] for x in b["done"]])
+            ok, msg = r.stop_batch()
+            self.assertFalse(ok)
+        finally:
+            shutil.rmtree(tmp, ignore_errors=True)
+
+    def test_unknown_group(self):
+        st, d = self.call("POST", "/batch", {"group": "everything"})
+        self.assertEqual(st, 409)
+        self.assertEqual(d["message"], "unknown queue")
+
+
+if __name__ == "__main__":
+    unittest.main()

forgetest/tests/test_responsiveness.py

@@ -184,6 +184,12 @@ class PageTests(unittest.TestCase):
         self.assertIn("function updateBench()", html)
         # the prompt buttons are rebuilt only when the prompt changes
         self.assertIn("if(pk!==promptKey)", html)
+        # the queue controls are static markup: a poll relabels them and
+        # flips disabled, it never replaces the node
+        for bid in ("q-unattended", "q-attended", "q-stop"):
+            self.assertIn("id='%s'" % bid, html)
+            self.assertNotIn("id='" + bid + "-", html)
+        self.assertIn("function renderQueue()", html)
 
     def test_page_is_self_contained_ascii(self):
         html = page.render("0" * 32)

forgetest/tests/test_server.py

@@ -167,25 +167,6 @@ class ServerTests(unittest.TestCase):
         st, d = self.call("GET", "/nope")
         self.assertEqual(st, 404)
 
-    def test_01b_state_is_conditional(self):
-        """The page polls; an unchanged state must cost a 304, and the
-        ETag must move as soon as the state does."""
-        st, d, hdrs = self.call_h("GET", "/state")
-        self.assertEqual(st, 200)
-        etag = hdrs.get("ETag")
-        self.assertTrue(etag and etag.startswith('"'), "no ETag on /state")
-        st, _, hdrs2 = self.call_h("GET", "/state", headers={"If-None-Match": etag})
-        self.assertEqual(st, 304)
-        self.assertEqual(hdrs2.get("ETag"), etag)
-        # a stale validator must not be honored
-        st, _, _ = self.call_h("GET", "/state", headers={"If-None-Match": '"stale"'})
-        self.assertEqual(st, 200)
-        # and the validator moves with the state
-        self.call("POST", "/invalidate", {"reason": "etag check"})
-        st, _, hdrs3 = self.call_h("GET", "/state", headers={"If-None-Match": etag})
-        self.assertEqual(st, 200)
-        self.assertNotEqual(hdrs3.get("ETag"), etag)
-
     def test_01c_connection_is_kept_alive(self):
         """A poll per second over a fresh TCP connection each time is
         waste the board does not need to pay."""
@@ -399,6 +380,31 @@ class ServerTests(unittest.TestCase):
         self.assertTrue(any("recovered" in m for m in r.messages))
         self.assertFalse(os.path.exists(marker))
 
+    def test_10_state_is_conditional(self):
+        """The page polls; an unchanged state must cost a 304, and the
+        ETag must move as soon as the state does.
+
+        Runs last because it invalidates: timestamps are whole seconds, and
+        a PASS stamped in the same second as an invalidate is deliberately
+        not inheritable, so an invalidate early in this class would decide
+        the inheritance the earlier tests are checking.
+        """
+        st, d, hdrs = self.call_h("GET", "/state")
+        self.assertEqual(st, 200)
+        etag = hdrs.get("ETag")
+        self.assertTrue(etag and etag.startswith('"'), "no ETag on /state")
+        st, _, hdrs2 = self.call_h("GET", "/state", headers={"If-None-Match": etag})
+        self.assertEqual(st, 304)
+        self.assertEqual(hdrs2.get("ETag"), etag)
+        # a stale validator must not be honored
+        st, _, _ = self.call_h("GET", "/state", headers={"If-None-Match": '"stale"'})
+        self.assertEqual(st, 200)
+        # and the validator moves with the state
+        self.call("POST", "/invalidate", {"reason": "etag check"})
+        st, _, hdrs3 = self.call_h("GET", "/state", headers={"If-None-Match": etag})
+        self.assertEqual(st, 200)
+        self.assertNotEqual(hdrs3.get("ETag"), etag)
+
 
 if __name__ == "__main__":
     unittest.main()