Let the page run a campaign's remaining tests as a queue

A campaign is mostly waiting for the next Start. The page now offers two
queues, and each takes every test of its kinds the campaign does not
already count as satisfied: Unattended for the auto tests, which need
nobody in the room, and Operator and live for the ones that need somebody
at the machine, since they prompt and they fire the laser. The buttons
say how many they would run and ask before starting; the live queue names
the tests that fire and takes the acknowledgment once, for all of them.

A queue runs one test at a time through the runner's single slot, in
prerequisite order. Registration order otherwise, so a run reads down the
page, but a prerequisite inside the queue always goes first. It stops on
the first result that is not a PASS: a FAIL closes the campaign, and
carrying on would only open a second one behind the operator's back. A
test the runner refuses to start is skipped with the reason on the page
and the rest carry on, which is what happens to an auto test waiting on an
operator one: run the attended queue, then the unattended one again.

The queue lives in the runner, not in the tab, so reloading the page or
closing it leaves the run alone. While one is up it holds the machine
between its tests as well as during them, so a single Start and the bench
tools are refused rather than cutting in. Stop the queue cancels what is
still waiting and lets the run in progress finish; Abort ends that one
too, and lands as the non-PASS that stops the queue. Every run a queue
starts records which one put it there.

Also moves the /state ETag test to the end of its class. It invalidates,
timestamps are whole seconds, and a PASS stamped in the same second as an
invalidate is deliberately not inheritable, so on a fast run it decided
the inheritance an earlier test was checking.

forgetest is a dev-only component and can never appear in a coverage map,
so this has no acceptance catalog consequence.

forgetest/forgetest/catalog.py

@@ -120,6 +120,35 @@ def get(id, registry=None):
     return (registry if registry is not None else REGISTRY).get(id)
 
 
+def order_by_requires(tests, selected):
+    """`selected` ids in an order that runs a prerequisite before the test
+    that names it.
+
+    Registration order otherwise, so a run reads down the page. A
+    prerequisite outside the selection places nothing: either it is
+    already satisfied, or the run that needs it will be refused and
+    recorded as skipped. validate() has ruled out cycles; the stack check
+    only keeps a malformed registry from recursing forever.
+    """
+    want = set(selected)
+    by_id = {t.id: t for t in tests}
+    out, placed = [], set()
+
+    def visit(tid, stack):
+        if tid in placed or tid not in want or tid not in by_id or tid in stack:
+            return
+        stack.add(tid)
+        for r in by_id[tid].requires:
+            visit(r, stack)
+        stack.discard(tid)
+        placed.add(tid)
+        out.append(tid)
+
+    for t in tests:
+        visit(t.id, set())
+    return out
+
+
 def validate(registry=None):
     """Every `requires` names a known test and there are no cycles."""
     reg = registry if registry is not None else REGISTRY

forgetest/forgetest/page.py

@@ -76,6 +76,16 @@ pre#log{background:#1d1e26;color:#d7dae0;font-family:ui-monospace,Consolas,monos
 .switch .on{color:var(--warn);font-weight:600}
 .req.over{color:var(--dim)}
 .grp{margin-top:6px}
+.queue{background:#f7f8fa;margin:10px 0 0;padding:10px 12px}
+.queue h2{margin-bottom:8px}
+.queue .hint{margin-top:8px}
+.qline{font-size:12.5px;color:var(--dim);line-height:1.7;margin-top:8px}
+.qline b{color:var(--txt)}
+.qbar{display:flex;height:6px;border-radius:3px;overflow:hidden;background:#e2e4e9;margin:6px 0}
+.qbar i{display:block}
+.qbar i.ok{background:var(--ok)}.qbar i.bad{background:var(--red)}
+.qbar i.skip{background:var(--warn)}.qbar i.now{background:var(--blue)}
+.badge.queued{background:#dde7f5;color:#24507f}
 .tool .argrow{display:flex;gap:8px;flex-wrap:wrap;margin:6px 0}
 .tool .argrow label{font-size:12px;color:var(--dim)}
 </style></head><body>
@@ -88,6 +98,19 @@ pre#log{background:#1d1e26;color:#d7dae0;font-family:ui-monospace,Consolas,monos
    <div class='banner'><div class='auth' id='auth'>?</div>
     <div class='kv' id='banner'></div></div>
    <div id='invnote'></div><div id='msgs'></div>
+   <div class='card queue'><h2>Run what is left</h2>
+    <div class='actions'>
+     <button class='pri' id='q-unattended' onclick='startBatch("unattended")'>Unattended</button>
+     <button class='pri' id='q-attended' onclick='startBatch("attended")'>Operator and live</button>
+     <button class='danger' id='q-stop' onclick='stopBatch()' disabled>Stop the queue</button>
+    </div>
+    <div id='qstate'></div><div id='qmsg'></div>
+    <p class='hint'>Each queue takes every test of its kind that the campaign does not
+     already count as satisfied, runs them one at a time in prerequisite order, and stops
+     on the first result that is not a PASS. The unattended queue needs nobody in the room.
+     The other one does: it prompts, and it fires the laser. Stop-the-queue cancels what is
+     still waiting and lets the run in progress finish; Abort ends that one too.</p>
+   </div>
    <div class='actions' style='margin-top:10px'>
     <button class='pri' id='exportbtn' onclick='doExport()'>Export release artifact</button>
     <a id='dljson' href='/export/acceptance.json' style='display:none'><button>acceptance.json</button></a>
@@ -159,7 +182,7 @@ function api(method,path,body,cb,hdrs,timeoutMs){var x=new XMLHttpRequest();x.op
 /* The state poll. It carries the last ETag, so an unchanged state costs
    a 304 and no re-render; a run gets a faster tick, and every action
    pulls the next poll forward instead of waiting out the interval. */
-function pollDelay(){return (state&&state.running)?1000:2000}
+function pollDelay(){return (state&&(state.running||batchActive()))?1000:2000}
 function schedule(ms){if(pollTimer)window.clearTimeout(pollTimer);pollTimer=window.setTimeout(poll,ms)}
 function kick(){schedule(120)}
 function poll(){if(polling){schedule(150);return}polling=true;
@@ -181,12 +204,14 @@ function setIgnoreReq(on){ignoreReq=!!on;try{window.localStorage.setItem('forget
  if(state&&catalog)renderGroups()}
 /* A start already sent but not yet seen in the state counts as busy, so
    the buttons grey out on the click rather than on the next poll. The
-   window is capped in case the answer never arrives. */
-function isBusy(){if(state&&state.running)return true;
+   window is capped in case the answer never arrives. A queue holds the
+   machine between its tests as well as during them. */
+function batchActive(){return !!(state&&state.batch&&!state.batch.finished)}
+function isBusy(){if(state&&(state.running||batchActive()))return true;
  return !!(pending&&(nowMs()-pending)<PENDING_MS)}
 function fmtTs(t){return t?t.replace('T',' ').replace('Z',' UTC'):'-'}
 function render(){if(!state||!catalog)return;
- if(state.running){pending=0;pendingId=null}
+ if(state.running||batchActive()){pending=0;pendingId=null}
  var m=state.manifest||{};setText($('hdrver'),(m.version||'?'));setText($('hdrsub'),m.image||'');
  var a=$('auth');setText(a,'Release authorized: '+(state.authorized?'YES':'NO'));setProp(a,'className','auth'+(state.authorized?' yes':''));
  var c=state.campaign,cn=state.counts||{};var b='';
@@ -197,7 +222,34 @@ function render(){if(!state||!catalog)return;
  setHtml($('banner'),b);
  var inv=state.invalidate;setHtml($('invnote'),inv?"<div class='note'>Full campaign required since "+esc(fmtTs(inv.ts))+": "+esc(inv.reason)+"</div>":'');
  var ms='';(state.messages||[]).forEach(function(x){ms+="<div class='note'>"+esc(x)+"</div>"});setHtml($('msgs'),ms);
- renderGroups();renderRun();if(bench)renderBench()}
+ renderQueue();renderGroups();renderRun();if(bench)renderBench()}
+/* The two queues: what each would run now, and how the running one is
+   getting on. Built from the state, so a reload picks the queue back up
+   exactly where it is - the queue lives in the runner, not in this tab. */
+var QUEUES=[['unattended','Unattended'],['attended','Operator and live']];
+function renderQueue(){var av=state.batch_available||{},b=state.batch,busy=isBusy();
+ QUEUES.forEach(function(p){var e=$('q-'+p[0]);if(!e)return;
+  var ids=av[p[0]]||[];
+  setText(e,ids.length?(p[1]+' ('+ids.length+')'):(p[1]+' (none left)'));
+  setDis(e,busy||!ids.length);
+  setProp(e,'title',!ids.length?('nothing left: every '+p[0]+' test is satisfied')
+   :(busy?'a run is in progress':('in order: '+ids.join(', '))))});
+ setDis($('q-stop'),!batchActive()||!!(b&&b.stopping));
+ if(!b){setHtml($('qstate'),'');return}
+ var total=b.order.length||1,ok=0,bad=0;
+ b.done.forEach(function(x){if(x.result==='PASS')ok++;else bad++});
+ function seg(cls,n){return n?("<i class='"+cls+"' style='width:"+(100*n/total)+"%'></i>"):''}
+ var h="<div class='qbar'>"+seg('ok',ok)+seg('bad',bad)+seg('skip',b.skipped.length)+
+   (b.current?seg('now',1):'')+"</div><div class='qline'><b>"+esc(b.group)+"</b> queue, opened "+
+   esc(fmtTs(b.ts))+" &middot; ";
+ h+=b.finished?('finished '+esc(fmtTs(b.finished))):(b.current?('running <b>'+esc(b.current)+'</b>'):
+   (b.stopping?'stopping':'starting'));
+ h+=" &middot; <b>"+ok+"</b> passed, <b>"+bad+"</b> not, <b>"+b.skipped.length+
+   "</b> skipped, <b>"+b.pending.length+"</b> waiting";
+ if(b.stopped)h+="<br><span class='req'>stopped: "+esc(b.stopped)+"</span>";
+ if(b.skipped.length)h+="<br>skipped: "+esc(b.skipped.map(function(x){return x.test+' ('+x.reason+')'}).join('; '));
+ if(b.pending.length)h+="<br>waiting: <span class='tid'>"+esc(b.pending.join(', '))+"</span>";
+ setHtml($('qstate'),h+"</div>")}
 /* The rows are built once for a given catalog and then only updated in
    place: a poll never rewrites the table, so a Start button survives the
    press that is landing on it. */
@@ -225,11 +277,13 @@ function buildGroups(){var groups={},order=[];
  catalog.forEach(function(t){rowEls[t.id]={st:$('st-'+t.id),last:$('last-'+t.id),btn:$('btn-'+t.id),
   unmet:$('unmet-'+t.id),note:$('note-'+t.id),detdyn:$('detdyn-'+t.id)}})}
 function updateGroups(){if(!rowEls||!state)return;var busy=isBusy();
+ var queued={};if(batchActive()){(state.batch.pending||[]).forEach(function(x){queued[x]=1})}
  catalog.forEach(function(t){var e=rowEls[t.id];if(!e)return;var s=state.tests[t.id]||{};
   var st;
   if(pendingId===t.id&&isBusy()&&!state.running){st="<span class='st running'>starting&hellip;</span>"}
   else{st="<span class='st "+esc(s.status)+"'>"+esc(s.status||'none')+"</span>";
    if(s.required&&s.status!=='running')st+="<br><span class='req'>required: "+esc(s.reason)+"</span>"}
+  if(queued[t.id])st+="<br><span class='badge queued'>queued</span>";
   setHtml(e.st,st);
   var last=s.last?(esc(s.last.result)+' '+esc(fmtTs(s.last.ts))):'-';
   if(s.status==='inherited'&&s.origin)last+="<br><span class='tid'>from "+esc(s.origin.campaign)+" on "+esc(s.origin.image)+"</span>";
@@ -275,7 +329,28 @@ function startTest(id){if(isBusy())return;var t=findTest(id);var body={test:id};
  api('POST','/start',body,function(s,d){
   if(s!==200){pending=0;pendingId=null;rowMsg[id]=d.message||d.error;updateGroups()}
   setMsg('actmsg',d.message||d.error,s!==200);kick()})}
-function confirmLive(){return window.confirm('LIVE LASER TEST.\n\nConfirm before starting:\n - eye protection on, everyone in the room\n - fire watch present, extinguisher at hand\n - exhaust running, lid closed, scrap in place\n - you will press the physical button to arm when prompted\n\nStart the test?')}
+function confirmLive(live){return window.confirm(
+ (live?('LIVE LASER QUEUE.\n\nThese fire the laser:\n - '+live.join('\n - ')+'\n'):'LIVE LASER TEST.\n')+
+ '\nConfirm before starting:\n - eye protection on, everyone in the room\n - fire watch present, extinguisher at hand\n - exhaust running, lid closed, scrap in place\n - you will press the physical button to arm when prompted\n\n'+
+ (live?'Start the queue?':'Start the test?'))}
+/* A queue takes the machine for a long stretch, so both what it will run
+   and the acknowledgment it needs are put in front of the operator once,
+   before anything starts. */
+function startBatch(group){if(isBusy())return;
+ var ids=(state&&state.batch_available&&state.batch_available[group])||[];
+ if(!ids.length)return;
+ var body={group:group};if(ignoreReq)body.ignore_requires=true;
+ var live=[];catalog.forEach(function(t){if(t.kind==='live'&&ids.indexOf(t.id)>=0)live.push(t.id)});
+ if(live.length){if(!confirmLive(live))return;body.ack_live=true}
+ else if(!window.confirm('Run these '+ids.length+' test(s), in this order?\n\n - '+ids.join('\n - ')))return;
+ pending=nowMs();pendingId=null;rowMsg={};updateGroups();renderQueue();
+ setMsg('qmsg','starting the '+group+' queue...');
+ api('POST','/batch',body,function(s,d){
+  if(s!==200){pending=0;updateGroups();renderQueue()}
+  setMsg('qmsg',d.message||d.error,s!==200);kick()})}
+function stopBatch(){var e=$('q-stop');if(e.disabled)return;setDis(e,true);
+ setMsg('qmsg','stopping the queue...');
+ api('POST','/batch/stop',{},function(s,d){setMsg('qmsg',d.message||d.error,s!==200);kick()})}
 function promptBusy(on){var b=$('promptb').getElementsByTagName('button');for(var i=0;i<b.length;i++)setDis(b[i],on)}
 function answerIdx(i){if(!curPrompt)return;var p=curPrompt,v=p.options[i];
  promptBusy(true);setMsg('actmsg','answer sent: '+v);

forgetest/forgetest/runner.py

@@ -10,6 +10,13 @@ the campaign rules (campaign.py) do the rest.
 Bench tools are subprocesses (bench.py registry): same single slot, same
 log pane, no campaign effect.
 
+A queue (BATCH_GROUPS) runs what a campaign still owes, one test at a
+time through that same single slot: the unattended tests for an empty
+room, the attended ones for an operator at the machine. It runs them in
+prerequisite order and stops on the first result that is not a PASS,
+because a FAIL closes the campaign and going on would quietly open a
+second one.
+
 Safety, in code rather than convention: a live test starts only with the
 operator's acknowledgment in the request; the runner never touches the
 laser latch; a takeover always ends with forgectrl started again, and a
@@ -31,6 +38,16 @@ from .log import now_ts, data_dir
 
 MAX_LINES = 4000
 
+# The two queues the page offers. A campaign is mostly waiting: the
+# unattended tests need nobody in the room, the attended ones need the
+# operator at the machine, and sorting them that way lets one set run
+# while the operator is elsewhere. Each queue takes everything of its
+# kinds the campaign does not already count as satisfied.
+BATCH_GROUPS = {
+    "unattended": ("auto",),
+    "attended": ("operator", "live"),
+}
+
 
 class Aborted(Exception):
     pass
@@ -308,6 +325,7 @@ class Runner:
         self._lock = threading.Lock()
         self.current = None
         self.last = None
+        self.batch = None
         self.messages = []
         self.boot_ref = None
         self.recover()
@@ -363,6 +381,10 @@ class Runner:
         st["running"] = r.snapshot() if r and not r.finished else None
         last = r if (r and r.finished) else self.last
         st["last_run"] = last.snapshot() if last else None
+        st["batch"] = self.batch_snapshot()
+        # what each queue would run if started now, so the page can label
+        # its buttons with the work rather than a bare verb
+        st["batch_available"] = {g: self.batch_selection(g, st) for g in BATCH_GROUPS}
         return st, records
 
     def busy(self):
@@ -407,19 +429,27 @@ class Runner:
         evidence records which prerequisites were unmet (the release gate
         needs every test satisfied anyway, so nothing is hidden - the
         record just says the order was the operator's)."""
+        if self.batch_active():
+            return False, "a queue is running"
+        ok, msg, _ = self._start_test(test_id, ack_live, ignore_requires)
+        return ok, msg
+
+    def _start_test(self, test_id, ack_live=False, ignore_requires=False, batch=None):
+        """As start_test, and hands back the Run so the queue driver can
+        follow it without racing another start for `current`."""
         t = _catalog.get(test_id, self.registry)
         if t is None:
-            return False, "unknown test"
+            return False, "unknown test", None
         with self._lock:
             if self.busy():
-                return False, "a run is in progress"
+                return False, "a run is in progress", None
             state, _ = self.state()
             ts = state["tests"][t.id]
             missing = list(ts["missing_requires"])
             if missing and not ignore_requires:
-                return False, "prerequisites not satisfied: %s" % ", ".join(missing)
+                return False, "prerequisites not satisfied: %s" % ", ".join(missing), None
             if t.kind == "live" and not ack_live:
-                return False, "live test: acknowledge eye protection, fire watch, and exhaust first"
+                return False, "live test: acknowledge eye protection, fire watch, and exhaust first", None
             campaign = self._open_campaign_if_needed(state)
             run = Run("test", t.id, t.title)
             self.last = self.current
@@ -430,10 +460,117 @@ class Runner:
             run.evidence["prerequisites"] = {"overridden": True, "missing": missing, "ts": now_ts()}
         if t.kind == "live":
             run.evidence["operator"] = {"ack_live": True, "ts": now_ts()}
+        if batch:
+            run.log("queued by the %s queue opened %s" % (batch["group"], batch["ts"]))
+            run.evidence["batch"] = {"group": batch["group"], "ts": batch["ts"]}
         th = threading.Thread(target=self._exec_test, args=(t, run, campaign), daemon=True,
                               name="forgetest-run")
         th.start()
-        return True, "started"
+        return True, "started", run
+
+    # -- the queue ----------------------------------------------------------
+    def batch_active(self):
+        b = self.batch
+        return bool(b and not b["finished"])
+
+    def batch_selection(self, group, state=None):
+        """The ids the given queue would run, in prerequisite order: every
+        test of those kinds that the campaign does not already count as
+        satisfied, which is exactly what is left to do."""
+        kinds = BATCH_GROUPS.get(group)
+        if kinds is None:
+            return None
+        if state is None:
+            state, _ = self.state()
+        tests = self.tests()
+        want = [t.id for t in tests
+                if t.kind in kinds and not state["tests"][t.id]["satisfied"]]
+        return _catalog.order_by_requires(tests, want)
+
+    def start_batch(self, group, ack_live=False, ignore_requires=False):
+        if group not in BATCH_GROUPS:
+            return False, "unknown queue", None
+        if self.batch_active():
+            return False, "a queue is already running", None
+        if self.busy():
+            return False, "a run is in progress", None
+        order = self.batch_selection(group)
+        if not order:
+            return False, "nothing to run: every %s test is already satisfied" % group, None
+        live = [tid for tid in order
+                if _catalog.get(tid, self.registry).kind == "live"]
+        if live and not ack_live:
+            return False, ("this queue fires the laser (%s): acknowledge eye protection, "
+                           "fire watch, and exhaust first" % ", ".join(live)), None
+        with self._lock:
+            if self.batch_active() or self.busy():
+                return False, "a run is in progress", None
+            self.batch = {"group": group, "ts": now_ts(), "order": list(order), "done": [],
+                          "skipped": [], "current": None, "stop": False, "stopped": None,
+                          "finished": None, "ack_live": bool(ack_live),
+                          "ignore_requires": bool(ignore_requires)}
+            b = self.batch
+        self._note("queue %s: %d test(s) to run: %s" % (group, len(order), ", ".join(order)))
+        threading.Thread(target=self._drive_batch, args=(b,), daemon=True,
+                         name="forgetest-queue").start()
+        return True, "queue started: %d test(s)" % len(order), list(order)
+
+    def stop_batch(self):
+        """Cancel what is still queued. The run in progress finishes and is
+        recorded; Abort is the lever that stops that one."""
+        b = self.batch
+        if not self.batch_active():
+            return False, "no queue is running"
+        left = len(self.batch_snapshot()["pending"])
+        b["stop"] = True
+        b["stopped"] = b["stopped"] or "stopped by the operator"
+        self._note("queue %s: stop requested, %d test(s) will not run" % (b["group"], left))
+        return True, "queue stopped; the run in progress finishes"
+
+    def _drive_batch(self, b):
+        """One test at a time, in order, until the queue empties or a run
+        comes back anything other than PASS. A FAIL closes the campaign, so
+        carrying on would only open a second one behind the operator's
+        back; an ABORTED means they asked it to stop."""
+        try:
+            for tid in b["order"]:
+                if b["stop"]:
+                    break
+                b["current"] = tid
+                ok, msg, run = self._start_test(tid, ack_live=b["ack_live"],
+                                                ignore_requires=b["ignore_requires"], batch=b)
+                if not ok:
+                    b["skipped"].append({"test": tid, "reason": msg})
+                    self._note("queue %s: skipped %s: %s" % (b["group"], tid, msg))
+                    continue
+                # stop_batch lets the run in progress finish; Abort is what
+                # ends this one, and lands here as a non-PASS result.
+                while run.finished is None:
+                    time.sleep(0.2)
+                result = run.finished["result"]
+                b["done"].append({"test": tid, "result": result})
+                if result != _campaign.PASS:
+                    b["stop"] = True
+                    b["stopped"] = "%s on %s" % (result, tid)
+                    self._note("queue %s: stopped on %s (%s)" % (b["group"], tid, result))
+                    break
+        except Exception as e:  # noqa: BLE001 - a broken queue must not wedge the runner
+            b["stopped"] = "%s: %s" % (type(e).__name__, e)
+            self._note("queue %s: driver errored: %s" % (b["group"], b["stopped"]))
+        finally:
+            b["current"] = None
+            b["finished"] = now_ts()
+
+    def batch_snapshot(self):
+        b = self.batch
+        if b is None:
+            return None
+        done_ids = set(x["test"] for x in b["done"]) | set(x["test"] for x in b["skipped"])
+        return {"group": b["group"], "ts": b["ts"], "order": list(b["order"]),
+                "done": list(b["done"]), "skipped": list(b["skipped"]),
+                "pending": [t for t in b["order"] if t not in done_ids and t != b["current"]],
+                "current": b["current"], "stopping": bool(b["stop"]) and not b["finished"],
+                "stopped": b["stopped"], "finished": b["finished"]}
 
     # -- baseline around every run -----------------------------------------
     def _baseline_pre(self, run):
@@ -490,6 +627,8 @@ class Runner:
 
     # -- bench tools ---------------------------------------------------------
     def start_bench(self, tool_id, args=None, ack_live=False):
+        if self.batch_active():
+            return False, "a queue is running"
         if self.bench is None:
             return False, "no bench registry"
         tool = self.bench.get(tool_id)

forgetest/forgetest/server.py

@@ -19,6 +19,9 @@ Routes
   GET  /log                 the raw JSONL
   GET  /export/acceptance.json | .md   the last export
   POST /start {test, ack_live, ignore_requires}   start an acceptance test
+  POST /batch {group, ack_live, ignore_requires}  run everything a queue
+                            still owes, in prerequisite order
+  POST /batch/stop          cancel what is still queued
   POST /bench/start {tool, args, ack_live}
   POST /answer {prompt_id, value}
   POST /abort
@@ -261,6 +264,14 @@ class Handler(BaseHTTPRequestHandler):
                 ok, msg = r.start_test(str(body.get("test", "")), ack_live=bool(body.get("ack_live")),
                                        ignore_requires=bool(body.get("ignore_requires")))
                 self._send(200 if ok else 409, {"ok": ok, "message": msg})
+            elif path == "/batch":
+                ok, msg, order = r.start_batch(str(body.get("group", "")),
+                                               ack_live=bool(body.get("ack_live")),
+                                               ignore_requires=bool(body.get("ignore_requires")))
+                self._send(200 if ok else 409, {"ok": ok, "message": msg, "order": order})
+            elif path == "/batch/stop":
+                ok, msg = r.stop_batch()
+                self._send(200 if ok else 409, {"ok": ok, "message": msg})
             elif path == "/bench/start":
                 args = body.get("args") or {}
                 if not isinstance(args, dict):