booth/persistent-memory.d/2026-09-23-the-browser-suite-is-flaky-under-load.md

The browser suite is flaky under load — UNRESOLVED, and owned by design-dev

2026-09-23 · booth

⚠ OPEN. Not fixed. Do not read a green suite as proof of anything without re-running it.

What is observed, with its limits

Three different browser tests have each failed once under full-suite load while passing repeatedly in isolation:

test	owner	isolation	full suite
test_the_keyboard_flag_actually_submits	booth-dev	5/5 pass	1 failure
test_a_failed_save_says_so_reloads_and_never_re_posts	design-dev	3/3 pass	1 failure
test_the_review_keys_judge_in_place_and_stay_out_of_the_note	design-dev	10/10 + 5/5 pass	1 in ~9

Three different tests points at the environment under contention, not at any one test. That is a direction, not a finding.

Two real defects found chasing it — NEITHER PROVEN TO BE THE CAUSE

1. A keypress race. The flag test fired ArrowRight and f back to back, assuming the first had finished — but focus() does a scrollIntoView, so under load f could arrive with no cursor set. Now waits for figure.item.is-cursor.
2. A port TOCTOU in BOTH browser fixtures. Each did bind → getsockname → CLOSE → hand uvicorn the port NUMBER, leaving a window for the kernel to give that port away — and this suite runs two browser files that each start a server per test, so the competitor is the other file. The bound socket is now passed to server.run(sockets=[sock]).

Since those fixes: one failure in three full runs. n=3 CANNOT distinguish that from the prior rate, and no claim of improvement is made.

Who owns it and what the method is

The operator ruled: "let him diagnose it properly." design-dev owns it. His method: a trace hook keeping a Playwright trace (screenshots + DOM snapshots) for every browser test that fails, captured from the run that fails, then full-suite runs until red and the artefact decides. Test-only infra, on his branch as its own commit.

⚠ The methodological trap, and why the artefact must come from the full suite: the failure only reproduces there. A narrowed repro that passes proves nothing and will be mistaken for a fix.

Addendum 2026-09-24 — the offline fix, and what 0/24 can and cannot say

design-dev's suspect was Google Fonts stalling "networkidle". A stalled font request reproduces the exact error (which shows sufficiency only). The fix landed in b92b002: the test browser has no internet, with a positive control per fixture. Traced runs: 0/15 (light) and 0/8 (heavy). No trace ever caught the stalled request. Untraced after the fix: 0 reds in 24, against a pre-fix rate of 1 in 8 that is itself one red in eight runs (95% CI roughly 0.3–53%). At a true rate of 1 in 20, 0/24 happens 29% of the time. So it's consistent with the fix and not a confirmation of the cause.