fix(manifest)!: the size cap opened a service-wide hang; close it

The diff-scoped bug-hunt panel, four arms, artifact-only. Its strongest
finding is one I created two hours earlier while hardening the reader.

`stat` reports size 0 for a FIFO and 0 for a symlink to /dev/zero, so both
sail under the byte cap added for the RecursionError round — and then
`read_text` either blocks in read() with no EOF, so the except never runs,
or allocates until the kernel intervenes. `list_booths` reads every booth
on every GET / and /healthz, so ONE such file stalls the front page for the
whole service, with no error and no recovery short of a restart.
Reproduced before believing it (timeout returned 124). S_ISREG is checked
BEFORE the size in both modules now; verified against the live service with
two FIFOs planted, which answered 200 in 36ms.

The shape worth carrying: st_size answers a different question than "can
this be read", and a bound that trusts it inherits everything it does not
mean. A hardening fix opened a worse hole than the one it closed.

THE UPLOAD PATH WROTE ABOVE ITS OWN CLEANUP GUARD (4/4)

A failed manifest write orphaned a .uploaded half-booth with no files in
it — and because the temp name now carries a random suffix, nothing ever
overwrote the leak, and .booth.json.<hex>.tmp is not a .lock, so
_newest_mtime counted it and kept that empty booth past every sweep. The
uniqueness fix from the previous round is what made the leak permanent.
Both writes moved inside the guard; the temp is removed on every exit path.

DAMAGED BYTES ARE KEPT, NOT REPLACED (4/4, INV-6)

Marks made this explicit in v0.2.1 and this write path contradicted it: a
manifest that failed on ONE field lost the others with it, including a why
the re-announcer may never have kept anywhere. It diverges from marks in
HOW it honours the rule — marks refuse and answer 409 because the
operator's judgment is not restatable; a manifest quarantines and proceeds,
because refusing would fail `booth add` and lose the files it was copying.

ONE OPENNESS PREDICATE, AS U2 SAID (2/4)

`booth answer` spelled out `if m.answer is None` while `booth marks` asked
`open_marks`, so a partially-answered pick read as done to one verb and
open to the other — at the same instant, on the same booth. U2's INV-2 put
openness in one function precisely so they could not drift. The mirror case
is fixed too: a pick that hydrates broken is refused by the web route, so
`answer --wait` polled an hour on a form nothing could ever land.

ALSO

- now_stamp was whole-second while the importer had moved to microseconds,
  and '-' sorts before '.', so a later mark came out ahead of an earlier
  import inside the same second. One format; the previous round's ordering
  fix had opened this one.
- `_broken` was the third of three directory-name fallbacks and the one
  still handing a raw name into a card's sub-line.
- An identical re-announce rewrote the file and reset the TTL. `booth link`
  does this on every post to the standing board.
- The importer's return went through the bare _hydrate, not _hydrate_safe.
- A marks document could be written larger than it can be read back, and
  then read as no marks at all. Refused at the write instead.
- `choice` reached the answer builder raw while `notes` beside it did not.

AND ONE FINDING DELIBERATELY NOT FULLY CLOSED

The mtime-restore race is real. The clean fix — ignore a booth directory's
own mtime whenever the booth holds anything — also silently retires the
documented rule that releasing a kept board resets its clock, which the CLI
header, the README and a deliberately-written test all pin. That is a TTL
doctrine change, not a bug fix, and an existing test caught the attempt.
The concrete half is fixed (a failing os.utime escaped and 500'd the
route); the race is stated in the code where the next reader will meet it.

341 tests. Live service restarted, 24/24 booth pages verified.
This commit is contained in:
Vuong Hoang
2026-09-22 02:27:18 -07:00
parent f3193fb054
commit 95beede3c3
11 changed files with 616 additions and 61 deletions
@@ -88,6 +88,7 @@ HANDLE_MAX, TITLE_MAX, WHY_MAX = 64, 120, 200
MANIFEST_MAX_BYTES = 64 * 1024
QUARANTINE_FILE = ".booth.json.broken"
def read_manifest(booth: Path) -> Manifest | None:
@@ -108,6 +109,14 @@ def read_manifest(booth: Path) -> Manifest | None:
huge document, `RecursionError` from a deeply nested one — are caught as
well, so that raising the bound one day cannot quietly re-open the hole.
REGULAR-FILE FIRST, THEN SIZE — and the order is the whole point. `st_size`
is 0 for a FIFO and 0 for a symlink to `/dev/zero`, so both sail under any
byte cap and then the read either blocks forever with no EOF or allocates
until the kernel intervenes. The bound is what made this reachable: a cap
that trusts `st_size` inherits everything `st_size` does not mean. One such
file stalls every `GET /` and `/healthz`, with no error and no recovery
short of a restart.
Absent -> None. Present but too large, unreadable, unparseable, not an
object, or missing `handle` -> a Manifest carrying `error`, so the card can
say `unreadable` rather than quietly showing the same thing as a booth that
@@ -133,6 +142,19 @@ def write_manifest(booth: Path, handle: str, *, title: str | None = None,
`""`, is treated as having no stamp to preserve and gets `now()`: a stamp
that is silently wrong is worse than one that is silently new.
A WRITE THAT CHANGES NOTHING IS NOT ACTIVITY and does not touch the file,
so it cannot reset the booth's TTL — the rule marks learned in v0.2.0,
needed here because `booth link` re-announces the standing board on every
single post to it.
BYTES THAT COULD NOT BE READ ARE KEPT, not replaced. See INV-6.
A FAILED WRITE LEAVES NOTHING BEHIND. The temp name carries a random suffix
so two writers cannot share it — which also means nothing ever overwrites an
orphan, and `.booth.json.<hex>.tmp` is not a `.lock`, so `_newest_mtime`
counts it and a leak would keep a dead booth alive forever. Cleaned up on
every exit path.
`title` falls back to the directory name, THROUGH the same normalizer the
explicit value gets — a directory name may legally carry a newline on POSIX
and may run to 255 bytes, and the fallback used to hand either straight
@@ -360,6 +382,22 @@ standard library and nothing from `booth.*` — a cross-import between two
stdlib-only modules is a second way for the repo rule to break. Relative
imports count; the AST walk sees them.
**INV-6 — bytes that could not be read are never destroyed.** When
`write_manifest` replaces a manifest whose read returned `error`, the old bytes
move to `QUARANTINE_FILE` first. This is the doctrine marks made explicit in
v0.2.1 — reads lenient, writes strict, damaged bytes stay on disk — and this
unit contradicted it by replacing outright, so a file that failed on ONE field
lost the others with it, including a `why` the re-announcer may never have kept
anywhere.
It diverges from marks in HOW it honours the rule, and the divergence is the
interesting part. Marks REFUSE the write and answer 409, because the operator's
judgment is not restatable. A manifest QUARANTINES and proceeds, because
refusing would fail `booth add` and lose the files it was mid-way through
copying — and a booth's own description is something its poster can say again.
One fixed quarantine name rather than a timestamped series: nothing prunes a
booth but the sweep, and the most recent damage is the only copy anyone opens.
**INV-5 — unannounced and unreadable render DIFFERENT TEXT.** Not merely
different styling: the words differ (`unannounced` / `unreadable`), so the
distinction survives a stylesheet change and a reader who cannot see colour. A