Read from docker inspect, nvidia-smi and each model's own config.json rather
than from compose files or the gateway config, so the document records what is
deployed rather than what was intended.
Findings worth naming:
- char-rp drift. The LiteLLM config documents char-rp as G4-MeroMero-v2-31B on
ana-ml2 GPU 0; :8016 actually serves gemma4-26b-a4b-it-nvfp4, a 26B 30-layer
128-expert stock Gemma-4, on fv-ml1 GPU 1. Host renaming explains part of it,
but 31B MeroMero-v2 and 26B stock gemma4 are different models and the comment
block carries sampler defaults keyed to the model it names.
- Two gateway aliases are dead: erp-tune-v2 (:8098) and gemma4-26b-a4b-it-base
(:8099). Nothing is listening on either; callers get a connection failure
rather than a clear retirement error.
- Three different speculative-decoding methods coexist -- dflash k=7 with a
separate drafter on mog-sec, qwen3_5_mtp k=3 on gen, mtp k=3 on flash-next.
The method string is architecture-specific; copying a --speculative-config
between seats without changing it will not work.
- Three seats run the floating :latest tag, so what they would pull on recreate
is not what is running and was never recorded.
- Seven aliases resolve to vllm-gen on :8015, which is why GPU 0 shows draw
during ordinary OpenWebUI use -- its auto-titling fires summarizer per turn.
Also commits the flash-next-seat MTP k=3 change made today: compose gains the
speculative-config with its measured rationale, and .env.example documents the
KV-cache reduction MTP requires (14 GiB OOMs, 10 GiB verified in production).
Container teardown latency here is unpredictable, not merely slow. Measured
2026-09-13 on the same container in the same session: once ~55s, once 0s.
Removing a container holding ~92 GB of GPU memory plus the offloaded PLE
mapping leaves an Exited record owning its name for that whole window, so any
fixed wait or retry budget is a guess that will eventually be wrong -- a 12x5s
retry lost arm k1 by roughly two attempts.
Arms now use $NAME_BASE-$arm, set in boot() after stop_bench has torn down the
previous arm. Names are never reused, so the collision is impossible by
construction and teardown latency stops being load-bearing. cleanup() reaps
every fn-mtp-bench-* container at exit.
stop_bench keeps the GPU-memory wait -- the next container genuinely needs the
device, and unlike the container record nvidia-smi clears promptly and reports
truthfully. It no longer waits on the container listing at all.
Three earlier hypotheses were wrong and are recorded so they are not retried:
the name is not held by a phantom (the container is present and Exited), the
removal does not fail (rm -f succeeds; it is asynchronous), and GPU memory
release does not track name release (memory frees first, by a wide margin).
The previous fix polled `docker ps -a` until the bench container's name
disappeared. That probe is useless here and the fix was a no-op: measured
2026-09-13, the container stops being listed while the daemon still holds the
name reservation, so the poll returns false early and `docker run` hits a
Conflict naming a container ID that `docker inspect` already reports as
'no such object'. Arm k1 was lost twice this way.
Two changes, each aimed at something actually observed:
- stop_bench waits on GPU memory falling below 1000 MiB rather than on the
container listing. That is the resource the next container genuinely needs,
and nvidia-smi reports it truthfully.
- boot() retries `docker run` while stderr matches 'already in use', up to 12
attempts at 5s. The daemon's name lag is transient, and forfeiting an arm
over it is silent data loss -- run_arm turns a failed boot into a skipped
arm that still lets the campaign report success.
Both probes verified against real artifacts before deploy: grep -c
'already in use' on the captured k1.runerr returns 1, and the nvidia-smi
query returns a bare integer that compares correctly under [ -gt ]. The
earlier fix passed bash -n and was still inert, so syntax is not the check
that matters here.
`docker rm -f` returns once removal is INITIATED, not once complete. The
bench container holds ~92 GB of GPU memory plus the offloaded PLE mapping, so
the name stays reserved for several seconds while the device is released. The
next `docker run --name` then dies with a name Conflict.
That failure was near-invisible: run_arm treats a failed boot as
`arm SKIPPED`, the campaign continues, and it still prints CAMPAIGN DONE.
Observed 2026-09-13 -- arm k1 lost the race after off_A's loaded container,
while k2 won it only because k1 had never started a container to tear down.
Every arm that follows one which actually ran is exposed, so k3 and off_B were
both on track to vanish from a run that would have reported success.
stop_bench now polls until the name is released (120s ceiling, warns and
continues). Added a completeness gate: the campaign asserts every expected
res-<arm>-rep<n>.json exists and reports CAMPAIGN INCOMPLETE naming each
missing result rather than DONE, so a gutted run cannot look like a clean one.
Verified the docker ps format string empirically -- an earlier draft nested
quotes so the template rendered as '{{.Names}}' with literal quotes, which
grep -qx could never match, making the wait a no-op that passed bash -n.
power.log lines begin '<HH:MM:SS> 0, <W> W, ...', so splitting the first
pipe-field on ', ' yields a[1] = '20:18:00 0' rather than '0'. GPU 0 was
therefore keyed by sample timestamp, emitting one bogus row per sample and
no recoverable peak, while GPUs 1-3 aggregated correctly.
Take the GPU index as the last whitespace-separated token of a[1]. Verified
against a two-sample fixture: the old parser emits a spurious row per
timestamp, the patched one reports 0/1/2/3 once each at the correct peaks.
The end-of-campaign summary is the GPU-side input to the fv-ml1 circuit
budget, so a silently wrong GPU 0 row is a measurement fault, not cosmetic.
The in-flight campaign still runs the old on-host copy (editing a running
bash script corrupts execution by byte offset); its summary will be
recomputed from the raw power.log.
A one-hour window deleted the handoff unread across any overnight gap -- which is
precisely the case the handoff exists for. Tonight's snapshot would have been discarded
before the operator returned from the site visit. 8 h also matches the global CLAUDE.md
and the /snapshot skill default, so the repo stops being the outlier.
The reasoning is kept inline in the header rather than only in this commit message,
because the next session reads the header and not the log.
Operator-invoked before heading to the Fountain Valley site visit.
Rewrites Current state / in-flight around the outage: FV dark since 06:56Z including the
BMC, will not self-recover, 19 of 30 gateway aliases down with no local fallback because
every free local model lived on fv-ml1 and irv-ml1 runs no chat seat. The section now
POINTS AT docs/runbooks/fv-site-dark-20260913.md rather than restating it, since the
runbook is the single source of truth for the visit.
Records what was built and verified before the power failed -- the Flash-Next seat on one
card with its 51B n-gram table in host RAM, gen-large on the gateway -- and what is
decided but unexecuted: the 250 W / 200 W power caps, the incomplete MTP campaign with one
off_A rep banked, and the ten stale Homepage labels that the staged bring-up fixes as a
side effect.
Closes a stale in-flight claim: the 'fv-ml1 cannot initiate to fleet LAN IPs' gap is
resolved, fixed by another session's scoped SNAT at 06:22Z with Beszel 18/18 verified.
Adds ana-ml3 as its own subsection, since conflating it with fv-ml1 caused two rounds of
wrong arithmetic today.
Auto-archival fired (index was 431 lines, over the ~300 cap) but reached only 2 entries.
The 14-day guard holds almost everything: this repo has had an exceptionally dense
fortnight and nearly every dated entry postdates 2026-08-30. Of the six entries old
enough, four carry open deferred-work pointers and were held back per the decision-loss
guard. The file stays over cap, which is the documented trade -- an over-cap file that
keeps live decisions beats a scannable one that lost one.
I claimed in 100670e that DCGM's config enforcement was plausibly gated to datacenter
SKUs and told the operator not to plan around it. That was a guess presented as a caveat
and it is wrong. Verified against NVIDIA's own documentation at the operator's request.
Supported platforms explicitly cover 'All NVIDIA Maxwell and newer non-datacenter (e.g.
NVIDIA GeForce or NVIDIA Quadro) GPUs', and the feature-overview table marks
Configuration Management as supported for Tesla, Titan, Quadro and GeForce alike --
where Configuration Management explicitly includes 'Power Limit: Set the maximum allowed
power consumption'. What is actually gated on non-datacenter cards is diagnostics: Level
1 only, against All Levels on Tesla. Configuration was never the restricted part.
One soft edge retained rather than papered over: the table says 'Quadro', the former name
for the professional line, and RTX 6000 Ada / RTX PRO 6000 are its successors, so placing
them in that column is inference rather than quotation. One command on the box settles it.
What does not change is the distribution question. DCGM_CONFIG_POWER_BUDGET_GROUP is
available to us, but the docs still never state how a group budget is divided, and the
NVML argument is untouched -- there is no bank-level register, so it resolves to per-GPU
writes either way and the likely finding is static even division, which is exactly
4 x 250 W. The experiment is therefore promoted from curiosity back to a real test.
DCGM is NVIDIA's own Data Center GPU Manager -- first-party, Apache-2.0, packaged as
datacenter-gpu-manager -- and it layers above NVML rather than beside it: nvidia-smi is a
thin CLI over NVML's per-GPU primitives, and DCGM is a daemon plus dcgmi adding health,
diagnostics, config enforcement, policy and group abstractions on top. Which is why its
group notion still resolves to N per-GPU writes underneath.
The caveat that matters, and it undercuts the experiment suggested in the previous
commit: DCGM is datacenter-oriented and parts of it are gated to datacenter SKUs of the
Tesla/A100/H100 class. Our cards are professional/workstation parts -- RTX PRO 6000
Blackwell Max-Q and RTX 6000 Ada -- and several DCGM capabilities are unsupported or
degraded outside that line, plausibly including config enforcement, which is precisely
the power path. So DCGM_CONFIG_POWER_BUDGET_GROUP may return 'unsupported on this
device'. Downgraded from 'worth testing' to five minutes of curiosity after the real
work, and explicitly not a planning assumption.
None of which touches the plan: nvidia-smi -pl 250 is plain NVML and works on these
cards. DCGM would only have bought the group-budget experiment and nicer telemetry, and
is probably not installed anyway since beszel-agent-nvidia shells out to nvidia-smi.
DCGM_CONFIG_POWER_BUDGET_GROUP ('the power budget for the entire group') exists
alongside DCGM_CONFIG_POWER_CAP_INDIVIDUAL, so the concept is first-class. The docs do
not state how a group budget is distributed, and the deduction is that it cannot be
anything exotic: the only enforcement primitive underneath is NVML's per-GPU
nvmlDeviceSetPowerManagementLimit and there is no bank-level register, so any group
budget resolves to N per-GPU writes. Static even division is one write each; 'each card
free until they are all loaded' requires continuous re-writing, which is a control loop
rather than a hardware feature. Worth a ten-minute test when the box returns, in case
NVIDIA already runs that loop.
Records the design constraint that matters more than the logic: power readings lag and
-pl application takes tens of milliseconds, so a reactive daemon overshoots during a load
ramp -- and the ramp is the dangerous moment, being the same all-cards-at-once shape as
this box's ten restart:unless-stopped containers starting together. So any such loop must
be safe-by-default and opportunistic upward: boot at budget/N, only ever raise after
observing idle neighbours. Inverted, it works for weeks and then fails on precisely the
event it existed to prevent.
And the reason to defer it: 4x250 W is already 1000 W, so the static cap is the
conservative floor of the dynamic scheme rather than an alternative. The daemon's entire
contribution is the one-card-busy case, worth perhaps 5% throughput, which is rare for a
serving fleet that puts one seat per card and common only for a training window.
Operator set fv-ml1 at 250 W per card (83% of TGP, ~5% throughput) and ana-ml3 at 200 W
(67%, ~10-15%). Records the term that decides whether 250 W actually clears a 15 A feed,
because it is easy to drop: a power limit bounds BOARD power, and the wall sees that
divided by PSU efficiency.
Four cards at 250 W is 1000 W of board; add 180-300 W of host components and divide by
~0.90 and the plug sees ~1310-1445 W, against a 15 A circuit's 1440 W NEC continuous
derating -- an inference box serving all day being a continuous load. So 250 W lands ON
the limit rather than under it, where 200 W would give ~1090-1220 W with real margin.
The deciding term is the host draw, which is still an estimate, so the procedure is: set
250 W, verify at the plug under four-card load, fall back to 200 W if it reads near
1440 W. A cap is a claim; the ammeter is the verification.
Two consequences recorded alongside. Caps bound sustained draw and not transients -- the
enforcement window is short but not instantaneous -- and while a breaker's
thermal-magnetic curve forgives brief overload, a UPS's overload protection does not. So
250 W implicitly commits the fv-ml1 chassis to the PDU rather than behind the 1500 VA
unit, which it exceeds even capped. And ana-ml3's 200 W across only two cards is
deliberately conservative at 400 W total, relaxable if Anaheim's measured headroom beats
its trip history.
Operator clarification, and it separates two boxes I had been conflating. fv-ml1 is
4x Blackwell RTX PRO 6000 Max-Q at 300 W each (Max-Q being the reduced-TGP SKU; the
Workstation Edition is the 600 W part), 391 GB VRAM, deployed and currently dark.
ana-ml3 is 2x Ada Generation RTX 6000 at 300 W, 96 GB VRAM, not yet deployed. The 200 W
cap directive is ana-ml3's.
With the TGP known, the outage stops being a vague 'undersized' and acquires a
mechanism: two Max-Q cards at 300 W is ~600 W of card, plus a host carrying 566 GB of
RAM, drives, fans and PSU conversion loss at perhaps 200-350 W, against an Eaton 1500
VA's real ~900-1200 W. That lands at or just over the rating, which is precisely what
explains a full day of service on one card and failure minutes into the second. The host
term is the only one being guessed; idle-at-the-plug measures it directly.
It also surfaces something that is not a UPS question at all. Four cards at 300 W plus
~300 W of host is ~1500 W against a 15 A circuit's 1440 W continuous derating, so four
cards uncapped is marginal on the breaker with no UPS in the path. Capping therefore
belongs at fv-ml1 as well as ana-ml3, or fv-ml1 needs a 20 A feed -- and worth noting
today's incident only ever had two of the four cards working.
ana-ml3's placement constraints sharpen too: sm_89 has native FP8 but no NVFP4, so the
in-house NVFP4 quants stay at FV, and at 96 GB total it cannot host the Flash-Next seat
at all -- that needs 74 GiB resident on a single card, and the offload moves the n-gram
table rather than the experts.
Corrects the SKU: RTX 6000 Ada, 300 W, not the ~600 W initially recalled. That makes
200 W a cap to 67% of TGP -- the favourable part of the concave perf/watt curve, roughly
10-15% of throughput -- rather than the severe 33% cap a 600 W part would have implied,
and it very likely sits above the card's enforceable floor, so the check becomes a
formality rather than a gate.
The protective value is worth stating: four cards at 300 W uncapped is ~1200 W, which is
roughly the neighbourhood that overwhelmed a 1500 VA unit at FV with only TWO Blackwell
cards drawing. Capping to 800 W makes a repeat of today a non-event.
Two consequences that follow from Ada independent of power, and both are placement
constraints rather than details. sm_89 has native FP8 but NOT NVFP4, which is
Blackwell-only -- so the in-house NVFP4 quants that most of this fleet runs will not be
accelerated on that colo's cards, and its seats want FP8 W8A8 builds or the NVFP4
checkpoints stay at FV. And it unparks the triton-backend item, which is a hard no on
Ampere because fp8e4nv is unsupported on sm_86 and was explicitly deferred to Ada;
sm_89 has what it needs.
VRAM is 4x48 = 192 GB against fv-ml1's 391 GB, so big-model placement stays at FV. The
Flash-Next seat needs 74 GiB resident on one card and would not fit a 48 GB Ada card
even with the n-gram table offloaded -- the offload moves the table, not the experts.
Operator directive, and the right generalisation of the FV outage: decide the power
envelope first and size the cards into it, rather than installing cards and discovering
the constraint by tripping it. Four cards at 200 W is 800 W, which fits a real circuit
with a real UPS and headroom.
Records three things to settle before it is a plan. First, 200 W may sit below the
card's enforceable floor: nvidia-smi -pl is bounded by Min Power Limit, often around
half of TGP on a high-TGP part, and a sub-floor request is refused -- quietly, depending
on how it is scripted. Run nvidia-smi -q -d POWER before any build planning depends on
the number.
Second, the 600 W figure wants confirming against the actual SKU. The Ada parts do not
land there -- RTX 6000 Ada is 300 W, L40/L40S 300/350 W, 4090 450 W -- while 600 W is
Blackwell RTX PRO 6000 Workstation territory, so these may be Blackwell or the figure
may be a two-card total. Read it off the device rather than a spec sheet.
Third, the workload asymmetry is in this fleet's favour: decode is
memory-bandwidth-bound and tolerates a cap far better than training does, with a concave
perf/watt curve where 60-70% of TGP costs roughly 10-15% of throughput. A cap to a third
of TGP is deeper into the steep region; measure it on the first card rather than
predicting, and expect prefill-heavy and training work to pay more than a serving seat.
And persist the cap. A hand-set limit holds until the next reboot and then silently
stops holding, which is the worst shape available given that the thing rebooting the box
is likely to be the power event the cap existed to prevent.
Operator: 'unless of course the thing trips the circuit anyway.' Correct, and it splits
into two halves with different answers.
A breaker trip is the event the split-power proposal survives: firewall + BMC is 25-40 W
on a 1500 VA unit, which is hours of battery, and on a trip the UPS stops being a
load-bearing supply and goes back to being what it is for. What it does NOT cover is the
colo's own handoff -- their switch, ONT or demarc. If that sits on the circuit we just
tripped, the outcome is a firewall running on battery with nothing upstream to talk to
and the drive happens anyway. Added as a question for the facility, because it decides
whether split power delivers remote diagnosis or merely feels like it does.
Records the case where none of it matters: removing an undersized UPS does not remove
the constraint, it promotes the next one -- UPS ~900-1200 W to circuit ~1800 W at 15 A
or ~2400 W at 20 A. Which side the four-card figure lands on decides everything, which
is what makes that single ammeter reading the load-bearing measurement of the visit.
Surfaces the lever that may avoid an electrician entirely: nvidia-smi -pl caps per-card
TGP, so the box can be made to fit its feed at a throughput cost rather than a rewiring
cost. Read nvidia-smi -q -d POWER for the enforced range before assuming how much room
the dial has, and persist any cap -- one that evaporates on reboot will hold right up
until the next power event and then silently stop holding.
Operator's reasoning, accepted and better than the hypothesis-space argument it
replaces: the NAT change went effective, was verified bidirectional, and then ran
correctly for twenty minutes before the site died the moment GPU load was applied. A
working config change does not spontaneously fail under an unrelated physical variable.
The load correlation is tight; the NAT correlation is merely adjacent in time.
Undersized UPS is the only candidate that explains the trigger. NAT material retained
as record, and the power.log/uptime check demoted from decision point to free
confirmation.
Adds the measurement protocol, since the operator is bringing a PDU and an ammeter.
The load-bearing caveat: power.log is GPU-ONLY -- nvidia-smi per-card, excluding CPU,
566 GB of RAM, drives, fans and PSU conversion losses -- so the ammeter at the plug is
the primary instrument and power.log only cross-checks the GPU share. Four states to
capture (idle, one card, two cards, four cards), and capture PEAK rather than average:
UPS overload protection responds to short-term overload, so an average-only reading
that hides transients will mis-size the replacement exactly the way the present unit
got mis-sized, and must be recorded as a floor rather than as the draw.
The four-card figure is earmarked for servers/fv-ml1/README.md, because it closes the
cutover's own open question -- that the FV circuit was likely specced against half the
real draw, back when every record still said the box had two GPUs.
Another session applied a scoped Tailscale SNAT rule to the FV gateway at ~06:22Z, 34
minutes before the site went dark (docs/runbooks/fv-to-ana-nat.md, not my work, left
uncommitted). That makes the UPS-overload theory a hypothesis rather than a finding,
and nobody should buy hardware on it until the discriminator below has been read.
On the evidence that change is the wrong shape to have caused this, and it is recorded
as such so the visit is not wasted chasing it: one OUTBOUND SNAT rule scoped to a
single source /32 and a single destination /16 cannot stop the gateway, the BMC or the
public WAN address from answering inbound; no routes, filter rules, WAN settings or
subnet advertisements were touched; pfctl -sr came back byte-identical; and it was
verified bidirectional afterwards including ANA->FV SSH with Beszel 18/18 up.
Their BMC datapoint used 10.251.50.50, which is not the BMC -- that is 10.251.250.50,
a different subnet. They correctly declined to claim BMC health, but the observation is
void rather than negative and should not be reasoned from.
The discriminator costs nothing and is already on disk: power.log is written locally to
/tank every 10 s by a shell loop on the box and does not depend on the network.
Entries past 06:56Z mean the machine never lost power, which makes this a routing fault
and the UPS innocent; entries stopping at 06:56Z confirm power. Cross-check with uptime
and journalctl --list-boots -- continuous uptime across 06:56Z kills the UPS theory
outright.
So the first action on site is now to READ, not to fix. The two hypotheses lead to
completely different remediations and only one of them needs a new UPS.
Operator-directed so the campaign could run in parallel with live gen-large testing.
Strips the production-restore tail entirely -- the only container the driver can create
or remove is fn-mtp-bench, and the sole remaining compose call is a read-only
config --format json to derive argv. Adds per-card power+memory sampling every 10 s to
power.log, because two cards under load was the risk being accepted and a record beats
an argument.
That power log is now the only surviving measurement of what the load drew before
Fountain Valley went dark; it lives on /tank rather than in the container.
Durable capture so tomorrow's session does not have to reconstruct either half.
Built and verified before the power failed: Qwen3.8-Flash-Next serving on a single
RTX PRO 6000 with its 51B n-gram table pinned in host RAM and read over CUDA UVA --
74.36 GiB weights resident, 14.00 GiB KV for 560,654 tokens at the full 262,144
context, 67 GiB host RSS -- plus a gen-large gateway alias verified end to end.
The five findings worth carrying: the offload is #54371 (UVA, merged) which supersedes
the paused worker-based #53899 and designs out its entire bug family;
text_config.ple_embedding_dtype is the load-or-fail discriminator for any community
build; --kv-cache-memory makes vLLM SKIP memory profiling and ignore
gpu-memory-utilization, which inverts the usual pin-bytes advice and let a 16 GiB pin
nearly OOM with no visible failure; MTP is off pending measurement here rather than
written off, because the recipe's number is cross-harness and tested k=3 only while the
head is one layer run autoregressively; and a container once reported (healthy) with no
published port at all, because the healthcheck runs inside the boundary it was trusted
to validate.
Then the outage. Records it as will-not-self-recover, so no session wastes effort
polling a dead site, and carries the three things that change the visit: bypass the UPS
rather than using its surge-only bank (both banks share one 12 A inlet -- the surge
bank bypasses the inverter, not the current rating), recover power.log before anything
else because it is the only load measurement that exists anywhere, and bring seats up
one at a time because ten restart:unless-stopped containers loading at once is the
largest transient the box can make into whatever just failed.
Also records what is still half-done: the stale homepage labels on the 10 containers
that died before they could be recreated, which the staged bring-up fixes as a side
effect, and the eight drifted stacks plus three untracked host-only stacks that were
deliberately left for a deliberate reconciliation.
Operator's read is that the UPS the box was plugged into overloaded and died, and it
fits better than the breaker-trip theory: a UPS's output rating sits far below the
circuit's, so it is the first protective device to give -- which explains why the site
let go at TWO cards loaded rather than four, and why the ~25 W firewall died with it.
Records the operationally important consequence: a tripped UPS resets, an overloaded
one can kill its output stage permanently. If it is dead, nothing on site can be reset
back to life, so the visit needs the means to BYPASS the UPS or it is wasted.
Elevates recovery of /tank/.../power.log to the first action on site. It sampled all
four cards every 10 s up to the cut, lives on /tank rather than in a container, and is
the only measurement of what the load actually drew -- without it a replacement UPS
gets sized by guesswork. Also states that no load figure exists yet, only idle.
Corrects an earlier claim of mine in this session: there is NO local fallback for the
19 dark aliases. Probed -- every free local model is on fv-ml1, and irv-ml1 runs no
chat seat at all, only TTS/ComfyUI/arbo/clipper work on two partly-occupied Ampere
cards. The only non-fv chat backends are paid. Any paid coverage must go under a new
opt-in alias name rather than a silent repoint of summarizer/gen/classifier.
Written while the site is down so recovery does not have to be reconstructed later.
Records what was measured rather than what is suspected: every FV address including
the BMC is unreachable while all three other sites answer, the campaign's last log
line was off_A rep 2 at 06:56:04Z, and the site was dark by 06:58:40Z.
Names three candidate causes with the evidence that would distinguish them, because
the instrument that could have settled it -- the per-card power log -- died with the
box. The two-card-load hypothesis fits the timing and the two prior Anaheim breaker
trips on this same chassis, but it is circumstantial and is recorded as such.
Carries the recovery hazard that matters: every seat on the box is
restart:unless-stopped, so resetting power alone brings ten vLLM containers up
loading at once -- the largest transient the box can produce, into a circuit that may
have just tripped. Staged sequence given, gen first and flash-next last.
Also records the OOB gap the outage exposes: OPNsense-as-subnet-router protects
against box-down/gateway-up, and not at all against the site-wide loss that actually
happened, because the BMC's only path out is through that same gateway.
Operator-directed: raise context to the model's native maximum and take as much KV
as the card safely allows, and expose the seat through LiteLLM as `gen-large`.
max_model_len 131,072 -> 262,144
KV cache 8.76 -> 14.00 GiB (332,721 -> 560,654 tokens)
concurrency 2.54x@128K -> 2.14x@262K
⚠ 16.00 GiB WAS TRIED FIRST AND IS TOO AGGRESSIVE. A 155,497-token non-repeating
prefill drove GPU 2 to 97,074 of 97,887 MiB and the caching allocator logged "OOM on
device 0 while trying to allocate 488636416 bytes (free: 422117376)" -- 466 MiB wanted
against 403 MiB free. The request completed, so nothing failed visibly; that is one
step before the shape that crashed stacks/mog-sec twice on 2026-09-10 (~1.04 GiB
wanted, ~600 MB free). Backed off to 14.00 GiB, which re-probes clean: zero allocator
warnings, a 155,557-token prefill in 14.2 s, and 2,085 MiB still free at peak.
The reason the first estimate was wrong is worth keeping, because it is not obvious
and it inverts the usual advice: --kv-cache-memory makes vLLM SKIP MEMORY PROFILING
ENTIRELY and ignore --gpu-memory-utilization. The profiler was the thing accounting
for deep-prefill activation, so pinning bytes switched off the protection that the
pin was supposed to formalise. vLLM's own "--kv-cache-memory=18745235968 (17.46 GiB)
to fully utilize gpu memory" line is computed from a profile measured at
max-num-batched-tokens depth and sits 3.5 GiB above what a 150K-token request
survives; open #54764 compounds it, since PLE short-conv prefill pads every request
in a batch to the batch-MAX query length.
max-num-batched-tokens stays at 8192 -- it is what bounds the activation peak, and
doubling max_model_len left the profiled peak unchanged at 1.65 GiB precisely because
the peak tracks chunk size, not context length.
Gateway: `gen-large` added to the LiteLLM model_list, pointing at fv-ml1:8022. One
alias on purpose -- a single alias cannot trip the shared-config enable_thinking
mutation footgun, which needs two over the same (model, api_base). Sampling is the
checkpoint's own declared set (temp 1.0 / top_p 0.95 / top_k 20); presence_penalty,
min_p and repetition_penalty are left unset because the checkpoint declares no
canonical value for them. Verified registered for both the infra-ops admin key and
the shared all-agents key, since a new model behind a scoped allowlist 403s silently.
Also adds services/flash-next-mtp-bench/ -- the MTP measurement campaign and its
rationale. MTP stays off, but on "not yet measured here" rather than on vLLM's
4xH100 recipe number, which is a cross-harness comparison and not evidence about a
TP=1 Blackwell seat.
Every fv-ml1 link on the Homepage dashboard was broken. Measured against the
live dashboard API before the fix: 16 entries pointing at the dead 10.250.50.54
and zero at the live 10.251.50.54, covering gen, M.O.G.-SEC, Scriberr, Embed,
Rerank, Reward, Coder, Dockge and six dormant seats.
The miss was structural, not careless. fv-ml1-rename-sweep.sh works from an
allowlist assembled from files that mention the HOST, and a homepage.href label
mentions only an IP -- so every stack whose sole stale reference was a label
fell outside it. The allowlist now covers those 24 files, and records how to
derive the list next time (grep the old address, subtract history) rather than
enumerating from memory.
History is still untouched, and the exclusions are now written down with the
reason each one keeps the old address: recorded benchmark results, whose
base_url is part of a measurement's provenance; the one LiteLLM comment
preserving a retired hand-test endpoint; and the cutover runbooks, where the old
address is the subject matter.
Two bugs found while applying it, both fixed here:
- deploy-stack.sh rejected any stack name containing a dot, so qwen3.5-122b,
qwopus3.5-122b and mistral-medium-3.5 could not be deployed by the script at
all. The check exists to stop path traversal, which means rejecting ".." and
"/" -- not every dot. Traversal is now rejected explicitly and tested.
- stacks/scriberr/.env.example allowed CORS only from the dead IP and from
scriberr.ana.internal, which no longer resolves; the box is at the fv site
and DNS already carries scriberr.fv.internal. The live .env had both stale
origins, i.e. an allowlist with nothing reachable in it.
Host side, applied separately: canonical pushed for the 16 stacks whose only
difference from the host was this renumber, and an in-place address-only fix for
the nine whose host copy has genuinely drifted or has no canonical copy, so that
drift survives for a deliberate reconciliation instead of being clobbered. Every
compose.yaml on fv-ml1 now reads 10.251.50.54. The labels themselves only take
effect at container creation, so the running containers still need recreating.
First seat whose weights do not fit its card. The model is 176B total -- a 125B
main model plus a 51B n-gram (PLE) lookup table -- at ~6B active per token. The
table is a pure embedding lookup, so it lives in pinned host RAM and the GPU
reads rows directly over CUDA UVA: ~78 GiB resident on a 95.6 GiB card, 47.7 GiB
pinned of 566 GB. GPU 2 and GPU 3 were both idle, so this displaced nothing.
Checkpoint dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99, pinned by
revision: NVFP4 W4A4 routed experts, FP8 PLE table, everything else at source
precision. Chosen over better-liked builds because its provenance states
protocols and repeat counts -- AIME26 pass@1 98.75% over 30x8 repeats with a
stated SEM, full-set GSM8K, and a byte-equality audit covering all 31 MTP
tensors -- and because it declares text_config.ple_embedding_dtype, which is the
field vLLM reads first when selecting the PLE weight format. Builds that ship an
FP8 table without that declaration resolve to the unquantized path and fail on
load; the README records the check.
Requires vLLM #54371 (UVA PLE-offload, merged 2026-09-09T14:32Z), verified by
ancestry: the pinned nightly is +150 commits / behind_by 0 from the merge commit.
Not in v0.29.0, cut six hours earlier. The older worker-based offload (#53899) is
paused upstream and is not the path here -- its deadlocks, ptrace gate and
stale-output-under-graphs bugs all came from the separate worker process that UVA
does not have.
Five deliberate departures from the other seats on this box, each from a
measurement rather than a preference, all annotated in place:
- no MTP: the vLLM recipe measured it worse at every concurrency on 4xH100
(8-36% less throughput, 32-173% more latency, ~36% acceptance)
- modelopt_fp4, not compressed-tensors: only the ModelOpt reader honours the
ignore list keeping attention, shared experts, PLE and MTP out of W4A4
- KV left at auto: fp8 KV on this model's QSA path is an unmerged RFC (#54426)
- mamba-cache-mode stated explicitly: the model raises on mode "all"
- 128K context and 8192 batched tokens, not the native 262K: #54764 and #54919
make depth the risky axis, and sizing to the KV pool has never fixed a
depth-driven crash on this hardware
Nothing is wired into LiteLLM. Pointing an alias at this seat changes what
existing callers receive and is a separate decision.
Work by a parallel session on 2026-09-12; committed here with the rest of the
day's changes. Rationale in persistent-memory.d/2026-09-12-esh-vm-db-restic-repair.md.
The only visible symptom was a systemd-failed unit from a Sep 6 repository
network timeout after boot. The real fault was quieter and much worse: the
pre-backup hook logged failures as WARN and returned zero, so pg_dumpall could
fail every single night -- it used TCP localhost and wanted a password nobody
supplied -- while restic dutifully backed up the stale April 23 dump still
sitting in the staging directory and reported success. Mongo was fine, which
is part of why it went unnoticed.
Postgres now dumps over the /var/run/postgresql socket with peer auth and -w,
and both database failures now fail the backup rather than masking it, while
still preserving any prior per-DB dump rather than truncating to nothing. An
ERRORS counter replaces the warn-and-continue path, and the staging directory
is overridable via RESTIC_STAGE_DIR so the new test can exercise it.
Adds backup.contract.md, retry.conf and test_pre_backup.py -- three red-green
regression tests covering the failure modes above. systemd drop-ins on both
jobs add network-online ordering plus Restart=on-failure with a 5m delay and a
3-per-hour limit, which addresses the original boot-timeout symptom.
Verified against a real run: snapshot bc5eeaff at 07:01 PDT with a fresh 3.46MB
PG dump, retrieved from the repository with decompression and completion marker
checked (not a full restore). Repository check passed, 99 snapshots. The old
hook and stale dump are preserved root-only at /var/lib/restic/repair-20260912.
Captures the Fountain Valley cutover as executed rather than planned: fv-ml1
live on 10.251/16 and serving, BMC recovered after finding it was tagging
802.1q VLAN 250 into an untagged port, and the corrected four-GPU / 391 GB
count that invalidates prior sizing assumptions.
Also records two things that cost real time and would cost it again: OPNsense
write APIs need an X-CSRFToken scraped from a script block, not a hidden form
input, and a 403 that coincides with the box going down looks exactly like a
successful reboot.
Auto-archived 8 settled entries (Recent decisions: 8). The index stays over
the 300-line cap at 385 because the guards hold -- nearly every remaining
dated entry is under 14 days old or carries open deferred work, and an
over-cap file that keeps live decisions beats a scannable one that lost any.
homepage-regroup, mog-sec-move-to-gpu0, pull-hf-repo and serve-qwen3.5-122b
all carried runnable 'scripts/elway ana-ml2 --playbook ...' instructions or
the old 10.250.50.54 address. Each would fail today against a dead name and a
dead IP, so these are corrections rather than cosmetics. homepage-regroup is
renamed to match; the other three keep their names, which never carried the
host.
A zsh glob failure in the staging step of 91bda3c meant the five renamed
playbooks went in as pure renames with their bodies still saying ana-ml2 and
10.250.50.54, and the homepage docker.yaml direct-connection edit from 17586ab
did not stage either. Same content, just the half that got dropped.
stacks/ana-ml2-proxy existed for one reason, stated in its own README: ESH
could not reach the box's Docker API cross-site because of a return-route gap,
so ana-docker relayed :2354 -> :2375 from the same subnet. The README named
its own retirement condition -- "once it has a working return route to ESH,
point Homepage back at it directly and compose down this stack".
That condition is met. FV is on the mesh, vb-gateway accepts routes, and the
NH3 UDM, ESH UDM and Anaheim FortiGate all carry 10.251.0.0/16 statics toward
their local scale nodes. Verified before removing anything: esh-docker-vm
opens a TCP connection to 10.251.50.54:2375 directly.
Homepage now points at 10.251.50.54:2375 and discovers 112 services across 20
groups, so nothing regressed. The socat container is stopped and removed from
ana-docker; the compose file and README are deleted here rather than renamed,
because renaming would preserve a moving part that no longer earns its place.
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16,
and serving inference again. This lands the repo half of that.
Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq
reservation so the address the runbook, DNS and LiteLLM all assume is the
address it actually has. Its headscale node is renamed too.
The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the
reason this diff touches current-state files and not the record. Dated
persistent-memory entries, archival-memory and incident notes still say
ana-ml2 in 31 and 62 places respectively, because that is what the box was
when those things happened. Rewriting them would make the history lie.
LiteLLM was the load-bearing piece and needed more than the api_base sed the
runbook describes. Twenty api_base entries repointed, but a grep-and-verify
pass also caught a LIVE pass_through_endpoints target for the scalar-judge
reward route still on the old address -- an api_base-only substitution would
have left it dead. Four prose references describing current state were
repointed as well; one historical note recording where a hand-test was run
is deliberately left pointing at 10.250.50.54.
Two facts in the server tables were wrong and are corrected here. The site is
Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell
Max-Q, not two -- verified by nvidia-smi -L and independently by PCI
enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196,
which changes what fits on it.
DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback
approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all
three resolvers. The BMC record carries a warning that its 802.1q VLAN tag
must stay disabled -- it shipped tagging VLAN 250 into an untagged port,
which made it invisible to every network-side diagnostic and is the reason
it appeared dead through several cable changes.
Verified end to end: summarizer and sec both answer through the Anaheim
gateway across the mesh to FV seats on different ports.
fv-ml1-cutover.md covers what to change; this covers the on-site hour, where the
binding constraints are limited time and not locking ourselves out of a site an
hour away.
The load-bearing correction is that fv-ml1 running tailscale is NOT a second
independent path. Its default route is the OPNsense LAN address, so its node
needs egress through OPNsense to reach the control plane; OPNsense down means
fv-ml1 is unreachable by construction. An earlier version of this plan claimed
otherwise and proposed proving it by rebooting the firewall, which would have
failed on site with the operator standing at the rack. He caught it.
What the fv-ml1 node does buy is narrower and still worth one command: it covers
control-plane lockout -- a bad admin rule, a WebGUI on the wrong interface, a
lost password -- where the box still routes but cannot be managed. It does not
cover data-plane failure. Those are different failure classes and conflating
them produced the wrong test. The gate is now a deny-admin-from-mesh test run
while the console is in arm's reach, which exercises the bypass that actually
exists.
Deliberate deviation from the cutover runbook: flat /24 with the BMC at
10.251.50.50 rather than a mgmt VLAN at 10.251.250.50. The VLAN needs OPNsense
config plus switch tagging during the scarcest hour, for something addable
remotely in ten minutes.
Also records the DNS chicken-and-egg (the mesh needs public resolvers because
the fleet AdGuard is unreachable until the mesh is up) and the interface
auto-assignment trap measured on the lab VM the same night.
worldtree-dev's U6 reader refuses at boot if it cannot append and read back
<memory root>/reader/canary.jsonl. Per-euid subdirectories are lazy and only
warn, so that root canary is the single boot-blocking check -- which makes the
memory root's writability by the container uid the precondition worth knowing
before a flip rather than during one.
Protocol agreed with worldtree-dev: neither memory.reader.enabled nor
memory.writer.enabled gets flipped on any deployment without infra-ops
confirming that writability first. Both ship dark until the operator schedules
the tracer skeleton.
Measured tonight on corviduo-dev, all three pass. Also retracts a wrong
prediction I sent earlier in the thread: personal runs as root, not uid 1000,
and pinned is the only uid-1000 deployment -- it passes regardless because
/data/state is owned 1000:1000. Recorded with the caveat that a permissions
reading is a claim about its own date, so the probe gets re-run immediately
before any flip rather than cited from tonight.
Beszel agents are installed and verified across the fleet but the artifacts that
produced them were never committed, so the deployment existed only on the hosts.
Adds the per-host agent environment files (PORT, NICS, EXTRA_FILESYSTEMS and the
hub's PUBLIC key), the systemd unit, the guest install script, the Synology
compose, and the elway playbooks for native, guest-stage, guest-install and
Synology paths. The two dated memory detail files covering the priority-1 and
priority-2 waves ship alongside, per the convention that memory lands with the
work it describes.
No credentials here. The KEY= value in every host env is the Beszel hub's public
ed25519 key, identical across all nine and public by design; the agent README
says so explicitly. The nh3-nas sudo password referenced in the runbook prose
lives in Vaultwarden and the helper scripts named there never contained it.
⚠ Overlapping VMIDs across hypervisors are a standing trap and are recorded in
the priority-2 notes: pfi-pve 105=postgres and 100=pbs-ana, nh3-pve 105=pbs-nh3.
⚠ PBS-NH3's export was ~75.5% used at capture; resource checks are not job
success monitoring and should not be read as such.
Plex never hardware-transcoded on esh-pve-nas LXC 105 despite correct passthrough,
cgroups, group membership, authenticated HuC firmware, a lifetime Plex Pass,
HardwareAcceleratedCodecs=1, and the Arc already selected as HardwareDevicePath.
Root cause sat below all of that: intel-media-va-driver 22.3.1 (Apr 2023, stock
jammy) predates Arc/DG2 support and exports only __vaDriverInit_1_14, against the
libva 2.22 that Plex bundles and loads via RPATH rather than the system one. A
half-finished prior attempt at the same fix was also present -- libva and libva-drm
hand-installed at 2.22 with libva-x11 left at 2.14, breaking every X11 VA-API
consumer on va_fool_postp.
Fixed with Intel's client-GPU repo: iHD 24.3.4 (__vaDriverInit_1_22, an exact ABI
match) plus a consistent libva 2.22.0.2-87 set, which also brings the orphaned
manual install back under dpkg. The repo track is rolling, so the six packages are
pinned in /etc/apt/preferences.d/intel-gpu-pin and apt-mark held; verified by a
simulated upgrade that moves 152 packages and touches none of them.
Two findings worth more than the fix:
pct snapshot refuses on a guest with a bind mount and still exits 0, so a script
guarding a change with it proceeds without the rollback point it believes it has.
The ZFS dataset snapshot is the working path, verified by reading it back.
A synthetic Plex Transcoder invocation is not a valid test of Plex's transcode
path. Plex bundles its own libc among 61 libraries; the harness produced three
distinct failure modes that were artifacts of not reproducing that runtime, and it
failed identically before and after a fix that worked. With no positive control its
negatives carried no information. Only a forced transcode settles it, and PASS is
recognisable by Plex naming the device. The original empty decoder/encoder line was
an absence of evidence rather than evidence of failure -- TranscodeSession was 0.
Jellyfin LXC 107 has the same stale stack and the same Arc available; left alone
per the operator, and it ships its own ffmpeg so this may not transfer verbatim.
Adds the Yarros-side evaluation the training exists to justify: does the adapter
move arbitrary prose toward Yarros, and can the instruct arm still expand a beat
to a paragraph on direction. Yarros-flavoured voice prompts (modern/neutral/
romantasy tiers so any Yarros voice in the modern tier is adapter-attributable,
not prompt-supplied) and a Yarros-register beat SYS on the chat generator.
voice_distance.py is the honest slice of adjudication that needs no seat: Burrows's
Delta over character bigrams against held-out Yarros. Its first cut mis-framed the
noise floor — it used the same-author distance (held-out vs itself) as the
between-arm significance threshold, which is the target, not the threshold. Fixed
to the measured floor: the within-arm seed spread, which is this metric's sampling
variance at this sample size, computed from the two seeds already generated rather
than assumed.
Result on the built corpus, ordering base-125-tuned < instruct-tuned <
base-unadapted, both adapters clearing the 0.046 measured floor (base +0.157,
instruct +0.076), and the ordering corroborating the independent held-out loss
ordering (Base below Instruct). One seed-pair per arm, so it corroborates rather
than settles; the full frozen adjudication still needs a romantasy control panel,
a second seed, and the gen seat for the beat-incumbent leg.
The recurring incident-time failure: during an outage the session reaches for
althing to notify infra-ops, and since ALTHING_HANDLE is infra-ops the send mails
itself, the herald pokes about new mail from infra-ops, and the session triages
its own words as a peer report. Documented the identity, the check
($ALTHING_HANDLE before addressing an ops message), and the one legitimate use of
the infra-ops thread during an incident (a durable memo to a future infra-ops
session on recovery, not a page to a peer).
Records the comparison (Sentinel-R3 is a real agentic-pentest SFT on the same
Qwen3.8-27B base vs M.O.G.-SEC's persona-on-stock-weights, on a tool surface that
matches our own harness), the HF check (M.O.G.-SEC repo unchanged), and the graft:
Sentinel ships no MTP head, so the verbatim base head was grafted in, which is the
correct lineage but leaves acceptance unverified because the head now sits on an
SFT-finetuned body rather than stock weights.
The quant is done and structurally verified; serving and the acceptance/A-B gate
are blocked on GPU space and are the operator's next call, along with the
proprietary-license posture. Detail in the artifact's PROVENANCE.txt.
VRAM said yes and the box said no. Two 4B LoRA jobs fit easily -- 36 of 121 GiB
with 98 free -- but both dropped from a 37.10 s/it solo baseline to ~85 s/it,
2.29x each, so combined throughput fell below serial. The GB10 is past its
memory-bandwidth roofline and pays a contention penalty on top of the split.
Worth keeping because the intuition it corrects is a common one: free GPU memory
is the number everyone reaches for, and on this box it does not predict whether a
second job is affordable. The control is what makes it a finding rather than an
anecdote -- killing the second job returned the first to 37 s/it on the next step.
One arm is not a comparison. On the Brontë corpus Base won held-out loss (2.814
against the instruct arm's 2.908) while instruct won instruction-following, and
that trade is the open question for Skaldsong. It has to be re-measured on this
corpus rather than carried over: BabyYarros is 12% larger and contemporary.
Two entry points because the right one depends on a measurement rather than an
argument. The box has 98 GiB free against an 18.4 GiB training footprint, so VRAM
is not the constraint -- but VRAM was never the binding constraint on a GB10 that
onboarding measured at 6x slower than ana-ml2 where compute predicts 2.7x, and
where batching was not a throughput lever. If the box is already at its bandwidth
roofline, a second job splits the same bandwidth and buys nothing.
So: launch-yarros-4b-base.sh takes an explicit --allow-shared-gpu that bypasses
the GPU-clear guard, and chain-yarros-4b-base.sh waits on the instruct run and
refuses if it produced no adapter. The bypass is an argument and never a default,
because the guard's normal job -- stopping a chain from firing into a live run --
is what keeps one lost run from becoming two. A shared-GPU launch stamps its own
log with a warning that its s/it is not comparable to a solo run, since the
harness is part of the number.
Both carry the same gate guard as the instruct arm: refuse to start unless the
leak gate report on disk says PASSED.
Records what the gate cost to build and what it found: a second typography defect
(small-caps epigraphs and drop caps, which the D1 "no unwrap needed" note was right
about in the wrong direction), back matter inside the prose of all five works naming
real people, a possessive-only entity that was unrenamed and unreported at once, a
character lost to an in-book article typeset in lowercase, and a phrase class the
unigram scan structurally cannot see.
Also records the two things that will matter next time: every new detector flag is
opt-in and the Brontë map was re-derived and diffed after each change, because one
intermediate version silently dropped four real names from Shirley; and the gate's
sensitivity floor is part of its result, not a footnote.
The gate is new. There was no committed instrument for "does any of the author's
own proper nouns survive the rename" -- the Brontë number was produced by hand
-- so leak_gate.py is now that instrument, and it runs both directions every
time: the same scan over the unrenamed source as a positive control, and a nonce
string as a negative one. A detector that only ever sees renamed text cannot
distinguish absent from blind.
Run against BabyYarros as built it reported 212 surviving entities, not the 86
recorded earlier, because it scans the whole corpus rather than each work
separately and it counts the sub-threshold entities rename never looked at.
Three findings came out of closing that.
The corpus had a typography defect of its own. The D1 notes correctly say no
unwrap was needed; a different defect was there instead. The Empyrean books set
their chapter epigraphs in small caps and the extractor rendered the run as
uppercase while leaving the large initial as a separate token, so the corpus
carried "M AJOR A FENDRA'S G UIDE TO THE R IDERS Q UADRANT" -- 106 lines, ~700
splits -- plus 52 drop caps like "T he flight field". That is where the entities
called IDERS, UADRANT, NAUTHORIZED and seventeen bare single letters came from.
A split initial next to an uppercased run is enough to recover the original
mixed case, so the restore is exact rather than approximate: a word with a split
initial was capitalised, an all-caps word without one was lowercase.
Back matter was inside the prose. The builder splits on chapter headings and
nothing follows the last one, so every work carried its acknowledgments,
newsletter pitches and cover-artist credits -- 4,555 words naming the author's
agent, editors and children, in a corpus whose entire purpose is that no
identifiable name survives.
And the gate passed at 0 of 314 while Afendra was still in every copy. The name
never appears unpossessed, so it keyed as an apostrophe form, and rename and the
gate both skip those as contractions -- unrenamed and unreported at once, which
is the worst failure shape available. Baxter escaped a different way: wilder
renders an in-book news article entirely in lowercase, putting the cap/lowercase
ratio at 0.13 against a 0.05 bar.
Then a second class the unigram scan structurally cannot see. Riders Quadrant,
Flame Section, War Games and Fourth Wing -- the book's own title -- are built
from ordinary words the detector correctly refuses to call names. The gate now
audits recurring capitalised 2-3grams against an explicit allow list, and
rename applies a phrase map after the entity pass.
Every new detector flag is opt-in and off by default, and the Brontë entity map
was re-derived after each change and confirmed identical in keys, surfaces and
every field. The stoplist was built by reading each surface in context, which is
why it is short: Violence is Xaden's nickname for Violet, and Continent,
Presentation, Barrens, Originals, Montserrat, Athena, Aura, Curator and Sage are
all in-world. A plausible-looking guess would have excluded most of them.
Final: 0 of 325 entities and 0 of 91 audited phrases survive in any of 30 copy
files, both controls passing. The sensitivity floor is stated in the gate's own
output -- 3 occurrences for a name, 5 for a phrase -- because a negative without
one is unfalsifiable.
vh