Operator's reasoning, accepted and better than the hypothesis-space argument it
replaces: the NAT change went effective, was verified bidirectional, and then ran
correctly for twenty minutes before the site died the moment GPU load was applied. A
working config change does not spontaneously fail under an unrelated physical variable.
The load correlation is tight; the NAT correlation is merely adjacent in time.
Undersized UPS is the only candidate that explains the trigger. NAT material retained
as record, and the power.log/uptime check demoted from decision point to free
confirmation.
Adds the measurement protocol, since the operator is bringing a PDU and an ammeter.
The load-bearing caveat: power.log is GPU-ONLY -- nvidia-smi per-card, excluding CPU,
566 GB of RAM, drives, fans and PSU conversion losses -- so the ammeter at the plug is
the primary instrument and power.log only cross-checks the GPU share. Four states to
capture (idle, one card, two cards, four cards), and capture PEAK rather than average:
UPS overload protection responds to short-term overload, so an average-only reading
that hides transients will mis-size the replacement exactly the way the present unit
got mis-sized, and must be recorded as a floor rather than as the draw.
The four-card figure is earmarked for servers/fv-ml1/README.md, because it closes the
cutover's own open question -- that the FV circuit was likely specced against half the
real draw, back when every record still said the box had two GPUs.
Another session applied a scoped Tailscale SNAT rule to the FV gateway at ~06:22Z, 34
minutes before the site went dark (docs/runbooks/fv-to-ana-nat.md, not my work, left
uncommitted). That makes the UPS-overload theory a hypothesis rather than a finding,
and nobody should buy hardware on it until the discriminator below has been read.
On the evidence that change is the wrong shape to have caused this, and it is recorded
as such so the visit is not wasted chasing it: one OUTBOUND SNAT rule scoped to a
single source /32 and a single destination /16 cannot stop the gateway, the BMC or the
public WAN address from answering inbound; no routes, filter rules, WAN settings or
subnet advertisements were touched; pfctl -sr came back byte-identical; and it was
verified bidirectional afterwards including ANA->FV SSH with Beszel 18/18 up.
Their BMC datapoint used 10.251.50.50, which is not the BMC -- that is 10.251.250.50,
a different subnet. They correctly declined to claim BMC health, but the observation is
void rather than negative and should not be reasoned from.
The discriminator costs nothing and is already on disk: power.log is written locally to
/tank every 10 s by a shell loop on the box and does not depend on the network.
Entries past 06:56Z mean the machine never lost power, which makes this a routing fault
and the UPS innocent; entries stopping at 06:56Z confirm power. Cross-check with uptime
and journalctl --list-boots -- continuous uptime across 06:56Z kills the UPS theory
outright.
So the first action on site is now to READ, not to fix. The two hypotheses lead to
completely different remediations and only one of them needs a new UPS.
Durable capture so tomorrow's session does not have to reconstruct either half.
Built and verified before the power failed: Qwen3.8-Flash-Next serving on a single
RTX PRO 6000 with its 51B n-gram table pinned in host RAM and read over CUDA UVA --
74.36 GiB weights resident, 14.00 GiB KV for 560,654 tokens at the full 262,144
context, 67 GiB host RSS -- plus a gen-large gateway alias verified end to end.
The five findings worth carrying: the offload is #54371 (UVA, merged) which supersedes
the paused worker-based #53899 and designs out its entire bug family;
text_config.ple_embedding_dtype is the load-or-fail discriminator for any community
build; --kv-cache-memory makes vLLM SKIP memory profiling and ignore
gpu-memory-utilization, which inverts the usual pin-bytes advice and let a 16 GiB pin
nearly OOM with no visible failure; MTP is off pending measurement here rather than
written off, because the recipe's number is cross-harness and tested k=3 only while the
head is one layer run autoregressively; and a container once reported (healthy) with no
published port at all, because the healthcheck runs inside the boundary it was trusted
to validate.
Then the outage. Records it as will-not-self-recover, so no session wastes effort
polling a dead site, and carries the three things that change the visit: bypass the UPS
rather than using its surge-only bank (both banks share one 12 A inlet -- the surge
bank bypasses the inverter, not the current rating), recover power.log before anything
else because it is the only load measurement that exists anywhere, and bring seats up
one at a time because ten restart:unless-stopped containers loading at once is the
largest transient the box can make into whatever just failed.
Also records what is still half-done: the stale homepage labels on the 10 containers
that died before they could be recreated, which the staged bring-up fixes as a side
effect, and the eight drifted stacks plus three untracked host-only stacks that were
deliberately left for a deliberate reconciliation.
Operator's read is that the UPS the box was plugged into overloaded and died, and it
fits better than the breaker-trip theory: a UPS's output rating sits far below the
circuit's, so it is the first protective device to give -- which explains why the site
let go at TWO cards loaded rather than four, and why the ~25 W firewall died with it.
Records the operationally important consequence: a tripped UPS resets, an overloaded
one can kill its output stage permanently. If it is dead, nothing on site can be reset
back to life, so the visit needs the means to BYPASS the UPS or it is wasted.
Elevates recovery of /tank/.../power.log to the first action on site. It sampled all
four cards every 10 s up to the cut, lives on /tank rather than in a container, and is
the only measurement of what the load actually drew -- without it a replacement UPS
gets sized by guesswork. Also states that no load figure exists yet, only idle.
Corrects an earlier claim of mine in this session: there is NO local fallback for the
19 dark aliases. Probed -- every free local model is on fv-ml1, and irv-ml1 runs no
chat seat at all, only TTS/ComfyUI/arbo/clipper work on two partly-occupied Ampere
cards. The only non-fv chat backends are paid. Any paid coverage must go under a new
opt-in alias name rather than a silent repoint of summarizer/gen/classifier.
Written while the site is down so recovery does not have to be reconstructed later.
Records what was measured rather than what is suspected: every FV address including
the BMC is unreachable while all three other sites answer, the campaign's last log
line was off_A rep 2 at 06:56:04Z, and the site was dark by 06:58:40Z.
Names three candidate causes with the evidence that would distinguish them, because
the instrument that could have settled it -- the per-card power log -- died with the
box. The two-card-load hypothesis fits the timing and the two prior Anaheim breaker
trips on this same chassis, but it is circumstantial and is recorded as such.
Carries the recovery hazard that matters: every seat on the box is
restart:unless-stopped, so resetting power alone brings ten vLLM containers up
loading at once -- the largest transient the box can produce, into a circuit that may
have just tripped. Staged sequence given, gen first and flash-next last.
Also records the OOB gap the outage exposes: OPNsense-as-subnet-router protects
against box-down/gateway-up, and not at all against the site-wide loss that actually
happened, because the BMC's only path out is through that same gateway.
vh