Two operator-authorised changes on the same afternoon.
## althing v3 (U9b flag day, one-way, no rollback)
The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.
73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.
Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.
nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.
The v2 database is left inert on disk. There is no import path and none
was improvised.
## sec onto GPU0
GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.
Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
forseti asked for journald kernel persistence plus sysstat, on the
premise that nh3-dev's three OOM events in 14 days left no evidence.
The premise was wrong. journald has been persistent all along: 15,068
kernel entries in the 82-day previous boot and 351 OOM records across
retained boots, full task tables included. `journalctl -b -1 -k`
returned one entry because it ran as a user in neither adm nor
systemd-journal, and journalctl shows only your own messages in that
case. The same artifact produced the "journal stops at 05:36:08 with no
shutdown sequence" claim -- the true boot -1 end is 05:47:04 with OOM
kills logged at 05:38, 05:40 and 05:42.
So the fix for "no evidence" is a group membership, not a logging
change: usermod -aG adm lkraven, which is the group Debian's journald
ACL names explicitly.
With the journal readable the attribution is already in it. The
versioned Claude Code binary lives at .local/share/claude/versions/,
so OOM victims named 2.1.220 / 2.1.177 / 2.1.168 are CC sessions, as
are those named claude. Every one of the twelve largest resident
processes ever recorded on this box is a CC session, topping out at
18.4 GB. Everything else killed is 30-55 MB collateral, which clears
the althing daemons by measurement rather than by their own sampling.
sysstat and atop are added because the journal records the moment of
the kill, not the ramp, and names the victim rather than the winner.
atop was not requested and is the one that matters: with a dozen panes
open, only a per-process timeseries says which session was growing.
Not done: a cgroup cap on CC sessions. It is the real mitigation and it
would kill long-running sessions mid-work, so it goes to the operator.
Operator supplied the topology: "the entire rack is on the same circuit,
public ip is served by firewall on the same circuit. load tripped
breaker, entire rack goes dark."
That inverts the argument I committed one commit ago in 3cc55b4. I held
the load attribution partly because a site-wide blackout looked like too
large a blast radius for one GPU box. It is not — it is exactly the
blast radius a shared circuit predicts. The observation I treated as
unexplained was the signature of the mechanism.
Kept the wrong version visible with a superseded marker rather than
editing it away, per the superseded-claims convention: the reasoning
circulated in two althing messages and a peer adopted it.
Holding the attribution as unmeasured was still correct; the argument
used to hold it was reasoning about wiring neither side had seen and
neither side asked for.
Records the finding that outlives the incident: the same circuit powers
ana-ml2, ana-gw, ana-wg, ana-nas and the BMCs, so the load takes every
recovery path with it. Composed with ana-gw's WAN admin being closed
(tunnel-only) and the tunnel terminating on ana-wg, there is no remote
management path to Anaheim during a rack power event, in-band or out.
Open question for the weekend triage: is any of that rack on UPS or a
second circuit?
The record read "power capacity is the open item" next to ana-ml2's
~600 W, which reads as a cause. It is not one. The trip and its timing
are measured; the attribution to the training load is the operator's
working read and the reason for the weekend triage.
The observation that makes the single-load story incomplete on its own
terms: a site-wide blackout is a larger blast radius than one GPU box
accounts for. If ana-ml2's draw were the whole story, ana-nas, ana-wg
and the public address would not have gone dark with it.
Shedding seats may still be the right first move and it is cheap. That
is not the same as having identified what loaded the circuit, and the
distinction matters going into a triage that will act on it.
brokkr-smithy-dev asked how many times 3c was launched rather than
reconstructing it, and their reading was three. It was two.
#1 17:53:33 PDT killed by the power loss at step 80/604
#2 20:58:41 PDT stopped deliberately at 21:07:40, healthy
The phantom third came from a report I wrote at 23:03 narrating the
21:07 kill in the present tense with no timestamp. Every fact in it was
accurate; it was unreadable in sequence against a correctly-observed
22:46 snapshot of an idle GPU.
Evidence is ZFS birth times (a `>` redirect truncates the log but keeps
its birth, so mtime alone cannot separate "rewritten" from "created"),
plus the absence of any mtime under /tank/erp-tune after 21:07:34 — a
relaunch would have rewritten three files there.
Also pins the outage window to 18:14:45-18:17:00 PDT and corrects the
downtime from "~90 minutes" to 1h58m: the last journald entry before a
hard power loss is the last time anything wanted to log, not the moment
of the loss, and here it was 20 minutes early.
Corrects the in-flight header (step 22 -> last-logged step 24, stop
deliberate) and its stale "as of" stamp.
Run 3 trained, gated and dispositioned do-not-serve on a measured 44pp self-harm
guardrail regression that its own preregistered rule passed -- a pooled preserve-list
test cannot see a single-axis collapse. Run 3c (lr 20x cut, single variable) launched,
killed by an Anaheim power-breaker trip at step 80, relaunched, then stopped by the
operator at step 22 pending a weekend power triage.
Also captured: the corpus mix was specified in a unit the optimiser never sees
(45.8% dialogue by context, 24.2% by loss); the dose-response says benefit and damage
are one direction in weight space, so the merge-back measures the problem rather than
fixing it; four guests including the storage SPOF had onboot unset and never came back
from the outage, now fixed with dependency ordering; and a transport failure that
enters a measurement as a value looks like whatever you hoped to find -- which found a
live defect in another agent's instrument an hour after it was reported.
Auto-archived 8 entries to archival-memory.md (Recent decisions: 8, Tried and
abandoned: 0); 4 held back on open deferred-work pointers.
Run 2 is finished, gated FAIL, and serving on the gateway at operator request.
Run 3's corpus was built to brokkr's first recipe and held before any GPU spend:
creative-writing-multiturn is a DECLARED MEGAMIX containing bluemoon, PIPPA,
LimaRP and stheno, and the remix promoted creative-writing AND bluemoon -- the
two roots that overlap, at median jaccard 0.873. Containment, not overlap.
Dedup direction reversed so the primary source survives rather than the copy
inside the bag: bluemoon 67 -> 126 conversations and 38.6% of loss signal, the
largest contributor. Wholly-human share up, megamix share down, total context
unchanged at 12.49M so the operator's settled mix arithmetic survived.
Two structural findings recorded because they outlive this recipe: F1 'excise
PIPPA' removes the ROOT and not the MATERIAL (F2's 250-word floor does that
work, since PIPPA turns cannot exceed 123 words wherever they live), and LimaRP
and stheno remain unchecked against any other root.
Also records the correction I published wrong twice: run 2 was never unstable.
All 46 flags were too_short, the collapse guards fired zero times, and it is the
left tail of a length distribution -- not new to run 2 either, so it is a
property of the recipe and a further base swap will not fix it.
Rewrites the in-flight section: run 1's seat is down, run 2 is up on :8098, and
the base decision the previous snapshot recorded as OPEN is resolved (stock
instruct, operator 2026-08-25).
Four new decisions, and the detail file carries the arc: the two operator calls
that produced run 2, all five gates, the harness commit chain, and the caveat
that its own provenance names a commit AHEAD of the code that ran.
Records three things a future session would otherwise get wrong:
- the mask is proven by the loss-token delta, NOT by the matching p50 step
times -- step time is insensitive to which positions carry loss, so that
check cannot go red on the axis I originally cited it for
- two bf16 26B arms do not fit on one 97.9 GB card (98 GB of weights before
any KV cache), so brokkr's both-arms-in-one-window requirement is a GPU
resourcing call, not a scheduling one
- erp-tune-v1 is still registered in the gateway and returns HTTP 500; the
fix needs a config edit plus a reload that interrupts fleet traffic, so it
is batched for morning rather than done at 2am
Run-01 completed in 7:21:52 (47% faster than the 13.85h round-1 projection),
lora_B gate 205/205 non-zero at median norm 1.708, and the acceptance gate says
it did the thing it was built for: diversity +0.178 against a 0.008 floor (22x),
attractor hit rate -11.3pt against a 2.0pt floor, memorisation 0.0000 on both
arms — which closes the R20 licensed-prose exposure on measurement rather than
argument.
Five new detail files carry the substance:
erp-tune-run2-complete the run, the gate, the noise-floor near-miss
(brokkr was one step from reporting a 13-point
T6 regression sitting inside twice his
instrument's own variance)
mfu-root-caused-attention 8.6% MFU was an accounting artifact; real
utilisation 17-20%, cost was attention on
AMPERE kernels. Two independent methods agreed
to 2.6 points.
nvfp4-serving-pipeline merged weights are MANDATORY — vLLM cannot
serve a LoRA on ANY Gemma-4 — plus the recipe
that silently misses all 11,520 expert tensors
refusal-retention-probe measured base 0/100 -> tuned 29/100, then had
to accept it was the wrong axis
worldtree-b188-b189-and-selene three arcs closed, and a #411 diagnosis I got
wrong twice before a directory probe settled it
Current state rewritten end to end — the previous snapshot had the run in
flight at ~17h with MFU unexplained. Both are now closed.
The open operator decision is run 2's base, deliberately unstaged and flagged
against being filed as a config knob: it is a reversal of the trainee-selection
decision, and the pretrained-base option removes the last non-lexical floor on
the CSAM axis given stage-2-detector-inert and contamination-scan-absent are
both already overridden.
Tried-and-abandoned gains four measured-dead throughput levers, the packing
correction (bucketing wins under sdpa and the conclusion flips under flex — do
not carry it past the backend decision), and the merge-back-undoes-abliteration
trap brokkr caught in his own advice.
Index stays at 291 lines, under the soft cap. No archival this run.
Captures an evening that ran from an OOM crash-loop to a measured trainee base.
The durable lessons, none of which CLAUDE.md can carry: --gpu-memory-utilization
sizes the KV cache and does not cover CUDA context or graphs, which is half of
why a seat that fit on the 21st stopped fitting on the 24th; the other half is
that gen's footprint GROWS WITH UPTIME (38.5 GiB fresh against 45.6 GiB after
three days, same container, same flag), so headroom arithmetic against a
long-running gen measures a moving number. The stale-chat-template trap turned
out to be endemic across third-party Gemma-4 derivatives rather than a one-off,
and it is silent in both directions — wrong prompt when serving, train/serve
skew when tuning. And a benchmark finding was retracted because 12% on a
five-option task is below the 20% chance floor: a below-chance score indicts the
instrument before the model, and a preflight can be thorough while aimed in the
wrong direction.
Records the serving decision for the tuned model with its history intact:
LoRA-on-NVFP4 is preferred if it works, merged weights the expected fallback,
but the archived root-cause says the objection was never NVFP4-specific — vLLM
0.24.0's LoRA application was a silent no-op proven quant-agnostic, and ana-ml2
now runs 0.26.0. Retest before designing around merge; the answer changes what
Eitri's harness must emit, and he is still early.
Auto-archival moved 5 entries (Recent decisions) to archival-memory.md; the
guards held back the rest of the 78 age-eligible candidates because their bodies
carry open deferred-work language, per the keep-when-unsure rule. Index sits at
286 lines, above the ~250 target and reported rather than forced.
The board was on the Australis TERMINAL palette, which is dark-only by design
("Always dark first. No light mode in this system"). Skyfall is the dual-theme
web derivative of the same science, and its bundle turned out to be sitting in
this repo's own git history: a predecessor vendored it on 2026-08-19 and a
later commit deleted it. `git show 45c1995:...` returns colors.css with both
`:root` (dark) and `[data-theme="light"]` (Skyfall Day) intact, plus the
calm-depth layout tokens, the typography scale and Supreme 400/500/700. So the
light ramp is canonical rather than derived, which was the entire objection to
building one.
The visual language moves with the palette. Depth is now the recipe and not a
choice — every elevated surface carries a 1px hairline AND a two-layer shadow,
never one without the other. Radii move to Skyfall's scale, cards at
--radius-lg. Widget stat values move from the display face to mono, because
Skyfall is explicit that numbers and telemetry are always --font-mono. The
full-width aurora ribbon under the tab bar is gone: Skyfall sanctions exactly
two accent expressions, the active rail and hero-only glows, and a decorative
gradient across the chrome is neither — so the colour it carried now lands on
the active tab as a 2px accent bar plus an --accent-soft fill, which is the
rail. Every binding is written against the semantic layer; there are no raw
family tokens and no colour literals left in our own file.
build.py now guards the vendoring instead of advising it. The three token files
are hashed and a mismatch FAILS the build — a vendored file is either
byte-identical to the bundle or it is a fork wearing the bundle's name, and the
theme this one replaces had to be torn out twice for exactly that.
⚠ Homepage's own theme toggle is unreachable, and reaching for it breaks the
dashboard. It renders only when settings.yaml leaves `theme:` unpinned, and
with the key absent the page's data loader throws and its catch branch serves
`initialSettings: {}` — no tab bar, no layout, no i18n. Six force-recreates
over seven minutes all came up empty; restoring `theme: dark` rendered
correctly on the next recreate in 12 seconds, while /api/services returned 200
with fully correct content the whole time. That is the first confirmed cause of
the long-running "tab bar goes missing after a recreate" symptom, and it also
retires the homepage.log-size lead recorded earlier today: rolling the log
aside did nothing during this episode, so that coincidence was intermittency.
So the toggle is ours. conf/custom.js renders it and stores the choice;
build.py re-emits each vendored light block twice, once for an explicit
`data-theme` and once inside a prefers-color-scheme media query scoped to
`html:not([data-theme="dark"]):not([data-theme="light"])` — that :not() pair is
what lets a stored dark choice survive a light-mode OS. Verified against both
OS preferences: load, click, click back, reload, all four correct. `data-theme`
is the control surface; Homepage's own `dark` class stays on <html> and does
not fight, because our rules carry !important on the surfaces Tailwind's
`dark:` variants would otherwise claim.
Two font substitutions, both documented rather than silent: Space Grotesk for
Bespoke Sans and JetBrains Mono for Victor Mono. Only Supreme was ever vendored
here and Skyfall's own notes call Victor Mono user-supplied, so this is a
two-line swap when the real faces arrive.
Dark and light, all four tabs: http://10.100.10.50:8090/b/homepage-skyfall/
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.
Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.
A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.
The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.
28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.
The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.
Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.
Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
The board's card width changed at every group boundary because `columns:` was
being tuned per group under the 2026-08-18 "columns = member count" rule. That
rule is retired: it sets `lg:grid-cols-N` for one group, so it fixes that
group's CARD WIDTH, not its density. Measured on the live board, Notes rendered
a single 1464px card, News and Media 728px, Eval & Retrieval 286px, everything
else 360px. All 20 groups are now `columns: 4` and every card renders at 360.
`.service-name` reserved a 78px status gutter with `padding-right` and relied
on `overflow: hidden` to hold it, but overflow clips at the PADDING box — the
gutter was spill room the title printed straight through. Six cards on the AI
tab rendered their name underneath their own status pill, measured by testing
the title text node's box against the status cluster's. The intended ellipsis
never fired either: it is painted by whichever block's own line overflows, and
that is the anonymous box around the bare title text node, which does not carry
`overflow`. The gutter holds by wrapping now, and the description opts back out
of it with a negative margin since the pill only ever covers the first line.
Scriberr's `homepage.group=AI Systems` named a group absent from `layout:`, so
it had no `tab:` and rendered on all four tabs — the same defect as UltraSeedbox
in 2026-08-18, arriving this time from a container label. Relabelled to
`AI - Audio Tools`, where the other ASR seats already live. The
`homepage.group=AI Systems` sample in the repo-root CLAUDE.md was the source of
the name and now carries the constraint, plus the fact that a labelled
container is discovered from any of the five configured engines and must not
also be listed in services.yaml.
Also: descriptions clamped to three lines so a five-line outlier stops dragging
its row 50px taller than its neighbours; icon ramp overridden off slate-400 ->
slate-700, which was sinking the bottom half of every glyph into the card fill;
bookmark groups and Jellyfin's trailing stream rows brought into the card and
eyebrow vocabulary; group gap 10px -> 22px now that width no longer separates
them.
The icon override sits on `html[class]`, not `:root`: Homepage sets the same
variables on `.theme-slate`, which is on the <html> element, and a class beats
`:root` on the same element.
Verified with Playwright against the live board — per-group card width, card
height spread, and a geometric title-vs-status collision check, before and
after. Before/after captures: http://10.100.10.50:8090/b/homepage-relayout/
Rewrites the in-flight section, which had gone stale in one place
(speaches is stopped, not live) and did not reflect a session in which
everything opened was also closed. Three detail files carry the bulk:
the FortiGate's public surface reduced to nothing including the ACME
listener, Scriberr's deployment and the three upstream defects it
required working around, and the ESH DNS repair that also made the
IPv6 naming scheme real on three hosts.
The tunnel entry in the decisions log described a cipher change as
proposed and pending; it has since been attempted, found impossible in
the form intended, and resolved differently, so the line now points at
the outcome rather than the intention.
New decisions cover the cipher adoption and the finding that the
per-flow ceiling belongs to the UniFi gateway's software cipher rather
than the firewall, the administrative closure, the transcription
deployment, the DNS repair, the retirement of an ASR service whose
only consumer was abandoned, and a database VM that until now had no
fleet identity and no vaulted credential at all.
Three approaches are recorded as abandoned: an AEAD cipher the far end
cannot express, an upstream DNS setting the gateway accepts and
ignores, and a scheme to claim unused delegated prefixes that founders
not on the prefixes but on having to rebuild a firewall policy to use
them.
The gateway has no IP-passthrough mode, confirmed by the operator with
admin access to it, so the inexpensive path of letting the UDM take the
delegation whole and carve it natively is unavailable at this site.
What remains is a separate DHCPv6 client presenting several identities,
which needs recabling to reach the gateway, splits routing so that one
device handles v4 and another v6, and above all moves IPv6 off the
UDM's zone firewall, leaving the entire policy to be rebuilt elsewhere
before any host could safely hold a globally routable address. The
operator has declined it and the LANs stay without v6.
The index entry also still asserted the superseded single-delegation
conclusion as fact; it now carries the corrected reading alongside the
ruling, so the finding and the decision not to act on it arrive
together rather than the first inviting a retry of the second.
The note concluded AT&T delegates a single /64 and that reading the
address pattern as a /60 was a mistake. A later finding in the same
session overturned that by reading the BGW's own statistics page, and
that correction was never written down; it survived only in the
transcript and surfaced again while assessing whether more prefixes
could be claimed.
The gateway holds the /60 and rations it, keeping half for itself and
delegating the top eight /64s one at a time. Both observations agree:
a prefix-id only carves within a delegation already held, so the
earlier test could not have moved a lone /64 regardless. The limit is
that UniFi solicits once, which makes the requester the ceiling rather
than the carrier.
The ACME client is unbound from wan1, which removes the HTTP-01
challenge listener that had been holding port 80 open irrespective of
allowaccess. An external sweep of fifty-five ports against the WAN
address now finds nothing open at all, while the internal interface
still serves the GUI and SSH and retains its certificate, which
remains valid until late October and simply stops renewing.
The previous note's claim that four virtual IPs were unrestricted
all-port static NATs is withdrawn. A FortiOS virtual IP can be scoped
either by an explicit forwarded port or by a service binding on the
object, and the earlier parse inspected only the former. All four use
the latter, and the custom services behind them are narrow. None of
the fourteen is unrestricted.
Ground truth from outside is recorded in place of further config
reading, listing what each public address actually exposes. Three
configured mappings answer nothing at all and are noted as tidy-up
candidates for the migration rather than as exposure.
The previous note attributed the open port to an ISP transparent
proxy. That was wrong. It terminates on the FortiGate: system acme is
bound to wan1, and FortiOS opens port 80 there to answer HTTP-01
challenges regardless of what allowaccess permits, which is why the
port stays open with the interface set to ping only. Every
non-challenge request returns a fixed 403 whose body reads ACME Access
Only. No DNAT is involved; of the fourteen virtual IPs only two land
on that address, neither on port 80.
The wrong conclusion came from a sniffer filtered on dst host, which
matches inbound packets alone and so excluded the replies being looked
for. Filtered bidirectionally the box is plainly seen emitting SYN-ACK
on port 80. The note records the rule.
Two consequences follow. The earlier warning that certificate renewal
would fail without http in allowaccess is retracted, since FortiOS
opens the challenge port itself. And the listener is not an
administrative surface, though its value is now marginal with WAN
administration closed, so the note records how to remove it and leaves
that decision open.
Also captured: four virtual IPs are all-port static NAT and map every
port of their external address.
Both tunnels now negotiate AES-128 for ESP, applied make-before-break
so neither dropped waiting on its far end: the FortiGate was widened
to accept the new cipher alongside the old one first, then each UniFi
gateway was flipped. Single-stream throughput moves from 245 to 270
on the NH3 tunnel and from 268 to 304 on the ESH tunnel. Both network
objects were diffed against pre-change snapshots and the only field
that moved on either is the ESP cipher.
The proposal lists are left accepting AES-256 as well. The peers offer
only AES-128 so the extra entries are inert, and retaining them means
a gateway reverting cannot strand a tunnel.
With that up, the WAN administrative surfaces are closed. The
interface is back to permitting only ping, and the infra-ops account
is again restricted to RFC1918 space. Ports 443 and 22 were confirmed
closed from two separate sites and management over the tunnel still
works. The close was issued over the tunnel rather than over the WAN,
since withdrawing SSH from the interface while connected through it
would sever the session mid-command. The box now has no out-of-band
path, which the memory records explicitly.
Also captured: the two UniFi vault items have different shapes, one a
bare key and one a documentation note requiring extraction, which
produces an opaque nginx rejection if missed, and the ESH key's first
confirmed write.
FortiOS has no WireGuard, so any WireGuard site-to-site has to
terminate behind the edge, and ana-wg already fills that role. The
earlier 767 Mbit/s figure was taken with traffic terminating on that
box, which left the forwarding case unmeasured and overstated what a
real deployment would see.
Measured properly, transit from NH3 through the tunnel and onward to
ana-docker returns 763.8 Mbit/s on one stream and 790.4 across eight,
so the forwarding hop costs nothing and the full threefold gain over
IPsec survives. The container uses around a fifth of four cores at
that rate, against the UniFi gateway spending a third of its own four
to move a third as much.
The note records what still needs deciding rather than presenting this
as ready to build: Anaheim hosts have to route to the terminator
directly or pay a hairpin through the edge, which was not measured;
the terminator introduces a failure mode the edge did not have, with a
fallback route as the mitigation; and the NH3 end needs its own
terminator, where a Linux host matches the measured figure while the
existing UniFi WireGuard server would likely land lower.
The test tunnel, its peer and the temporary route were all removed.
Varying only the cipher settles what inference could not. AES-128 has
ten rounds against AES-256's fourteen, so a software-bound path must
speed up when the cipher gets cheaper while a hardware-offloaded one
will not move. Run as A/B/A on a single stream, AES-256 returns 232,
245 and 243 Mbit/s and AES-128 returns 282 and 275, a gain of about a
fifth at identical CPU. The FortiGate's offload engine is therefore
not the constraint, and the operator's reading of the UniFi side as
software crypto without acceleration is correct.
Two earlier claims of mine were wrong and are corrected in place. The
observation that the UDM sat at seven percent CPU was a sampling
artifact, since UniFi refreshes that statistic on the device report
interval and the windows used were four seconds; a sustained run reads
thirty-five percent, around one and a half of four cores. The
per-session offload hypothesis is refuted rather than merely unproven,
because it predicts no response to a cipher change.
The remaining figures follow from this. One stream is one core, eight
streams engage about three, and aggregate stops responding to cipher
choice once several cores are working because the path bounds it. CBC
chaining is what makes this expensive, as it serialises blocks and
prevents the AES instructions from pipelining, which is also why the
same gateway manages far more over WireGuard.
The cipher was restored and the network object verified unchanged
against its pre-test snapshot.
The operator asked for a WireGuard test between ana-wg and NH3 across
the public internet, which is the arrangement that separates the path
from the crypto since neither gateway performs encryption in it. A
single stream returns 767 Mbit/s against 245 for IPsec over the same
wire, and eight streams return 763, so one flow already saturates the
path and there is no per-flow penalty whatsoever.
That settles several things at once. The limit is not the circuit,
the NH3 uplink, the ISP or the physical path, all of which carry 767
on one flow. Ranking the implementations, Linux WireGuard shows no
per-flow penalty, UniFi's WireGuard shows roughly 1.4x, and IPsec on
this pair shows 2.8x. Latency under a single bulk stream tells the
same story, rising to 12.7 ms on WireGuard against 102 ms on IPsec.
This overturns the earlier conclusion that changing transport was not
worth pursuing, which compared eight-stream figures and so understated
the gap for single-stream work by a factor of three. A WireGuard
site-to-site terminated on ana-wg now looks worth considering, and the
note records the open questions around failover and policy.
Attribution between the FortiGate and the UniFi gateway remains
unresolved, since both perform IPsec in the slow measurements, but the
remedy does not depend on which one is responsible.
The test tunnel was removed, ana-wg is back to its original three
peers and the generated keys were shredded.
The operator asked whether the 80F is simply out of capacity. It is
not. Routing a single flow between two Anaheim VLANs through the same
box, with no tunnel involved, sustains 940 Mbit/s, which saturates the
link, and eight flows over that path return the same figure. There is
therefore no per-session ceiling in the plain forwarding path and the
roughly 250 Mbit/s per-flow limit belongs to the IPsec datapath alone.
Measuring the second tunnel with the same probe supports this. The ESH
side, terminating on different gateway hardware at nearly half the
round-trip time, returns 268 Mbit/s on one stream against NH3's 245,
and 715 against 692 across eight. A window-limited path would have
been substantially faster at the shorter round-trip, so the agreement
between two dissimilar far ends is further evidence of a rate cap.
This also corrects the previous note, which named the FortiGate on
reasoning that could not separate the box from the protocol, since
every slow path was both. That separation is still not established:
both tunnels terminate on UniFi gateways running common firmware, so
the cap could belong to either side. The note records the tunnel
topology that would decide it and flags the bearing on the pending
replacement.
The earlier note described the constraint as per-flow serialisation
without evidence. Pinning SO_SNDBUF and sweeping it shows what it
actually is: throughput holds flat between 224 and 247 Mbit/s across
a thirteenfold range of in-flight data while round-trip time scales
with the buffer instead, from 7.8 ms up to 107 ms. Retransmissions
are absent throughout. That is a fixed service rate with a standing
queue ahead of it, so socket buffer, window scaling and congestion
control are not the lever and should not be pursued.
The same measurements surface something with wider consequences than
throughput: a single bulk stream lifts tunnel latency from 6.9 ms to
102 ms average, 136 ms peak, with no loss. Interactive traffic
sharing the Anaheim link degrades sharply whenever anything moves
bulk data, and parallelising transfers makes that worse rather than
better.
Localisation rests on the FortiGate appearing in both slow paths and
in neither fast one, with aggregate throughput over the same security
association reaching 692 Mbit/s. Per-session NPU offload fits the
shape but was not confirmed on the device; the note records the test
that would settle it and flags the bearing this has on the pending
FortiGate replacement.
The AES-GCM cutover was authorised and attempted, NH3 side first. It
cannot be completed: UniFi's manual site-to-site IPsec implements no
AEAD cipher. Eight GCM spellings were rejected with
api.err.InvalidPayload while an otherwise identical body carrying
aes256 returned rc:ok, which isolates the enum rather than the
request as the cause. The accepted set is aes128/aes192/aes256/3des.
Both Anaheim tunnels terminate on UniFi gateways, so this blocks the
ESH tunnel on the same grounds.
Measuring while testing also retires the premise. NH3's uplink is a
1 Gbps link, so Anaheim's 2 Gbps circuit was never the relevant
ceiling, and the tunnel sustains 692 Mbit/s across 8 streams rather
than the ~550 previously recorded from a 4-stream run. Compared with
WireGuard over the same gateway and uplink, the deficit narrows from
2.3x at one stream to 15% at eight, so moving this link onto a
different transport is not worth doing.
The constraint is per-stream, around 245 Mbit/s, with both endpoints
idle under load. Parallelising bulk transfers remains the mitigation
and is worth 2.8x at no cost; NFS nconnect is the equivalent lever
for single-stream mounts such as /mnt/smithy on ana-ml2.
FortiGate phase2 for pfi-ana-nh3 keeps the widened proposal list,
which is inert while the peer offers only CBC and avoids a further
renegotiation. The UDM network object was diffed field-by-field
against its pre-change snapshot and is unchanged.
Session captured for a context reset. Six new detail entries.
THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~550 Mbit/s aggregate against a
circuit measured at 2,153 Mbit/s. Not WireGuard (it is IPsec on ana-gw), not
CPU (idle), not crypto exhaustion (NPU-offloaded), not the fibre. Both tunnels
negotiate aes256-sha1; AES-GCM proposed. Operator signalled authorization;
execution pending, untracked by operator choice.
Also recorded: selene retired after losing a head-to-head on its own job with
chat-judge moved to gen and the model name left to 404; the 7-alias collision
on the gen seat that makes cross-alias corroboration an echo; hrafn adopted and
its CI found to have been reporting green while deploying nothing for its whole
life; all three Worldtree instances de-armed from a 69-day-stale :latest and
the Matrix homeserver re-plumbed to personal; every secret-bearing .env on
ana-docker tightened to 0600; the pfi org closing the repo-creation half of the
credential-migration directive; booth kept-board deletion and link pruning.
Two entries under Tried and abandoned: the CI checkout assertion that broke the
pipeline twice and was removed, and my proposal to alias a retired model name
at a different model, which the operator correctly overruled.
Index 271 -> 282 lines, under the 300 cap, so no archival fired.
Operator approved after real-use testing. The experimental standalone
container is retired and stacks/mog-sec is canonical again, with
restart: unless-stopped so the configuration survives a reboot.
Cutover verified against the container it replaces: KV pool 526,617 tokens
at 1.10x concurrency, identical; zero restarts; both gateway aliases
serving; DFlash2 confirmed drafting at k=7 with 231 draft tokens over 33
drafts; vision working at 2048x2048.
One variable was deliberately dropped rather than carried over. The previous
stack hardcoded PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, the
validated container never set it, and the quant playbook records
expandable_segments corrupting retained tensors in another context. The
compose now defaults it empty via MOG_ALLOC_CONF. Promoting the stack as it
stood would have shipped a variable the tested configuration did not have.
The speculative config moves into a single MOG_SPEC_CONFIG carrying the
whole JSON, because the two shapes are not interchangeable: dflash requires
a model pointing at the drafter and MTP must not have one, so a
method-plus-tokens template cannot express both. Also parameterised:
MOG_DRAFT_MODEL, MOG_MM_PROCESSOR_KWARGS, MOG_MAX_NUM_BATCHED_TOKENS.
The mm-processor image cap is now mandatory rather than incidental. The
model's own preprocessor declares 4096x4096, which expands to 16384 image
tokens and kills startup on builds that enforce the image-token count check.
Adds the .env.example this stack never had, carrying the measured rationale
for each value and the one-line rollback.
Operator ruling: the multi-turn degeneration lives in the un-fixed vLLM, not
in the weights. The hypothesis that sec's stock-graft MTP head causes it is
withdrawn.
Two failures produced it. First, a false dichotomy treated as a deduction:
having verified gen and sec run an identical engine, I concluded config was
eliminated and therefore the weights were responsible. That does not follow.
An engine bug present in both seats is not exonerated by the seats being
identical; it only means the engine cannot explain a difference between
them. It can still explain the failure.
Second, and more instructive, the difference being explained may not exist.
The premise was a single operator observation made during a session with
many concurrent changes. That cannot carry a causal claim, and it became the
load-bearing support for a root-cause narrative it could not hold.
The same caveat now attaches to the coherent-to-10k observation on the new
build: same n, same uncontrolled conditions, opposite direction. The
comparison is weak at both ends, so the file no longer presents either
sighting as a result.
What survives as measured fact is unchanged and still recorded: sec's MTP
head is byte-identical to the uncensored base across all 15 tensors, gen's
was abliterated in-band, and acceptance differs slightly. None of that is
shown to cause degeneration.
Adds the generalisable lesson: an observation made while many things are
changing cannot support a causal conclusion. It is the inverse of the
warning already in the gen-seat compose file, which guards against trusting
a negative result from a synthetic probe; this guards against trusting a
positive sighting from an uncontrolled session.
Records the 2026-08-22 session with measured results, hypotheses, and wrong
turns explicitly separated -- the operator held this back while it was in
flight specifically so conjecture would not enter the record as fact.
MEASURED. DFlash2 works on an abliterated + NVFP4 compressed-tensors target
on Blackwell sm_120, which the model card does not claim (it tests stock
BF16 on H200). gen 2.753 -> 3.254 accepted tok/forward and 114.9 -> 131.9
tok/s; sec 2.676 -> 3.252 and 110.5 -> 130.0. The drafter is model-agnostic
across two different finetunes to 0.06%, but is EAGLE3-style coupled to its
target's hidden states, so the weights file is shareable while the 3.85 GB
of VRAM is per-seat.
The k=7 MTP control is the load-bearing result: raising MTP depth improves
acceptance and collapses throughput to 74.0 tok/s, because a single-module
head run autoregressively costs one forward pass per draft token. Without
that control the obvious recommendation would have been wrong.
CONFOUNDED. sec no longer degenerates at 2k, but the engine advanced 259
commits and the drafter changed at the same time. Isolating it means running
MTP k=3 on the new build. Also recorded: #51113 is present in both builds
and is therefore necessary but insufficient, since sec ran it and still
degenerated.
HYPOTHESES, labelled as such: that sec's stock-graft MTP head causes the
degeneration, and that NVFP4 explains the gap against published acceptance
figures. Neither is proven.
WRONG TURNS, recorded so they are not re-derived: version strings are not
lineage, Docker Hub push timestamps are not source freshness, and the claim
that 1M context needs YaRN absent from config is false for the sec quant.
Operationally important: sec is serving from a standalone container rather
than its compose stack, which is stopped but unmodified. Rollback is two
commands and is written down.
Live GEN_MODEL is now qwen38-27b-coldfusion-h300-nvfp4-mixed (ana-ml2 GPU0
:8015). Served-name left unchanged so all 7 LiteLLM aliases route without a
gateway edit.
Verification:
KV pool 401,550 tok / 1.53x (baseline 403k / 1.54x)
LiteLLM aliases 7/7 green
vision 3/3 shapes, colour+form+position correct
MTP acceptance 59.7% median @ 118.37 tok/s
quality gens 4/4 correct
abliteration 4/4 compliance
PPL NOT measured (see below)
The roadmap predicted ~47% acceptance for a pristine MTP graft versus L35's
59.1% in-band edit. Measured 59.7% on the same harness: there is no acceptance
penalty, which removes the throughput argument for reimplementing MPOA.
A single long-prose generation read 47.5% off the same counters -- below the
8-run minimum of 49.0% -- and would have "confirmed" the prediction by
coincidence. Acceptance must be read from quickbench.py, never one sample.
PPL is blocked on VRAM, not on the model: eval_quality.py aborts with
"prompt_logprobs look uniform" under --speculative-config, and the probe-seat
workaround needs ~22 GB while both cards sit at ~96% committed.
Also normalizes the quant dir from root:0600 to llmuser:llmuser 0664 to match
every other model dir, and records that config.json sha256 is byte-identical
across the h300 and L35 quants and is therefore useless for confirming which
weights are mounted (mtime and a head-hash are the discriminating views).
Rollback is one line to .env.bak-pre-h300-20260820.
Ran Heretic v1.4.0's 300-trial TPE search on Cold-Fusion. Best trial scores
8/100 refusals at KL 0.0136 against a 98/100 base, versus absolute-heresy at
29/100 and our hand-tuned Robinson L35 at 72/100 / KL 0.0116 — i.e. 64 fewer
refusals for the same damage. Hand-verified coherent: correct arithmetic with
shown working, clean code, 66-167 word prose across nine probes.
Durable findings:
- direction_scope=0 (single shared direction) is decisive on this merged base:
n=129, best 8/100. Per-layer directions n=131 never beat 52/100 despite a
better median. Points against the multi-direction intuition for a diffuse
direction (our two-template |cos| is 0.62 vs Robinson's 0.99 on stock).
- Aggression is not the lever. r(KL, refusals) = -0.561 over 261 trials; the
KL<0.02 band contains both the worst results (median 87/100) and the single
best. A KL 0.3554 trial scored worse than one at 0.0193.
- PR #317 confirmed: Heretic silently drops the MTP head on save. Source 1199
tensors -> export 1184, all 15 mtp.* gone, vision 333/333 intact, exit 0, no
warning. This is also why absolute-heresy ships a byte-identical MTP head —
a bug, not a design choice. Always diff tensor keys after a Heretic export.
- Heretic's recovered direction carries 6.18% of its energy in sink dim 3994,
versus 0.094% for our L35 and 1.97% for the L39 we rejected as brick-inducing.
It survives that only because of magnitude-preserving ablation
(row_normalization=FULL); our plain projection has no such protection, so the
sink screen correctly refused the in-band MTP graft. Same direction, different
operation. MPOA is the prerequisite for in-band MTP on a Heretic trunk.
- Heretic's edit is recoverable from weights: delta is rank-1 (s2/s1 ~ 0.010),
SVD gives the direction, norms give per-layer weights (1.08 -> 1.34, i.e.
over-projection). Cross-layer |cos| agreement 0.9903 independently confirms
the single-direction result.
New tooling in services/coldfusion-abliteration/:
kl_divergence.py first-token KL, class-split, zero noise floor
catatonia_gate.py 12 probes x 220 tokens, prints every completion
heretic_export.py PTY driver; selects by measured value, never by menu
position — Heretic's resume prompt puts "delete the
checkpoint and all results" one arrow-key from the target
graft_mtp.py recovers the trunk direction by SVD; --pristine for the
safe path when the sink screen refuses
Also adds quant playbook 3.13: the NVFP4 recipe sets observer="imatrix_mse" but
llm-compressor has always silently fallen back to uniform MSE for want of
importance data — on this build and on the incumbent. Existing A/B comparisons
stay valid since every build shares the fallback. Parked as id 42.
Guardrail note: this build has lost the self-harm guardrail that the Robinson
L35 build retained. Restoration is the operator's own work item.
Restoring the two GPU0 seats with `start meromero; sleep 10; start gen` put
meromero into a 7-restart crash-loop:
ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less
than desired GPU memory utilization (0.52, 49.38 GiB).
The previous commit's README claimed restore order "is not actually load-bearing"
on the grounds that both seats pass --gpu-memory-utilization as a fraction of
total VRAM. That is half right and the wrong half mattered: the fraction sets the
target, but vLLM gates startup on FREE VRAM and refuses to start unless the whole
target is available. GPU0 runs at ~96.4/97.9 GB with roughly 0.4 GiB of slack, so
the seats coexist only in the order they were originally brought up, and meromero
is the one that does not fit in the remainder. The pre-existing auto-memory note
("gen takes a fraction of free VRAM at startup and will starve meromero") was
pointing at the real effect.
Also: "first" means healthy, not ten seconds earlier. A sleep 10 against a
two-to-three minute weight load is simultaneity, not ordering — gate on observed
state.
Recovery applied: stop gen, wait for meromero healthy, start gen. Verified
against the pre-window baseline rather than against "both green":
gen KV 14.36 GiB / 403,065 tok / 1.54x -> 14.34 GiB / 401,550 tok / 1.53x
meromero KV 542,202 tok -> 542,202 tok
RestartCount 0 on both; summarizer smoke-tested through LiteLLM
Note for the next reader: raw nvidia-smi used-MiB is the wrong check here. It
reads 89,503 now vs 96,376 before, which looks like a 6.9 GB regression and is
allocator slack — serving capacity is unchanged. The anomalous boots were the
high ones (34.95 GiB KV), where gen came up on an empty card mid-window.
Adds `kl_divergence.py`: first-token KL(stock || abliterated) over the full
248,320-token vocabulary, bf16 vs bf16, scored separately for held-out harmless
and reserved-harmful prompts.
Result (L35, 256 harmless / 104 harmful, answer mode):
harmless median 0.0211 mean 0.0364 top-1 agreement 89.8%
harmful median 0.5996 mean 0.6992 top-1 agreement 55.8%
selectivity 28.4x (72.8x in think mode)
Self-KL noise floor is exactly 0.0, and all 720 per-prompt values are
bit-identical between a single-process and a two-process run, so the figures are
signal rather than bf16 jitter. Reverse KL on harmful/answer is 1.43 vs forward
0.70 — the mass-where-stock-had-none asymmetry expected of a refusal-direction
removal. Against the Heretic reference figures (0.1191 prior seat, 0.0759 the
live absolute-heresy seat) this is materially gentler, but those are the other
tool's optimizer output on a different base with its own harmless set and
template — order-of-magnitude, not head-to-head. KL remains a fidelity number;
the viability gate is still MTP acceptance (59.1%).
Method notes:
- Prompt classes are reported separately by design. A single averaged KL over a
mixed corpus is close to meaningless, since the metric is meant to be large on
harmful prompts and small on benign ones; the ratio carries the information.
- The harmless evaluation set is drawn from the alpaca pool minus calibration's
own draw, reconstructed by replaying that draw rather than remembered, and
asserted disjoint on text. The harmful set is the reserved test split.
- `render` is imported from abliterate.py rather than copied, so the measurement
cannot drift from the rendering the direction was captured against.
- Batch size 1 with logits_to_keep=1: no padding semantics, ~0.6 MB of logits.
Three corrections to the runbook, each of which cost time:
- "bf16 is 50 GB, only gen must go" was 50.10 GiB mislabelled. Text-only weights
are 51,300 MiB; freeing either GPU0 seat alone leaves ~50,933 MiB. Both must
stop. VRAM is now sized from the safetensors headers at run time.
- A 27B model cannot be released in-process: `del` + gc + empty_cache left free
VRAM at 45,287 MiB, and so did confining the model to an inner frame that
exits. Only process exit returned the card (96,689 MiB). The first run
completed only because the allocator hit OOM, collected, and retried. Each
model now gets its own process, handing log-probs to disk between stages.
- The residency gate read hf_device_map, which transformers leaves empty when the
model fits on one device — it reported "(unsharded)" whether or not anything
was wrong, so it could never fail. It now reads parameter devices directly.
Model-agnostic lessons promoted to the quant playbook (new 3.12).
Flip the in-flight status from 'capture done, calibration expansion next' to
'landed, works'. New detail file captures the three corrected diagnoses (layer-
selection metric, sharding/allocator misdiagnosis, corpus-size falsified) and the
verify/quant work still owed. Supersedes the -capture.md detail file's framing.
Captures the session's real work as the in-flight focus: abliterating
DavidAU Cold-Fusion with the Robinson formula. fp32 capture succeeded
(finite direction, layer 22, sink-clean) but two-template agreement is
0.59 vs Robinson's 0.99 — calibration-set expansion is the next step.
New detail file records the full saga including the transformers/DeltaNet
bf16-NaN fight (fp32 fix, the causal-conv1d kernel gap, the seat-restart
VRAM-greed gotcha). Supersedes the earlier "watch for DavidAU's heretic
build" posture — we abliterate it ourselves.
Auto-archived 4 closed entries (Recent decisions: Booth-3-features 08-05,
worldtree-sdk 07-31; Tried and abandoned: containerd-race 08-03,
mv-rename 08-02) to archival-memory.md; the rest of the over-cap entries
are held back by the <14-day and open-deferred guards. Index 331 -> 327.
esh-pve hard-froze at 03:34 on 2026-08-19 and stayed frozen ~4.5 hours
until a manual power cycle. No panic, no OOM, no MCE — the journal stops
mid-operation. The whole ESH site lost DNS with it, because esh-userland
(VLAN 10, the PVC SSID and wired userland LAN) was handed exactly one
resolver: 10.0.50.45, AdGuard on esh-docker-vm, on a different VLAN, with
no secondary. Internet and routing were healthy throughout.
Two fixes.
1. DNS: 10.0.10.1 (the gateway, verified resolving) added as secondary on
esh-userland via the UDM Classic API. Note this is degradation cover,
not clean failover — clients that query resolvers in parallel will
bypass AdGuard for a share of lookups.
2. Watchdog: softdog -> iTCO_wdt under systemd (RuntimeWatchdogSec=60),
watchdog-mux masked. The box looked watchdog-protected and was not: a
software watchdog cannot fire when the kernel it lives in is wedged,
and watchdog-mux only pets the device while an HA client is connected,
which never happens on a cluster with no HA resources. Firmware does
not block the TCO timer here, checked before committing to it.
Also pins VM 102 off (onboot: 0). It starts with full GPU passthrough and
vfio-pci enabling that device is the last thing the kernel logged, 39
minutes before the freeze. The other suspect is the kernel itself: the
host ran 4.5 months on 6.8.12-16, took 6.8.12-42 in an apt batch on
08-18, and died 20 hours into the first boot on it. 6.8.12-16 is still
installed as the rollback.
The playbook is idempotent — a second run skips all six steps and passes
all six verifies. The watchdog is confirmed armed (identity=iTCO_wdt,
state=active, held by PID 1) but has NOT been observed firing; proving
that needs a deliberate wedge.
Memory also corrects two wrong mid-incident calls: the mgmt VLAN is
routed over the site tunnel and is not firewalled off — both symptoms
were the dead host generating ICMP unreachables.
One update.sh run on irv-ml1 carried both open upstream PRs, per the
operator's green-light on the job-store fix:
- #5 (464dfc2) declares cupy-cuda12x[ctk] on the gpu extra and takes uv
out of the render path (sys.executable -m waterland.cli), retiring the
runtime prune trap at the source.
- #6 (b72425b) rehydrates the job index from the data volume at startup,
fixing the unbounded store growth reported from this side.
Verified after the update rather than assumed: healthy on backend cupy;
/api/jobs went 1 -> 16 against 16 directories on disk, so API and volume
agree for the first time; nothing wrongly reclaimed, correct since 16 is
under RETAIN=40 and adoption only makes them visible; a real 256^2 plate
render completes warm, so the kernel-cache volume survived the image swap.
A subsequent render took both counts to 17.
The image keeps its explicit [ctk] install and UV_NO_SYNC/UV_OFFLINE pins
even though both are now redundant. The header requirement is a property
of this slim base, not of the upstream extra, and the cost is measured
rather than assumed: uv sync satisfies it first, so the line reports
"Audited 1 package" and adds 0.3s to the build. The env pins are now
cheap defence-in-depth against any future path that re-enters uv.
Docs corrected in place: the README's upstream-finding section is now a
resolved-finding record, and the two "bounded ~500 MB" claims say which
commit made that bound hold across restarts rather than only within a
process. Comment-side changes pushed to the live compose dir; no restart
was needed for them.
waterland-dev merged PR #5 (main now 464dfc2), fixing both landmines at
source: the gpu extra declares cupy-cuda12x[ctk], and the renderer spawns
sys.executable -m waterland.cli instead of re-entering uv mid-job.
The running container deliberately stays on 8025366. Its own [ctk] install
and UV_NO_SYNC/UV_OFFLINE pins already neutralise both defects, so a rebuild
would buy reliability that is already present — and the project is in
wind-down. Both guards are kept rather than dropped: the header requirement
is a property of this slim image, not of the upstream extra, and the uv pins
are now cheap defence-in-depth against any future path that re-enters uv.
Also records waterland-dev's confirmation of the unbounded job-store growth
and the operator's green-light on their startup-rehydrate fix. That PR
merging is the rebuild trigger: one update.sh run lands the rehydrate and
464dfc2 together. Marks the inbox drained.
Captures a long infra session: fleet *.internal DNS (git-sourced, 42 names,
three resolvers including a new colo one), waterland studio containerised on
irv-ml1, Homepage cleaned up and themed with Australis Skyfall over an
Arbo-generated background, and four unmanaged stacks adopted into stacks/.
Four detail files added. Auto-archived 4 entries to archival-memory.md
(Recent decisions 2, Tried and abandoned 2); 5 held back by the open-deferred
guard rather than moved.
Also records three operator-owned open items: the colo DNS repoint, the
static-v6 convention, and the deliberately belayed AI-tab Dormant regrouping.
Session captured: ESH cut over to Cityside 2Gb symmetric fiber and was
fully provisioned on it; esh-pve-nas completed its ZFS-root migration off
the USB DOM and took its 225-package security backlog with the reboot
deferred; ESH<->colo IPsec was rebuilt as a dialup tunnel with NAT-T after
CGNAT broke the statically-pinned one; IPv6 was mapped across all three
sites.
Auto-archival fired at the soft cap: 7 entries moved to
archival-memory.md (Recent decisions 3, Tried and abandoned 4), all
verified-complete arcs, with two detail files moved and removed. The
remaining pre-Aug-05 entries were held back by the open-deferred-work
guard, so the index stays slightly over cap at 313 lines rather than
losing a live pointer.
Also records the one self-inflicted outage of the session (missing
--make-rslave on a chroot rbind) and that three long-dead things
surfaced incidentally: pvestatd down 82 days, a vzdump hung 126 days,
and a VM sitting in prelaunch for four months.
Everything but the reboot. Two rerunnable elway playbooks; the host is
still running from the ext4 root and its boot path is byte-identical to
the last 140 days, because grub-install is deliberately held back to the
cutover window.
Phase 1 (esh-pve-nas-stage-zfs-root.yaml): carve a 512 MB /boot LV out
of the 768 MB swap LV, populate it, rsync the 4.3 GB ext4 root into
nvme/ROOT/pve-1, write the copy's fstab.
Phase 2 (esh-pve-nas-stage-bootloader.yaml): ZFS initramfs, grub.cfg,
explicit pve-zfs-root and pve-ext4-rollback entries with stable ids,
grubenv pinned to the rollback so cutover's grub-reboot is a one-shot.
Three landmines the plan did not predict, all caught by verify steps
asserting effective state rather than by reading the plan:
- The /boot LV had nowhere to live. VG pve had 4 MB free and mounted
ext4 cannot shrink; freeing space from root needs a rescue boot, which
costs the one-reboot property. Space came from swap (768M -> 256M).
- The runbook's `zpool set cachefile=... nvme` would have broken the
NAS. Populating a cachefile flips the host from import-by-scan to
import-by-cache, so a one-pool cache leaves ssd and tank unimported --
and CT 103 esh-nas has twelve bind mounts spanning all three pools.
Set on all three instead, verified in the resulting cache.
- update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, which
boots to an initramfs prompt. Debian's 10_linux builds ${rpool}${bootfs}
and rpool comes from grub-probe --target=fs_label, which returns empty
because GRUB's ZFS reader cannot open a pool with encryption,
large_dnode and zstd_compress -- the same feature set that forced /boot
to stay ext4. The probe failure is swallowed by `2>/dev/null || true`.
Fixed with a /etc/default/grub.d drop-in plus explicit menu entries.
The transferable lesson: the original verify grepped for the correct
root= string appearing somewhere in grub.cfg, which passes while every
menu entry is still broken. Assert the effective value, not the presence
of a substring.
Durable capture ahead of the ESH fiber install (2026-08-18) that puts the
house behind CGNAT and breaks Site Magic on IPv4 -- IPv6 becomes the
escape hatch and the likely first consumer of fleet v6.
Topology verified rather than assumed: Site Magic between UniFi units,
IPsec IKEv2 colo<->UniFi, and WireGuard as a remote-access convention
only, host-based on ana-wg behind a FortiGate UDP VIP. The FortiGate
port-forwards and never terminates WireGuard, so FortiOS 7.2's lack of
native WG is a non-issue.
IPv6 today: NH3 WAN live, colo and ESH none. AT&T delegates exactly one
/64 at NH3 -- established by forcing the prefix ID from auto to 0 and
observing the subnet not move, since the c110/c11f pattern otherwise
reads as a /60. PD enabled on nh3-iot to measure, then reverted; all
five NH3 LANs are back to ipv6_interface_type=none.
Also fixed on ana-wg: wg0.conf, keys/*_priv, keys/*_psk and the client
configs were mode 644 with private key material in them. Now 600, with
keys/ and configs/ at 700. wg-quick@wg0 stayed active, three peers
intact.
Corrects two stale in-flight rows: the DS regeneration is retired, not
queued, and SPEC-ds-regeneration.md is deleted rather than untracked.
Current state rewritten to session-end: secrets-broker LIVE (secret CLI + 25-item
backfill + rm/warn), stonehenge-park /park service repo stood up (vor-plan+vor-ui),
dots-tts extracted to tts-stack; NEXT PROJECT = wgtunnel (WireGuard over a plane's
wifi). Two detail files added; 2 oldest T&A entries archived. Handoff written to
/tmp/infra-ops-handoff.md for the post-clear pickup.
Session (2026-08-05): 3 Booth features live+tagged (verbatim-wrap chip, .md/.txt
doc-viewer, image prev/next arrows); worldtree herald re-nudge bug -> forseti
althing-core v2.1.2; fleet-CI-resilience flip attempted end-to-end and PARKED on
an act_runner->gitea action-fetch auth blocker (infra-ops to research, deferred).
Archived the 2026-07-15/16 recent-decisions batch (8 entries) to keep the index
under the soft cap.
vh