Operator suggested checking the BGW on its 192.x management address,
which turned out to give the whole picture from unauthenticated status
pages.
The CPE is a BGW210-700 on firmware 4.28.7 at 192.168.1.254. AT&T does
hand it a /60 -- c110 through c11f. The BGW keeps c110-c117 for itself
and re-delegates up to eight individual /64s on c118-c11f, top-down. Our
UDM holds c11f, delegation number eight.
So the earlier conclusion that AT&T only grants a /64 was right about
the symptom and wrong about the cause. Seven further /64s are available
and simply never solicited, because UniFi exposes a single
wan_dhcpv6_pd_size integer with no field for how many prefixes to
request. The documented workaround is repeated -P flags to dhclient,
which the UniFi UI cannot express.
This also settles that an AT&T ticket cannot help: the rationing is CPE
firmware behaviour, not provisioning. Records the two real options --
accept one /64, or bypass the BGW entirely with wpa_supplicant EAP-TLS
on the UDM to negotiate the full /60 -- with the warning that the latter
modifies NH3's only uplink and needs a planned window.
AT&T support guessed 'I do not believe att will do that' from a DNS
provisioning desk. The guess was correct, but it needed proving rather
than accepting, so the UDM solicited DHCPv6-PD at /48, /56 and /60. All
three returned the same single /64. This is not a case of nobody having
asked -- the ask was made three ways.
Proven by temporarily enabling PD on nh3-iot, the only NH3 VLAN with
zero clients, then setting ipv6_pd_prefixid to 0, 15 and 16. All three
returned an identical c11f prefix, which only happens when exactly one
/64 is delegated; with a larger block the prefix-id moves the LAN within
it.
Records a mistake worth not repeating: I first read the gap between the
WAN address (c110) and the delegated prefix (c11f) as evidence of a /60.
It is not -- AT&T assigns those from different parts of their pool and
the spread means nothing.
NH3 was fully restored afterwards, with rollback artifacts kept. Also
captures the concrete ask for AT&T Business, phrased as something their
provisioning team can verify against their own DHCPv6 logs, and the
consequence if refused: NH3 can host exactly one v6 segment against
ESH's 256.
Operator opened a ticket with Cogent for v6 at Anaheim, which closes the
one thing tonight's investigation could not resolve from our side.
Captures the diagnosis so the ticket has evidence behind it: a single RA
in 90 seconds of sniffing wan1, from fe80::ea0a:b9ff:fe3b:2c16, proving
an IPv6-capable router sits one hop away on the circuit terminating
38.120.12.42/29 -- but SLAAC obtained no global address across multiple
RA intervals and ping6 to Cloudflare and Google both returned 100% loss.
Router present, circuit unprovisioned.
Also records that the FortiGate v6 config was fully reverted after
testing, the FortiOS gotcha that SLAAC is 'set autoconf enable' rather
than an ip6-mode, and the ask to make when it lands: a /56 or better,
since NH3 only receives a single /64 from AT&T.
Notes the consequence worth planning around -- once provisioned, the
colo becomes the only site with both a static public v4 and routable
v6, which makes it the natural v6 hub given ESH is CGNAT'd and NH3 is
prefix-constrained.
Operator's framing, and it is a better argument than the terminology
correction that preceded it. Under v4, exposing a host needed two
affirmative acts -- a DNAT and an accept rule -- so missing either left
the host dark. There is no v4 misconfiguration that exposes an internal
host by accident. NAT was load-bearing security whether or not anyone
designed it that way.
v6 removes the first control entirely. The path exists inherently, so
the firewall is the only thing left, and the failure mode inverts from
fail-closed to fail-open. Rule-ordering slips, rulesets that silently
match only one address family, new VLANs added without policy, and
re-delegated prefixes unmatching address-literal rules all become
exposure events rather than no-ops.
Records the practical consequences: key rules on interface/zone rather
than address literals, treat enabling v6 on a segment as requiring
policy to exist first, and verify default-deny from off-net rather than
by reading the ruleset -- which is lesson 3's assert-the-effective-value
discipline applied to firewall policy.
Also corrects my own claim from the previous commit that the pending
firewall pass was 'smaller' than I had implied. It is not smaller, it is
different in kind.
I wrote that enabling SLAAC would give LAN devices 'globally reachable
addresses'. Wrong word, and the wrong word in a persistent-memory entry
a future session inherits as fact.
Addressable is a property of the address. Reachable is a policy decision
the firewall makes. v6 removes NAT; it does not remove the firewall, and
treating those as the same thing is exactly how v6 gets mischaracterised
as automatic exposure.
Records the operator's position while correcting it: no 1:1 inbound
pass-through. The pending firewall-policy pass is about writing explicit
default-deny inbound rules per v6 segment, not about deciding what to
expose.
esh-iot keeps the identical eight digits -- 4DBAD107, rendering
4dba:d107 -- and only the reading changes: 4 is FOR rather than A, so it
parses 'FOR DA BAD IOT', which describes what the segment is actually
for.
esh-mgmt genuinely changes: 115D:B055 becomes 15DA:B055. The leading I
is dropped and DA is spelled in full, giving 'IS DA BOSS' with the
network as subject rather than speaker. Still eight digits. DA written
out needs no substitution since D and A are both native hex; spelling it
as a single D the way esh-iot does would have yielded seven digits and
broken the house pattern.
'FOR ALL DA BAD' -- 4=FOR, 411=ALL, D=DA, BAD=BAD. Eight digits, house
style, renders 4411:dbad.
No DMZ network exists on the ESH UDM today; this is a name claimed
against the day one is built. Pairs deliberately with esh-iot's
4DBA:D107 -- IoT is 'a da bad', the DMZ is 'for all da bad', which is
the correct relationship between the two segments.
'FOR ALL BIOS' -- 4=FOR, 411=ALL, B105=BIOS. Eight digits like the rest.
Completes the set. All six ESH networks now carry an 8-hex-digit phrase
in a consistent first-person/declarative voice:
Default 4BA5:3417 A BASE FOR IT
esh-mgmt 115D:B055 I IS DA BOSS
esh-server 4411:B105 FOR ALL BIOS
esh-userland CAFE:4411 CAFE FOR ALL
esh-iot 4DBA:D107 A DA BAD IOT
esh-cameras 1533:FACE5 I SEE FACES
Still a documentation convention rather than wire-level configuration --
UniFi has no static-v6 client assignment and the gateway address is
platform-fixed -- but these are the values to use whenever ESH LAN v6 is
switched on and hosts get hand-assigned addresses.
Same phrase, 'A BASE FOR IT', but written as a plain 8-digit string
rather than forcing the article into its own group. 4=A, BA53=BASE,
4=FOR, 17=IT renders as 4ba5:3417 -- two groups, matching every other
network in the scheme, with the words straddling the colon exactly the
way 4DBA:D107 does.
Corrects the previous commit, which claimed this needed nine digits and
a third group. It is eight, and always was.
'A BASE FOR IT' -- the article makes it a full sentence, matching the
voice of the other five. BA53 uses 3=E rather than the 5E spelling used
in the previous BA5E version.
Nine hex digits rather than eight, so unlike the others it does not fit
two groups and renders across three as a:ba53:0417.
'BASE FOR IT' -- BA5E=BASE, 4=FOR, 17=IT. The foundation segment
everything else hangs off, which is what the default network is, and it
doubles as 'base for IT'.
Note the trailing group zero-pads: it renders as ba5e:0417, not
ba5e:417.
'CAFE FOR ALL' -- CAFE, 4=FOR, 411=ALL. Eight digits like the others,
and the only one so far that splits on its own phrase boundary, so it
renders legibly as cafe:4411.
Bonus reading: 411 is US directory assistance, which is a fitting second
joke for the segment the humans actually live on.
'A DA BAD IOT' -- 4=A, D=DA, BAD=BAD, 107=IOT. Eight hex digits to match
the cameras and mgmt picks.
Worth noting it renders as 4dba:d107, so unlike the other two the phrase
does not split on its word boundaries and reads as noise unless you know
it is there -- which is arguably right for the untrusted segment.
'I IS DA BOSS' -- 1=I, 15=IS, D=DA, B055=BOSS. Eight hex digits like
the cameras pick, so it renders as 2607:73c0:402:1d??::115d:b055 with
the same two-group split and room for host numbering.
Pairs structurally with 1533:FACE5 on cameras: both eight digits, both
first-person, and the network that is actually in charge gets to say so
to the one that is merely watching.
'I SEE FACES' -- 1->I, 5->S, 3->E, 3->E then FACES. Eight hex digits
splitting cleanly across two groups, so it renders as
2607:73c0:402:1d00::1533:face5 with room left for host numbering.
Still a documentation convention rather than anything on the wire, per
the constraints recorded in the same entry, but this one is good enough
that it should survive to whenever ESH LAN v6 actually gets switched on.
Picked six hexspeak names for the ESH LANs during a wind-down moment
(FACE/B055/B105/CAFE/DEAD/BASE), then checked whether any of it could
actually land on the wire before implementing anything.
It can't, for three independent reasons: a network's only nameable slot
is its /64 prefix id, which is 2 hex digits and can't spell a 4-char
word; the gateway's own address is fixed at ::1 by the UniFi platform
with no field to override it; and UniFi has no IPv6 equivalent of
use_fixedip/fixed_ip, confirmed directly against the client schema, so
individual devices can't be pinned to a chosen v6 address either --
SLAAC devices self-assign via EUI-64 or privacy extension.
So this stays a documentation mnemonic. Recorded as such rather than
implied as something live, since I'd already started suggesting a
static-camera-assignment plan that the schema check ruled out.
The ESH<->colo tunnel is restored and the FortiGate end is permanently
address-agnostic, but the UniFi end still needs a literal ipsec_local_ip
and so drops on any ESH WAN change -- Cox reclaiming WAN1, the fiber
cutover, or a DHCP renewal.
Operator's call not to build the self-healing watcher yet, which is
right: it would be written against the 5G failover address, which is
about to be replaced, and the fiber may reshape the topology anyway.
Parked as self-healing-ipsec-local-ip-watcher-for-the-esh with the
trigger recorded, plus the follow-up to retire the old ana-to-eshudm
tunnel whose distance-10 route would otherwise reclaim traffic if Cox
returned on the old address.
Records the manual stopgap in persistent memory so the gap is cheap to
cover by hand in the meantime: read wan_ip from the UDM's health
endpoint, PUT it into esh-ana.ipsec_local_ip.
The link died when ESH lost its public IP during the fiber cutover. Two
independent causes, and the second would have defeated the obvious fix:
- phase1 ana-to-eshudm was type static, pinned to 70.181.90.232, an
address that no longer exists.
- nattraversal was disable, so ESP could not have crossed NAT even with
the peer IP corrected. pfi-ana-nh3 shares that setting and survives
only because NH3 is publicly addressed, which is why the two tunnels
diverged.
FortiOS refuses `set type dynamic` on an existing tunnel -- "Cannot
change tunnel type once configured" -- and rolled back cleanly, so the
fix could not be an edit. Rather than delete and recreate, which
cascades into the phase2, two static routes and ten policies, the
replacement was built alongside: new phase1+phase2 ana-eshudm-dyn
(type dynamic, ikev2, aes256-sha1, dh14, NAT-T on, PSK read from the ESH
UDM API so neither side needed a new key), static route id 10 at
distance 20, and two consolidated multi-zone policies 73/74. The old
tunnel is left in place, dead and harmless, as rollback.
Verified up: ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1 -- the _0
suffix is a dialup child, :4500 is NAT-T, and the address is the
carrier's, which is precisely what could never have been pinned. ESH
reaches all four colo hosts at 40-56ms, the colo reaches all three ESH
hosts, and traceroute drops from eight hops leaking into the carrier
network to three hops fully encapsulated.
Config was backed up before any write (1.17MB, 36903 lines, off-box).
Residual fragility recorded: the UDM's ipsec_local_ip demands a literal
address -- empty is rejected as api.err.InvalidPayload -- so it still
needs updating when the fiber changes ESH's WAN address. The gateway end
is now address-agnostic; the UniFi end is not.
The prior edit missed its anchor and left the over-broad version in
place. The entry now separates the two inter-site links rather than
treating them as one: Site Magic (WireGuard, NH3<->ESH) survives
arbitrary NAT and is proven to; IPsec (colo<->ESH via ana-gw) does not
and is currently down, with traffic leaking unencapsulated to the
carrier. IPv6 keeps its justification on the IPsec link specifically.
Correcting an over-generalisation from earlier today. Proving that NAT
does not break Site Magic, I wrote it up as "no addressing outcome
threatens the inter-site tunnel." That is wrong: the fleet has two
inter-site links with opposite NAT behaviour.
- NH3<->ESH is Site Magic, i.e. WireGuard. It survives arbitrary NAT,
proven live on RFC1918 double-NAT (192.168.200.111) with nh3-dev and
nh3-docker reachable at ~40ms. It dials out to NH3's public edge and
never needs inbound reachability.
- colo<->ESH is IPsec on the ana-gw FortiGate, and it is broken right
now under those same conditions. ana-docker, pfi-pve and pbs-ana all
fail from esh-pve-nas, and traceroute shows packets for 10.250.x
leaving the UDM to the 5G modem and then wandering the carrier network
before dying -- not encapsulated at all, so no SA is up and the
traffic falls through to the default route. Site-to-site IPsec pins a
peer IP and ESH no longer has a routable one.
So the IPv6 work keeps its justification, but on the IPsec link
specifically rather than on the tunnels generally. Operator caught the
over-generalisation.
Adds lesson 8 -- a result proven for one protocol does not transfer to
another -- and corrects the superseded-claims row rather than replacing
it, since the original claim was half right and the halves are the
point. Also records my own over-broad claim as its own superseded row.
Seeded by the Site Magic / CGNAT premise, which justified a body of IPv6
work and turned out to be false the first time anything actually tested
it. The mechanism was discoverable in advance: Site Magic is WireGuard
and the far side has a public endpoint, so the NAT'd side dials out and
never needs inbound reachability. NAT breaks inbound; it does not break
outbound-initiated tunnels with keepalives.
Also fills the first row of the superseded-claims table, which is what
that table exists for -- the claim is corrected with a date rather than
quietly deleted, so older references to it resolve instead of misleading.
The fleet IPv6 work was justified primarily by the expectation that ESH
fiber landing behind CGNAT would break Site Magic on IPv4, making v6 the
escape hatch. The fiber cutover provided a free natural experiment and
the premise does not hold.
Cox was unplugged, ESH failed over to the 5G WAN (already configured
failover-only, so this needed no intervention), and the resulting WAN
address is 192.168.200.111 -- RFC1918, double-NAT, no inbound path at
all, which is strictly worse than the CGNAT that was feared. Site Magic
stayed up throughout: all four ESH hosts reachable, ssh and command exec
working, 20MB pulled over the tunnel, latency 15ms -> ~46ms as expected
for cable to 5G.
The mechanism is visible on the device: magic_site_to_site_vpn holds
only `enabled` plus a WireGuard keypair, with peer orchestration in the
UniFi cloud and no WAN binding of any kind. NH3's edge is publicly
reachable, so the NAT'd side dials out and never needs reachability.
Consequence: no addressing outcome on the new fiber -- public, CGNAT or
double-NAT -- threatens the inter-site tunnel. IPv6 stays worth doing on
its own merits but stops being urgent, and stops gating anything.
Also worth recording that Site Magic cannot be pinned to a WAN. It rides
whichever uplink is active, so the only lever is failover priority --
which moves all site traffic, not just the tunnel. The existing
failover-only config on WAN2 already handles a primary-WAN outage
correctly and needed no change.
pve-manager 8.4.11 -> 8.4.20, corosync 3.1.9 -> 3.1.10-pve2, and kernel
6.8.12-42 staged on the /boot LV. dpkg clean, nothing outstanding for
apt -f install, all PVE services active, cluster quorate, no unapplied
conffiles. Reboot deliberately deferred at operator request, so the host
still runs 6.8.12-13 until a chosen window.
This validates the GRUB fix from 061c4b7 under the exact condition it
was written for. update-grub regenerated entries for the new kernel and
entry 0 -- what GRUB_DEFAULT=0 selects -- is now
/vmlinuz-6.8.12-42-pve with root=ZFS=nvme/ROOT/pve-1, supplied by the
grub.d drop-in since grub-mkconfig cannot derive the pool name itself.
The old kernel keeps correct entries as a fallback and the ext4 rollback
entry is untouched. Had the fix not landed first, saved_entry would
still be pinned to 6.8.12-13 and the host would boot the old kernel
indefinitely -- 161 security updates installed and never run.
/boot holds both kernel sets at 176M used of 488M, confirming the 512M
LV carved out of swap was sized correctly.
Adds a ZFS snapshot step to the upgrade playbook, taken automatically on
ZFS-root nodes before any package lands. That is the first real use of
the boot-environment upside the migration was meant to unlock: rollback
for this upgrade is now `zfs rollback -r
nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z && reboot` rather than
archaeology in dpkg. Also documents that the corosync bump restarts
corosync mid-upgrade, which on a 2-node cluster is a brief quorum event.
Sibling to model-quantization-playbook.md, and it exists for the same
reason that one does: hard-won lessons were dying inside per-host
runbooks where nobody finds them until after repeating the mistake.
Six entries seeded from the esh-pve-nas migration, all of which would
bite identically on any other host:
1. mount --rbind into a chroot needs --make-rslave, and losing cgroup2
impersonates failing root-disk I/O closely enough that it was
misdiagnosed as exactly that.
2. A reboot is not confirmed until the host is observed DOWN; "never
rebooted" and "rebooted fast" are indistinguishable otherwise.
3. Assert the effective value, not the presence of a substring. Grep
proves presence; only evaluation proves effect.
4. Ask the server who its clients are -- documented dependent lists rot.
Plus the corollary that an idle hard NFS mount blocks and resumes, so
quiescing means stopping consumers, not always unmounting.
5. The scoped-looking command can be the dangerous one; setting a ZFS
cachefile on one pool of three would have stopped the other two from
importing at boot.
6. Long uptime hides breakage, and a forced look is worth more than it
appears -- one migration surfaced an 82-day-dead pvestatd, a 126-day
hung vzdump, a VM in prelaunch for four months, and an undocumented
cluster, none of them caused by the work.
Carries a superseded-claims table so corrections are dated rather than
silently edited, same discipline as the quantization playbook. The ESH
runbook now links here so the general rules are reachable from the
specific story and vice versa.
171 packages, pve-manager 8.4.11 -> 8.4.20, kernel 6.8.12-16 -> 6.8.12-42,
corosync 3.1.9 -> 3.1.10-pve2. dpkg clean, no unapplied conffiles, no
failed units, cluster quorate with both nodes visible after the reboot.
Adds a reusable pve-node-upgrade playbook (upgrade only -- reboot stays a
separate deliberate step, since it has cluster and NFS consequences the
playbook cannot see). It guards on quorum and free space, snapshots
/etc/pve and friends first, uses --force-confdef/--force-confold, and
surfaces any .dpkg-dist files that policy left unapplied so they are not
silently ignored.
The reboot needed a forced guest stop, operator-authorised after the risk
was surfaced. Two obstacles, only one of them ours:
- A vzdump had been hung since 14 April -- 126 days, stalled at 0% of
256 GiB -- holding lock: backup on VM 102, which had therefore been
sitting in QEMU prelaunch that entire time. Killed by explicit PID; 102
is now cleanly stopped rather than half-alive.
- esh-vm-db would not shut down: its guest agent had died and ACPI went
unanswered. Most likely ours -- it hard-mounts /mnt/backup from CT 103,
which we deliberately left mounted through the NAS reboots.
PostgreSQL survived the hard stop. It had checkpointed five minutes
prior, so recovery replayed 56 bytes of WAL in 0.02s and came up ready;
all four databases present and queryable. That was lucky timing as much
as anything -- a hard stop mid-checkpoint on a busy database would not
read the same way.
The reboot also repaired esh-vm-db, which had silently lost sshd, mongod
and its guest agent. All three are back.
The cutover left saved_entry=pve-zfs-root, a hand-authored 40_custom
entry hardcoding /vmlinuz-6.8.12-13-pve. The pending upgrade installs
proxmox-kernel-6.8.12-42, which made that a trap with two exits: if -13
were autoremoved the default entry would point at a missing kernel and
the host would need console recovery it has no IPMI for; if -13 survived
the host would silently keep booting the old kernel, so 161 security
updates including a kernel would install and never run.
That entry was written as a one-time cutover target. It was never fit to
be the standing default across kernel upgrades, and this is remediation
of that, caught before the upgrade rather than after.
Fix is to stop hand-authoring the ZFS entry: GRUB_DEFAULT=0 boots the
first auto-generated entry, which grub-mkconfig regenerates for the
newest kernel on every install, and which /etc/default/grub.d/zfs-root.cfg
already corrects to the pool-qualified root=ZFS=nvme/ROOT/pve-1. grubenv
is cleared so nothing overrides it.
The rollback entry stays pinned, which is correct rather than an
oversight: it boots the untouched ext4 root on the DOM, whose /boot is
never regenerated because update-initramfs writes only to the /boot LV.
That kernel genuinely never changes.
Also adds a reusable safe-reboot playbook for this host, carrying the
constraints that are easy to forget: quiesce the hard-NFS clients first,
the other cluster node goes read-only while this one is down (quorum 2,
no qdevice), and a failed boot has no auto-fallback and no remote
console.
Two findings from chasing node `pve` showing dark in the UI.
esh-pve and esh-pve-nas are a 2-node cluster (esh-pve-cluster, expected
votes 2, quorum 2, no qdevice). This was undocumented, and last night's
migration rebooted one of the members without accounting for it. Nothing
broke -- quorum is intact and both nodes report the same ring id -- but
that was luck. Rebooting either node drops the survivor below quorum and
makes its /etc/pve read-only until the partner returns. It matters for
the pending confirmation reboot and the 225-package upgrade, both of
which take a node down.
The dark tile itself was NOT last night's doing. pvestatd SEGV'd on
2026-05-28 and had been dead 82 days; the journal has nothing between
that crash and the restart today. It is only the reporting daemon, so
the node stayed quorate and healthy with all services active and all
three guests running the whole time -- the UI simply had nothing telling
it the node was alive. Fourth SEGV in that unit's history, so treat a
recurrence as expected and consider a watchdog: nothing alerts on it,
and the sole symptom is cosmetic enough to go unnoticed for months.
Root is now nvme/ROOT/pve-1. The USB DOM keeps the ESP and /boot but is
out of the runtime I/O path, so a bus reset can no longer drop root from
under a running hypervisor. All five guests healthy, three pools ONLINE,
system running, ext4 pve-root intact and unmounted as the rollback with
its own kernel and initrd. zfs-import-cache is now the active import
path -- the all-three-pools cachefile fix doing its job.
The window cost an unplanned outage, and the cause was this repo's own
tooling rather than the migration.
The staging chroot ran `mount --rbind /dev` and /sys with no
--make-rslave. On systemd `/` has shared propagation, so the cutover's
`umount -R` propagated back into the live host and removed the real
/sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone systemd-logind
could not create a session: ping fine, TCP fine, SSH authentication
succeeded, resident daemons kept serving -- and every new exec hung,
including /sbin/reboot, so the reboot never ran at all.
It impersonates failing root-disk I/O almost perfectly, and I called it
as the DOM dying. That was wrong. dmesg had the answer throughout: the
DOM attached cleanly with no errors, and the last log timestamp was
12114881s -- 140 days -- meaning this was still the original boot. A
down-detector had also never reported the host down, which I read as a
fast reboot rather than as no reboot.
Fixes and guards:
- --make-rslave after every rbind, plus a guard that refuses to proceed
while any chroot bind still reports shared propagation.
- Confirm a reboot by observing the host DOWN, not by watching for it to
come back. Those two states are indistinguishable otherwise.
- Blast radius now measured from the server: `ss` inside CT 103 found
five NFS clients, not the two documented. The new one that mattered is
esh-vm-db, hard-mounted and unreachable by ssh. Left mounted on
purpose and it came through read-write.
- grub-reboot's one-shot does NOT work here: grubenv sits on an LVM LV
which GRUB reads but cannot write, so next_entry survived the boot
that consumed it. Steady state is saved_entry=pve-zfs-root with no
next_entry. There is no auto-fallback on this host and no IPMI.
Recovery needed no console: an idempotent cgroup2/devpts/shm remount
landed in the brief windows where exec succeeded. No data was lost, and
neither the DOM nor any pool was ever at risk.
Operator ruled 2026-08-18 on the 5-character PARK_API_KEY flagged during
the v1.0.0-beta.2 deploy: leave it. The henge is LAN/WG-internal and
never internet-exposed. Written down so the next audit does not re-raise
a question that has already been answered.
Operator-directed request from park-dev. Rebuilt from tag v1.0.0-beta.2
(commit 2c258f7) and redeployed; park-data volume preserved (28 items,
9 comments verified present after the recreate).
Build source is now exported per-tag to
~/deploy-src/stonehenge-park-v1.0.0-beta.2 rather than overwriting the
single mirror, so the previous tag's tree stays on the host as a
rollback. The mirror was never a git checkout, so the source comes from
`git archive <tag>` against a box that has the repo -- which also leaves
park-dev's working tree untouched.
Verification, and two things worth writing down:
- HEAD 401s on EVERY route, including /healthz and /. So park-dev's
suggested check `curl -sI .../ui/assets/favicon.svg` reports a false
failure. GET is 200 with content-type image/svg+xml; the packaging is
fine and all nine assets are in the wheel. App-wide and pre-existing,
not a beta.2 regression -- /healthz predates this release.
- /park/due-count returning 0 is not a data-loss signal; it counts what
is due now, and /park/due is empty across overdue/today/stale. Items
survived: GET /park returns all 28. README now says to check that
instead of the due counters.
Also corrected the stack README, which still told the reader to build on
nh3-docker and verify against 10.100.50.40 -- the host decommissioned for
this stack on 2026-08-13 and the one park-dev explicitly asked us not to
deploy to.
Everything but the reboot. Two rerunnable elway playbooks; the host is
still running from the ext4 root and its boot path is byte-identical to
the last 140 days, because grub-install is deliberately held back to the
cutover window.
Phase 1 (esh-pve-nas-stage-zfs-root.yaml): carve a 512 MB /boot LV out
of the 768 MB swap LV, populate it, rsync the 4.3 GB ext4 root into
nvme/ROOT/pve-1, write the copy's fstab.
Phase 2 (esh-pve-nas-stage-bootloader.yaml): ZFS initramfs, grub.cfg,
explicit pve-zfs-root and pve-ext4-rollback entries with stable ids,
grubenv pinned to the rollback so cutover's grub-reboot is a one-shot.
Three landmines the plan did not predict, all caught by verify steps
asserting effective state rather than by reading the plan:
- The /boot LV had nowhere to live. VG pve had 4 MB free and mounted
ext4 cannot shrink; freeing space from root needs a rescue boot, which
costs the one-reboot property. Space came from swap (768M -> 256M).
- The runbook's `zpool set cachefile=... nvme` would have broken the
NAS. Populating a cachefile flips the host from import-by-scan to
import-by-cache, so a one-pool cache leaves ssd and tank unimported --
and CT 103 esh-nas has twelve bind mounts spanning all three pools.
Set on all three instead, verified in the resulting cache.
- update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, which
boots to an initramfs prompt. Debian's 10_linux builds ${rpool}${bootfs}
and rpool comes from grub-probe --target=fs_label, which returns empty
because GRUB's ZFS reader cannot open a pool with encryption,
large_dnode and zstd_compress -- the same feature set that forced /boot
to stay ext4. The probe failure is swallowed by `2>/dev/null || true`.
Fixed with a /etc/default/grub.d drop-in plus explicit menu entries.
The transferable lesson: the original verify grepped for the correct
root= string appearing somewhere in grub.cfg, which passes while every
menu entry is still broken. Assert the effective value, not the presence
of a substring.
infra-ops aliased tts-1, tts-1-hd and gpt-4o-mini-tts onto ext-tts's upstream
and extended the lobe-chat-esh key allow-list 20 -> 23 models, so the manual
"set the TTS model to ext-tts, per browser" step this file described a few
hours ago is obsolete. Lobe's stock three-field payload now returns 200
audio/mpeg — verified from the host with this stack's own .env.
Adds the coupling that the fix introduces: the three new names are independent
LiteLLM DB rows carrying their own copy of the upstream URL, so a future
repoint of ext-tts must move all four or stock clients land on a dead engine
without any error on the gateway side.
Comment/doc only — no functional change, no redeploy.
Lobe's TTS had never worked. It sends model:"tts-1" and LiteLLM resolves the
model name before routing, so it 403'd against the scoped key's allow-list and
never reached :8198 -- our belief that an unknown model routes to the gateway
default was true of the gateway and false of the LiteLLM path, which is what
hid it.
Fixed at the gateway rather than the client: tts-1, tts-1-hd and gpt-4o-mini-tts
aliased to the same upstream as ext-tts, and added to the lobe-chat-esh
allow-list. Verified with Lobe's exact payload on Lobe's own key. The obsolete
'one-time human UI pass' follow-up is dropped.
Banks two durable facts: those aliases are independent DB rows that must move if
ext-tts repoints, and the infra-ops key has admin rights for /model/new and
/key/update so this class of work does not need sk-corvid.
The operator-visible symptom is that PVE cannot be updated on this box for lack
of room. Measured: 225 packages pending, 161 carrying deb12uN/Debian-Security
bumps including ssh, against esh-pve's 8.4.14 versus this host's 8.4.11 and 20
weeks of uptime.
Records the ordering explicitly -- migrate first, upgrade after. The pending set
includes proxmox-kernel-6.8.12-42-pve-signed, roughly 250 MB of kernel plus
initramfs landing in /boot which is on root with 1.3 GB free. Unpacking 225
packages including dpkg and perl into that headroom risks filling the disk
mid-transaction and wedging dpkg on a hypervisor running five guests.
Notes the apt archive-dir redirect as a partial escape hatch if patching cannot
wait, and that zfs-initramfs 2.2.8 is fully capable of root-on-ZFS so there is
no need to upgrade ZFS before migrating.
Operator's proposal, and it is strictly better than the reinstall plan.
Boot and root do not have to share a device. Keep the ESP and /boot on the DOM
as ext4 -- so GRUB never has to read ZFS, which matters because the nvme pool
has encryption, large_dnode and zstd_compress enabled and GRUB cannot read
those -- and move root to nvme/ROOT/pve-1. The initramfs imports the pool and
pivots.
What this buys over the reinstall: the nvme pool survives, so no guest
migration, no export/import of ssd and tank, no reinstall. Downtime is one
reboot rather than half a day. Rollback is a GRUB menu entry, because the ext4
root stays on the DOM untouched. And it retires the actual top risk -- with
root on NVMe, a USB bus reset mid-run no longer takes the running system down;
the DOM becomes read-mostly, written only on kernel updates.
Preconditions verified and already met: UEFI with grub-efi, zfs-initramfs
2.2.8-pve1 installed with 76 ZFS files already in the running initrd, root only
4.3 GB to copy, swap negligible against 125 GB RAM.
Two traps recorded: canmount=noauto on the root dataset or ZFS tries to mount
over the running root; and cachefile is currently none with a 0-byte
zpool.cache, so the pool imports by scan today and must be given a cachefile
before the initramfs is rebuilt.
The reinstall plan is retained as the fallback.
PVE root on esh-pve-nas is a USB Disk-on-Module: 6 GB ext4 with the host's only
ESP. A DOM is SLC/pSLC so wear is not the driver -- the problems are that it is
on the USB bus (a reset drops root under a running hypervisor), has no headroom,
and is unmirrored while 928 GB of mirrored NVMe sits 96% empty.
Runbook targets a fresh PVE install to ZFS RAID1 across both NVMes. In-place
conversion is unsupported, and adding an ESP to the existing NVMes is impossible
-- both are whole-disk ZFS members with 1.7 MiB free and proxmox-boot-tool
manages nothing today.
The headline risk is not on the host being rebuilt: CT 103 esh-nas IS the NAS
at 10.0.50.50, and both esh-docker-vm and esh-pve mount it hard. Taking this box
down stalls esh-pve's storage layer and wedges esh-docker-vm into the D-state
whose only remedy is a host reboot -- the incident shape already on record.
Quiescing those clients is step one of the window, and the README now warns
against casual reboots.
Config snapshot captured off-box to nh3-dev (0600) with /etc/pve, network and
fstab config plus zpool/zfs/disk-by-id/guest state; the newest on-disk copy
before this was June 2024.
Two claims in this stack's docs were reasoned from the wrong hop, and one of
them hid a dead feature since deploy. Re-measured from esh-docker-vm against
the live .env:
1. "An unknown `model` routes to the gateway default" — true of :8198, false of
the path Lobe takes. LiteLLM resolves the model name first, so Lobe's default
`tts-1` returns 403 (`key not allowed to access model`) and never reaches the
gateway. `ext-tts` returns 200 + audio. The endpoint inheriting
OPENAI_PROXY_URL is necessary but not sufficient: Settings -> TTS -> OpenAI
TTS model -> `ext-tts` is a required one-time step per browser, and removing
it needs a LiteLLM alias plus a key allow-list entry (both master-key, so
infra-ops).
2. "`response_format: mp3` ... set it in the UI" — not possible. Lobe's OpenAI
TTS client sends `{input, model, voice}` and nothing else (server bundle
chunks/29685.js), so format is not selectable from this stack at any level.
The deploy gets the fleet gateway's default (WAV, ~23.5 MB for a 245 s turn),
relabelled `audio/mpeg` by LiteLLM. That is tts-dev's fence, not this one's.
Comment/doc only — no functional change, so the host copy needs no redeploy.
HOMEPAGE_ALLOWED_HOSTS now carries the IP:port form; direct access to
http://10.0.50.45:5100/ returns 200 and the host-validation errors are gone
from the container log.
The .env was mode 644 holding the Plex and Jellyfin API keys; now 600. It is
root-owned, so editing it needs the infra-ops identity -- lkraven has only
password-sudo on that host.
Also records that Homepage renders client-side, so grepping the served HTML
to verify a config change is the wrong instrument (it gave a stale prerender
and then an empty page). GET /api/services is the honest check, and config
changes need a recreate rather than a restart.
Homepage on esh-docker-vm:5100 was the one stack whose config lived only on
the host, edited in place. Its version history was six hand-rolled
services.yaml.bak-* files. Now canonical here and deployed with
deploy-stack.sh like everything else; the .bak files are gone.
Corrections from the audit:
- ANA-Firewall described a 'Fortigate 81F'. It is a FortiGate-80F running
FortiOS 7.2.10, verified live against the device.
- NH3-Ansible pointed at 10.100.50.42 as an 'Ansible control node'. That host
is nh3-extdev, the manager/external-dev successor after nh3-ansible was
retired. Renamed and re-described.
- Dropped the UltraSeedbox layout group: nothing provides it, so it only ever
rendered empty.
Adds .env.example and a README documenting the two-path service model (docker
label discovery across five engines vs manual entries), the labels-only-apply-
on-recreate rule, and the foot-guns found: HOMEPAGE_ALLOWED_HOSTS matches
host AND port so a bare IP does not cover IP:port; :2375 is plaintext and
unauthenticated on all five engines; ping: cards can only be judged from the
dashboard host.
Verified after deploy via /api/services: 105 cards across 19 groups, both
corrections live, ana-docker discovery intact.
Records the one signal the synthetic gates cannot provide -- multi-turn
degeneration is stochastic and invisible to probes, and four synthetic tests
once validated three non-fixes on this exact seat.
Not yet the 60k-token bar the prior seat cleared, so the rollback weights
stay in place.
Cold-Fusion-GAIN V1.1 examined and not adopted -- it is a capability
finetune of stock Qwen3.8 and every bench row is labelled [non heretic], so
adopting it would reintroduce base refusals the current seat does not have.
Records why it reads as uncensored at a glance: DavidAU's back catalog is
almost entirely Uncensored-Heretic builds, so the naming pattern implies it.
The heretic stage for this one is still in progress from base, and that is
the release worth watching.
Also banks what makes it interesting when the heretic build lands -- real
third-party benchmark gains over stock, claimed MTP acceptance well above
ours, thinking tokens cut to a fraction -- and the two caveats: the MTP
numbers are GGUF/llama.cpp not vLLM, and a trained MTP head means the free
CPU-hash gate would not apply.
Also drops the now-stale 'primary until the DavidAU Qwen3.8 lands' clause
from the superseded seat entry.
MuXodious/Qwen3.8-27B-absolute-heresy (Heretic v1.4.0 + SOMPOA, trial T377,
pin c2374593) quantized through our mixed NVFP4+FP8 recipe and promoted after
passing the full gate on the probe port.
Gate vs incumbent -- MTP acceptance 47.2% (48.2%), decode 103.5 tok/s (96.4),
prefill 6618/5403 at 6.7k/27k (6334/5085), TTFT 27k 5.00s (5.31s), perplexity
6.910 (7.059, 2.1% better), surface 6/6, abliteration compliance 4/4. On our
battery-instruct arm -- the framing that actually elicits refusals -- 0/55 with
zero EMPTY, so no catatonia at the hard edge.
Speed deltas are image-confounded: the probe ran the seat's pinned nightly
while the incumbent's stored numbers came from an earlier image. Read as not
worse. Acceptance, perplexity, surface and refusal are apples-to-apples.
All 7 LiteLLM aliases verified end-to-end. GPU0 at 91.3/97.9 GB with meromero
healthy -- more headroom than the previous build. Incumbent weights untouched
and .env.bak-heresy-20260817 in place for rollback.
Candidate is a 2-day-old RC1 with ~348 downloads; watch real multi-turn use.
post_quant assumed the source ships a standalone model-mtp.safetensors, which
is how JonathanColetti's grafted head is packaged. MuXodious/absolute-heresy is
an unmodified full checkpoint, so its mtp.* lives in model-00012-of-00012 --
the copy silently did nothing while the index was still rewritten to point at
model-mtp.safetensors, leaving 15 unresolvable tensors. Tensor counts looked
correct; the checkpoint would have failed at load.
The existing FAILED-CHECKS assertion caught it, which is the design working.
Now extracts from the numbered shard when the standalone file is absent.
Verified on the heresy build: 1968 tensors, all resolvable, 15 mtp, 333 visual,
no missing shards, no orphans.
Three defects, each of which produced a false read on the candidate:
1. Hardcoded vllm/vllm-openai:latest. The Qwen3.8-27B gen seat is pinned to a
nightly carrying the #51113 qwen3_5_mtp x GDN fix; probing on :latest
reproduces the multi-turn corruption we already diagnosed and reads as a
candidate failure. Now PROBE_IMAGE, defaulting to :latest for older seats.
2. --speculative-config JSON died twice on quoting. The inner double quotes are
stripped by the outer double-quoted ssh string, and then bash BRACE EXPANSION
splits {"a":1,"b":2} on the comma. Needs escaped quotes AND remote-side
single quotes; both traps documented inline.
3. No --tool-call-parser/--enable-auto-tool-choice/--reasoning-parser. Without
them surface_test reported tool calling as a 400 and measured a thinking split
of reasoning=0ch -- both probe-config artifacts, not model defects. Re-running
with the seat's flags took the candidate from 5/6 to 6/6.
Also adds PROBE_MAXLEN; the hardcoded 32768 rejected prefill_bench's ~27k prompt.
numpy has no bfloat16, so .numpy().tobytes() raised
'TypeError: Got unsupported ScalarType BFloat16' on real checkpoints.
Flatten then view(torch.uint8) before hashing.
Result on the heresy candidate: VERDICT IDENTICAL -- all 15 mtp.* tensors
byte-identical to the incumbent's verbatim base graft, the head already
measured at 47.7% acceptance in production. The ~56 GB bf16 acceptance gate
is redundant, so no second seat comes down.
Operator ruled the probe port for validation; runbook updated to match.
The bf16 MTP acceptance gate is ~56 GB resident, which on a full 97.9 GB card
means downing meromero-charrp as well as gen -- freeing gen's 0.43 (~42 GB)
alone is not enough. Two seats down to answer one question.
compare_mtp_head.py answers the common case for free. The
Qwen3_5ForConditionalGeneration wrapper never loads the MTP head, so PEFT
merges, Heretic runs, and llm-compressor passes all leave mtp.* as it came
from the base. It hashes a candidate's 15 mtp.* tensors against the
incumbent's grafted-verbatim head -- the one measured at 47.7% acceptance in
production through this exact pipeline. Identical means the acceptance
question is already answered; different means the head was edited and the
real gate is warranted; missing means it was dropped.
CPU only, reads just the shard holding mtp.*. The runbook states the residual
risk plainly: an identical head proves the head is intact, not that the
abliterated body still drafts well with it -- which the Stage-3 acceptance
measurement on the 22 GB quantized build catches anyway.
Candidate MuXodious/Qwen3.8-27B-absolute-heresy (Heretic v1.4.0 + SOMPOA,
trial T377), pinned c2374593. Beats the incumbent on both axes: refusals
2/101 vs 12/100, first-token KL 0.0759 vs 0.1191. Structurally a clean full
checkpoint (1199 tensors, 15 mtp.*, 333 visual.*, lm_head), so the existing
mixed NVFP4+FP8 recipe applies with no graft-and-reconstruct.
Runbook carries the bf16 MTP-acceptance gate before any quant spend, the
llm-compressor ignore-pruning foot-gun, the three measurement traps
(cache-busting, unseeded prefill nonce, PPL with spec off), the .env 0600
sudo trap, the GPU0 co-tenant starvation risk, and rollback.
Flags that the candidate is a 2-day-old RC1 whose own card carries a broken
GGUF benchmark block (RC1 and RC2 report identical at-chance scores across
three benchmarks), so its numbers are claims rather than measurements.
worldtree-dev closed#401 on our demo verification. Records the two-layer
state (their e41b139 compose pin verified on demo, covered-not-verified on
personal/pinned; our daemon floor staged), the measured fact that
default-ulimits is not SIGHUP-reloadable on Docker 29.4.3, the explicit
no-dockerd-restart decision, live-restore parked as a separate call, and
the one ping we still owe once worldtree-personal recreates.
Worldtree #401: a slow fd accrual in worldtree-personal hit the 1024 soft
nofile ceiling and converted into a hard deadlock. Operator authorized the
raise 2026-08-17 (relayed via worldtree-dev); sizing 65536 agreed.
Applied at the daemon layer rather than compose because /opt/worldtree-*/
compose.yaml on corviduo-dev is written by the team CI deploy identity -- a
host-side compose edit reverts on the next deploy and would leave a false
'raised' record. Daemon config is infra-ops-owned and covers all 13
containers on the box. worldtree-dev shipped a redundant compose-level pin
(e41b139) as the belt to this braces.
daemon.json is written and valid, but the floor is STAGED, NOT ACTIVE:
default-ulimits is not in dockerd's SIGHUP-reloadable set. Measured on
29.4.3 -- the post-reload 'Reloaded configuration' log enumerates the live
config without default-ulimits, and a fresh container still reports
ulimit -n 1024. Activation needs a full dockerd restart, which bounces every
container; not taken, since #401 is not urgent at fd ~100 and the compose
pin already covers worldtree. The playbook documents this and its verify
step 3 fails by design until a restart happens.
vh