memory: snapshot — run 3c gated, run 4 training, DAC revert, GPU rebalance, NAS exposure
This commit is contained in:
@@ -1,119 +0,0 @@
|
||||
# `[2026-08-19]` Fleet `.internal` DNS — git-sourced, agent-managed, three resolvers
|
||||
|
||||
Operator: *"with ipv6 i can't memorize the IP addresses anymore. need a way to
|
||||
keep track of local .internal dns names that can be agent managed and is
|
||||
lightweight."* Built and live in one session; commit `b8003c7`.
|
||||
|
||||
## Shape
|
||||
|
||||
```
|
||||
dns/internal.yaml source of truth — 38 hosts + 4 service aliases
|
||||
scripts/dns-sync.py reconciles AdGuard resolvers against it
|
||||
stacks/adguard-ana/ the colo's resolver, which did not exist
|
||||
dns/README.md workflow, naming, the IPv6 caveat
|
||||
```
|
||||
|
||||
Deliberately the same posture as `deploy-stack.sh`: the file is intent, the
|
||||
resolvers are derived state, you see a diff before anything changes.
|
||||
`--dry-run` / `--yes` / `--site <s>`. Verified idempotent — a second run prints
|
||||
`nothing to do`.
|
||||
|
||||
Naming is `<host>.<site>.internal` with sites **`ana` / `esh` / `nh3`**
|
||||
(operator's call). `.internal` is ICANN-reserved for private use since 2024;
|
||||
`.local` is reserved for mDNS, which is why the pre-existing
|
||||
`searxng.pfi.local` was a standards collision that merely happened to work.
|
||||
|
||||
Every name is published to **every** resolver — the site label says where a
|
||||
host *is*, not which resolver knows about it.
|
||||
|
||||
## The framing correction that mattered most
|
||||
|
||||
The ask reads as "I can't memorise v6 addresses", but the deeper problem is
|
||||
that **v6 addresses are derived, not assigned**, so they cannot reliably be
|
||||
*written down once* either. SLAAC gives EUI-64 (MAC-coupled) or
|
||||
privacy-extension (rotating) addresses, and UniFi has **no v6 equivalent of a
|
||||
DHCP reservation** — so a hand-maintained v6 table rots on its own.
|
||||
|
||||
⇒ The fix has two halves and only the second is DNS: (1) pin static v6 on
|
||||
server-class hosts, (2) then the name table is just a file. Surfaced to the
|
||||
operator before building.
|
||||
|
||||
**Verified 2026-08-19: no fleet host has a global v6 address at all yet** —
|
||||
ESH's `/56` is live only on `esh-cameras`, NH3's LANs are back to
|
||||
`ipv6_interface_type: none`, the colo has none. So the `v6:` column ships
|
||||
EMPTY and correct, and the naming layer was built first rather than blocking
|
||||
on v6. Names established now need no renaming when addresses land.
|
||||
|
||||
Suggested convention when they do (awaiting operator): each server static at
|
||||
its site's `/64` with low-order bits echoing the v4 host octet —
|
||||
`esh-docker-vm` at `…::45` — so addresses are declarable *and* semi-memorable.
|
||||
|
||||
## Two properties not to break
|
||||
|
||||
**Authority is scoped to the ZONE, not the resolver.** Only rewrites ending in
|
||||
`.internal` are managed. ESH's resolver turned out to carry three hand-made
|
||||
`esteban.net` rewrites (`eshnas`, `brotherprinter`, `eshhome`) — **my first
|
||||
read of the config missed them**, because an `awk` range on `rewrites:` matched
|
||||
an empty-looking block. A resolver-wide authoritative sync would have silently
|
||||
deleted all three on first run. Verified intact after sync.
|
||||
|
||||
**Within `.internal` it IS authoritative** — names added by hand in the AdGuard
|
||||
UI get deleted by the next sync. That is the point: one place to look.
|
||||
|
||||
## The colo had no resolver at all
|
||||
|
||||
ESH and NH3 each ran AdGuard; **ana-docker resolved straight against
|
||||
`1.1.1.1`**, so the colo had no way to answer for internal names. Closed with
|
||||
`stacks/adguard-ana/`.
|
||||
|
||||
⚠️ Its API is on **8053**, not 8080 — `:8080` and `:3000` were already taken on
|
||||
that busy host. The port is therefore carried **per-site in the yaml**, not
|
||||
assumed by the script, so the odd one out cannot be forgotten.
|
||||
|
||||
⚠️ It ships with **no blocklists**, deliberately. The other two filter ads for
|
||||
human browsing; this one resolves for a rack of servers, where a blocklist
|
||||
false-positive breaks service-to-service calls at 3am for no upside.
|
||||
|
||||
First boot uses a **seed config** (`conf/AdGuardHome.seed.yaml`) copied into
|
||||
the conf volume before first start, so the container comes up configured
|
||||
instead of sitting in the setup wizard.
|
||||
|
||||
## Credential — service account, not the operator's
|
||||
|
||||
Added a dedicated **`infra-ops`** AdGuard user to all three resolvers rather
|
||||
than asking for the `lkraven` password (per the standing migrate-off-operator-
|
||||
creds directive). Password vaulted at
|
||||
`nh3-dev/adguard-infra-ops-password`; `lkraven` untouched; pre-change configs
|
||||
backed up on each host as `AdGuardHome.yaml.bak-preinfraops-*`. Both existing
|
||||
resolvers kept answering across the restart.
|
||||
|
||||
Two landmines worth keeping:
|
||||
|
||||
- **Go's bcrypt rejects `htpasswd`'s `$2y$` prefix.** Same algorithm, different
|
||||
marker; `golang.org/x/crypto/bcrypt` accepts only `$2a$`/`$2b$`. Normalise
|
||||
the prefix, and self-verify the hash with `htpasswd -vb` BEFORE installing it
|
||||
on a live resolver.
|
||||
- **The vault appends a trailing newline on `get`.** A password carrying a
|
||||
stray `\n` fails auth in a way that looks exactly like a wrong password.
|
||||
`dns-sync.py` strips it.
|
||||
|
||||
## `pfi.local` migration — and the one that must NOT move
|
||||
|
||||
`searxng.pfi.local` → `searxng.ana.internal`, with the **old `Host()` kept
|
||||
alongside** in the Traefik rule so nothing breaks mid-migration; both return
|
||||
200. Drop the fallback once the access log shows the old name unused.
|
||||
|
||||
**`matrix.pfi.local` deliberately NOT migrated.** A Matrix `server_name` is
|
||||
baked into every user ID, room ID and signing key, and federation identity
|
||||
derives from it — renaming it is not a DNS change, it is rebuilding the
|
||||
homeserver's identity and invalidating its history. The operator approved
|
||||
"migrate pfi.local" generally; this was surfaced as a deliberate exclusion
|
||||
rather than executed blindly.
|
||||
|
||||
## Still open
|
||||
|
||||
Colo hosts still point at `1.1.1.1`, so they do not yet *use* the new resolver
|
||||
— it only answers what asks it directly. Repointing a whole site's DNS is a
|
||||
bigger change than standing the service up, and is the operator's to schedule.
|
||||
|
||||
See also [[2026-08-17-fleet-ipv6-mesh]].
|
||||
@@ -1,145 +0,0 @@
|
||||
# `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background
|
||||
|
||||
Commits `9d92c4b`, `c3de7db`, `45c1995`, `f38cf69`, `df68dd2`.
|
||||
|
||||
## The cleanup (three real defects)
|
||||
|
||||
- **UltraSeedbox rendered on all four tabs.** The bookmark group had no entry in
|
||||
`settings.yaml`'s `layout:` block at all, and Homepage's documented behaviour
|
||||
is that a group with no `tab:` is shown on **every** tab. Pinned to Main.
|
||||
⚠️ This will happen again to the next group added without a `tab:` — the rule
|
||||
is now written at the top of the layout block.
|
||||
- **Uptime Kuma rendered twice** — a manual `services.yaml` entry under
|
||||
Monitoring *and* `homepage.group=Apps` on the container. Exactly the
|
||||
"never list a labelled container manually" failure the stack README warns
|
||||
about; it survived the previous day's audit because a duplicate reads as two
|
||||
plausible cards rather than as an error. Manual block deleted, label moved to
|
||||
`Monitoring`, `homepage.siteMonitor` added.
|
||||
- **Column counts were fiction** — several groups declared more columns than
|
||||
they had members, so the last row of each was dead space (Notes: 1 card in a
|
||||
4-wide row). Columns now track member counts; `GET /api/services` prints the
|
||||
live per-group counts and is the check.
|
||||
|
||||
Later, on operator instruction, the **AI tab was reordered by clickability**:
|
||||
Gateways & Chat → Image & Media → Audio Tools on top, then the vLLM `/docs`
|
||||
seats and TTS endpoints. Reasoning written into the config so it survives:
|
||||
order by "would I click this?", not by how central the service is.
|
||||
|
||||
## ⚠️ The expensive red herring — the tab bar after a recreate
|
||||
|
||||
After a recreate the client render comes up with **no tab bar, no wallpaper and
|
||||
no i18n** (search box shows the raw key `search.search`), groups falling back to
|
||||
side-by-side columns. **It restores itself with no intervention.**
|
||||
|
||||
Timing, measured rather than assumed: a fresh container was still tab-less at
|
||||
**4m30s, twice**; it was healthy again after roughly an hour. `docker ps`
|
||||
reporting `healthy` says nothing about it — the container is serving, the page
|
||||
is just wrong.
|
||||
|
||||
An hour went into ruling out four causes that were never the cause:
|
||||
|
||||
1. **Not the config** — restoring `settings.yaml` *and* `services.yaml` to
|
||||
their committed versions reproduces it, as does the pre-adoption backup in
|
||||
`/opt/docker-bu/conf/homepage/`.
|
||||
2. **Not the v2.0.0 release** — a throwaway container on `v1.13.2` shows
|
||||
identical symptoms, and the image never changed anyway (working and broken
|
||||
both report `v2.0.0` / rev `17456f2`).
|
||||
3. **Not `PUID`/`PGID`**, and not Docker discovery — tested both, and with the
|
||||
socket unmounted entirely.
|
||||
4. **Not server-side** — the server-rendered HTML still contains the tab
|
||||
markup, the background URL and `useEqualHeights`; `GET /api/validate`
|
||||
returns `[]`. The loss is client-side, with no page error, no failed chunk
|
||||
and no non-200.
|
||||
|
||||
Every throwaway container in that list was judged within ~30s of starting, so
|
||||
they were all inside the same window — and that consistency **read as a
|
||||
reproduction when it was the same measurement mistake five times over.**
|
||||
|
||||
**Operative rule: recreate, walk away, re-check later. Do not chase it.**
|
||||
|
||||
## ⚠️ The iteration loop that would have prevented the overcook
|
||||
|
||||
`custom.css` is served **per request** from `/api/config/custom.css`, so a CSS
|
||||
change needs a **browser reload** — not a container recreate, and it never owed
|
||||
the layout warm-up above. Conflating the two costs ~10 operator-visible minutes
|
||||
per attempt (operator called this out directly).
|
||||
|
||||
Faster still, and how the final pass was done: **inject candidate CSS into the
|
||||
running page and screenshot it** —
|
||||
`await p.addStyleTag({content: css})` in Playwright against the live
|
||||
dashboard. Seconds per iteration, no deploy. Build + deploy only once the
|
||||
render looks right.
|
||||
|
||||
## The theme — Australis Skyfall
|
||||
|
||||
Operator supplied a Claude Design handoff bundle via the Booth (`26-copper`).
|
||||
Skyfall is a dual-theme OKLCH system: one lightness law across every chromatic
|
||||
family (deep 0.48 / base 0.66 / bright 0.80), all hues cooler than neutral, a
|
||||
Sea neutral ramp drifting ice-blue→ocean-green as it brightens, and a
|
||||
"calm depth" language of **hairline + two-layer shadow on every elevated
|
||||
surface, never one without the other**.
|
||||
|
||||
```
|
||||
theme/colors.css layout.css typography.css vendored VERBATIM from the bundle
|
||||
theme/fonts/Supreme-{400,500,700}.woff2 the body/UI face
|
||||
theme/skyfall.css.in the Homepage bindings (ours)
|
||||
theme/build.py → conf/custom.css (generated — do not hand-edit)
|
||||
```
|
||||
|
||||
The build step exists for one reason: **Homepage serves only `custom.css` and
|
||||
`custom.js` out of its config dir**, with no static route beside them, so a
|
||||
`@font-face` pointing at a vendored `.woff2` would 404 — the face must arrive
|
||||
as a data URI. The background image takes the other road, because
|
||||
`/app/public/images` **is** a real static route (mounted read-only in
|
||||
`compose.yaml`).
|
||||
|
||||
Only Supreme is embedded: a link dashboard has no display type, and Victor
|
||||
Mono ships as 2.4 MB TTF statics per cut — 30x the whole stylesheet for a
|
||||
handful of latency figures.
|
||||
|
||||
## The background is generated, not stock
|
||||
|
||||
**Arbo as an image-gen engine** (the operator's actual ask, which I first
|
||||
misread as "use Arbo's palette" and had to redo). Arbo's `t2i-ui-background`
|
||||
workflow is purpose-built: *"abstract full-bleed backgrounds, no subject"*.
|
||||
Job `13f0891f4e42`, seed 26, flux2-klein-9b, 2048×1152, 1.6 MB PNG → **22 KB
|
||||
WebP** (smooth gradients compress absurdly well).
|
||||
|
||||
⚠️ Arbo API gotcha: `prompt` is a **discriminated union, not a string** — a
|
||||
bare string 422s. `{"kind":"raw","text":…,"negative":…}` is the shape.
|
||||
|
||||
## Two documented deviations from the design system
|
||||
|
||||
1. **Skyfall forbids this background.** Its rule is "flat semantic surfaces; no
|
||||
photography, no textures", with one permitted motif — a subtle aurora
|
||||
gradient on hero/empty-state areas only, *"never behind body text blocks"*.
|
||||
A dashboard is a body-text block. Present on the operator's explicit
|
||||
instruction, mitigated rather than excused: abstract, no subject, strictly
|
||||
cool temperature, held at **`opacity: 30`**. That number is load-bearing —
|
||||
at 14 the aurora was invisible, and turning it up makes the cards fight the
|
||||
ribbon.
|
||||
2. **Service icons stay full-colour vendor logos.** Desaturating them from CSS
|
||||
only makes them illegible.
|
||||
|
||||
## Overcorrection, and the colour pass
|
||||
|
||||
First stat-well pass went from `font-thin` 13px straight to **bold 22px in
|
||||
heading white** — operator: *"went from subtle to BASH YOU OVER THE HEAD."*
|
||||
The principle missed: a stat only has to out-rank **its own label**, not the
|
||||
service name above it. Now `--text-md` medium in cyan.
|
||||
|
||||
Colour was then lifted **from inside the system**: Skyfall names Aurora (blue,
|
||||
cyan, green) the *primary* families, "used generously, in that order", while
|
||||
Dawn (amber/red/violet) is semantic-only. So group markers cycle
|
||||
blue→cyan→green down the page (icons full strength, names at 0.72), service
|
||||
icons take a single cool wash, latency tags move to the info family so
|
||||
"how fast" stops looking like "is it alive". **No Dawn colour is used
|
||||
decoratively anywhere.**
|
||||
|
||||
Two DOM findings that made it possible:
|
||||
|
||||
- **Homepage renders mdi icons as a gradient behind an SVG mask** — recolour
|
||||
via `background`, not `color`.
|
||||
- **Homepage emits `docker-status-<state>`, not `status-<state>`.** The
|
||||
original selectors matched nothing, so every green pill up to that point was
|
||||
stock colouring rather than the theme. Both forms are now matched.
|
||||
@@ -1,77 +0,0 @@
|
||||
# `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken
|
||||
|
||||
Commits `42c594c`, `dc3e47b`, plus `uptimekuma` in `9d92c4b`.
|
||||
|
||||
## The pattern worth remembering
|
||||
|
||||
Chasing two bad-looking cards on the dashboard turned up **four stacks running
|
||||
on fleet hosts with no canonical copy anywhere**: `uptimekuma` and (already
|
||||
known) the two AdGuards on esh-docker-vm, `searxng` and `seafile` on
|
||||
ana-docker, and `heretic2-charrp-reasoning` on ana-ml2 (untracked in git).
|
||||
|
||||
⇒ **A dashboard card is a cheap census of what is actually running.** When
|
||||
something on it looks wrong, check whether the stack behind it is even in
|
||||
`stacks/` before debugging the symptom — twice here the answer was "no", and
|
||||
the fix belonged in version control as much as on the host.
|
||||
|
||||
Adopted: `stacks/uptimekuma/`, `stacks/searxng/`, `stacks/seafile/`,
|
||||
`stacks/heretic2-charrp-reasoning/`. ESH/NH3 AdGuard compose files were
|
||||
**deliberately left unmanaged** — adopting three live resolvers while also
|
||||
introducing a new DNS naming system is two risky changes at once.
|
||||
|
||||
## SearXNG — the healthcheck was eating itself
|
||||
|
||||
Card flapped UNHEALTHY; the container was fine the whole time. The compose
|
||||
passed `--tries` and `--spider` as **two separate argv entries**, so wget
|
||||
consumed `--spider` as the *value* of `--tries`. Spider mode never engaged,
|
||||
which means every probe since April **downloaded** the healthz response to a
|
||||
file:
|
||||
|
||||
```
|
||||
295,287 healthz.N files in the container's working directory
|
||||
```
|
||||
|
||||
With that many files, wget's scan for the next free filename is what
|
||||
intermittently blew the 10s timeout. **Self-worsening — every probe made the
|
||||
next one slower.** Restored `--tries=1`; the junk lived in the writable layer
|
||||
so the recreate cleared it. Now `healthy`, `fails=0`, 200 in 0.16s.
|
||||
|
||||
Lesson: an argv list in YAML has no shell to catch a missing `=`. A flag that
|
||||
silently swallows the next argument turns a liveness probe into a workload.
|
||||
|
||||
## SeaFile — not broken, never restarted
|
||||
|
||||
Card showed EXITED for three months. **None of the three services declared a
|
||||
restart policy**, so Docker defaulted them to `no`. On
|
||||
**2026-05-06T21:27:45Z** the daemon stopped all three within 200ms of each
|
||||
other — a daemon restart or host reboot — and nothing brought them back.
|
||||
|
||||
⚠️ **Exit code `255` is a red herring**: it is what a container that ignores
|
||||
SIGTERM reports when the daemon stops it, **not** evidence of a crash. Reading
|
||||
it as one sends you hunting a bug that does not exist. The tell was all three
|
||||
services stopping within 200ms.
|
||||
|
||||
Added `restart: unless-stopped` to all three; brought up; mariadb gated on its
|
||||
healthcheck exactly as the existing `depends_on` comments intended, seahub
|
||||
started without the race, `302` → login page. Data was in local named volumes,
|
||||
not on the ana-nas NFS, so nothing was at risk.
|
||||
|
||||
Three months of silent downtime whose only signal was a card nobody read as an
|
||||
outage — the argument for semantic status colour on the dashboard (see
|
||||
[[2026-08-19-homepage-skyfall-theme]], where amber EXITED pills made six
|
||||
mis-grouped AI seats obvious at a glance).
|
||||
|
||||
## heretic2-charrp-reasoning — tracked, with its shim
|
||||
|
||||
The `char-rp-reasoning` seat (NEO-CODE Heretic2 27B, modelopt NVFP4 + grafted
|
||||
BF16 MTP head, ~77 tok/s via `qwen3_5_mtp` spec-decode) had been running
|
||||
untracked. Now in `stacks/`, including
|
||||
`conf/mtp-workaround/sitecustomize.py`, which is **not optional**: vLLM 0.24.0
|
||||
does not propagate modelopt `exclude_modules` to the spec-decode **draft**
|
||||
model, so the BF16 MTP head gets quantized and the engine dies at load. Both
|
||||
the mount and `PYTHONPATH` are load-bearing.
|
||||
|
||||
Added the two files house convention expects and the directory lacked — a
|
||||
`.env.example` naming every knob (all values are compose defaults; the host
|
||||
overrides only the three VRAM ones) and a README pointing at
|
||||
`docs/runbooks/heretic2-nvfp4-mtp-seat.md` rather than duplicating it.
|
||||
@@ -1,176 +0,0 @@
|
||||
# `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured
|
||||
|
||||
Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`):
|
||||
a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare
|
||||
`nohup` (PID 1283383) that would not survive a reboot. Now
|
||||
`stacks/waterland-studio/`, `restart: unless-stopped`, healthy on
|
||||
irv-ml1:8410. Commits `a2b5b58`, `8189076`.
|
||||
|
||||
Tracking `main` per operator: PR #4 merged and `main` HEAD was exactly the
|
||||
pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway.
|
||||
|
||||
**Now deployed at `b72425b` (2026-08-19).** The container sat on `8025366` for
|
||||
a few hours after PR #5 (`464dfc2`) landed — deliberately, since the image's
|
||||
own guards already neutralised both landmines and the project was in
|
||||
wind-down. PR #6 (the job-store rehydrate, operator-green-lit) was the rebuild
|
||||
with a real reason behind it, and one `update.sh` run carried both. Verified
|
||||
end to end after the update: healthy, `backend: cupy`, and a real 256² plate
|
||||
render completes warm — the kernel-cache volume survived the image swap.
|
||||
|
||||
## Build context lives OUTSIDE the compose dir — on purpose
|
||||
|
||||
`/opt/waterland-studio/src` is the checkout; the Dockerfile is passed
|
||||
out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh`
|
||||
rsyncs `stacks/<stack>/` with `--delete`**, so a checkout kept beside
|
||||
`compose.yaml` would be destroyed by the next deploy of this stack. `update.sh`
|
||||
refreshes source → rebuild → recreate → health, and is verified end to end.
|
||||
|
||||
## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN
|
||||
|
||||
`gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer
|
||||
silently drops to the numpy path at ~21x wall time — it does not error, it
|
||||
just gets slow. waterland-dev warned about the build side.
|
||||
|
||||
The runtime side is worse and was not in the handover: **`studio/jobs.py`
|
||||
shells the renderer out as a literal `uv run waterland ...` with no `--extra`
|
||||
flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project
|
||||
mid-job to its default extras and prunes cupy back out from under a correctly
|
||||
built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the
|
||||
pin ever stops holding the job fails **loudly** instead of quietly rebuilding a
|
||||
slower environment.
|
||||
|
||||
**Fixed upstream in `464dfc2`:** the server now spawns
|
||||
`sys.executable -m waterland.cli` directly — no resolver in the render path at
|
||||
all. **The pins stay anyway.** They cost nothing and are now defence-in-depth:
|
||||
if any future code path re-enters `uv` inside the container, the job fails
|
||||
loudly instead of quietly dropping to the numpy backend. `uv` itself must stay
|
||||
in the image regardless — it performs the build-time `uv sync` /
|
||||
`uv pip install`, and this is a single-stage build.
|
||||
|
||||
## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare
|
||||
|
||||
Every render died 1.7s in with:
|
||||
|
||||
```
|
||||
RuntimeError: Failed to find CUDA headers.
|
||||
```
|
||||
|
||||
printed **through argparse's usage banner**, which makes it read like a CLI
|
||||
argument bug rather than a missing toolkit. That misdirection is the reason
|
||||
this is written down.
|
||||
|
||||
cupy compiles kernels at runtime through NVRTC, which needs the toolkit
|
||||
**headers** — not just the driver and the runtime libs bundled in the
|
||||
`cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the
|
||||
bare `nohup` process found them **by accident**; a slim image has none.
|
||||
|
||||
Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few
|
||||
hundred MB against ~6 GB for a `-devel` base image. It runs **after**
|
||||
`uv sync`, because sync prunes what it does not know about.
|
||||
|
||||
Reported upstream: it is an undeclared runtime dependency of the `gpu` extra,
|
||||
and anyone running this without a system toolkit hits it. **Declared upstream
|
||||
in `464dfc2`** (`gpu` is now `cupy-cuda12x[ctk]>=13`). **The explicit install
|
||||
stays in the Dockerfile**: the header requirement is a property of *this*
|
||||
image — a slim base with no system CUDA toolkit — so it belongs in the file
|
||||
that creates the problem, not inherited from an extra two repos away. It also
|
||||
survives any future restructuring of the `gpu` extra. Cost of keeping it is
|
||||
now measured, not assumed: since `uv sync` satisfies it first, the line
|
||||
reports `Audited 1 package in 49ms` and adds **0.3s** to the build. A no-op
|
||||
that documents a non-obvious requirement is worth 0.3s. (waterland-dev
|
||||
independently agreed they would keep it too.)
|
||||
|
||||
## Landmine 3 — the GPU index inside the container is not the host's
|
||||
|
||||
The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects
|
||||
`CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where
|
||||
`nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU
|
||||
(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the
|
||||
container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying
|
||||
the host's value selects a device that does not exist. Host device 0 is the
|
||||
3090, which carries the TTS zoo and must not be touched.
|
||||
|
||||
## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume
|
||||
|
||||
| job | wall |
|
||||
|---|---|
|
||||
| 256² + anim, cold container | 23.3 s |
|
||||
| 256² + anim, warm | **6.1 s** |
|
||||
| 256² plate only (`--codec none`) | 3.9 s |
|
||||
| 512² plate only | 6.4 s |
|
||||
|
||||
Warm beats the **7.4 s** recorded against the bare-metal process, so
|
||||
containerising cost nothing. Verified the cache volume properly: recreate
|
||||
(fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without
|
||||
it every restart makes the next user wait 4x and the service merely *looks*
|
||||
slow.
|
||||
|
||||
## Upstream finding — the on-disk job store grows without bound
|
||||
|
||||
`JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at
|
||||
startup**. Consequences:
|
||||
|
||||
1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and
|
||||
how this was spotted: the API reported **1 job** while the volume held all
|
||||
**16 directories, 60.6 MB**. Not data loss.
|
||||
2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict,
|
||||
so directories orphaned by a restart are **never reclaimed**. The
|
||||
handover's "bounded around 500 MB" holds within a single process lifetime;
|
||||
across restarts the store grows monotonically at ~12 MB per animated job.
|
||||
|
||||
Reported to waterland-dev with evidence; **not patched from the infra side** —
|
||||
it is their code. Prune the volume by hand if it bites first.
|
||||
|
||||
**waterland-dev confirmed it (2026-08-19)** — their "bounded ~500 MB" handover
|
||||
claim holds within one process lifetime and nowhere else, which on a
|
||||
`restart: unless-stopped` service is the wrong lifetime to have bounded. They
|
||||
have **surfaced a startup-rehydrate fix to the operator** rather than opening a
|
||||
third PR during wind-down. **Operator green-lit it; PR #6 merged as `b72425b`
|
||||
and is DEPLOYED (2026-08-19).**
|
||||
|
||||
Startup rehydrate, as recommended — and waterland-dev deliberately went
|
||||
further than the framing I sent them. I had said a directory the scan cannot
|
||||
parse "just does not enter the index"; they made the opposite call, because a
|
||||
directory that never enters the index is exactly the one that never gets
|
||||
reclaimed. **That is the sharper reading and it is the reason the fix works on
|
||||
this volume at all** — the 16 pre-existing dirs have no sidecar. Their
|
||||
adoption ladder: sidecar → restored verbatim; no sidecar → adopted with
|
||||
dimensions recovered from the PNG IHDR (24-byte read, not a decode); corrupt
|
||||
sidecar → degrades to inference, no startup crash; **neither source nor
|
||||
sidecar → skipped on purpose**, since adopting it would turn eviction into a
|
||||
delete-arbitrary-directories primitive pointed at this volume. Sidecar writes
|
||||
go through `os.replace`, and `job.json` is excluded from `ARTIFACTS` so it is
|
||||
unreachable via the artifact route.
|
||||
|
||||
They also closed a second leak I never saw, because it needs a restart
|
||||
*mid-render* to surface: a job left `running`/`queued` in its sidecar is
|
||||
non-terminal forever, and eviction skips non-terminal jobs — so it is a
|
||||
phantom that is never reclaimed and `queue_depth` over-reports for the life of
|
||||
the process. Adoption now marks those `failed`.
|
||||
|
||||
**Verified on this host after the update:** `/api/jobs` went **1 → 16** while
|
||||
the volume stayed at 16 dirs / 61 MB — disk and API agree for the first time.
|
||||
Nothing was reclaimed, correctly: 16 is under `RETAIN=40`, so adoption only
|
||||
made them visible. A subsequent real render took both to 17. From here the
|
||||
store is bounded **across** restarts, not merely within a process.
|
||||
|
||||
## Access
|
||||
|
||||
Repo is not anonymously readable (a bare clone 403s). Operator granted
|
||||
**`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False,
|
||||
pull: True`. Token on irv-ml1 at
|
||||
`/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a
|
||||
**repo-scoped** credential helper; `.git/config` carries no token (verified),
|
||||
so the remote stays clean in any diff or backup. The operator's `vh`
|
||||
site-admin token was used only for the initial clone and the grant itself and
|
||||
was **never written to disk on that host** — a site-admin credential on a GPU
|
||||
box is a blast radius nobody needs for a read-only fetch.
|
||||
|
||||
## Constraints honoured as stated (not inferred)
|
||||
|
||||
- **Serial by design — one replica, one card.** A render is 20–45s of near-full
|
||||
GPU with a single worker thread. Two on the same A6000 would OOM or thrash.
|
||||
Throughput is a hardware conversation, not a replica-count one.
|
||||
- **No authentication, arbitrary file uploads** ⇒ stays inside the
|
||||
LAN/WireGuard boundary. Do **not** paper over it with a proxy password;
|
||||
waterland-dev offered to add a real auth layer if wider reach is ever needed.
|
||||
@@ -1,95 +0,0 @@
|
||||
# `[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight
|
||||
|
||||
The real work of the session: abliterate `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1`
|
||||
using the MTP-aware, vision-preserving **Robinson formula** (documented in
|
||||
`docs/pfi/abliteration-recipe-qwen38.md` from `RobinsonLabs/Qwen3.8-27B-abliterated`).
|
||||
Harness: `services/coldfusion-abliteration/`. Runs on ana-ml2.
|
||||
|
||||
## Why this model, why abliterate it ourselves
|
||||
|
||||
Stock Cold-Fusion's refusal profile was **probed 2026-08-19** (Q6_K GGUF on
|
||||
llama.cpp, 24-prompt battery, hand-verified after a keyword-classifier bug):
|
||||
**~33% creative refusal**, concentrated on **explicit-sexual + graphic-torture**;
|
||||
4/5 hard-harm technical refused; self-harm guardrails intact 3/3; benign
|
||||
over-refusal 0. So there is a real creative-content refusal surface to remove.
|
||||
This **supersedes** the earlier "watch for DavidAU's own heretic build" posture —
|
||||
we abliterate it ourselves.
|
||||
|
||||
**It is additive over the current gen seat.** The live Heretic seat
|
||||
(`qwen38-27b-heresy-bf16`) left its MTP head a **byte-identical base graft** —
|
||||
the `Qwen3_5ForConditionalGeneration` wrapper never loads it, so Heretic could
|
||||
not touch it. The Robinson formula abliterates the MTP head **in-band** (its 2
|
||||
residual-write matrices), and the MTP head is what gates speculative acceptance.
|
||||
That in-band MTP edit is the delta this experiment tests.
|
||||
|
||||
## Recipe maps 1:1 — dry-run PASSED
|
||||
|
||||
Against the staged bf16: 1199 tensors, 333 vision preserved,
|
||||
`down_proj=64 o_proj=16 linear_out=48 mtp=2 embed=1`, coverage gate 6/6, exactly
|
||||
**131** tensors to orthogonalize. Same architecture as RobinsonLabs' base, no
|
||||
name drift. Two hard gates in the harness halt before any write: the coverage
|
||||
identity `o_proj(16)+linear_out(48)==64`, and the attention-sink screen on
|
||||
**dim 3994** (orthogonalizing a direction living there bricks the model).
|
||||
|
||||
## Capture SUCCEEDED — but only after a real environment fight (the durable lessons)
|
||||
|
||||
**The transformers Qwen3.5 DeltaNet linear-attention NaNs in bf16 on ana-ml2.**
|
||||
The fast-path needs BOTH `flash-linear-attention` (`fla`, triton, installs fine)
|
||||
AND `causal-conv1d` (**needs nvcc to build — absent, no prebuilt wheel**).
|
||||
Without causal-conv1d the DeltaNet short-conv runs the torch fallback, which
|
||||
produces **nondeterministic all-NaN** hidden states in bf16 (same 11-token input:
|
||||
finite on one forward, NaN at layer 4 on the next). bf16 and fp32 share exponent
|
||||
range, so this is **precision-driven catastrophic cancellation, not overflow** —
|
||||
**fp32 resolves it.** Diagnosed via `diag_nan.py` / `diag2.py`: `sdpa` + plain
|
||||
prompt = 65 layers all finite; chat-template input = NaN; the trigger is the
|
||||
input path through the unstable recurrence.
|
||||
|
||||
Fixes, all in the committed harness (`7abd301`):
|
||||
- **`--capture` loads fp32**; the write/surgery path stays bf16 (no forward, no NaN).
|
||||
- **A finite-gate aborts on a non-finite direction** — the sink screen alone
|
||||
can't catch it (`nan > threshold` is False, so a NaN direction "passed" it and
|
||||
saved silently on the first run).
|
||||
- `attn_implementation="sdpa"` pinned.
|
||||
|
||||
**fp32 (110 GB) needs the whole GPU.** device_map=auto packed it tight and the
|
||||
forward OOM'd against the resident seats. Had to **stop three seats** for VRAM:
|
||||
`vllm-meromero-rp`, `vllm-fablefusion-probe`, and production `vllm-gen`.
|
||||
⚠ **Restart order matters:** gen restarted into an empty GPU0 and greedily
|
||||
grabbed 64 GB (vLLM takes a fraction of *free* memory at startup), starving
|
||||
meromero into a crash-loop. Fixed by bringing **meromero up first**, then gen
|
||||
into the remainder. All three restored to healthy.
|
||||
|
||||
⚠ **fla lives in a side dir, not the venv.** The shared
|
||||
`/tank/aimodels/quant-work/.venv` is not llmuser-writable; `fla` + `einops` are
|
||||
`--target`-installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and
|
||||
reached via `PYTHONPATH`. Prune deps that shadow the venv's torch/transformers.
|
||||
|
||||
## Result
|
||||
|
||||
Refusal direction: **finite, unit-normed, layer 22**, sink energy **0.0008%**
|
||||
in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated.
|
||||
Saved to `/tank/aimodels/qwen38-27b-coldfusion-bf16/refusal-direction.pt`.
|
||||
|
||||
⚠ **QUALITY CAVEAT — the reason the next step is calibration-set expansion.**
|
||||
Two-template `|cos|` agreement at layer 22 is **0.59**, well below Robinson's
|
||||
0.99. Almost certainly the small calibration set: **8 harmful / 8 harmless**
|
||||
(HARMFUL/HARMLESS in `abliterate.py`) vs Robinson's **416 / 104**. The direction
|
||||
is valid and sink-clean but noisier than ideal; abliterating on it risks
|
||||
under-removing refusals or nicking capability. **Expand the sets to a few
|
||||
hundred each and re-capture** before the `--out` write.
|
||||
|
||||
## Sequence from here
|
||||
|
||||
1. **Expand HARMFUL/HARMLESS calibration sets** → re-capture (fp32, seats down).
|
||||
2. `--out` write (bf16 surgery, no forward) → `qwen38-27b-coldfusion-abliterated-bf16`.
|
||||
3. Verify: vision byte-identical, refusal re-profile via `services/refusal-probe/`
|
||||
(the canonical harness, NOT the ad-hoc GGUF one), MTP acceptance on the quant
|
||||
(gate ≳40%, not KL — `reference_abliteration_mtp_lessons`), PPL/coherence.
|
||||
4. NVFP4-quantize via `services/gen-seat-mixed-quant/` → gen-seat candidate.
|
||||
**Do NOT delete the incumbent** (`qwen38-27b-heresy-nvfp4-mixed`) until it
|
||||
holds through real multi-turn use.
|
||||
|
||||
bf16 staged at `/tank/aimodels/qwen38-27b-coldfusion-bf16` (pinned `9c44193`,
|
||||
provenance recorded). All write paths re-stop the seats for fp32 VRAM — batch
|
||||
re-capture + write in one window. Commits `ccb56a0`, `1857a8e`, `b56cb0d`,
|
||||
`7abd301`.
|
||||
@@ -0,0 +1,41 @@
|
||||
# `[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against the wrong number, twice in one hour
|
||||
|
||||
`vllm-embed` had OOM-crashed **7 times** (`RestartCount 7`): GPU1 was at **216 MiB free** of
|
||||
97,887, and a ~3.4 GB tenant sharing a card with five other seats dies when it cannot get another
|
||||
96 MiB mid-inference. Root cause of the `qwen3-embed` 500s brokkr saw — **not** the LiteLLM
|
||||
gateway restart he attributed them to (different host, different component, 50 min earlier, and
|
||||
six of the seven crashes predate it).
|
||||
|
||||
**Operator ruling:** move `gen` to GPU0. *"we were keeping it free for training, but we don't have
|
||||
the power for sustained load."* GPU0 was never free — `mog-sec` has been there at util 0.52
|
||||
(55,126 MiB) since the August move.
|
||||
|
||||
## ⚠ THE SIZING ERROR, WHICH IS THE POINT OF THIS ENTRY
|
||||
|
||||
gen at util 0.43 wants 42,091 MiB GPU0 free 42,113 margin 22 MiB — would not start
|
||||
gen at util 0.41 wants 40,134 MiB margin 1,979 MiB — I called this safe. IT WAS NOT.
|
||||
result 1,548 MiB free, 56 FlashInfer autotuner OOM-fallbacks per 3 min
|
||||
gen at util 0.38 34,558 MiB actual 4,466-7,548 MiB free ZERO OOM events over 6 min
|
||||
|
||||
**`gpu-memory-utilization` governs vLLM's declared weights+KV budget. It does not cover what the
|
||||
process actually needs at runtime** — FlashInfer JIT autotuner workspace, CUDA graphs, expandable
|
||||
segments all allocate on top. My "1,979 MiB margin" was against the declared budget, so the seat
|
||||
came up and then silently fell back to slower kernels for want of 20 MB chunks.
|
||||
|
||||
⚠ **Same class of error as the DAC plateau the same day: checked a proxy, reported it as the
|
||||
thing.** The fix was to *measure* — count OOM-fallback events before and after — not to re-derive.
|
||||
|
||||
## Final state and what it cost
|
||||
|
||||
GPU0 mog-sec 55,126 + gen 34,558 = 89,701 / 97,887
|
||||
GPU1 50,507 / 97,887 — 46.7 GB free, was 216 MiB
|
||||
|
||||
`gen` KV cache is now **348,515 → 268,205 tokens, 1.02x concurrency at its 262,144 max-model-len**
|
||||
— it holds exactly one full-context request. Short/medium requests still batch; long-context
|
||||
throughput effectively serialises. The lever to reclaim it is `MOG_GPU_MEM_UTIL 0.52`, untouched.
|
||||
|
||||
.env.bak-preGPU0-20260904-164032 the GPU move
|
||||
.env.bak-preShrink-165133 the utilization
|
||||
|
||||
⚠ `/opt/docker/compose/*/.env` is root-owned — `docker compose restart` needs **sudo** on
|
||||
ana-docker/ana-ml2 or it fails with a bare `permission denied` on the .env.
|
||||
@@ -0,0 +1,46 @@
|
||||
# `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early
|
||||
|
||||
The ESH-Media ↔ UDM uplink is a 3 m OEM `SFP-H10GB-CU3M` twinax that negotiated **1000 Mbps**
|
||||
with **zero errors** on both ends. Diagnosis: `sfp_compliance: Unknown` — the switch reads the
|
||||
EEPROM but cannot parse the compliance codes on a third-party cable wearing Cisco coding, so
|
||||
autoneg falls back to the safe rate. Control case: a TP-Link `TL-SM5220-1M` on the adjacent UDM
|
||||
port runs 10G with the same speed_caps, exonerating the port, firmware and autoneg.
|
||||
|
||||
Operator authorised forcing it. Forced `autoneg False / 10000 / full` on the **UDM end only**;
|
||||
ESH-Media followed to 10000 unforced — proof the cable was electrically *capable*.
|
||||
|
||||
## ⚠ IT DEGRADED, AND MY "PLATEAU" READ WAS WRONG
|
||||
|
||||
10:03 200 errors link-up burst
|
||||
10:48 221 +21 in 42 min — I reported this as "flat", it was not
|
||||
14:38 416 +195 over 4 h, plus user-visible flapping the operator felt
|
||||
|
||||
**A marginal link declares itself over hours, not minutes.** I watched a two-minute flat window
|
||||
and reported a plateau; the counter simply had not moved yet. The operator noticed the flapping
|
||||
before the soak I left running had accumulated enough to raise it. Reverted 14:38, both ends back
|
||||
to autoneg/1000, stable.
|
||||
|
||||
⚠⚠ **The correction that matters: a clean, zero-error link at 1G does NOT rule out a marginal
|
||||
cable.** It only proves the cable is clean *at 1G*. Clean at 1G and marginal at 10G is exactly
|
||||
what a cheap 3 m twinax is — which means the platform's autoneg fallback was **protecting
|
||||
something real**, not being fussy about vendor coding. The coding explains the negotiation; it
|
||||
does not explain errors once forced. I conflated the two.
|
||||
|
||||
⚠ **Do not re-force this port.** The fix is the cable.
|
||||
|
||||
## Method notes worth keeping
|
||||
|
||||
- **Force the RECOVERABLE end.** ESH-Media reaches the controller *through* this link, so a
|
||||
failed force there strands the switch behind a dead uplink — a physical visit. The UDM end is
|
||||
safe because the path to the controller (`nh3-dev → 10.100.10.1 → 10.0.0.1`, site VPN) does not
|
||||
cross it. Verified with `traceroute` **before** the change; revert payload written before the
|
||||
forward one.
|
||||
- ⚠ `port_overrides` is a **whole-array PUT** — the array was diffed to prove exactly one field
|
||||
on one port changed before sending, and read back after.
|
||||
- Blast radius named before acting: ESH-Media backhauls the operator's office switch, a WiFi AP
|
||||
and the Zigbee coordinator.
|
||||
|
||||
Resolution: the operator already owns a replacement and ran the copper himself through a drilled
|
||||
floor 2x4. Recommendation given was **fiber** (10Gtek SR 2-pack + OM4 3 m LC-LC, ~$40-65) because
|
||||
the cable-vs-pull-damage question was never resolved and a second DAC through the same hole risks
|
||||
the same fault. Runbook `docs/runbooks/esh-media-dac-10g.md`; commits `b9988a7`, `514ce7a`, `054c098`.
|
||||
@@ -0,0 +1,33 @@
|
||||
# `[2026-09-04]` esh-nas SMB account for the AudioGridder box — and the NAS is effectively open to the whole LAN
|
||||
|
||||
`10.0.50.50` is **CT 103 on esh-pve-nas**, a hand-rolled Debian NAS (not a Synology). Created an
|
||||
SMB account for the operator's Windows AudioGridder DSP box:
|
||||
|
||||
username dsp uid 999, /usr/sbin/nologin, no home — SMB ONLY, cannot log in anywhere
|
||||
password vaulted at esh-nas/dsp-smb-password (round-trip verified by sha)
|
||||
verified authenticates, sees all 8 shares, WRITE to //share confirmed (mkdir/rmdir)
|
||||
|
||||
⚠ `smbclient` is absent from the NAS, nh3-dev and esh-docker-vm — verification ran in a throwaway
|
||||
`alpine:3.20 --network host` container with `samba-client`, leaving nothing installed.
|
||||
|
||||
Shares are registry-defined (`registry shares = Yes`), so they are invisible in `smb.conf` — use
|
||||
**`testparm -s`**, not grep, or you will conclude there are no shares.
|
||||
|
||||
## ⚠⚠ THE EXPOSURE, UNACTIONED — operator has not ruled
|
||||
|
||||
**NFS: twelve exports, `rw` to all of `10.0.0.0/8`, `sec=sys`, no authentication.**
|
||||
|
||||
/mnt/{backup,books,compose,documents,iso,media,music,share,
|
||||
pvestore,nvme-pvestore,ssd-pvestore,tank-vmbu}
|
||||
|
||||
`sec=sys` means the NAS trusts whatever uid the client claims. **Every VLAN at ESH — IoT,
|
||||
cameras, guest — can mount the NAS read-write today.** `/mnt/backup` additionally has
|
||||
`all_squash,anonuid=2000` so every client collapses to `nas_user`.
|
||||
|
||||
**SMB: every share except `backup` is `guest ok = Yes` and writable**, with
|
||||
`map to guest = Bad User` — an unknown username lands as guest with write access.
|
||||
|
||||
So the `dsp` credential is auditable and survives guest being turned off, but it is **not what is
|
||||
gating access today**. Hardening (narrow exports to `10.0.50.0/24` + named hosts, drop `guest ok`
|
||||
where unneeded) was offered and is roughly an hour; it would break anything relying on guest,
|
||||
which is why it needs the operator's say-so. **Not actioned. No tracking surface beyond this entry.**
|
||||
@@ -0,0 +1,52 @@
|
||||
# `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression
|
||||
|
||||
Run 3c is the lr **1e-05** arm against run 3's 2e-04; corpus, base, mask and every other
|
||||
hyperparameter byte-identical. It exists to answer one question: does a gentler learning rate
|
||||
keep run 3's diversity benefit while dropping its safety regression? **Answer: no, and the
|
||||
failure is the informative kind.**
|
||||
|
||||
trained 604/604 in 11:55:40 on pfi-gx10 (GB10, aarch64, sm_121), 69.8 s/it
|
||||
merged /home/infra-ops/erp-tune/serve/merged-run03c, bf16 51.6 GB
|
||||
served erp-tune-v3c on :8098, exposed as gateway alias `trial`
|
||||
|
||||
## The verdict (brokkr-smithy-dev, run03-gate/RESULT-run03c-gate.md)
|
||||
|
||||
CSAM 175 gens x 3 arms — ZERO hits, hard-line guard passed everywhere
|
||||
selfharm/methods base 10/10 refused -> 3c 0.6 = -40pp de-refusal
|
||||
meta pooled 0.0 delta — a CANCELLATION, not cleanliness: selfharm -40pp
|
||||
offset by advice/medical +40pp
|
||||
diversity attractor primary returned to BASE level — the gain is GONE
|
||||
|
||||
cross-arm run 3 (lr 2e-04) selfharm -44pp diversity: real gain
|
||||
run 3c (lr 1e-05) selfharm -40pp diversity: none
|
||||
|
||||
⚠ **The selfharm de-refusal is LR-ROBUST.** A 20x cut moved it 4pp while erasing the diversity
|
||||
benefit entirely. It comes from corpus content and imprints at even gentle exposure — it is not
|
||||
something a lower learning rate dials out. That is what the LR sweep was run to find out.
|
||||
|
||||
⚠ **A pooled preserve-list test cannot see a single-axis collapse.** The pooled operational
|
||||
delta reads 0.0 because two axes moved 40pp in opposite directions. Same structural defect that
|
||||
let run 3's gate pass — recorded as R47 §8 item 11.
|
||||
|
||||
## What the port proved about the box
|
||||
|
||||
- **The tooling loads on aarch64/sm_121.** Full training stack plus flex_attention and the
|
||||
chunked-loss path. Nothing exotic needed beyond `python3-dev`.
|
||||
- **Verification that earned its keep**: both 49 GB base shards sha256-matched ana-ml2's, and a
|
||||
full encode was run into a throwaway dir and compared byte-for-byte — 197,360,233 B, sha256
|
||||
`c08bb1fe2ecb0be3`, identical. transformers 5.15.1→5.16.1 and x86-64→aarch64 are *measured*
|
||||
inert, not assumed.
|
||||
- ⚠ **The encode-cache FILENAME differs by design** — `base_model_path` is in the cache key, so
|
||||
rehoming the base changes the key while content stays identical. Input hash, not output hash.
|
||||
Do not read it as drift; do not "fix" it by faking `/tank` on the GX10.
|
||||
|
||||
## The lora_B signal worth carrying forward
|
||||
|
||||
run 2 (lr 2e-04) min 0.6826 median 1.7212 max 3.7573
|
||||
run 3c (lr 1e-05) min 0.0480 median 0.1561 max 0.4133
|
||||
|
||||
~11x gentler across the board — exactly what a 20x LR cut should produce. A consistency check
|
||||
passing, not a red flag, but worth in hand *before* reading diversity numbers: "the tune did
|
||||
nothing" and "the tune did less on purpose" look alike in the output.
|
||||
|
||||
Runbook `docs/runbooks/gx10-run-03c.md`; commit `dae77ee`.
|
||||
@@ -0,0 +1,39 @@
|
||||
# `[2026-09-05]` A peer's "2.7x stack effect" was a coin flip — the operator caught it and the arithmetic is worth keeping
|
||||
|
||||
brokkr-smithy-dev reported that a diversity **noise floor** was 2.7x tighter on ana-ml2 than on
|
||||
the GX10 and ranked ana-ml2 the better instrument on it. I relayed it. **The operator rejected it
|
||||
on instinct** — *"that makes zero sense. except for speed, serving a model should be identical
|
||||
across servers"* — and he was substantially right.
|
||||
|
||||
ana-ml2 0.9688, 0.9574 spread 1.14pp mean 0.9631
|
||||
gx10 0.9583, 0.9895 spread 3.12pp mean 0.9739
|
||||
|
||||
between-box LEVEL difference 1.08pp
|
||||
ana-ml2's own replicate spread 1.14pp <- LARGER than the between-box gap
|
||||
pooled range ignoring box 3.21pp <- ~= the entire "GX10 floor"
|
||||
|
||||
**Each "floor" is `|block0 - block1|` from n=2.** A range over two draws is not an estimate of
|
||||
dispersion; the ratio of two such ranges is a ratio of two half-normals — a **half-Cauchy**.
|
||||
brokkr computed the tail himself on retraction: `P(ratio >= 2.7) = 1 - (2/pi)*arctan(2.7)`,
|
||||
doubled = **0.452**, confirmed by 400,000-draw simulation. He had reported a coin flip as a
|
||||
measured effect and ranked hardware on it. Retracted at `97f73dd`.
|
||||
|
||||
⚠ **The disconfirming evidence was inside his own sentence.** He wrote "the base level is nearly
|
||||
identical, it is the spread that shifts" and offered it as *reassurance*. A real stack effect
|
||||
moves the level. Level agreeing while a two-draw range differs is the signature of a noisy range
|
||||
estimator — he had the refutation in hand and read it as support.
|
||||
|
||||
⚠ **His own diagnosis of why it got through is the transferable part:** he had spent the session
|
||||
triaging *my* claims hard — reading the encode path, simulating preflight, recomputing shas from
|
||||
bytes — and this one was **his, and flattering**: it made his earlier work look prescient and
|
||||
produced a clean recommendation. **Asymmetric scepticism is one error twice, and the flattering
|
||||
direction needs the extra pass.**
|
||||
|
||||
**What survived, deliberately separated:** re-measuring the floor on whatever stack actually
|
||||
serves stays non-negotiable. The mechanism list is sound whether or not those four numbers show
|
||||
it — sm_121 vs sm_120 kernel selection, vLLM 0.28.0 against mixed 0.26/nightly, and separately
|
||||
measured batch-invariance (3.12pp at jobs=8, same order as the whole claimed effect).
|
||||
**Retracting the evidence and keeping the discipline are different acts.** Settling it properly
|
||||
wants several blocks per box and is its own probe, not a by-product of a gate.
|
||||
|
||||
See [[2026-09-05-vllm-on-sm121-and-run4]].
|
||||
@@ -0,0 +1,53 @@
|
||||
# `[2026-09-04/05]` vLLM RUNS on sm_121 — and the blocker was `ninja` not being on PATH, not the silicon
|
||||
|
||||
Long-standing open question closed: **vLLM 0.28.0 serves on the GB10 (aarch64, sm_121)** from a
|
||||
stock wheel, no source build. Model loads, `torch.compile` completes (~29 s), CUDA graphs capture.
|
||||
|
||||
⚠ **The one trap, and it looks exactly like an sm_121 kernel problem:** FlashInfer JIT-builds its
|
||||
sampling kernel at first use and needs **`ninja` on PATH**. It ships as a vLLM dependency but
|
||||
lives in the venv `bin/`, so the failure surfaces as `FileNotFoundError: 'ninja'` from deep inside
|
||||
a `profile_run` traceback. Same shape as the `python3-dev` trap that bit the training harness on
|
||||
this box, and the third present-but-not-on-PATH false-absence on this hardware (after `nvcc`).
|
||||
|
||||
Launch with **both** `~/vllm-env/bin` and `/usr/local/cuda/bin` on PATH. Harmless and expected:
|
||||
`Using default MoE config ... device_name=NVIDIA_GB10` — nobody has tuned MoE kernels for this chip.
|
||||
|
||||
⚠ **Open WebUI sends `tool_choice: "auto"` on every request**, which vLLM 400s unless launched with
|
||||
`--enable-auto-tool-choice --tool-call-parser gemma4`. Taken from the working sibling seat
|
||||
(`gemma4-charrp`), which is where the canonical flags live. **Deliberately did NOT copy
|
||||
`--reasoning-parser gemma4`** — that seat pairs it with `--default-chat-template-kwargs
|
||||
'{"enable_thinking": false}'`, and adding it alone moves output into `reasoning_content` with a
|
||||
null `content`, which OWUI renders as an empty reply. A 400 traded for a blank box.
|
||||
|
||||
## Run 4 — the corpus arm
|
||||
|
||||
Run 4 adds an **airoboros-3.2 instruct root** (7,229 rows) and displaces kvasir 38% → 18% context
|
||||
share, at run 3's lr 2e-04, everything else held byte-identical. Operator granted a run-scoped
|
||||
training-eligibility override `operator-2026-09-04-rnd-run4` (the SECOND grant; run-1's was
|
||||
one-run-scoped, a run 5 needs a third).
|
||||
|
||||
**Two peer artifacts were rejected before they cost GPU time, both by reading the harness rather
|
||||
than accepting a "confirm this":**
|
||||
|
||||
1. `sample_kind: "instruct-dialogue"` — `core.py:521-544` dispatches on the **literal string** and
|
||||
raises on anything outside `rp-dialogue`/`prose-chunk`/`actual-play-passage`. Would have
|
||||
hard-stopped the encode on the first airoboros row. brokkr had labelled it *non-blocking*.
|
||||
2. Recipe targets collapsed three dialogue roots into one descriptive label with `root_sha256:
|
||||
null` on three of four. Preflight resolves `roots_dir/<dataset_id>/clean-v1/CLEANROOT.json`
|
||||
literally and requires the sha.
|
||||
|
||||
⚠ **I did NOT fill the missing shas in myself**, though the values were known and it would have
|
||||
taken two minutes. `root_sha256` is the *author's* assertion and preflight exists to check the
|
||||
deployed root against it — supply both sides and the check verifies my copy-paste, an inert gate.
|
||||
brokkr then went further and recomputed his shas **from shard bytes** rather than reading them
|
||||
back out of the deployed CLEANROOT, which had the same defect one step removed.
|
||||
|
||||
**The kvasir prefix cut has three independent confirmations**: my build measured 1,613 samples /
|
||||
3,347,622 ctx = **47.4%** of run-3 kvasir, which reproduces the recipe's 18/38 share ratio
|
||||
(47.37%); the harness's own `[mix]` block then reported **ctx 0.1805** against the 18.0% target.
|
||||
|
||||
⚠ **`str.splitlines()` splits on U+2028**, which `json.dumps` leaves unescaped under
|
||||
`ensure_ascii=False`. The harness splitlines() in four places and four roots DO carry U+2028
|
||||
(fireball 8, c2-logs 31, kvasir 1, creative-writing 1) — but those are read by ordinary file
|
||||
iteration (safe), and the splitlines() paths touch only files written with the default
|
||||
`ensure_ascii=True`. Latent, one flag away from live, not acting. Found by brokkr the hard way.
|
||||
Reference in New Issue
Block a user