memory: snapshot — run 3c gated, run 4 training, DAC revert, GPU rebalance, NAS exposure
This commit is contained in:
@@ -3410,3 +3410,634 @@ strictly better than softdog; treat a real firing as unconfirmed until tested.
|
||||
- **`rest/user` `last_seen` is not maintained** (it read ~203 days for hosts
|
||||
that are demonstrably online). `stat/sta` is the live view.
|
||||
_Archived 2026-09-03._
|
||||
|
||||
- `[2026-08-21]` **speaches ASR live on irv-ml1 for Eyra — and `no_speech_prob` alone is a weak hallucination gate.** Silence and room tone both hallucinated "Thank you." under 0.11; `avg_logprob` separates ~6× better. Consumers should gate on a composite. (`aa5863c`, `c7e2187`)
|
||||
|
||||
_Archived 2026-09-05._
|
||||
|
||||
# `[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight
|
||||
|
||||
The real work of the session: abliterate `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1`
|
||||
using the MTP-aware, vision-preserving **Robinson formula** (documented in
|
||||
`docs/pfi/abliteration-recipe-qwen38.md` from `RobinsonLabs/Qwen3.8-27B-abliterated`).
|
||||
Harness: `services/coldfusion-abliteration/`. Runs on ana-ml2.
|
||||
|
||||
## Why this model, why abliterate it ourselves
|
||||
|
||||
Stock Cold-Fusion's refusal profile was **probed 2026-08-19** (Q6_K GGUF on
|
||||
llama.cpp, 24-prompt battery, hand-verified after a keyword-classifier bug):
|
||||
**~33% creative refusal**, concentrated on **explicit-sexual + graphic-torture**;
|
||||
4/5 hard-harm technical refused; self-harm guardrails intact 3/3; benign
|
||||
over-refusal 0. So there is a real creative-content refusal surface to remove.
|
||||
This **supersedes** the earlier "watch for DavidAU's own heretic build" posture —
|
||||
we abliterate it ourselves.
|
||||
|
||||
**It is additive over the current gen seat.** The live Heretic seat
|
||||
(`qwen38-27b-heresy-bf16`) left its MTP head a **byte-identical base graft** —
|
||||
the `Qwen3_5ForConditionalGeneration` wrapper never loads it, so Heretic could
|
||||
not touch it. The Robinson formula abliterates the MTP head **in-band** (its 2
|
||||
residual-write matrices), and the MTP head is what gates speculative acceptance.
|
||||
That in-band MTP edit is the delta this experiment tests.
|
||||
|
||||
## Recipe maps 1:1 — dry-run PASSED
|
||||
|
||||
Against the staged bf16: 1199 tensors, 333 vision preserved,
|
||||
`down_proj=64 o_proj=16 linear_out=48 mtp=2 embed=1`, coverage gate 6/6, exactly
|
||||
**131** tensors to orthogonalize. Same architecture as RobinsonLabs' base, no
|
||||
name drift. Two hard gates in the harness halt before any write: the coverage
|
||||
identity `o_proj(16)+linear_out(48)==64`, and the attention-sink screen on
|
||||
**dim 3994** (orthogonalizing a direction living there bricks the model).
|
||||
|
||||
## Capture SUCCEEDED — but only after a real environment fight (the durable lessons)
|
||||
|
||||
**The transformers Qwen3.5 DeltaNet linear-attention NaNs in bf16 on ana-ml2.**
|
||||
The fast-path needs BOTH `flash-linear-attention` (`fla`, triton, installs fine)
|
||||
AND `causal-conv1d` (**needs nvcc to build — absent, no prebuilt wheel**).
|
||||
Without causal-conv1d the DeltaNet short-conv runs the torch fallback, which
|
||||
produces **nondeterministic all-NaN** hidden states in bf16 (same 11-token input:
|
||||
finite on one forward, NaN at layer 4 on the next). bf16 and fp32 share exponent
|
||||
range, so this is **precision-driven catastrophic cancellation, not overflow** —
|
||||
**fp32 resolves it.** Diagnosed via `diag_nan.py` / `diag2.py`: `sdpa` + plain
|
||||
prompt = 65 layers all finite; chat-template input = NaN; the trigger is the
|
||||
input path through the unstable recurrence.
|
||||
|
||||
Fixes, all in the committed harness (`7abd301`):
|
||||
- **`--capture` loads fp32**; the write/surgery path stays bf16 (no forward, no NaN).
|
||||
- **A finite-gate aborts on a non-finite direction** — the sink screen alone
|
||||
can't catch it (`nan > threshold` is False, so a NaN direction "passed" it and
|
||||
saved silently on the first run).
|
||||
- `attn_implementation="sdpa"` pinned.
|
||||
|
||||
**fp32 (110 GB) needs the whole GPU.** device_map=auto packed it tight and the
|
||||
forward OOM'd against the resident seats. Had to **stop three seats** for VRAM:
|
||||
`vllm-meromero-rp`, `vllm-fablefusion-probe`, and production `vllm-gen`.
|
||||
⚠ **Restart order matters:** gen restarted into an empty GPU0 and greedily
|
||||
grabbed 64 GB (vLLM takes a fraction of *free* memory at startup), starving
|
||||
meromero into a crash-loop. Fixed by bringing **meromero up first**, then gen
|
||||
into the remainder. All three restored to healthy.
|
||||
|
||||
⚠ **fla lives in a side dir, not the venv.** The shared
|
||||
`/tank/aimodels/quant-work/.venv` is not llmuser-writable; `fla` + `einops` are
|
||||
`--target`-installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and
|
||||
reached via `PYTHONPATH`. Prune deps that shadow the venv's torch/transformers.
|
||||
|
||||
## Result
|
||||
|
||||
Refusal direction: **finite, unit-normed, layer 22**, sink energy **0.0008%**
|
||||
in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated.
|
||||
Saved to `/tank/aimodels/qwen38-27b-coldfusion-bf16/refusal-direction.pt`.
|
||||
|
||||
⚠ **QUALITY CAVEAT — the reason the next step is calibration-set expansion.**
|
||||
Two-template `|cos|` agreement at layer 22 is **0.59**, well below Robinson's
|
||||
0.99. Almost certainly the small calibration set: **8 harmful / 8 harmless**
|
||||
(HARMFUL/HARMLESS in `abliterate.py`) vs Robinson's **416 / 104**. The direction
|
||||
is valid and sink-clean but noisier than ideal; abliterating on it risks
|
||||
under-removing refusals or nicking capability. **Expand the sets to a few
|
||||
hundred each and re-capture** before the `--out` write.
|
||||
|
||||
## Sequence from here
|
||||
|
||||
1. **Expand HARMFUL/HARMLESS calibration sets** → re-capture (fp32, seats down).
|
||||
2. `--out` write (bf16 surgery, no forward) → `qwen38-27b-coldfusion-abliterated-bf16`.
|
||||
3. Verify: vision byte-identical, refusal re-profile via `services/refusal-probe/`
|
||||
(the canonical harness, NOT the ad-hoc GGUF one), MTP acceptance on the quant
|
||||
(gate ≳40%, not KL — `reference_abliteration_mtp_lessons`), PPL/coherence.
|
||||
4. NVFP4-quantize via `services/gen-seat-mixed-quant/` → gen-seat candidate.
|
||||
**Do NOT delete the incumbent** (`qwen38-27b-heresy-nvfp4-mixed`) until it
|
||||
holds through real multi-turn use.
|
||||
|
||||
bf16 staged at `/tank/aimodels/qwen38-27b-coldfusion-bf16` (pinned `9c44193`,
|
||||
provenance recorded). All write paths re-stop the seats for fp32 VRAM — batch
|
||||
re-capture + write in one window. Commits `ccb56a0`, `1857a8e`, `b56cb0d`,
|
||||
`7abd301`.
|
||||
|
||||
_Archived 2026-09-05._
|
||||
|
||||
# `[2026-08-19]` Fleet `.internal` DNS — git-sourced, agent-managed, three resolvers
|
||||
|
||||
Operator: *"with ipv6 i can't memorize the IP addresses anymore. need a way to
|
||||
keep track of local .internal dns names that can be agent managed and is
|
||||
lightweight."* Built and live in one session; commit `b8003c7`.
|
||||
|
||||
## Shape
|
||||
|
||||
```
|
||||
dns/internal.yaml source of truth — 38 hosts + 4 service aliases
|
||||
scripts/dns-sync.py reconciles AdGuard resolvers against it
|
||||
stacks/adguard-ana/ the colo's resolver, which did not exist
|
||||
dns/README.md workflow, naming, the IPv6 caveat
|
||||
```
|
||||
|
||||
Deliberately the same posture as `deploy-stack.sh`: the file is intent, the
|
||||
resolvers are derived state, you see a diff before anything changes.
|
||||
`--dry-run` / `--yes` / `--site <s>`. Verified idempotent — a second run prints
|
||||
`nothing to do`.
|
||||
|
||||
Naming is `<host>.<site>.internal` with sites **`ana` / `esh` / `nh3`**
|
||||
(operator's call). `.internal` is ICANN-reserved for private use since 2024;
|
||||
`.local` is reserved for mDNS, which is why the pre-existing
|
||||
`searxng.pfi.local` was a standards collision that merely happened to work.
|
||||
|
||||
Every name is published to **every** resolver — the site label says where a
|
||||
host *is*, not which resolver knows about it.
|
||||
|
||||
## The framing correction that mattered most
|
||||
|
||||
The ask reads as "I can't memorise v6 addresses", but the deeper problem is
|
||||
that **v6 addresses are derived, not assigned**, so they cannot reliably be
|
||||
*written down once* either. SLAAC gives EUI-64 (MAC-coupled) or
|
||||
privacy-extension (rotating) addresses, and UniFi has **no v6 equivalent of a
|
||||
DHCP reservation** — so a hand-maintained v6 table rots on its own.
|
||||
|
||||
⇒ The fix has two halves and only the second is DNS: (1) pin static v6 on
|
||||
server-class hosts, (2) then the name table is just a file. Surfaced to the
|
||||
operator before building.
|
||||
|
||||
**Verified 2026-08-19: no fleet host has a global v6 address at all yet** —
|
||||
ESH's `/56` is live only on `esh-cameras`, NH3's LANs are back to
|
||||
`ipv6_interface_type: none`, the colo has none. So the `v6:` column ships
|
||||
EMPTY and correct, and the naming layer was built first rather than blocking
|
||||
on v6. Names established now need no renaming when addresses land.
|
||||
|
||||
Suggested convention when they do (awaiting operator): each server static at
|
||||
its site's `/64` with low-order bits echoing the v4 host octet —
|
||||
`esh-docker-vm` at `…::45` — so addresses are declarable *and* semi-memorable.
|
||||
|
||||
## Two properties not to break
|
||||
|
||||
**Authority is scoped to the ZONE, not the resolver.** Only rewrites ending in
|
||||
`.internal` are managed. ESH's resolver turned out to carry three hand-made
|
||||
`esteban.net` rewrites (`eshnas`, `brotherprinter`, `eshhome`) — **my first
|
||||
read of the config missed them**, because an `awk` range on `rewrites:` matched
|
||||
an empty-looking block. A resolver-wide authoritative sync would have silently
|
||||
deleted all three on first run. Verified intact after sync.
|
||||
|
||||
**Within `.internal` it IS authoritative** — names added by hand in the AdGuard
|
||||
UI get deleted by the next sync. That is the point: one place to look.
|
||||
|
||||
## The colo had no resolver at all
|
||||
|
||||
ESH and NH3 each ran AdGuard; **ana-docker resolved straight against
|
||||
`1.1.1.1`**, so the colo had no way to answer for internal names. Closed with
|
||||
`stacks/adguard-ana/`.
|
||||
|
||||
⚠️ Its API is on **8053**, not 8080 — `:8080` and `:3000` were already taken on
|
||||
that busy host. The port is therefore carried **per-site in the yaml**, not
|
||||
assumed by the script, so the odd one out cannot be forgotten.
|
||||
|
||||
⚠️ It ships with **no blocklists**, deliberately. The other two filter ads for
|
||||
human browsing; this one resolves for a rack of servers, where a blocklist
|
||||
false-positive breaks service-to-service calls at 3am for no upside.
|
||||
|
||||
First boot uses a **seed config** (`conf/AdGuardHome.seed.yaml`) copied into
|
||||
the conf volume before first start, so the container comes up configured
|
||||
instead of sitting in the setup wizard.
|
||||
|
||||
## Credential — service account, not the operator's
|
||||
|
||||
Added a dedicated **`infra-ops`** AdGuard user to all three resolvers rather
|
||||
than asking for the `lkraven` password (per the standing migrate-off-operator-
|
||||
creds directive). Password vaulted at
|
||||
`nh3-dev/adguard-infra-ops-password`; `lkraven` untouched; pre-change configs
|
||||
backed up on each host as `AdGuardHome.yaml.bak-preinfraops-*`. Both existing
|
||||
resolvers kept answering across the restart.
|
||||
|
||||
Two landmines worth keeping:
|
||||
|
||||
- **Go's bcrypt rejects `htpasswd`'s `$2y$` prefix.** Same algorithm, different
|
||||
marker; `golang.org/x/crypto/bcrypt` accepts only `$2a$`/`$2b$`. Normalise
|
||||
the prefix, and self-verify the hash with `htpasswd -vb` BEFORE installing it
|
||||
on a live resolver.
|
||||
- **The vault appends a trailing newline on `get`.** A password carrying a
|
||||
stray `\n` fails auth in a way that looks exactly like a wrong password.
|
||||
`dns-sync.py` strips it.
|
||||
|
||||
## `pfi.local` migration — and the one that must NOT move
|
||||
|
||||
`searxng.pfi.local` → `searxng.ana.internal`, with the **old `Host()` kept
|
||||
alongside** in the Traefik rule so nothing breaks mid-migration; both return
|
||||
200. Drop the fallback once the access log shows the old name unused.
|
||||
|
||||
**`matrix.pfi.local` deliberately NOT migrated.** A Matrix `server_name` is
|
||||
baked into every user ID, room ID and signing key, and federation identity
|
||||
derives from it — renaming it is not a DNS change, it is rebuilding the
|
||||
homeserver's identity and invalidating its history. The operator approved
|
||||
"migrate pfi.local" generally; this was surfaced as a deliberate exclusion
|
||||
rather than executed blindly.
|
||||
|
||||
## Still open
|
||||
|
||||
Colo hosts still point at `1.1.1.1`, so they do not yet *use* the new resolver
|
||||
— it only answers what asks it directly. Repointing a whole site's DNS is a
|
||||
bigger change than standing the service up, and is the operator's to schedule.
|
||||
|
||||
See also [[2026-08-17-fleet-ipv6-mesh]].
|
||||
|
||||
_Archived 2026-09-05._
|
||||
|
||||
# `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured
|
||||
|
||||
Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`):
|
||||
a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare
|
||||
`nohup` (PID 1283383) that would not survive a reboot. Now
|
||||
`stacks/waterland-studio/`, `restart: unless-stopped`, healthy on
|
||||
irv-ml1:8410. Commits `a2b5b58`, `8189076`.
|
||||
|
||||
Tracking `main` per operator: PR #4 merged and `main` HEAD was exactly the
|
||||
pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway.
|
||||
|
||||
**Now deployed at `b72425b` (2026-08-19).** The container sat on `8025366` for
|
||||
a few hours after PR #5 (`464dfc2`) landed — deliberately, since the image's
|
||||
own guards already neutralised both landmines and the project was in
|
||||
wind-down. PR #6 (the job-store rehydrate, operator-green-lit) was the rebuild
|
||||
with a real reason behind it, and one `update.sh` run carried both. Verified
|
||||
end to end after the update: healthy, `backend: cupy`, and a real 256² plate
|
||||
render completes warm — the kernel-cache volume survived the image swap.
|
||||
|
||||
## Build context lives OUTSIDE the compose dir — on purpose
|
||||
|
||||
`/opt/waterland-studio/src` is the checkout; the Dockerfile is passed
|
||||
out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh`
|
||||
rsyncs `stacks/<stack>/` with `--delete`**, so a checkout kept beside
|
||||
`compose.yaml` would be destroyed by the next deploy of this stack. `update.sh`
|
||||
refreshes source → rebuild → recreate → health, and is verified end to end.
|
||||
|
||||
## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN
|
||||
|
||||
`gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer
|
||||
silently drops to the numpy path at ~21x wall time — it does not error, it
|
||||
just gets slow. waterland-dev warned about the build side.
|
||||
|
||||
The runtime side is worse and was not in the handover: **`studio/jobs.py`
|
||||
shells the renderer out as a literal `uv run waterland ...` with no `--extra`
|
||||
flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project
|
||||
mid-job to its default extras and prunes cupy back out from under a correctly
|
||||
built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the
|
||||
pin ever stops holding the job fails **loudly** instead of quietly rebuilding a
|
||||
slower environment.
|
||||
|
||||
**Fixed upstream in `464dfc2`:** the server now spawns
|
||||
`sys.executable -m waterland.cli` directly — no resolver in the render path at
|
||||
all. **The pins stay anyway.** They cost nothing and are now defence-in-depth:
|
||||
if any future code path re-enters `uv` inside the container, the job fails
|
||||
loudly instead of quietly dropping to the numpy backend. `uv` itself must stay
|
||||
in the image regardless — it performs the build-time `uv sync` /
|
||||
`uv pip install`, and this is a single-stage build.
|
||||
|
||||
## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare
|
||||
|
||||
Every render died 1.7s in with:
|
||||
|
||||
```
|
||||
RuntimeError: Failed to find CUDA headers.
|
||||
```
|
||||
|
||||
printed **through argparse's usage banner**, which makes it read like a CLI
|
||||
argument bug rather than a missing toolkit. That misdirection is the reason
|
||||
this is written down.
|
||||
|
||||
cupy compiles kernels at runtime through NVRTC, which needs the toolkit
|
||||
**headers** — not just the driver and the runtime libs bundled in the
|
||||
`cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the
|
||||
bare `nohup` process found them **by accident**; a slim image has none.
|
||||
|
||||
Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few
|
||||
hundred MB against ~6 GB for a `-devel` base image. It runs **after**
|
||||
`uv sync`, because sync prunes what it does not know about.
|
||||
|
||||
Reported upstream: it is an undeclared runtime dependency of the `gpu` extra,
|
||||
and anyone running this without a system toolkit hits it. **Declared upstream
|
||||
in `464dfc2`** (`gpu` is now `cupy-cuda12x[ctk]>=13`). **The explicit install
|
||||
stays in the Dockerfile**: the header requirement is a property of *this*
|
||||
image — a slim base with no system CUDA toolkit — so it belongs in the file
|
||||
that creates the problem, not inherited from an extra two repos away. It also
|
||||
survives any future restructuring of the `gpu` extra. Cost of keeping it is
|
||||
now measured, not assumed: since `uv sync` satisfies it first, the line
|
||||
reports `Audited 1 package in 49ms` and adds **0.3s** to the build. A no-op
|
||||
that documents a non-obvious requirement is worth 0.3s. (waterland-dev
|
||||
independently agreed they would keep it too.)
|
||||
|
||||
## Landmine 3 — the GPU index inside the container is not the host's
|
||||
|
||||
The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects
|
||||
`CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where
|
||||
`nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU
|
||||
(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the
|
||||
container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying
|
||||
the host's value selects a device that does not exist. Host device 0 is the
|
||||
3090, which carries the TTS zoo and must not be touched.
|
||||
|
||||
## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume
|
||||
|
||||
| job | wall |
|
||||
|---|---|
|
||||
| 256² + anim, cold container | 23.3 s |
|
||||
| 256² + anim, warm | **6.1 s** |
|
||||
| 256² plate only (`--codec none`) | 3.9 s |
|
||||
| 512² plate only | 6.4 s |
|
||||
|
||||
Warm beats the **7.4 s** recorded against the bare-metal process, so
|
||||
containerising cost nothing. Verified the cache volume properly: recreate
|
||||
(fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without
|
||||
it every restart makes the next user wait 4x and the service merely *looks*
|
||||
slow.
|
||||
|
||||
## Upstream finding — the on-disk job store grows without bound
|
||||
|
||||
`JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at
|
||||
startup**. Consequences:
|
||||
|
||||
1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and
|
||||
how this was spotted: the API reported **1 job** while the volume held all
|
||||
**16 directories, 60.6 MB**. Not data loss.
|
||||
2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict,
|
||||
so directories orphaned by a restart are **never reclaimed**. The
|
||||
handover's "bounded around 500 MB" holds within a single process lifetime;
|
||||
across restarts the store grows monotonically at ~12 MB per animated job.
|
||||
|
||||
Reported to waterland-dev with evidence; **not patched from the infra side** —
|
||||
it is their code. Prune the volume by hand if it bites first.
|
||||
|
||||
**waterland-dev confirmed it (2026-08-19)** — their "bounded ~500 MB" handover
|
||||
claim holds within one process lifetime and nowhere else, which on a
|
||||
`restart: unless-stopped` service is the wrong lifetime to have bounded. They
|
||||
have **surfaced a startup-rehydrate fix to the operator** rather than opening a
|
||||
third PR during wind-down. **Operator green-lit it; PR #6 merged as `b72425b`
|
||||
and is DEPLOYED (2026-08-19).**
|
||||
|
||||
Startup rehydrate, as recommended — and waterland-dev deliberately went
|
||||
further than the framing I sent them. I had said a directory the scan cannot
|
||||
parse "just does not enter the index"; they made the opposite call, because a
|
||||
directory that never enters the index is exactly the one that never gets
|
||||
reclaimed. **That is the sharper reading and it is the reason the fix works on
|
||||
this volume at all** — the 16 pre-existing dirs have no sidecar. Their
|
||||
adoption ladder: sidecar → restored verbatim; no sidecar → adopted with
|
||||
dimensions recovered from the PNG IHDR (24-byte read, not a decode); corrupt
|
||||
sidecar → degrades to inference, no startup crash; **neither source nor
|
||||
sidecar → skipped on purpose**, since adopting it would turn eviction into a
|
||||
delete-arbitrary-directories primitive pointed at this volume. Sidecar writes
|
||||
go through `os.replace`, and `job.json` is excluded from `ARTIFACTS` so it is
|
||||
unreachable via the artifact route.
|
||||
|
||||
They also closed a second leak I never saw, because it needs a restart
|
||||
*mid-render* to surface: a job left `running`/`queued` in its sidecar is
|
||||
non-terminal forever, and eviction skips non-terminal jobs — so it is a
|
||||
phantom that is never reclaimed and `queue_depth` over-reports for the life of
|
||||
the process. Adoption now marks those `failed`.
|
||||
|
||||
**Verified on this host after the update:** `/api/jobs` went **1 → 16** while
|
||||
the volume stayed at 16 dirs / 61 MB — disk and API agree for the first time.
|
||||
Nothing was reclaimed, correctly: 16 is under `RETAIN=40`, so adoption only
|
||||
made them visible. A subsequent real render took both to 17. From here the
|
||||
store is bounded **across** restarts, not merely within a process.
|
||||
|
||||
## Access
|
||||
|
||||
Repo is not anonymously readable (a bare clone 403s). Operator granted
|
||||
**`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False,
|
||||
pull: True`. Token on irv-ml1 at
|
||||
`/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a
|
||||
**repo-scoped** credential helper; `.git/config` carries no token (verified),
|
||||
so the remote stays clean in any diff or backup. The operator's `vh`
|
||||
site-admin token was used only for the initial clone and the grant itself and
|
||||
was **never written to disk on that host** — a site-admin credential on a GPU
|
||||
box is a blast radius nobody needs for a read-only fetch.
|
||||
|
||||
## Constraints honoured as stated (not inferred)
|
||||
|
||||
- **Serial by design — one replica, one card.** A render is 20–45s of near-full
|
||||
GPU with a single worker thread. Two on the same A6000 would OOM or thrash.
|
||||
Throughput is a hardware conversation, not a replica-count one.
|
||||
- **No authentication, arbitrary file uploads** ⇒ stays inside the
|
||||
LAN/WireGuard boundary. Do **not** paper over it with a proxy password;
|
||||
waterland-dev offered to add a real auth layer if wider reach is ever needed.
|
||||
|
||||
_Archived 2026-09-05._
|
||||
|
||||
# `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background
|
||||
|
||||
Commits `9d92c4b`, `c3de7db`, `45c1995`, `f38cf69`, `df68dd2`.
|
||||
|
||||
## The cleanup (three real defects)
|
||||
|
||||
- **UltraSeedbox rendered on all four tabs.** The bookmark group had no entry in
|
||||
`settings.yaml`'s `layout:` block at all, and Homepage's documented behaviour
|
||||
is that a group with no `tab:` is shown on **every** tab. Pinned to Main.
|
||||
⚠️ This will happen again to the next group added without a `tab:` — the rule
|
||||
is now written at the top of the layout block.
|
||||
- **Uptime Kuma rendered twice** — a manual `services.yaml` entry under
|
||||
Monitoring *and* `homepage.group=Apps` on the container. Exactly the
|
||||
"never list a labelled container manually" failure the stack README warns
|
||||
about; it survived the previous day's audit because a duplicate reads as two
|
||||
plausible cards rather than as an error. Manual block deleted, label moved to
|
||||
`Monitoring`, `homepage.siteMonitor` added.
|
||||
- **Column counts were fiction** — several groups declared more columns than
|
||||
they had members, so the last row of each was dead space (Notes: 1 card in a
|
||||
4-wide row). Columns now track member counts; `GET /api/services` prints the
|
||||
live per-group counts and is the check.
|
||||
|
||||
Later, on operator instruction, the **AI tab was reordered by clickability**:
|
||||
Gateways & Chat → Image & Media → Audio Tools on top, then the vLLM `/docs`
|
||||
seats and TTS endpoints. Reasoning written into the config so it survives:
|
||||
order by "would I click this?", not by how central the service is.
|
||||
|
||||
## ⚠️ The expensive red herring — the tab bar after a recreate
|
||||
|
||||
After a recreate the client render comes up with **no tab bar, no wallpaper and
|
||||
no i18n** (search box shows the raw key `search.search`), groups falling back to
|
||||
side-by-side columns. **It restores itself with no intervention.**
|
||||
|
||||
Timing, measured rather than assumed: a fresh container was still tab-less at
|
||||
**4m30s, twice**; it was healthy again after roughly an hour. `docker ps`
|
||||
reporting `healthy` says nothing about it — the container is serving, the page
|
||||
is just wrong.
|
||||
|
||||
An hour went into ruling out four causes that were never the cause:
|
||||
|
||||
1. **Not the config** — restoring `settings.yaml` *and* `services.yaml` to
|
||||
their committed versions reproduces it, as does the pre-adoption backup in
|
||||
`/opt/docker-bu/conf/homepage/`.
|
||||
2. **Not the v2.0.0 release** — a throwaway container on `v1.13.2` shows
|
||||
identical symptoms, and the image never changed anyway (working and broken
|
||||
both report `v2.0.0` / rev `17456f2`).
|
||||
3. **Not `PUID`/`PGID`**, and not Docker discovery — tested both, and with the
|
||||
socket unmounted entirely.
|
||||
4. **Not server-side** — the server-rendered HTML still contains the tab
|
||||
markup, the background URL and `useEqualHeights`; `GET /api/validate`
|
||||
returns `[]`. The loss is client-side, with no page error, no failed chunk
|
||||
and no non-200.
|
||||
|
||||
Every throwaway container in that list was judged within ~30s of starting, so
|
||||
they were all inside the same window — and that consistency **read as a
|
||||
reproduction when it was the same measurement mistake five times over.**
|
||||
|
||||
**Operative rule: recreate, walk away, re-check later. Do not chase it.**
|
||||
|
||||
## ⚠️ The iteration loop that would have prevented the overcook
|
||||
|
||||
`custom.css` is served **per request** from `/api/config/custom.css`, so a CSS
|
||||
change needs a **browser reload** — not a container recreate, and it never owed
|
||||
the layout warm-up above. Conflating the two costs ~10 operator-visible minutes
|
||||
per attempt (operator called this out directly).
|
||||
|
||||
Faster still, and how the final pass was done: **inject candidate CSS into the
|
||||
running page and screenshot it** —
|
||||
`await p.addStyleTag({content: css})` in Playwright against the live
|
||||
dashboard. Seconds per iteration, no deploy. Build + deploy only once the
|
||||
render looks right.
|
||||
|
||||
## The theme — Australis Skyfall
|
||||
|
||||
Operator supplied a Claude Design handoff bundle via the Booth (`26-copper`).
|
||||
Skyfall is a dual-theme OKLCH system: one lightness law across every chromatic
|
||||
family (deep 0.48 / base 0.66 / bright 0.80), all hues cooler than neutral, a
|
||||
Sea neutral ramp drifting ice-blue→ocean-green as it brightens, and a
|
||||
"calm depth" language of **hairline + two-layer shadow on every elevated
|
||||
surface, never one without the other**.
|
||||
|
||||
```
|
||||
theme/colors.css layout.css typography.css vendored VERBATIM from the bundle
|
||||
theme/fonts/Supreme-{400,500,700}.woff2 the body/UI face
|
||||
theme/skyfall.css.in the Homepage bindings (ours)
|
||||
theme/build.py → conf/custom.css (generated — do not hand-edit)
|
||||
```
|
||||
|
||||
The build step exists for one reason: **Homepage serves only `custom.css` and
|
||||
`custom.js` out of its config dir**, with no static route beside them, so a
|
||||
`@font-face` pointing at a vendored `.woff2` would 404 — the face must arrive
|
||||
as a data URI. The background image takes the other road, because
|
||||
`/app/public/images` **is** a real static route (mounted read-only in
|
||||
`compose.yaml`).
|
||||
|
||||
Only Supreme is embedded: a link dashboard has no display type, and Victor
|
||||
Mono ships as 2.4 MB TTF statics per cut — 30x the whole stylesheet for a
|
||||
handful of latency figures.
|
||||
|
||||
## The background is generated, not stock
|
||||
|
||||
**Arbo as an image-gen engine** (the operator's actual ask, which I first
|
||||
misread as "use Arbo's palette" and had to redo). Arbo's `t2i-ui-background`
|
||||
workflow is purpose-built: *"abstract full-bleed backgrounds, no subject"*.
|
||||
Job `13f0891f4e42`, seed 26, flux2-klein-9b, 2048×1152, 1.6 MB PNG → **22 KB
|
||||
WebP** (smooth gradients compress absurdly well).
|
||||
|
||||
⚠️ Arbo API gotcha: `prompt` is a **discriminated union, not a string** — a
|
||||
bare string 422s. `{"kind":"raw","text":…,"negative":…}` is the shape.
|
||||
|
||||
## Two documented deviations from the design system
|
||||
|
||||
1. **Skyfall forbids this background.** Its rule is "flat semantic surfaces; no
|
||||
photography, no textures", with one permitted motif — a subtle aurora
|
||||
gradient on hero/empty-state areas only, *"never behind body text blocks"*.
|
||||
A dashboard is a body-text block. Present on the operator's explicit
|
||||
instruction, mitigated rather than excused: abstract, no subject, strictly
|
||||
cool temperature, held at **`opacity: 30`**. That number is load-bearing —
|
||||
at 14 the aurora was invisible, and turning it up makes the cards fight the
|
||||
ribbon.
|
||||
2. **Service icons stay full-colour vendor logos.** Desaturating them from CSS
|
||||
only makes them illegible.
|
||||
|
||||
## Overcorrection, and the colour pass
|
||||
|
||||
First stat-well pass went from `font-thin` 13px straight to **bold 22px in
|
||||
heading white** — operator: *"went from subtle to BASH YOU OVER THE HEAD."*
|
||||
The principle missed: a stat only has to out-rank **its own label**, not the
|
||||
service name above it. Now `--text-md` medium in cyan.
|
||||
|
||||
Colour was then lifted **from inside the system**: Skyfall names Aurora (blue,
|
||||
cyan, green) the *primary* families, "used generously, in that order", while
|
||||
Dawn (amber/red/violet) is semantic-only. So group markers cycle
|
||||
blue→cyan→green down the page (icons full strength, names at 0.72), service
|
||||
icons take a single cool wash, latency tags move to the info family so
|
||||
"how fast" stops looking like "is it alive". **No Dawn colour is used
|
||||
decoratively anywhere.**
|
||||
|
||||
Two DOM findings that made it possible:
|
||||
|
||||
- **Homepage renders mdi icons as a gradient behind an SVG mask** — recolour
|
||||
via `background`, not `color`.
|
||||
- **Homepage emits `docker-status-<state>`, not `status-<state>`.** The
|
||||
original selectors matched nothing, so every green pill up to that point was
|
||||
stock colouring rather than the theme. Both forms are now matched.
|
||||
|
||||
_Archived 2026-09-05._
|
||||
|
||||
# `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken
|
||||
|
||||
Commits `42c594c`, `dc3e47b`, plus `uptimekuma` in `9d92c4b`.
|
||||
|
||||
## The pattern worth remembering
|
||||
|
||||
Chasing two bad-looking cards on the dashboard turned up **four stacks running
|
||||
on fleet hosts with no canonical copy anywhere**: `uptimekuma` and (already
|
||||
known) the two AdGuards on esh-docker-vm, `searxng` and `seafile` on
|
||||
ana-docker, and `heretic2-charrp-reasoning` on ana-ml2 (untracked in git).
|
||||
|
||||
⇒ **A dashboard card is a cheap census of what is actually running.** When
|
||||
something on it looks wrong, check whether the stack behind it is even in
|
||||
`stacks/` before debugging the symptom — twice here the answer was "no", and
|
||||
the fix belonged in version control as much as on the host.
|
||||
|
||||
Adopted: `stacks/uptimekuma/`, `stacks/searxng/`, `stacks/seafile/`,
|
||||
`stacks/heretic2-charrp-reasoning/`. ESH/NH3 AdGuard compose files were
|
||||
**deliberately left unmanaged** — adopting three live resolvers while also
|
||||
introducing a new DNS naming system is two risky changes at once.
|
||||
|
||||
## SearXNG — the healthcheck was eating itself
|
||||
|
||||
Card flapped UNHEALTHY; the container was fine the whole time. The compose
|
||||
passed `--tries` and `--spider` as **two separate argv entries**, so wget
|
||||
consumed `--spider` as the *value* of `--tries`. Spider mode never engaged,
|
||||
which means every probe since April **downloaded** the healthz response to a
|
||||
file:
|
||||
|
||||
```
|
||||
295,287 healthz.N files in the container's working directory
|
||||
```
|
||||
|
||||
With that many files, wget's scan for the next free filename is what
|
||||
intermittently blew the 10s timeout. **Self-worsening — every probe made the
|
||||
next one slower.** Restored `--tries=1`; the junk lived in the writable layer
|
||||
so the recreate cleared it. Now `healthy`, `fails=0`, 200 in 0.16s.
|
||||
|
||||
Lesson: an argv list in YAML has no shell to catch a missing `=`. A flag that
|
||||
silently swallows the next argument turns a liveness probe into a workload.
|
||||
|
||||
## SeaFile — not broken, never restarted
|
||||
|
||||
Card showed EXITED for three months. **None of the three services declared a
|
||||
restart policy**, so Docker defaulted them to `no`. On
|
||||
**2026-05-06T21:27:45Z** the daemon stopped all three within 200ms of each
|
||||
other — a daemon restart or host reboot — and nothing brought them back.
|
||||
|
||||
⚠️ **Exit code `255` is a red herring**: it is what a container that ignores
|
||||
SIGTERM reports when the daemon stops it, **not** evidence of a crash. Reading
|
||||
it as one sends you hunting a bug that does not exist. The tell was all three
|
||||
services stopping within 200ms.
|
||||
|
||||
Added `restart: unless-stopped` to all three; brought up; mariadb gated on its
|
||||
healthcheck exactly as the existing `depends_on` comments intended, seahub
|
||||
started without the race, `302` → login page. Data was in local named volumes,
|
||||
not on the ana-nas NFS, so nothing was at risk.
|
||||
|
||||
Three months of silent downtime whose only signal was a card nobody read as an
|
||||
outage — the argument for semantic status colour on the dashboard (see
|
||||
[[2026-08-19-homepage-skyfall-theme]], where amber EXITED pills made six
|
||||
mis-grouped AI seats obvious at a glance).
|
||||
|
||||
## heretic2-charrp-reasoning — tracked, with its shim
|
||||
|
||||
The `char-rp-reasoning` seat (NEO-CODE Heretic2 27B, modelopt NVFP4 + grafted
|
||||
BF16 MTP head, ~77 tok/s via `qwen3_5_mtp` spec-decode) had been running
|
||||
untracked. Now in `stacks/`, including
|
||||
`conf/mtp-workaround/sitecustomize.py`, which is **not optional**: vLLM 0.24.0
|
||||
does not propagate modelopt `exclude_modules` to the spec-decode **draft**
|
||||
model, so the BF16 MTP head gets quantized and the engine dies at load. Both
|
||||
the mount and `PYTHONPATH` are load-bearing.
|
||||
|
||||
Added the two files house convention expects and the directory lacked — a
|
||||
`.env.example` naming every knob (all values are compose defaults; the host
|
||||
overrides only the three VRAM ones) and a README pointing at
|
||||
`docs/runbooks/heretic2-nvfp4-mtp-seat.md` rather than duplicating it.
|
||||
|
||||
_Archived 2026-09-05._
|
||||
|
||||
Reference in New Issue
Block a user