memory: snapshot — .internal DNS live, waterland containerised, homepage themed
Captures a long infra session: fleet *.internal DNS (git-sourced, 42 names, three resolvers including a new colo one), waterland studio containerised on irv-ml1, Homepage cleaned up and themed with Australis Skyfall over an Arbo-generated background, and four unmanaged stacks adopted into stacks/. Four detail files added. Auto-archived 4 entries to archival-memory.md (Recent decisions 2, Tried and abandoned 2); 5 held back by the open-deferred guard rather than moved. Also records three operator-owned open items: the colo DNS repoint, the static-v6 convention, and the deliberately belayed AI-tab Dormant regrouping.
This commit is contained in:
@@ -2465,3 +2465,33 @@ _Archived 2026-08-18._
|
||||
|
||||
- `[2026-07-30]` **magpie TTS serving — evaluated, ABANDONED.** Pulled `magpie_tts_multilingual_357m` (the one real repo of brokkr's batch) to NFS, stood it up on irv-ml1 (ephemeral NeMo-Speech-`main` container — stock PyPI/NGC NeMo can't load v2607), A/B'd vs Zonos → Zonos wins expressive English decisively, multilingual not needed. Not served; `magpie-nemo` torn down. `.nemo` KEPT on NFS as brokkr's fine-tuning base. auto-memory `project_magpie_tts_eval_rejected`.
|
||||
_Archived 2026-08-18._
|
||||
|
||||
## Recent decisions (archived 2026-08-19 batch)
|
||||
|
||||
`[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap; fix relayed to heid, NOT applied gateway-side.**
|
||||
|
||||
heid reported that `kimi-k3` (the primary route = Kimi Code coding endpoint `openai/k3` @ `api.kimi.com/coding/v1`) silently degraded its cross-frontier panel: on large/reasoning-heavy dispatches, `completion_tokens: 16381` **exactly**, `content` empty, `reasoning_content` ~64KB, `finish_reason: **stop**` (a truncation mislabeled as a clean stop). `max_tokens: 100000` in the request was not honored.
|
||||
|
||||
**Investigation arc (a clean cross-frontier-triage + verify-on-the-wire case):**
|
||||
1. My first read: a flat ~16384 OUTPUT cap; fix = a LiteLLM `stop→length` relabel callback (heid's fallback ask). Confirmed the cap isn't in our LiteLLM config (no `max_tokens` clamp on the route).
|
||||
2. Operator routed a fix-research pass to **dvalin-smithy-dev + bil-smithy-dev** (independent). Both CONVERGED (docs-based): `max_tokens` is a deprecated alias on Kimi/Moonshot; the canonical field is `max_completion_tokens` (default 131072, max 1M); the coding endpoint defaults output to 16384; fix = send `max_completion_tokens` + `reasoning_effort` via `extra_body` (drop_params-safe).
|
||||
3. **heid's live data REFUTED the docs hypothesis:** a later dispatch hit `completion_tokens: 18455` (ABOVE 16384) cleanly, with `reasoning_tokens: 16198` (just under 16384) and content present. So COMPLETION is uncapped; the bound is on **REASONING at ~16384**. When a hard task's thinking exhausts that budget, nothing's left for content → empty answer under `stop`.
|
||||
4. **I proved it on the wire** — ran heid's real 500KB failing bundle direct at both endpoints (bypassing LiteLLM so `reasoning_effort` isn't dropped): default effort → 504/timeout (the failure); **`reasoning_effort: low` → reasoning ~12–13.5k (under the ceiling), content returns (6–7.6k chars)**, on BOTH coding AND general endpoints. So re-routing to the general endpoint buys nothing — the fix is the effort param, and it works on the wire.
|
||||
|
||||
**THE FIX (caller-side, no shared-gateway change/restart):** send `reasoning_effort` via **`extra_body`** on kimi-k3 dispatches (`low` for large bundles). LiteLLM `drop_params: true` strips the top-level `reasoning_effort` — which is exactly why heid's earlier `reasoning_effort: low` was a no-op. `extra_body` survives drop_params (the house GLM-thinking pattern). Tradeoff: low effort = shallower reasoning, but a complete answer beats today's empty one.
|
||||
|
||||
**Relayed to heid to validate on a real round** (the one unconfirmed hop is whether `extra_body` survives OUR LiteLLM). **Backstop if it doesn't:** add `allowed_openai_params: ["reasoning_effort"]` to the `kimi-k3` route in the gateway config — that IS a shared-gateway change + a ~10s restart (blips all consumers), so it needs a heads-up.
|
||||
|
||||
Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/litellm/conf/config.yaml` (see Recent-decisions `[2026-07-25]` Kimi K3 wiring). No gateway change was made this session. Failing dispatch on record: `01KYTASKTY3T` (jackdaw-dev bug-hunt).
|
||||
_Archived 2026-08-19._
|
||||
|
||||
- `[2026-07-25]` **bil-smithy-dev wired as an althing zellij-window-ping (pane route).** She's a `driver: human` dwarf peer (pane `bil-smithy` already live alongside eitri/dvalin/regin-smithy in the `Claude` zellij session) but had no delivery route → smoke messages posted to the bus but never reached her window. **Mechanism (reusable for any pane-route handle):** `~/.althing/config.yaml` → `zellij_sessions.Claude.agents[]` maps `handle` → `target` (a zellij pane **TITLE**, matched via `list-panes -j` in `althing/zellij.py:resolve_pane_id`) → `command` (herald `write-chars` + CR into that pane). The **herald loads config ONCE at startup** (`herald.py main()`), so **`systemctl --user restart althing-herald.service`** after editing. Added bil (`target: bil-smithy`), restarted, verified: herald delivered the pending smoke `01KYD7W7CF…` (available→attempted→**delivered**). ⚠️ Noticed pre-existing pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range", empty msg_ids — likely `render_command messages[0]` on an empty list; NOT caused by this change, bil works) — worth a herald look.
|
||||
_Archived 2026-08-19._
|
||||
|
||||
## Tried and abandoned (archived 2026-08-19 batch)
|
||||
|
||||
- `[2026-08-02]` **`docker exec` into worldtree containers defaults to ROOT — root writes contaminate the uid-1000 (vh) KB tree.** My `sudo docker exec … --reindex` on personal ran as ROOT (muninn app = uid 1000); its wing git-commit + atomic note-swap left root-owned files in the `worldtree-personal_worldtree-kb` volume: a root-owned `.old-<job>` backup dir (blocked the uid-1000 retry's `rmtree` → Errno 13, because unlink needs write on the DIR and it was root:root 755) AND **60 root-owned loose git objects** in `.git/objects/`. Fix (host-side, corviduo-dev): `sudo rm -rf` the superseded `.old-` dir (tar'd aside to /tmp first) + `sudo find … -user 0 -exec chown 1000:1000` the objects (ownership-only, git-content-safe; the `.git/objects/XX/` dirs were vh-owned so these weren't a hard blocker, but violated "clean tree"). **RUNBOOK RULE (worldtree-dev, ADOPTED):** any `docker exec` into worldtree containers that WRITES pipeline state runs **`-u 1000`**, never default-root — same genus as the mv footgun (acting without matching the target's constraints; 3rd such slip in one session). **GOTCHA that hid the scope:** `find … -user 0 | head -20` TRUNCATED (the `.old-` dir alone had 153 files, so the first page was all `.old-`) → I "verified clean" off a partial list. Never `head` a scope-defining find; count first (`| wc -l`). **Related blind-spot (muninn-dev):** a root-owned job SUBDIR passes every requeue guard (job_row/dispatch/list_jobs render fine) AND `/health` (contract's `os.access(ingestion_root, W_OK)` tests only the ROOT dir, so a foreign-owned subdir under `pending/` still reports `ingestion_root_writable: true`) — then the uid-1000 gate can't write into it. "Clean board + green /health + failure at next mutation." muninn-dev added an OWNERSHIP column to the standing post-move check to catch it; two green signals both miss a foreign-owned subdir otherwise.
|
||||
_Archived 2026-08-19._
|
||||
|
||||
- `[2026-07-25]` **Chaining the althing wake-listener arm orphans it.** `reply && althing-wake-listener &` (or spawning `althing-wake-listener` with `&` *inside* a `run_in_background` task) → the `&`-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a *plain operator turn* (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn `althing-wake-listener` as its OWN `run_in_background` task, and re-arm ONLY after a real fire (`<task-notification> completed rc0`). Reclaim an orphan with `althing-cli stop-monitor` then re-arm.
|
||||
_Archived 2026-08-19._
|
||||
|
||||
@@ -1,15 +0,0 @@
|
||||
`[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap; fix relayed to heid, NOT applied gateway-side.**
|
||||
|
||||
heid reported that `kimi-k3` (the primary route = Kimi Code coding endpoint `openai/k3` @ `api.kimi.com/coding/v1`) silently degraded its cross-frontier panel: on large/reasoning-heavy dispatches, `completion_tokens: 16381` **exactly**, `content` empty, `reasoning_content` ~64KB, `finish_reason: **stop**` (a truncation mislabeled as a clean stop). `max_tokens: 100000` in the request was not honored.
|
||||
|
||||
**Investigation arc (a clean cross-frontier-triage + verify-on-the-wire case):**
|
||||
1. My first read: a flat ~16384 OUTPUT cap; fix = a LiteLLM `stop→length` relabel callback (heid's fallback ask). Confirmed the cap isn't in our LiteLLM config (no `max_tokens` clamp on the route).
|
||||
2. Operator routed a fix-research pass to **dvalin-smithy-dev + bil-smithy-dev** (independent). Both CONVERGED (docs-based): `max_tokens` is a deprecated alias on Kimi/Moonshot; the canonical field is `max_completion_tokens` (default 131072, max 1M); the coding endpoint defaults output to 16384; fix = send `max_completion_tokens` + `reasoning_effort` via `extra_body` (drop_params-safe).
|
||||
3. **heid's live data REFUTED the docs hypothesis:** a later dispatch hit `completion_tokens: 18455` (ABOVE 16384) cleanly, with `reasoning_tokens: 16198` (just under 16384) and content present. So COMPLETION is uncapped; the bound is on **REASONING at ~16384**. When a hard task's thinking exhausts that budget, nothing's left for content → empty answer under `stop`.
|
||||
4. **I proved it on the wire** — ran heid's real 500KB failing bundle direct at both endpoints (bypassing LiteLLM so `reasoning_effort` isn't dropped): default effort → 504/timeout (the failure); **`reasoning_effort: low` → reasoning ~12–13.5k (under the ceiling), content returns (6–7.6k chars)**, on BOTH coding AND general endpoints. So re-routing to the general endpoint buys nothing — the fix is the effort param, and it works on the wire.
|
||||
|
||||
**THE FIX (caller-side, no shared-gateway change/restart):** send `reasoning_effort` via **`extra_body`** on kimi-k3 dispatches (`low` for large bundles). LiteLLM `drop_params: true` strips the top-level `reasoning_effort` — which is exactly why heid's earlier `reasoning_effort: low` was a no-op. `extra_body` survives drop_params (the house GLM-thinking pattern). Tradeoff: low effort = shallower reasoning, but a complete answer beats today's empty one.
|
||||
|
||||
**Relayed to heid to validate on a real round** (the one unconfirmed hop is whether `extra_body` survives OUR LiteLLM). **Backstop if it doesn't:** add `allowed_openai_params: ["reasoning_effort"]` to the `kimi-k3` route in the gateway config — that IS a shared-gateway change + a ~10s restart (blips all consumers), so it needs a heads-up.
|
||||
|
||||
Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/litellm/conf/config.yaml` (see Recent-decisions `[2026-07-25]` Kimi K3 wiring). No gateway change was made this session. Failing dispatch on record: `01KYTASKTY3T` (jackdaw-dev bug-hunt).
|
||||
@@ -0,0 +1,119 @@
|
||||
# `[2026-08-19]` Fleet `.internal` DNS — git-sourced, agent-managed, three resolvers
|
||||
|
||||
Operator: *"with ipv6 i can't memorize the IP addresses anymore. need a way to
|
||||
keep track of local .internal dns names that can be agent managed and is
|
||||
lightweight."* Built and live in one session; commit `b8003c7`.
|
||||
|
||||
## Shape
|
||||
|
||||
```
|
||||
dns/internal.yaml source of truth — 38 hosts + 4 service aliases
|
||||
scripts/dns-sync.py reconciles AdGuard resolvers against it
|
||||
stacks/adguard-ana/ the colo's resolver, which did not exist
|
||||
dns/README.md workflow, naming, the IPv6 caveat
|
||||
```
|
||||
|
||||
Deliberately the same posture as `deploy-stack.sh`: the file is intent, the
|
||||
resolvers are derived state, you see a diff before anything changes.
|
||||
`--dry-run` / `--yes` / `--site <s>`. Verified idempotent — a second run prints
|
||||
`nothing to do`.
|
||||
|
||||
Naming is `<host>.<site>.internal` with sites **`ana` / `esh` / `nh3`**
|
||||
(operator's call). `.internal` is ICANN-reserved for private use since 2024;
|
||||
`.local` is reserved for mDNS, which is why the pre-existing
|
||||
`searxng.pfi.local` was a standards collision that merely happened to work.
|
||||
|
||||
Every name is published to **every** resolver — the site label says where a
|
||||
host *is*, not which resolver knows about it.
|
||||
|
||||
## The framing correction that mattered most
|
||||
|
||||
The ask reads as "I can't memorise v6 addresses", but the deeper problem is
|
||||
that **v6 addresses are derived, not assigned**, so they cannot reliably be
|
||||
*written down once* either. SLAAC gives EUI-64 (MAC-coupled) or
|
||||
privacy-extension (rotating) addresses, and UniFi has **no v6 equivalent of a
|
||||
DHCP reservation** — so a hand-maintained v6 table rots on its own.
|
||||
|
||||
⇒ The fix has two halves and only the second is DNS: (1) pin static v6 on
|
||||
server-class hosts, (2) then the name table is just a file. Surfaced to the
|
||||
operator before building.
|
||||
|
||||
**Verified 2026-08-19: no fleet host has a global v6 address at all yet** —
|
||||
ESH's `/56` is live only on `esh-cameras`, NH3's LANs are back to
|
||||
`ipv6_interface_type: none`, the colo has none. So the `v6:` column ships
|
||||
EMPTY and correct, and the naming layer was built first rather than blocking
|
||||
on v6. Names established now need no renaming when addresses land.
|
||||
|
||||
Suggested convention when they do (awaiting operator): each server static at
|
||||
its site's `/64` with low-order bits echoing the v4 host octet —
|
||||
`esh-docker-vm` at `…::45` — so addresses are declarable *and* semi-memorable.
|
||||
|
||||
## Two properties not to break
|
||||
|
||||
**Authority is scoped to the ZONE, not the resolver.** Only rewrites ending in
|
||||
`.internal` are managed. ESH's resolver turned out to carry three hand-made
|
||||
`esteban.net` rewrites (`eshnas`, `brotherprinter`, `eshhome`) — **my first
|
||||
read of the config missed them**, because an `awk` range on `rewrites:` matched
|
||||
an empty-looking block. A resolver-wide authoritative sync would have silently
|
||||
deleted all three on first run. Verified intact after sync.
|
||||
|
||||
**Within `.internal` it IS authoritative** — names added by hand in the AdGuard
|
||||
UI get deleted by the next sync. That is the point: one place to look.
|
||||
|
||||
## The colo had no resolver at all
|
||||
|
||||
ESH and NH3 each ran AdGuard; **ana-docker resolved straight against
|
||||
`1.1.1.1`**, so the colo had no way to answer for internal names. Closed with
|
||||
`stacks/adguard-ana/`.
|
||||
|
||||
⚠️ Its API is on **8053**, not 8080 — `:8080` and `:3000` were already taken on
|
||||
that busy host. The port is therefore carried **per-site in the yaml**, not
|
||||
assumed by the script, so the odd one out cannot be forgotten.
|
||||
|
||||
⚠️ It ships with **no blocklists**, deliberately. The other two filter ads for
|
||||
human browsing; this one resolves for a rack of servers, where a blocklist
|
||||
false-positive breaks service-to-service calls at 3am for no upside.
|
||||
|
||||
First boot uses a **seed config** (`conf/AdGuardHome.seed.yaml`) copied into
|
||||
the conf volume before first start, so the container comes up configured
|
||||
instead of sitting in the setup wizard.
|
||||
|
||||
## Credential — service account, not the operator's
|
||||
|
||||
Added a dedicated **`infra-ops`** AdGuard user to all three resolvers rather
|
||||
than asking for the `lkraven` password (per the standing migrate-off-operator-
|
||||
creds directive). Password vaulted at
|
||||
`nh3-dev/adguard-infra-ops-password`; `lkraven` untouched; pre-change configs
|
||||
backed up on each host as `AdGuardHome.yaml.bak-preinfraops-*`. Both existing
|
||||
resolvers kept answering across the restart.
|
||||
|
||||
Two landmines worth keeping:
|
||||
|
||||
- **Go's bcrypt rejects `htpasswd`'s `$2y$` prefix.** Same algorithm, different
|
||||
marker; `golang.org/x/crypto/bcrypt` accepts only `$2a$`/`$2b$`. Normalise
|
||||
the prefix, and self-verify the hash with `htpasswd -vb` BEFORE installing it
|
||||
on a live resolver.
|
||||
- **The vault appends a trailing newline on `get`.** A password carrying a
|
||||
stray `\n` fails auth in a way that looks exactly like a wrong password.
|
||||
`dns-sync.py` strips it.
|
||||
|
||||
## `pfi.local` migration — and the one that must NOT move
|
||||
|
||||
`searxng.pfi.local` → `searxng.ana.internal`, with the **old `Host()` kept
|
||||
alongside** in the Traefik rule so nothing breaks mid-migration; both return
|
||||
200. Drop the fallback once the access log shows the old name unused.
|
||||
|
||||
**`matrix.pfi.local` deliberately NOT migrated.** A Matrix `server_name` is
|
||||
baked into every user ID, room ID and signing key, and federation identity
|
||||
derives from it — renaming it is not a DNS change, it is rebuilding the
|
||||
homeserver's identity and invalidating its history. The operator approved
|
||||
"migrate pfi.local" generally; this was surfaced as a deliberate exclusion
|
||||
rather than executed blindly.
|
||||
|
||||
## Still open
|
||||
|
||||
Colo hosts still point at `1.1.1.1`, so they do not yet *use* the new resolver
|
||||
— it only answers what asks it directly. Repointing a whole site's DNS is a
|
||||
bigger change than standing the service up, and is the operator's to schedule.
|
||||
|
||||
See also [[2026-08-17-fleet-ipv6-mesh]].
|
||||
@@ -0,0 +1,145 @@
|
||||
# `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background
|
||||
|
||||
Commits `9d92c4b`, `c3de7db`, `45c1995`, `f38cf69`, `df68dd2`.
|
||||
|
||||
## The cleanup (three real defects)
|
||||
|
||||
- **UltraSeedbox rendered on all four tabs.** The bookmark group had no entry in
|
||||
`settings.yaml`'s `layout:` block at all, and Homepage's documented behaviour
|
||||
is that a group with no `tab:` is shown on **every** tab. Pinned to Main.
|
||||
⚠️ This will happen again to the next group added without a `tab:` — the rule
|
||||
is now written at the top of the layout block.
|
||||
- **Uptime Kuma rendered twice** — a manual `services.yaml` entry under
|
||||
Monitoring *and* `homepage.group=Apps` on the container. Exactly the
|
||||
"never list a labelled container manually" failure the stack README warns
|
||||
about; it survived the previous day's audit because a duplicate reads as two
|
||||
plausible cards rather than as an error. Manual block deleted, label moved to
|
||||
`Monitoring`, `homepage.siteMonitor` added.
|
||||
- **Column counts were fiction** — several groups declared more columns than
|
||||
they had members, so the last row of each was dead space (Notes: 1 card in a
|
||||
4-wide row). Columns now track member counts; `GET /api/services` prints the
|
||||
live per-group counts and is the check.
|
||||
|
||||
Later, on operator instruction, the **AI tab was reordered by clickability**:
|
||||
Gateways & Chat → Image & Media → Audio Tools on top, then the vLLM `/docs`
|
||||
seats and TTS endpoints. Reasoning written into the config so it survives:
|
||||
order by "would I click this?", not by how central the service is.
|
||||
|
||||
## ⚠️ The expensive red herring — the tab bar after a recreate
|
||||
|
||||
After a recreate the client render comes up with **no tab bar, no wallpaper and
|
||||
no i18n** (search box shows the raw key `search.search`), groups falling back to
|
||||
side-by-side columns. **It restores itself with no intervention.**
|
||||
|
||||
Timing, measured rather than assumed: a fresh container was still tab-less at
|
||||
**4m30s, twice**; it was healthy again after roughly an hour. `docker ps`
|
||||
reporting `healthy` says nothing about it — the container is serving, the page
|
||||
is just wrong.
|
||||
|
||||
An hour went into ruling out four causes that were never the cause:
|
||||
|
||||
1. **Not the config** — restoring `settings.yaml` *and* `services.yaml` to
|
||||
their committed versions reproduces it, as does the pre-adoption backup in
|
||||
`/opt/docker-bu/conf/homepage/`.
|
||||
2. **Not the v2.0.0 release** — a throwaway container on `v1.13.2` shows
|
||||
identical symptoms, and the image never changed anyway (working and broken
|
||||
both report `v2.0.0` / rev `17456f2`).
|
||||
3. **Not `PUID`/`PGID`**, and not Docker discovery — tested both, and with the
|
||||
socket unmounted entirely.
|
||||
4. **Not server-side** — the server-rendered HTML still contains the tab
|
||||
markup, the background URL and `useEqualHeights`; `GET /api/validate`
|
||||
returns `[]`. The loss is client-side, with no page error, no failed chunk
|
||||
and no non-200.
|
||||
|
||||
Every throwaway container in that list was judged within ~30s of starting, so
|
||||
they were all inside the same window — and that consistency **read as a
|
||||
reproduction when it was the same measurement mistake five times over.**
|
||||
|
||||
**Operative rule: recreate, walk away, re-check later. Do not chase it.**
|
||||
|
||||
## ⚠️ The iteration loop that would have prevented the overcook
|
||||
|
||||
`custom.css` is served **per request** from `/api/config/custom.css`, so a CSS
|
||||
change needs a **browser reload** — not a container recreate, and it never owed
|
||||
the layout warm-up above. Conflating the two costs ~10 operator-visible minutes
|
||||
per attempt (operator called this out directly).
|
||||
|
||||
Faster still, and how the final pass was done: **inject candidate CSS into the
|
||||
running page and screenshot it** —
|
||||
`await p.addStyleTag({content: css})` in Playwright against the live
|
||||
dashboard. Seconds per iteration, no deploy. Build + deploy only once the
|
||||
render looks right.
|
||||
|
||||
## The theme — Australis Skyfall
|
||||
|
||||
Operator supplied a Claude Design handoff bundle via the Booth (`26-copper`).
|
||||
Skyfall is a dual-theme OKLCH system: one lightness law across every chromatic
|
||||
family (deep 0.48 / base 0.66 / bright 0.80), all hues cooler than neutral, a
|
||||
Sea neutral ramp drifting ice-blue→ocean-green as it brightens, and a
|
||||
"calm depth" language of **hairline + two-layer shadow on every elevated
|
||||
surface, never one without the other**.
|
||||
|
||||
```
|
||||
theme/colors.css layout.css typography.css vendored VERBATIM from the bundle
|
||||
theme/fonts/Supreme-{400,500,700}.woff2 the body/UI face
|
||||
theme/skyfall.css.in the Homepage bindings (ours)
|
||||
theme/build.py → conf/custom.css (generated — do not hand-edit)
|
||||
```
|
||||
|
||||
The build step exists for one reason: **Homepage serves only `custom.css` and
|
||||
`custom.js` out of its config dir**, with no static route beside them, so a
|
||||
`@font-face` pointing at a vendored `.woff2` would 404 — the face must arrive
|
||||
as a data URI. The background image takes the other road, because
|
||||
`/app/public/images` **is** a real static route (mounted read-only in
|
||||
`compose.yaml`).
|
||||
|
||||
Only Supreme is embedded: a link dashboard has no display type, and Victor
|
||||
Mono ships as 2.4 MB TTF statics per cut — 30x the whole stylesheet for a
|
||||
handful of latency figures.
|
||||
|
||||
## The background is generated, not stock
|
||||
|
||||
**Arbo as an image-gen engine** (the operator's actual ask, which I first
|
||||
misread as "use Arbo's palette" and had to redo). Arbo's `t2i-ui-background`
|
||||
workflow is purpose-built: *"abstract full-bleed backgrounds, no subject"*.
|
||||
Job `13f0891f4e42`, seed 26, flux2-klein-9b, 2048×1152, 1.6 MB PNG → **22 KB
|
||||
WebP** (smooth gradients compress absurdly well).
|
||||
|
||||
⚠️ Arbo API gotcha: `prompt` is a **discriminated union, not a string** — a
|
||||
bare string 422s. `{"kind":"raw","text":…,"negative":…}` is the shape.
|
||||
|
||||
## Two documented deviations from the design system
|
||||
|
||||
1. **Skyfall forbids this background.** Its rule is "flat semantic surfaces; no
|
||||
photography, no textures", with one permitted motif — a subtle aurora
|
||||
gradient on hero/empty-state areas only, *"never behind body text blocks"*.
|
||||
A dashboard is a body-text block. Present on the operator's explicit
|
||||
instruction, mitigated rather than excused: abstract, no subject, strictly
|
||||
cool temperature, held at **`opacity: 30`**. That number is load-bearing —
|
||||
at 14 the aurora was invisible, and turning it up makes the cards fight the
|
||||
ribbon.
|
||||
2. **Service icons stay full-colour vendor logos.** Desaturating them from CSS
|
||||
only makes them illegible.
|
||||
|
||||
## Overcorrection, and the colour pass
|
||||
|
||||
First stat-well pass went from `font-thin` 13px straight to **bold 22px in
|
||||
heading white** — operator: *"went from subtle to BASH YOU OVER THE HEAD."*
|
||||
The principle missed: a stat only has to out-rank **its own label**, not the
|
||||
service name above it. Now `--text-md` medium in cyan.
|
||||
|
||||
Colour was then lifted **from inside the system**: Skyfall names Aurora (blue,
|
||||
cyan, green) the *primary* families, "used generously, in that order", while
|
||||
Dawn (amber/red/violet) is semantic-only. So group markers cycle
|
||||
blue→cyan→green down the page (icons full strength, names at 0.72), service
|
||||
icons take a single cool wash, latency tags move to the info family so
|
||||
"how fast" stops looking like "is it alive". **No Dawn colour is used
|
||||
decoratively anywhere.**
|
||||
|
||||
Two DOM findings that made it possible:
|
||||
|
||||
- **Homepage renders mdi icons as a gradient behind an SVG mask** — recolour
|
||||
via `background`, not `color`.
|
||||
- **Homepage emits `docker-status-<state>`, not `status-<state>`.** The
|
||||
original selectors matched nothing, so every green pill up to that point was
|
||||
stock colouring rather than the theme. Both forms are now matched.
|
||||
@@ -0,0 +1,77 @@
|
||||
# `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken
|
||||
|
||||
Commits `42c594c`, `dc3e47b`, plus `uptimekuma` in `9d92c4b`.
|
||||
|
||||
## The pattern worth remembering
|
||||
|
||||
Chasing two bad-looking cards on the dashboard turned up **four stacks running
|
||||
on fleet hosts with no canonical copy anywhere**: `uptimekuma` and (already
|
||||
known) the two AdGuards on esh-docker-vm, `searxng` and `seafile` on
|
||||
ana-docker, and `heretic2-charrp-reasoning` on ana-ml2 (untracked in git).
|
||||
|
||||
⇒ **A dashboard card is a cheap census of what is actually running.** When
|
||||
something on it looks wrong, check whether the stack behind it is even in
|
||||
`stacks/` before debugging the symptom — twice here the answer was "no", and
|
||||
the fix belonged in version control as much as on the host.
|
||||
|
||||
Adopted: `stacks/uptimekuma/`, `stacks/searxng/`, `stacks/seafile/`,
|
||||
`stacks/heretic2-charrp-reasoning/`. ESH/NH3 AdGuard compose files were
|
||||
**deliberately left unmanaged** — adopting three live resolvers while also
|
||||
introducing a new DNS naming system is two risky changes at once.
|
||||
|
||||
## SearXNG — the healthcheck was eating itself
|
||||
|
||||
Card flapped UNHEALTHY; the container was fine the whole time. The compose
|
||||
passed `--tries` and `--spider` as **two separate argv entries**, so wget
|
||||
consumed `--spider` as the *value* of `--tries`. Spider mode never engaged,
|
||||
which means every probe since April **downloaded** the healthz response to a
|
||||
file:
|
||||
|
||||
```
|
||||
295,287 healthz.N files in the container's working directory
|
||||
```
|
||||
|
||||
With that many files, wget's scan for the next free filename is what
|
||||
intermittently blew the 10s timeout. **Self-worsening — every probe made the
|
||||
next one slower.** Restored `--tries=1`; the junk lived in the writable layer
|
||||
so the recreate cleared it. Now `healthy`, `fails=0`, 200 in 0.16s.
|
||||
|
||||
Lesson: an argv list in YAML has no shell to catch a missing `=`. A flag that
|
||||
silently swallows the next argument turns a liveness probe into a workload.
|
||||
|
||||
## SeaFile — not broken, never restarted
|
||||
|
||||
Card showed EXITED for three months. **None of the three services declared a
|
||||
restart policy**, so Docker defaulted them to `no`. On
|
||||
**2026-05-06T21:27:45Z** the daemon stopped all three within 200ms of each
|
||||
other — a daemon restart or host reboot — and nothing brought them back.
|
||||
|
||||
⚠️ **Exit code `255` is a red herring**: it is what a container that ignores
|
||||
SIGTERM reports when the daemon stops it, **not** evidence of a crash. Reading
|
||||
it as one sends you hunting a bug that does not exist. The tell was all three
|
||||
services stopping within 200ms.
|
||||
|
||||
Added `restart: unless-stopped` to all three; brought up; mariadb gated on its
|
||||
healthcheck exactly as the existing `depends_on` comments intended, seahub
|
||||
started without the race, `302` → login page. Data was in local named volumes,
|
||||
not on the ana-nas NFS, so nothing was at risk.
|
||||
|
||||
Three months of silent downtime whose only signal was a card nobody read as an
|
||||
outage — the argument for semantic status colour on the dashboard (see
|
||||
[[2026-08-19-homepage-skyfall-theme]], where amber EXITED pills made six
|
||||
mis-grouped AI seats obvious at a glance).
|
||||
|
||||
## heretic2-charrp-reasoning — tracked, with its shim
|
||||
|
||||
The `char-rp-reasoning` seat (NEO-CODE Heretic2 27B, modelopt NVFP4 + grafted
|
||||
BF16 MTP head, ~77 tok/s via `qwen3_5_mtp` spec-decode) had been running
|
||||
untracked. Now in `stacks/`, including
|
||||
`conf/mtp-workaround/sitecustomize.py`, which is **not optional**: vLLM 0.24.0
|
||||
does not propagate modelopt `exclude_modules` to the spec-decode **draft**
|
||||
model, so the BF16 MTP head gets quantized and the engine dies at load. Both
|
||||
the mount and `PYTHONPATH` are load-bearing.
|
||||
|
||||
Added the two files house convention expects and the directory lacked — a
|
||||
`.env.example` naming every knob (all values are compose defaults; the host
|
||||
overrides only the three VRAM ones) and a README pointing at
|
||||
`docs/runbooks/heretic2-nvfp4-mtp-seat.md` rather than duplicating it.
|
||||
@@ -0,0 +1,118 @@
|
||||
# `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured
|
||||
|
||||
Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`):
|
||||
a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare
|
||||
`nohup` (PID 1283383) that would not survive a reboot. Now
|
||||
`stacks/waterland-studio/`, `restart: unless-stopped`, healthy on
|
||||
irv-ml1:8410. Commits `a2b5b58`, `8189076`.
|
||||
|
||||
Tracking `main` per operator: PR #4 merged and `main` HEAD is exactly the
|
||||
pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway.
|
||||
|
||||
## Build context lives OUTSIDE the compose dir — on purpose
|
||||
|
||||
`/opt/waterland-studio/src` is the checkout; the Dockerfile is passed
|
||||
out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh`
|
||||
rsyncs `stacks/<stack>/` with `--delete`**, so a checkout kept beside
|
||||
`compose.yaml` would be destroyed by the next deploy of this stack. `update.sh`
|
||||
refreshes source → rebuild → recreate → health, and is verified end to end.
|
||||
|
||||
## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN
|
||||
|
||||
`gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer
|
||||
silently drops to the numpy path at ~21x wall time — it does not error, it
|
||||
just gets slow. waterland-dev warned about the build side.
|
||||
|
||||
The runtime side is worse and was not in the handover: **`studio/jobs.py`
|
||||
shells the renderer out as a literal `uv run waterland ...` with no `--extra`
|
||||
flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project
|
||||
mid-job to its default extras and prunes cupy back out from under a correctly
|
||||
built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the
|
||||
pin ever stops holding the job fails **loudly** instead of quietly rebuilding a
|
||||
slower environment.
|
||||
|
||||
## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare
|
||||
|
||||
Every render died 1.7s in with:
|
||||
|
||||
```
|
||||
RuntimeError: Failed to find CUDA headers.
|
||||
```
|
||||
|
||||
printed **through argparse's usage banner**, which makes it read like a CLI
|
||||
argument bug rather than a missing toolkit. That misdirection is the reason
|
||||
this is written down.
|
||||
|
||||
cupy compiles kernels at runtime through NVRTC, which needs the toolkit
|
||||
**headers** — not just the driver and the runtime libs bundled in the
|
||||
`cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the
|
||||
bare `nohup` process found them **by accident**; a slim image has none.
|
||||
|
||||
Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few
|
||||
hundred MB against ~6 GB for a `-devel` base image. It runs **after**
|
||||
`uv sync`, because sync prunes what it does not know about.
|
||||
|
||||
Reported upstream: it is an undeclared runtime dependency of the `gpu` extra,
|
||||
and anyone running this without a system toolkit hits it.
|
||||
|
||||
## Landmine 3 — the GPU index inside the container is not the host's
|
||||
|
||||
The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects
|
||||
`CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where
|
||||
`nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU
|
||||
(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the
|
||||
container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying
|
||||
the host's value selects a device that does not exist. Host device 0 is the
|
||||
3090, which carries the TTS zoo and must not be touched.
|
||||
|
||||
## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume
|
||||
|
||||
| job | wall |
|
||||
|---|---|
|
||||
| 256² + anim, cold container | 23.3 s |
|
||||
| 256² + anim, warm | **6.1 s** |
|
||||
| 256² plate only (`--codec none`) | 3.9 s |
|
||||
| 512² plate only | 6.4 s |
|
||||
|
||||
Warm beats the **7.4 s** recorded against the bare-metal process, so
|
||||
containerising cost nothing. Verified the cache volume properly: recreate
|
||||
(fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without
|
||||
it every restart makes the next user wait 4x and the service merely *looks*
|
||||
slow.
|
||||
|
||||
## Upstream finding — the on-disk job store grows without bound
|
||||
|
||||
`JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at
|
||||
startup**. Consequences:
|
||||
|
||||
1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and
|
||||
how this was spotted: the API reported **1 job** while the volume held all
|
||||
**16 directories, 60.6 MB**. Not data loss.
|
||||
2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict,
|
||||
so directories orphaned by a restart are **never reclaimed**. The
|
||||
handover's "bounded around 500 MB" holds within a single process lifetime;
|
||||
across restarts the store grows monotonically at ~12 MB per animated job.
|
||||
|
||||
Reported to waterland-dev with evidence; **not patched from the infra side** —
|
||||
it is their code. Prune the volume by hand if it bites first.
|
||||
|
||||
## Access
|
||||
|
||||
Repo is not anonymously readable (a bare clone 403s). Operator granted
|
||||
**`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False,
|
||||
pull: True`. Token on irv-ml1 at
|
||||
`/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a
|
||||
**repo-scoped** credential helper; `.git/config` carries no token (verified),
|
||||
so the remote stays clean in any diff or backup. The operator's `vh`
|
||||
site-admin token was used only for the initial clone and the grant itself and
|
||||
was **never written to disk on that host** — a site-admin credential on a GPU
|
||||
box is a blast radius nobody needs for a read-only fetch.
|
||||
|
||||
## Constraints honoured as stated (not inferred)
|
||||
|
||||
- **Serial by design — one replica, one card.** A render is 20–45s of near-full
|
||||
GPU with a single worker thread. Two on the same A6000 would OOM or thrash.
|
||||
Throughput is a hardware conversation, not a replica-count one.
|
||||
- **No authentication, arbitrary file uploads** ⇒ stays inside the
|
||||
LAN/WireGuard boundary. Do **not** paper over it with a proxy password;
|
||||
waterland-dev offered to add a real auth layer if wider reach is ever needed.
|
||||
+22
-12
@@ -109,7 +109,13 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-08-19 — **ESH fiber (Cityside 2Gb symmetric) is LIVE and the site is fully provisioned on it**; `esh-pve-nas` finished its ZFS-root migration AND took its 225-package backlog (reboot deferred); `ESH↔colo` IPsec rebuilt as dialup+NAT-T after CGNAT broke it; ESH has a `/56` of IPv6. Operator is moving to **Lobe Chat** work next — nothing from this session blocks it. ⚠ This session caused ONE self-inflicted outage (missing `--make-rslave`, see ops-lessons-playbook §1) and three long-dead things surfaced on their own (pvestatd 82d, a vzdump 126d, a VM in prelaunch 4mo)._
|
||||
_As of 2026-08-19 (late) — a long infra session, all of it landed and committed: **fleet `*.internal` DNS is live** (git-sourced, 42 names, three resolvers incl. a NEW colo one), **waterland studio containerised** on irv-ml1, **Homepage cleaned up and themed** with Australis Skyfall + an Arbo-generated background, and **four unmanaged stacks adopted** into `stacks/` (two of which turned out to be quietly broken — SearXNG's healthcheck, SeaFile down 3 months). ⚠ 17 commits unpushed. ⚠ 2 unread althing messages from waterland-dev left deliberately undrained — operator deferred the inbox to after the `/clear`._
|
||||
|
||||
- **🟢 FLEET `.internal` DNS — LIVE 2026-08-19.** `<host>.<site>.internal`, sites `ana`/`esh`/`nh3`. `dns/internal.yaml` is the source of truth; `scripts/dns-sync.py` reconciles the three AdGuard resolvers (diff → prompt → apply, idempotent). 42 names resolving from all three sites. **Colo got its first resolver ever** (`stacks/adguard-ana/`, API on **8053** not 8080, no blocklists by design) — before this, ana-docker resolved straight against `1.1.1.1`. Auth = a dedicated `infra-ops` AdGuard user, password vaulted `nh3-dev/adguard-infra-ops-password`. **⏳ TWO OPEN, both operator's to schedule:** (a) colo hosts still point at `1.1.1.1` so they do not yet *use* the new resolver — repointing a site's DNS is a separate change; (b) the static-v6 convention (each server at its site's `/64` with low bits echoing the v4 octet, `esh-docker-vm` → `…::45`) is **proposed, not ruled on**. The `v6:` column is empty and correct — no fleet host has a global v6 address yet. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md`
|
||||
|
||||
- **🟢 WATERLAND STUDIO — containerised + live, irv-ml1:8410.** Displaced a bare `nohup` (PID 1283383). Warm 256²+anim render **6.1s**, beating the 7.4s bare-metal figure. `claude-bot` now has **read** on `vh/waterland` (operator-granted); token on irv-ml1 root-owned 0600, repo-scoped helper, nothing in `.git/config`. **⏳ Reported upstream, NOT patched by us:** the on-disk job store grows without bound across restarts (`RETAIN=40` eviction only sees the in-memory dict). → `persistent-memory.d/2026-08-19-waterland-studio-containerised.md`
|
||||
|
||||
- **🟢 HOMEPAGE — cleaned + themed (Australis Skyfall).** Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by **Arbo** (`t2i-ui-background`, job `13f0891f4e42`). ⚠ **After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own — do not chase it.** ⚠ CSS is served per-request: a theme change needs a **reload**, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md`
|
||||
|
||||
- **🟢 GEN SEAT — SWAPPED to `absolute-heresy` 2026-08-17 (validated, promoted).** Live gen = `/tank/aimodels/qwen38-27b-heresy-nvfp4-mixed` — **MuXodious/Qwen3.8-27B-absolute-heresy** (Heretic v1.4.0 + **SOMPOA**, trial T377, pin `c2374593`) put through our own mixed NVFP4+FP8 recipe. Chosen because it beats the incumbent on **both** axes at once: author refusals 2/101 vs 12/100, first-token KL 0.0759 vs 0.1191. **Gate (probe :8017, pinned nightly, seat-matched flags): MTP 47.2% (inc. 48.2%), decode 103.5 tok/s (96.4), prefill 6618/5403 @6.7k/27k (6334/5085), PPL 6.910 (7.059 — 2.1% BETTER), surface 6/6, abliteration 4/4, and 0/55 refusals on our battery-instruct arm with ZERO EMPTY (no catatonia).** ⚠ speed deltas are **image-confounded** (probe on the pinned nightly, incumbent numbers from an earlier image) — read as "not worse", not a clean win. All 7 LiteLLM aliases verified end-to-end; GPU0 at 91.3/97.9 GB with meromero healthy (more headroom than the old build's 96.8). ⚠ **RC1, 2 days old, ~348 downloads.** **Operator-confirmed "working very well" in real use 2026-08-17**, same evening as the cutover — the signal the synthetic gates structurally cannot give (multi-turn degeneration is stochastic; four synthetic tests once validated three non-fixes). Not yet the 60k-token bar the prior seat cleared, so **keep watching and do NOT delete the rollback weights yet**. **ROLLBACK:** `sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-20260817 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen`; incumbent weights UNTOUCHED at `qwen38-27b-uncensored-nvfp4-mixed` — **do NOT delete** until this holds. Runbook `services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md`.
|
||||
|
||||
@@ -154,10 +160,22 @@ _As of 2026-08-19 — **ESH fiber (Cityside 2Gb symmetric) is LIVE and the site
|
||||
|
||||
- **althing monitor** ARMED (handle `infra-ops`). ⚠️ Re-arm ONLY after a real FIRE (rc0), never after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, never chained with `&` (orphans it — hit this twice 2026-08-17, `stop-monitor` reclaims).
|
||||
|
||||
- **eshpfi push state:** operator pushes **manually**; last push `50d13f5..8be8a51`. ⚠ **7 commits unpushed as of 2026-08-19** (`707a8cb`..this snapshot) — all docs/memory from the v6 investigation. Push over the INTERNAL gitea route: `git push ssh://git@10.250.50.70:222/vh/esh-pfi-infrastructure.git main:main` (`origin` resolves the public edge which fail2bans fleet-host egress, and stays stale locally as a result). `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED.
|
||||
- **eshpfi push state:** operator pushes **manually**; last push `50d13f5..8be8a51`. ⚠ **17 commits unpushed as of 2026-08-19** (`707a8cb`..`b8003c7`, plus this snapshot commit) — nothing is on the remote from this session.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-08-19]` **Fleet `.internal` DNS built and live — git-sourced, agent-managed, three resolvers.** Zone-scoped authority (ESH's hand-made `esteban.net` rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md`
|
||||
|
||||
- `[2026-08-19]` **waterland studio containerised on irv-ml1 — three landmines, all measured.** cupy needs CUDA *headers* the host had by accident; `uv run` re-syncs and prunes cupy at RUNTIME; the A6000 is container-index 0, not the host's 1. → `persistent-memory.d/2026-08-19-waterland-studio-containerised.md`
|
||||
|
||||
- `[2026-08-19]` **Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background.** Includes the hour lost to a self-healing tab-bar red herring, and the CSS-iteration loop that prevents it recurring. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md`
|
||||
|
||||
- `[2026-08-19]` **Four unmanaged stacks found on live hosts — two quietly broken.** A dashboard card is a cheap census of what is actually running; check whether the stack is even in `stacks/` before debugging the symptom. → `persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md`
|
||||
|
||||
- `[2026-08-19]` **`claude-bot` granted read on `vh/waterland`** (operator-empowered, verified `admin:false push:false pull:true`) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit `8189076`.
|
||||
|
||||
- `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
|
||||
|
||||
- `[2026-08-18]` **esh-pve-nas migration STAGED — and staging is where three landmines surfaced, none of which the plan predicted.** (1) The runbook's `/boot` LV had **nowhere to live**: VG `pve` had 4 MB free and mounted ext4 cannot shrink, so the space came from the 768 MB swap LV (operator's call: shrink to 256 MB, not drop). (2) The runbook's `zpool set cachefile=… nvme` would have **broken the NAS** — populating a cache flips the host to import-by-cache, and a one-pool cache leaves `ssd`+`tank` unimported under CT 103's twelve bind mounts. (3) **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`**, because GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` and the probe failure is swallowed. All three were caught by *verify steps that asserted effective state*, not by reading the plan. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`
|
||||
|
||||
- `[2026-08-17]` **esh-pve-nas PVE root is on a USB DOM — mitigated, and the migration replanned to split boot from root.** Operator's design beats my reinstall plan; wear was never the issue, blocked patching is. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`
|
||||
@@ -271,15 +289,11 @@ _As of 2026-08-19 — **ESH fiber (Cityside 2Gb symmetric) is LIVE and the site
|
||||
- `[2026-07-31]` **worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred.** memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on `vh/worldtree-sdk` (source pulled via the **Gitea API archive** — git-HTTP 403s on that repo); publishing to the vh USER namespace **can't be delegated** (401 `reqPackageAccess` even with `write:package`) so it needs an owner token — operator saved a **FULL vh site-admin token at `~/.config/gitea/vh-token` (0600)** for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
|
||||
|
||||
|
||||
- `[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap.** heid's cross-frontier panel was silently degraded (empty content, `finish_reason: stop`). dvalin+bil researched (docs said deprecated-max_tokens); heid's live data refuted that (completion hit 18455) → it's a reasoning ceiling. **Proven on the wire against heid's real 500KB bundle:** `reasoning_effort: low` drops reasoning under the ceiling → content returns, on BOTH coding + general endpoints. Fix is CALLER-side (no gateway change): send `reasoning_effort` via **`extra_body`** (LiteLLM `drop_params: true` strips the top-level param — why heid's earlier attempt no-op'd). Relayed to heid to validate; backstop = `allowed_openai_params` on the route. → `persistent-memory.d/2026-07-31-kimi-k3-reasoning-cap.md`
|
||||
|
||||
|
||||
|
||||
|
||||
- `[2026-07-25]` **bil-smithy-dev wired as an althing zellij-window-ping (pane route).** She's a `driver: human` dwarf peer (pane `bil-smithy` already live alongside eitri/dvalin/regin-smithy in the `Claude` zellij session) but had no delivery route → smoke messages posted to the bus but never reached her window. **Mechanism (reusable for any pane-route handle):** `~/.althing/config.yaml` → `zellij_sessions.Claude.agents[]` maps `handle` → `target` (a zellij pane **TITLE**, matched via `list-panes -j` in `althing/zellij.py:resolve_pane_id`) → `command` (herald `write-chars` + CR into that pane). The **herald loads config ONCE at startup** (`herald.py main()`), so **`systemctl --user restart althing-herald.service`** after editing. Added bil (`target: bil-smithy`), restarted, verified: herald delivered the pending smoke `01KYD7W7CF…` (available→attempted→**delivered**). ⚠️ Noticed pre-existing pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range", empty msg_ids — likely `render_command messages[0]` on an empty list; NOT caused by this change, bil works) — worth a herald look.
|
||||
|
||||
|
||||
_204 older entries archived to archival-memory.md._
|
||||
_206 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
@@ -291,8 +305,6 @@ _204 older entries archived to archival-memory.md._
|
||||
- `[2026-08-03]` **corviduo-dev shared containerd: a concurrent-pull race fails ONE instance's deploy; DON'T "prune to fix" — the image is in-use by the instance that won the race.** b169 personal deploy failed at `docker compose pull` (`Lchown … no such file or directory` on the big torch layer → looked like disk pressure / corrupt snapshot). ACTUAL: NOT disk (56G free, inodes 7%). demo + personal + pinned share ONE `/var/lib/containerd` on corviduo-dev; demo (from main) and personal (from staging tag) extracted b169's shared torch layer simultaneously → personal's hit a partial snapshot mid-race and aborted while demo's completed. The image `6e34a87` was FULLY VALID — demo was RUNNING it healthy. Fix = just re-run the failed deploy (image already materialized; compose pull finds it present). **NEAR-MISS:** worldtree-dev's suggested "prune unused images/snapshots" would have rmi'd `6e34a87` = the image the running demo depends on → demo outage. **Lesson: before any prune/rmi "cleanup," `docker ps` the running images — an "unused" image may be a co-tenant's live one; and verify the failure's REAL cause (disk? inode? in-use? race?) before applying the suggested remedy.** (Pipeline fix, deferred: serialize demo-from-main + personal-from-staging, or a per-image pull lock, to avoid the shared-layer extraction race.)
|
||||
|
||||
|
||||
- `[2026-08-02]` **`docker exec` into worldtree containers defaults to ROOT — root writes contaminate the uid-1000 (vh) KB tree.** My `sudo docker exec … --reindex` on personal ran as ROOT (muninn app = uid 1000); its wing git-commit + atomic note-swap left root-owned files in the `worldtree-personal_worldtree-kb` volume: a root-owned `.old-<job>` backup dir (blocked the uid-1000 retry's `rmtree` → Errno 13, because unlink needs write on the DIR and it was root:root 755) AND **60 root-owned loose git objects** in `.git/objects/`. Fix (host-side, corviduo-dev): `sudo rm -rf` the superseded `.old-` dir (tar'd aside to /tmp first) + `sudo find … -user 0 -exec chown 1000:1000` the objects (ownership-only, git-content-safe; the `.git/objects/XX/` dirs were vh-owned so these weren't a hard blocker, but violated "clean tree"). **RUNBOOK RULE (worldtree-dev, ADOPTED):** any `docker exec` into worldtree containers that WRITES pipeline state runs **`-u 1000`**, never default-root — same genus as the mv footgun (acting without matching the target's constraints; 3rd such slip in one session). **GOTCHA that hid the scope:** `find … -user 0 | head -20` TRUNCATED (the `.old-` dir alone had 153 files, so the first page was all `.old-`) → I "verified clean" off a partial list. Never `head` a scope-defining find; count first (`| wc -l`). **Related blind-spot (muninn-dev):** a root-owned job SUBDIR passes every requeue guard (job_row/dispatch/list_jobs render fine) AND `/health` (contract's `os.access(ingestion_root, W_OK)` tests only the ROOT dir, so a foreign-owned subdir under `pending/` still reports `ingestion_root_writable: true`) — then the uid-1000 gate can't write into it. "Clean board + green /health + failure at next mutation." muninn-dev added an OWNERSHIP column to the standing post-move check to catch it; two green signals both miss a foreign-owned subdir otherwise.
|
||||
|
||||
|
||||
- `[2026-08-02]` **`mv <job> complete/ → failed/` RENAMED the job to `failed` because failed/ didn't exist.** worldtree-dev's round-2 unblock command (`mv /data/state/ingestion/complete/<job> /data/state/ingestion/failed/`) assumed `failed/` existed; on PERSONAL muninn it did NOT (fresh instance — root was `active/ complete/ pending/ sources/`, no `failed/`). `mv src nonexistent/` **renames** src→nonexistent, so job1 became the `failed` dir and job2 nested inside it. Caught on post-move `ls` (failed/ held job *contents*, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt `failed/` (worldtree:worldtree 755) — NO data loss. **Lessons:** (1) before `mv X into-dir/`, verify the dir EXISTS (`[ -d dir ]`) — an empty `ls dir/ 2>/dev/null` is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is **`mv -t <targetdir> <src>`** (`--target-directory`): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a **trailing slash does NOT protect** — `mv src failed/` with `failed/` missing STILL silently renames to `failed` (rc=0); "just add the slash" is a false guard. (`mkdir -p failed/` first also works, but `mv -t` inverts the failure from silent-wrong to loud-safe in one flag.) Container `sh` is dash — no `(` in echo strings. **SILENT failure mode (muninn-dev carry-forward):** a misplaced ingestion-state move doesn't crash anything — `list_jobs()` stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (`job_row`→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (`job_row` found + guards pass), don't trust mv exit codes — and confirm `job.dispatch.json` survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. **DON'T TIDY (round-2 pending):** both DCC + P&P jobs currently REST in personal `failed/` with manifests reading `state: complete` until round-2 requeue runs — deliberate + load-bearing (`requeue` keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. **Round-2 sequencing:** the requeue is **mimir-dev's** browser flow (pending their operator's board-vs-API ruling); **muninn-dev** is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); **infra-ops** = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z. **ARC COMPLETE (2026-08-03 ~05:49):** both books terminal — DCC `mimir-6351554e8e8f` 705 concepts + P&P `mimir-f3887c9b97b7` 667, extracted AND indexed, 5/5 phases, 0 failures/truncations (validates the #385 budget fix vs April's 785 control); **#381 restart-after-ingest FIRED** (personal api, healthz/readyz 200 ~25s), retrieval-visibility confirmed (search_library returns DCC+P&P from fiction post-restart); handed ratatoskr-verify go to worldtree-dev. **Delete-sweep precondition NOW MET** — the stale DCC rows in `main` are genuine duplicates of live `fiction` rows, so worldtree-dev's supervised sweep of the ~785 April orphans is unblocked (still comes to me supervised: snapshot + operator-in-loop).
|
||||
|
||||
@@ -305,8 +317,6 @@ _204 older entries archived to archival-memory.md._
|
||||
|
||||
|
||||
|
||||
- `[2026-07-25]` **Chaining the althing wake-listener arm orphans it.** `reply && althing-wake-listener &` (or spawning `althing-wake-listener` with `&` *inside* a `run_in_background` task) → the `&`-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a *plain operator turn* (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn `althing-wake-listener` as its OWN `run_in_background` task, and re-arm ONLY after a real fire (`<task-notification> completed rc0`). Reclaim an orphan with `althing-cli stop-monitor` then re-arm.
|
||||
|
||||
|
||||
|
||||
_139 older entries archived to archival-memory.md._
|
||||
_141 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user