From bc3aada73a143cc60144f10b17731bd2d3d40910 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 19 Aug 2026 01:20:53 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20.internal=20DN?= =?UTF-8?q?S=20live,=20waterland=20containerised,=20homepage=20themed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures a long infra session: fleet *.internal DNS (git-sourced, 42 names, three resolvers including a new colo one), waterland studio containerised on irv-ml1, Homepage cleaned up and themed with Australis Skyfall over an Arbo-generated background, and four unmanaged stacks adopted into stacks/. Four detail files added. Auto-archived 4 entries to archival-memory.md (Recent decisions 2, Tried and abandoned 2); 5 held back by the open-deferred guard rather than moved. Also records three operator-owned open items: the colo DNS repoint, the static-v6 convention, and the deliberately belayed AI-tab Dormant regrouping. --- archival-memory.md | 30 ++++ .../2026-07-31-kimi-k3-reasoning-cap.md | 15 -- .../2026-08-19-fleet-internal-dns.md | 119 ++++++++++++++ .../2026-08-19-homepage-skyfall-theme.md | 145 ++++++++++++++++++ ...-08-19-unmanaged-stacks-searxng-seafile.md | 77 ++++++++++ ...26-08-19-waterland-studio-containerised.md | 118 ++++++++++++++ persistent-memory.md | 34 ++-- 7 files changed, 511 insertions(+), 27 deletions(-) delete mode 100644 persistent-memory.d/2026-07-31-kimi-k3-reasoning-cap.md create mode 100644 persistent-memory.d/2026-08-19-fleet-internal-dns.md create mode 100644 persistent-memory.d/2026-08-19-homepage-skyfall-theme.md create mode 100644 persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md create mode 100644 persistent-memory.d/2026-08-19-waterland-studio-containerised.md diff --git a/archival-memory.md b/archival-memory.md index 26b2ba4..f0c8266 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -2465,3 +2465,33 @@ _Archived 2026-08-18._ - `[2026-07-30]` **magpie TTS serving — evaluated, ABANDONED.** Pulled `magpie_tts_multilingual_357m` (the one real repo of brokkr's batch) to NFS, stood it up on irv-ml1 (ephemeral NeMo-Speech-`main` container — stock PyPI/NGC NeMo can't load v2607), A/B'd vs Zonos → Zonos wins expressive English decisively, multilingual not needed. Not served; `magpie-nemo` torn down. `.nemo` KEPT on NFS as brokkr's fine-tuning base. auto-memory `project_magpie_tts_eval_rejected`. _Archived 2026-08-18._ + +## Recent decisions (archived 2026-08-19 batch) + +`[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap; fix relayed to heid, NOT applied gateway-side.** + +heid reported that `kimi-k3` (the primary route = Kimi Code coding endpoint `openai/k3` @ `api.kimi.com/coding/v1`) silently degraded its cross-frontier panel: on large/reasoning-heavy dispatches, `completion_tokens: 16381` **exactly**, `content` empty, `reasoning_content` ~64KB, `finish_reason: **stop**` (a truncation mislabeled as a clean stop). `max_tokens: 100000` in the request was not honored. + +**Investigation arc (a clean cross-frontier-triage + verify-on-the-wire case):** +1. My first read: a flat ~16384 OUTPUT cap; fix = a LiteLLM `stop→length` relabel callback (heid's fallback ask). Confirmed the cap isn't in our LiteLLM config (no `max_tokens` clamp on the route). +2. Operator routed a fix-research pass to **dvalin-smithy-dev + bil-smithy-dev** (independent). Both CONVERGED (docs-based): `max_tokens` is a deprecated alias on Kimi/Moonshot; the canonical field is `max_completion_tokens` (default 131072, max 1M); the coding endpoint defaults output to 16384; fix = send `max_completion_tokens` + `reasoning_effort` via `extra_body` (drop_params-safe). +3. **heid's live data REFUTED the docs hypothesis:** a later dispatch hit `completion_tokens: 18455` (ABOVE 16384) cleanly, with `reasoning_tokens: 16198` (just under 16384) and content present. So COMPLETION is uncapped; the bound is on **REASONING at ~16384**. When a hard task's thinking exhausts that budget, nothing's left for content → empty answer under `stop`. +4. **I proved it on the wire** — ran heid's real 500KB failing bundle direct at both endpoints (bypassing LiteLLM so `reasoning_effort` isn't dropped): default effort → 504/timeout (the failure); **`reasoning_effort: low` → reasoning ~12–13.5k (under the ceiling), content returns (6–7.6k chars)**, on BOTH coding AND general endpoints. So re-routing to the general endpoint buys nothing — the fix is the effort param, and it works on the wire. + +**THE FIX (caller-side, no shared-gateway change/restart):** send `reasoning_effort` via **`extra_body`** on kimi-k3 dispatches (`low` for large bundles). LiteLLM `drop_params: true` strips the top-level `reasoning_effort` — which is exactly why heid's earlier `reasoning_effort: low` was a no-op. `extra_body` survives drop_params (the house GLM-thinking pattern). Tradeoff: low effort = shallower reasoning, but a complete answer beats today's empty one. + +**Relayed to heid to validate on a real round** (the one unconfirmed hop is whether `extra_body` survives OUR LiteLLM). **Backstop if it doesn't:** add `allowed_openai_params: ["reasoning_effort"]` to the `kimi-k3` route in the gateway config — that IS a shared-gateway change + a ~10s restart (blips all consumers), so it needs a heads-up. + +Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/litellm/conf/config.yaml` (see Recent-decisions `[2026-07-25]` Kimi K3 wiring). No gateway change was made this session. Failing dispatch on record: `01KYTASKTY3T` (jackdaw-dev bug-hunt). + _Archived 2026-08-19._ + +- `[2026-07-25]` **bil-smithy-dev wired as an althing zellij-window-ping (pane route).** She's a `driver: human` dwarf peer (pane `bil-smithy` already live alongside eitri/dvalin/regin-smithy in the `Claude` zellij session) but had no delivery route → smoke messages posted to the bus but never reached her window. **Mechanism (reusable for any pane-route handle):** `~/.althing/config.yaml` → `zellij_sessions.Claude.agents[]` maps `handle` → `target` (a zellij pane **TITLE**, matched via `list-panes -j` in `althing/zellij.py:resolve_pane_id`) → `command` (herald `write-chars` + CR into that pane). The **herald loads config ONCE at startup** (`herald.py main()`), so **`systemctl --user restart althing-herald.service`** after editing. Added bil (`target: bil-smithy`), restarted, verified: herald delivered the pending smoke `01KYD7W7CF…` (available→attempted→**delivered**). ⚠️ Noticed pre-existing pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range", empty msg_ids — likely `render_command messages[0]` on an empty list; NOT caused by this change, bil works) — worth a herald look. + _Archived 2026-08-19._ + +## Tried and abandoned (archived 2026-08-19 batch) + +- `[2026-08-02]` **`docker exec` into worldtree containers defaults to ROOT — root writes contaminate the uid-1000 (vh) KB tree.** My `sudo docker exec … --reindex` on personal ran as ROOT (muninn app = uid 1000); its wing git-commit + atomic note-swap left root-owned files in the `worldtree-personal_worldtree-kb` volume: a root-owned `.old-` backup dir (blocked the uid-1000 retry's `rmtree` → Errno 13, because unlink needs write on the DIR and it was root:root 755) AND **60 root-owned loose git objects** in `.git/objects/`. Fix (host-side, corviduo-dev): `sudo rm -rf` the superseded `.old-` dir (tar'd aside to /tmp first) + `sudo find … -user 0 -exec chown 1000:1000` the objects (ownership-only, git-content-safe; the `.git/objects/XX/` dirs were vh-owned so these weren't a hard blocker, but violated "clean tree"). **RUNBOOK RULE (worldtree-dev, ADOPTED):** any `docker exec` into worldtree containers that WRITES pipeline state runs **`-u 1000`**, never default-root — same genus as the mv footgun (acting without matching the target's constraints; 3rd such slip in one session). **GOTCHA that hid the scope:** `find … -user 0 | head -20` TRUNCATED (the `.old-` dir alone had 153 files, so the first page was all `.old-`) → I "verified clean" off a partial list. Never `head` a scope-defining find; count first (`| wc -l`). **Related blind-spot (muninn-dev):** a root-owned job SUBDIR passes every requeue guard (job_row/dispatch/list_jobs render fine) AND `/health` (contract's `os.access(ingestion_root, W_OK)` tests only the ROOT dir, so a foreign-owned subdir under `pending/` still reports `ingestion_root_writable: true`) — then the uid-1000 gate can't write into it. "Clean board + green /health + failure at next mutation." muninn-dev added an OWNERSHIP column to the standing post-move check to catch it; two green signals both miss a foreign-owned subdir otherwise. + _Archived 2026-08-19._ + +- `[2026-07-25]` **Chaining the althing wake-listener arm orphans it.** `reply && althing-wake-listener &` (or spawning `althing-wake-listener` with `&` *inside* a `run_in_background` task) → the `&`-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a *plain operator turn* (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn `althing-wake-listener` as its OWN `run_in_background` task, and re-arm ONLY after a real fire (` completed rc0`). Reclaim an orphan with `althing-cli stop-monitor` then re-arm. + _Archived 2026-08-19._ diff --git a/persistent-memory.d/2026-07-31-kimi-k3-reasoning-cap.md b/persistent-memory.d/2026-07-31-kimi-k3-reasoning-cap.md deleted file mode 100644 index 27be84e..0000000 --- a/persistent-memory.d/2026-07-31-kimi-k3-reasoning-cap.md +++ /dev/null @@ -1,15 +0,0 @@ -`[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap; fix relayed to heid, NOT applied gateway-side.** - -heid reported that `kimi-k3` (the primary route = Kimi Code coding endpoint `openai/k3` @ `api.kimi.com/coding/v1`) silently degraded its cross-frontier panel: on large/reasoning-heavy dispatches, `completion_tokens: 16381` **exactly**, `content` empty, `reasoning_content` ~64KB, `finish_reason: **stop**` (a truncation mislabeled as a clean stop). `max_tokens: 100000` in the request was not honored. - -**Investigation arc (a clean cross-frontier-triage + verify-on-the-wire case):** -1. My first read: a flat ~16384 OUTPUT cap; fix = a LiteLLM `stop→length` relabel callback (heid's fallback ask). Confirmed the cap isn't in our LiteLLM config (no `max_tokens` clamp on the route). -2. Operator routed a fix-research pass to **dvalin-smithy-dev + bil-smithy-dev** (independent). Both CONVERGED (docs-based): `max_tokens` is a deprecated alias on Kimi/Moonshot; the canonical field is `max_completion_tokens` (default 131072, max 1M); the coding endpoint defaults output to 16384; fix = send `max_completion_tokens` + `reasoning_effort` via `extra_body` (drop_params-safe). -3. **heid's live data REFUTED the docs hypothesis:** a later dispatch hit `completion_tokens: 18455` (ABOVE 16384) cleanly, with `reasoning_tokens: 16198` (just under 16384) and content present. So COMPLETION is uncapped; the bound is on **REASONING at ~16384**. When a hard task's thinking exhausts that budget, nothing's left for content → empty answer under `stop`. -4. **I proved it on the wire** — ran heid's real 500KB failing bundle direct at both endpoints (bypassing LiteLLM so `reasoning_effort` isn't dropped): default effort → 504/timeout (the failure); **`reasoning_effort: low` → reasoning ~12–13.5k (under the ceiling), content returns (6–7.6k chars)**, on BOTH coding AND general endpoints. So re-routing to the general endpoint buys nothing — the fix is the effort param, and it works on the wire. - -**THE FIX (caller-side, no shared-gateway change/restart):** send `reasoning_effort` via **`extra_body`** on kimi-k3 dispatches (`low` for large bundles). LiteLLM `drop_params: true` strips the top-level `reasoning_effort` — which is exactly why heid's earlier `reasoning_effort: low` was a no-op. `extra_body` survives drop_params (the house GLM-thinking pattern). Tradeoff: low effort = shallower reasoning, but a complete answer beats today's empty one. - -**Relayed to heid to validate on a real round** (the one unconfirmed hop is whether `extra_body` survives OUR LiteLLM). **Backstop if it doesn't:** add `allowed_openai_params: ["reasoning_effort"]` to the `kimi-k3` route in the gateway config — that IS a shared-gateway change + a ~10s restart (blips all consumers), so it needs a heads-up. - -Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/litellm/conf/config.yaml` (see Recent-decisions `[2026-07-25]` Kimi K3 wiring). No gateway change was made this session. Failing dispatch on record: `01KYTASKTY3T` (jackdaw-dev bug-hunt). diff --git a/persistent-memory.d/2026-08-19-fleet-internal-dns.md b/persistent-memory.d/2026-08-19-fleet-internal-dns.md new file mode 100644 index 0000000..fb9baeb --- /dev/null +++ b/persistent-memory.d/2026-08-19-fleet-internal-dns.md @@ -0,0 +1,119 @@ +# `[2026-08-19]` Fleet `.internal` DNS — git-sourced, agent-managed, three resolvers + +Operator: *"with ipv6 i can't memorize the IP addresses anymore. need a way to +keep track of local .internal dns names that can be agent managed and is +lightweight."* Built and live in one session; commit `b8003c7`. + +## Shape + +``` +dns/internal.yaml source of truth — 38 hosts + 4 service aliases +scripts/dns-sync.py reconciles AdGuard resolvers against it +stacks/adguard-ana/ the colo's resolver, which did not exist +dns/README.md workflow, naming, the IPv6 caveat +``` + +Deliberately the same posture as `deploy-stack.sh`: the file is intent, the +resolvers are derived state, you see a diff before anything changes. +`--dry-run` / `--yes` / `--site `. Verified idempotent — a second run prints +`nothing to do`. + +Naming is `..internal` with sites **`ana` / `esh` / `nh3`** +(operator's call). `.internal` is ICANN-reserved for private use since 2024; +`.local` is reserved for mDNS, which is why the pre-existing +`searxng.pfi.local` was a standards collision that merely happened to work. + +Every name is published to **every** resolver — the site label says where a +host *is*, not which resolver knows about it. + +## The framing correction that mattered most + +The ask reads as "I can't memorise v6 addresses", but the deeper problem is +that **v6 addresses are derived, not assigned**, so they cannot reliably be +*written down once* either. SLAAC gives EUI-64 (MAC-coupled) or +privacy-extension (rotating) addresses, and UniFi has **no v6 equivalent of a +DHCP reservation** — so a hand-maintained v6 table rots on its own. + +⇒ The fix has two halves and only the second is DNS: (1) pin static v6 on +server-class hosts, (2) then the name table is just a file. Surfaced to the +operator before building. + +**Verified 2026-08-19: no fleet host has a global v6 address at all yet** — +ESH's `/56` is live only on `esh-cameras`, NH3's LANs are back to +`ipv6_interface_type: none`, the colo has none. So the `v6:` column ships +EMPTY and correct, and the naming layer was built first rather than blocking +on v6. Names established now need no renaming when addresses land. + +Suggested convention when they do (awaiting operator): each server static at +its site's `/64` with low-order bits echoing the v4 host octet — +`esh-docker-vm` at `…::45` — so addresses are declarable *and* semi-memorable. + +## Two properties not to break + +**Authority is scoped to the ZONE, not the resolver.** Only rewrites ending in +`.internal` are managed. ESH's resolver turned out to carry three hand-made +`esteban.net` rewrites (`eshnas`, `brotherprinter`, `eshhome`) — **my first +read of the config missed them**, because an `awk` range on `rewrites:` matched +an empty-looking block. A resolver-wide authoritative sync would have silently +deleted all three on first run. Verified intact after sync. + +**Within `.internal` it IS authoritative** — names added by hand in the AdGuard +UI get deleted by the next sync. That is the point: one place to look. + +## The colo had no resolver at all + +ESH and NH3 each ran AdGuard; **ana-docker resolved straight against +`1.1.1.1`**, so the colo had no way to answer for internal names. Closed with +`stacks/adguard-ana/`. + +⚠️ Its API is on **8053**, not 8080 — `:8080` and `:3000` were already taken on +that busy host. The port is therefore carried **per-site in the yaml**, not +assumed by the script, so the odd one out cannot be forgotten. + +⚠️ It ships with **no blocklists**, deliberately. The other two filter ads for +human browsing; this one resolves for a rack of servers, where a blocklist +false-positive breaks service-to-service calls at 3am for no upside. + +First boot uses a **seed config** (`conf/AdGuardHome.seed.yaml`) copied into +the conf volume before first start, so the container comes up configured +instead of sitting in the setup wizard. + +## Credential — service account, not the operator's + +Added a dedicated **`infra-ops`** AdGuard user to all three resolvers rather +than asking for the `lkraven` password (per the standing migrate-off-operator- +creds directive). Password vaulted at +`nh3-dev/adguard-infra-ops-password`; `lkraven` untouched; pre-change configs +backed up on each host as `AdGuardHome.yaml.bak-preinfraops-*`. Both existing +resolvers kept answering across the restart. + +Two landmines worth keeping: + +- **Go's bcrypt rejects `htpasswd`'s `$2y$` prefix.** Same algorithm, different + marker; `golang.org/x/crypto/bcrypt` accepts only `$2a$`/`$2b$`. Normalise + the prefix, and self-verify the hash with `htpasswd -vb` BEFORE installing it + on a live resolver. +- **The vault appends a trailing newline on `get`.** A password carrying a + stray `\n` fails auth in a way that looks exactly like a wrong password. + `dns-sync.py` strips it. + +## `pfi.local` migration — and the one that must NOT move + +`searxng.pfi.local` → `searxng.ana.internal`, with the **old `Host()` kept +alongside** in the Traefik rule so nothing breaks mid-migration; both return +200. Drop the fallback once the access log shows the old name unused. + +**`matrix.pfi.local` deliberately NOT migrated.** A Matrix `server_name` is +baked into every user ID, room ID and signing key, and federation identity +derives from it — renaming it is not a DNS change, it is rebuilding the +homeserver's identity and invalidating its history. The operator approved +"migrate pfi.local" generally; this was surfaced as a deliberate exclusion +rather than executed blindly. + +## Still open + +Colo hosts still point at `1.1.1.1`, so they do not yet *use* the new resolver +— it only answers what asks it directly. Repointing a whole site's DNS is a +bigger change than standing the service up, and is the operator's to schedule. + +See also [[2026-08-17-fleet-ipv6-mesh]]. diff --git a/persistent-memory.d/2026-08-19-homepage-skyfall-theme.md b/persistent-memory.d/2026-08-19-homepage-skyfall-theme.md new file mode 100644 index 0000000..682d1ac --- /dev/null +++ b/persistent-memory.d/2026-08-19-homepage-skyfall-theme.md @@ -0,0 +1,145 @@ +# `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background + +Commits `9d92c4b`, `c3de7db`, `45c1995`, `f38cf69`, `df68dd2`. + +## The cleanup (three real defects) + +- **UltraSeedbox rendered on all four tabs.** The bookmark group had no entry in + `settings.yaml`'s `layout:` block at all, and Homepage's documented behaviour + is that a group with no `tab:` is shown on **every** tab. Pinned to Main. + ⚠️ This will happen again to the next group added without a `tab:` — the rule + is now written at the top of the layout block. +- **Uptime Kuma rendered twice** — a manual `services.yaml` entry under + Monitoring *and* `homepage.group=Apps` on the container. Exactly the + "never list a labelled container manually" failure the stack README warns + about; it survived the previous day's audit because a duplicate reads as two + plausible cards rather than as an error. Manual block deleted, label moved to + `Monitoring`, `homepage.siteMonitor` added. +- **Column counts were fiction** — several groups declared more columns than + they had members, so the last row of each was dead space (Notes: 1 card in a + 4-wide row). Columns now track member counts; `GET /api/services` prints the + live per-group counts and is the check. + +Later, on operator instruction, the **AI tab was reordered by clickability**: +Gateways & Chat → Image & Media → Audio Tools on top, then the vLLM `/docs` +seats and TTS endpoints. Reasoning written into the config so it survives: +order by "would I click this?", not by how central the service is. + +## ⚠️ The expensive red herring — the tab bar after a recreate + +After a recreate the client render comes up with **no tab bar, no wallpaper and +no i18n** (search box shows the raw key `search.search`), groups falling back to +side-by-side columns. **It restores itself with no intervention.** + +Timing, measured rather than assumed: a fresh container was still tab-less at +**4m30s, twice**; it was healthy again after roughly an hour. `docker ps` +reporting `healthy` says nothing about it — the container is serving, the page +is just wrong. + +An hour went into ruling out four causes that were never the cause: + +1. **Not the config** — restoring `settings.yaml` *and* `services.yaml` to + their committed versions reproduces it, as does the pre-adoption backup in + `/opt/docker-bu/conf/homepage/`. +2. **Not the v2.0.0 release** — a throwaway container on `v1.13.2` shows + identical symptoms, and the image never changed anyway (working and broken + both report `v2.0.0` / rev `17456f2`). +3. **Not `PUID`/`PGID`**, and not Docker discovery — tested both, and with the + socket unmounted entirely. +4. **Not server-side** — the server-rendered HTML still contains the tab + markup, the background URL and `useEqualHeights`; `GET /api/validate` + returns `[]`. The loss is client-side, with no page error, no failed chunk + and no non-200. + +Every throwaway container in that list was judged within ~30s of starting, so +they were all inside the same window — and that consistency **read as a +reproduction when it was the same measurement mistake five times over.** + +**Operative rule: recreate, walk away, re-check later. Do not chase it.** + +## ⚠️ The iteration loop that would have prevented the overcook + +`custom.css` is served **per request** from `/api/config/custom.css`, so a CSS +change needs a **browser reload** — not a container recreate, and it never owed +the layout warm-up above. Conflating the two costs ~10 operator-visible minutes +per attempt (operator called this out directly). + +Faster still, and how the final pass was done: **inject candidate CSS into the +running page and screenshot it** — +`await p.addStyleTag({content: css})` in Playwright against the live +dashboard. Seconds per iteration, no deploy. Build + deploy only once the +render looks right. + +## The theme — Australis Skyfall + +Operator supplied a Claude Design handoff bundle via the Booth (`26-copper`). +Skyfall is a dual-theme OKLCH system: one lightness law across every chromatic +family (deep 0.48 / base 0.66 / bright 0.80), all hues cooler than neutral, a +Sea neutral ramp drifting ice-blue→ocean-green as it brightens, and a +"calm depth" language of **hairline + two-layer shadow on every elevated +surface, never one without the other**. + +``` +theme/colors.css layout.css typography.css vendored VERBATIM from the bundle +theme/fonts/Supreme-{400,500,700}.woff2 the body/UI face +theme/skyfall.css.in the Homepage bindings (ours) +theme/build.py → conf/custom.css (generated — do not hand-edit) +``` + +The build step exists for one reason: **Homepage serves only `custom.css` and +`custom.js` out of its config dir**, with no static route beside them, so a +`@font-face` pointing at a vendored `.woff2` would 404 — the face must arrive +as a data URI. The background image takes the other road, because +`/app/public/images` **is** a real static route (mounted read-only in +`compose.yaml`). + +Only Supreme is embedded: a link dashboard has no display type, and Victor +Mono ships as 2.4 MB TTF statics per cut — 30x the whole stylesheet for a +handful of latency figures. + +## The background is generated, not stock + +**Arbo as an image-gen engine** (the operator's actual ask, which I first +misread as "use Arbo's palette" and had to redo). Arbo's `t2i-ui-background` +workflow is purpose-built: *"abstract full-bleed backgrounds, no subject"*. +Job `13f0891f4e42`, seed 26, flux2-klein-9b, 2048×1152, 1.6 MB PNG → **22 KB +WebP** (smooth gradients compress absurdly well). + +⚠️ Arbo API gotcha: `prompt` is a **discriminated union, not a string** — a +bare string 422s. `{"kind":"raw","text":…,"negative":…}` is the shape. + +## Two documented deviations from the design system + +1. **Skyfall forbids this background.** Its rule is "flat semantic surfaces; no + photography, no textures", with one permitted motif — a subtle aurora + gradient on hero/empty-state areas only, *"never behind body text blocks"*. + A dashboard is a body-text block. Present on the operator's explicit + instruction, mitigated rather than excused: abstract, no subject, strictly + cool temperature, held at **`opacity: 30`**. That number is load-bearing — + at 14 the aurora was invisible, and turning it up makes the cards fight the + ribbon. +2. **Service icons stay full-colour vendor logos.** Desaturating them from CSS + only makes them illegible. + +## Overcorrection, and the colour pass + +First stat-well pass went from `font-thin` 13px straight to **bold 22px in +heading white** — operator: *"went from subtle to BASH YOU OVER THE HEAD."* +The principle missed: a stat only has to out-rank **its own label**, not the +service name above it. Now `--text-md` medium in cyan. + +Colour was then lifted **from inside the system**: Skyfall names Aurora (blue, +cyan, green) the *primary* families, "used generously, in that order", while +Dawn (amber/red/violet) is semantic-only. So group markers cycle +blue→cyan→green down the page (icons full strength, names at 0.72), service +icons take a single cool wash, latency tags move to the info family so +"how fast" stops looking like "is it alive". **No Dawn colour is used +decoratively anywhere.** + +Two DOM findings that made it possible: + +- **Homepage renders mdi icons as a gradient behind an SVG mask** — recolour + via `background`, not `color`. +- **Homepage emits `docker-status-`, not `status-`.** The + original selectors matched nothing, so every green pill up to that point was + stock colouring rather than the theme. Both forms are now matched. diff --git a/persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md b/persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md new file mode 100644 index 0000000..1f33da0 --- /dev/null +++ b/persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md @@ -0,0 +1,77 @@ +# `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken + +Commits `42c594c`, `dc3e47b`, plus `uptimekuma` in `9d92c4b`. + +## The pattern worth remembering + +Chasing two bad-looking cards on the dashboard turned up **four stacks running +on fleet hosts with no canonical copy anywhere**: `uptimekuma` and (already +known) the two AdGuards on esh-docker-vm, `searxng` and `seafile` on +ana-docker, and `heretic2-charrp-reasoning` on ana-ml2 (untracked in git). + +⇒ **A dashboard card is a cheap census of what is actually running.** When +something on it looks wrong, check whether the stack behind it is even in +`stacks/` before debugging the symptom — twice here the answer was "no", and +the fix belonged in version control as much as on the host. + +Adopted: `stacks/uptimekuma/`, `stacks/searxng/`, `stacks/seafile/`, +`stacks/heretic2-charrp-reasoning/`. ESH/NH3 AdGuard compose files were +**deliberately left unmanaged** — adopting three live resolvers while also +introducing a new DNS naming system is two risky changes at once. + +## SearXNG — the healthcheck was eating itself + +Card flapped UNHEALTHY; the container was fine the whole time. The compose +passed `--tries` and `--spider` as **two separate argv entries**, so wget +consumed `--spider` as the *value* of `--tries`. Spider mode never engaged, +which means every probe since April **downloaded** the healthz response to a +file: + +``` +295,287 healthz.N files in the container's working directory +``` + +With that many files, wget's scan for the next free filename is what +intermittently blew the 10s timeout. **Self-worsening — every probe made the +next one slower.** Restored `--tries=1`; the junk lived in the writable layer +so the recreate cleared it. Now `healthy`, `fails=0`, 200 in 0.16s. + +Lesson: an argv list in YAML has no shell to catch a missing `=`. A flag that +silently swallows the next argument turns a liveness probe into a workload. + +## SeaFile — not broken, never restarted + +Card showed EXITED for three months. **None of the three services declared a +restart policy**, so Docker defaulted them to `no`. On +**2026-05-06T21:27:45Z** the daemon stopped all three within 200ms of each +other — a daemon restart or host reboot — and nothing brought them back. + +⚠️ **Exit code `255` is a red herring**: it is what a container that ignores +SIGTERM reports when the daemon stops it, **not** evidence of a crash. Reading +it as one sends you hunting a bug that does not exist. The tell was all three +services stopping within 200ms. + +Added `restart: unless-stopped` to all three; brought up; mariadb gated on its +healthcheck exactly as the existing `depends_on` comments intended, seahub +started without the race, `302` → login page. Data was in local named volumes, +not on the ana-nas NFS, so nothing was at risk. + +Three months of silent downtime whose only signal was a card nobody read as an +outage — the argument for semantic status colour on the dashboard (see +[[2026-08-19-homepage-skyfall-theme]], where amber EXITED pills made six +mis-grouped AI seats obvious at a glance). + +## heretic2-charrp-reasoning — tracked, with its shim + +The `char-rp-reasoning` seat (NEO-CODE Heretic2 27B, modelopt NVFP4 + grafted +BF16 MTP head, ~77 tok/s via `qwen3_5_mtp` spec-decode) had been running +untracked. Now in `stacks/`, including +`conf/mtp-workaround/sitecustomize.py`, which is **not optional**: vLLM 0.24.0 +does not propagate modelopt `exclude_modules` to the spec-decode **draft** +model, so the BF16 MTP head gets quantized and the engine dies at load. Both +the mount and `PYTHONPATH` are load-bearing. + +Added the two files house convention expects and the directory lacked — a +`.env.example` naming every knob (all values are compose defaults; the host +overrides only the three VRAM ones) and a README pointing at +`docs/runbooks/heretic2-nvfp4-mtp-seat.md` rather than duplicating it. diff --git a/persistent-memory.d/2026-08-19-waterland-studio-containerised.md b/persistent-memory.d/2026-08-19-waterland-studio-containerised.md new file mode 100644 index 0000000..168fadf --- /dev/null +++ b/persistent-memory.d/2026-08-19-waterland-studio-containerised.md @@ -0,0 +1,118 @@ +# `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured + +Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`): +a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare +`nohup` (PID 1283383) that would not survive a reboot. Now +`stacks/waterland-studio/`, `restart: unless-stopped`, healthy on +irv-ml1:8410. Commits `a2b5b58`, `8189076`. + +Tracking `main` per operator: PR #4 merged and `main` HEAD is exactly the +pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway. + +## Build context lives OUTSIDE the compose dir — on purpose + +`/opt/waterland-studio/src` is the checkout; the Dockerfile is passed +out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh` +rsyncs `stacks//` with `--delete`**, so a checkout kept beside +`compose.yaml` would be destroyed by the next deploy of this stack. `update.sh` +refreshes source → rebuild → recreate → health, and is verified end to end. + +## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN + +`gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer +silently drops to the numpy path at ~21x wall time — it does not error, it +just gets slow. waterland-dev warned about the build side. + +The runtime side is worse and was not in the handover: **`studio/jobs.py` +shells the renderer out as a literal `uv run waterland ...` with no `--extra` +flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project +mid-job to its default extras and prunes cupy back out from under a correctly +built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the +pin ever stops holding the job fails **loudly** instead of quietly rebuilding a +slower environment. + +## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare + +Every render died 1.7s in with: + +``` +RuntimeError: Failed to find CUDA headers. +``` + +printed **through argparse's usage banner**, which makes it read like a CLI +argument bug rather than a missing toolkit. That misdirection is the reason +this is written down. + +cupy compiles kernels at runtime through NVRTC, which needs the toolkit +**headers** — not just the driver and the runtime libs bundled in the +`cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the +bare `nohup` process found them **by accident**; a slim image has none. + +Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few +hundred MB against ~6 GB for a `-devel` base image. It runs **after** +`uv sync`, because sync prunes what it does not know about. + +Reported upstream: it is an undeclared runtime dependency of the `gpu` extra, +and anyone running this without a system toolkit hits it. + +## Landmine 3 — the GPU index inside the container is not the host's + +The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects +`CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where +`nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU +(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the +container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying +the host's value selects a device that does not exist. Host device 0 is the +3090, which carries the TTS zoo and must not be touched. + +## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume + +| job | wall | +|---|---| +| 256² + anim, cold container | 23.3 s | +| 256² + anim, warm | **6.1 s** | +| 256² plate only (`--codec none`) | 3.9 s | +| 512² plate only | 6.4 s | + +Warm beats the **7.4 s** recorded against the bare-metal process, so +containerising cost nothing. Verified the cache volume properly: recreate +(fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without +it every restart makes the next user wait 4x and the service merely *looks* +slow. + +## Upstream finding — the on-disk job store grows without bound + +`JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at +startup**. Consequences: + +1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and + how this was spotted: the API reported **1 job** while the volume held all + **16 directories, 60.6 MB**. Not data loss. +2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict, + so directories orphaned by a restart are **never reclaimed**. The + handover's "bounded around 500 MB" holds within a single process lifetime; + across restarts the store grows monotonically at ~12 MB per animated job. + +Reported to waterland-dev with evidence; **not patched from the infra side** — +it is their code. Prune the volume by hand if it bites first. + +## Access + +Repo is not anonymously readable (a bare clone 403s). Operator granted +**`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False, +pull: True`. Token on irv-ml1 at +`/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a +**repo-scoped** credential helper; `.git/config` carries no token (verified), +so the remote stays clean in any diff or backup. The operator's `vh` +site-admin token was used only for the initial clone and the grant itself and +was **never written to disk on that host** — a site-admin credential on a GPU +box is a blast radius nobody needs for a read-only fetch. + +## Constraints honoured as stated (not inferred) + +- **Serial by design — one replica, one card.** A render is 20–45s of near-full + GPU with a single worker thread. Two on the same A6000 would OOM or thrash. + Throughput is a hardware conversation, not a replica-count one. +- **No authentication, arbitrary file uploads** ⇒ stays inside the + LAN/WireGuard boundary. Do **not** paper over it with a proxy password; + waterland-dev offered to add a real auth layer if wider reach is ever needed. diff --git a/persistent-memory.md b/persistent-memory.md index 2f99503..9effc51 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -109,7 +109,13 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-08-19 — **ESH fiber (Cityside 2Gb symmetric) is LIVE and the site is fully provisioned on it**; `esh-pve-nas` finished its ZFS-root migration AND took its 225-package backlog (reboot deferred); `ESH↔colo` IPsec rebuilt as dialup+NAT-T after CGNAT broke it; ESH has a `/56` of IPv6. Operator is moving to **Lobe Chat** work next — nothing from this session blocks it. ⚠ This session caused ONE self-inflicted outage (missing `--make-rslave`, see ops-lessons-playbook §1) and three long-dead things surfaced on their own (pvestatd 82d, a vzdump 126d, a VM in prelaunch 4mo)._ +_As of 2026-08-19 (late) — a long infra session, all of it landed and committed: **fleet `*.internal` DNS is live** (git-sourced, 42 names, three resolvers incl. a NEW colo one), **waterland studio containerised** on irv-ml1, **Homepage cleaned up and themed** with Australis Skyfall + an Arbo-generated background, and **four unmanaged stacks adopted** into `stacks/` (two of which turned out to be quietly broken — SearXNG's healthcheck, SeaFile down 3 months). ⚠ 17 commits unpushed. ⚠ 2 unread althing messages from waterland-dev left deliberately undrained — operator deferred the inbox to after the `/clear`._ + +- **🟢 FLEET `.internal` DNS — LIVE 2026-08-19.** `..internal`, sites `ana`/`esh`/`nh3`. `dns/internal.yaml` is the source of truth; `scripts/dns-sync.py` reconciles the three AdGuard resolvers (diff → prompt → apply, idempotent). 42 names resolving from all three sites. **Colo got its first resolver ever** (`stacks/adguard-ana/`, API on **8053** not 8080, no blocklists by design) — before this, ana-docker resolved straight against `1.1.1.1`. Auth = a dedicated `infra-ops` AdGuard user, password vaulted `nh3-dev/adguard-infra-ops-password`. **⏳ TWO OPEN, both operator's to schedule:** (a) colo hosts still point at `1.1.1.1` so they do not yet *use* the new resolver — repointing a site's DNS is a separate change; (b) the static-v6 convention (each server at its site's `/64` with low bits echoing the v4 octet, `esh-docker-vm` → `…::45`) is **proposed, not ruled on**. The `v6:` column is empty and correct — no fleet host has a global v6 address yet. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md` + +- **🟢 WATERLAND STUDIO — containerised + live, irv-ml1:8410.** Displaced a bare `nohup` (PID 1283383). Warm 256²+anim render **6.1s**, beating the 7.4s bare-metal figure. `claude-bot` now has **read** on `vh/waterland` (operator-granted); token on irv-ml1 root-owned 0600, repo-scoped helper, nothing in `.git/config`. **⏳ Reported upstream, NOT patched by us:** the on-disk job store grows without bound across restarts (`RETAIN=40` eviction only sees the in-memory dict). → `persistent-memory.d/2026-08-19-waterland-studio-containerised.md` + +- **🟢 HOMEPAGE — cleaned + themed (Australis Skyfall).** Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by **Arbo** (`t2i-ui-background`, job `13f0891f4e42`). ⚠ **After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own — do not chase it.** ⚠ CSS is served per-request: a theme change needs a **reload**, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md` - **🟢 GEN SEAT — SWAPPED to `absolute-heresy` 2026-08-17 (validated, promoted).** Live gen = `/tank/aimodels/qwen38-27b-heresy-nvfp4-mixed` — **MuXodious/Qwen3.8-27B-absolute-heresy** (Heretic v1.4.0 + **SOMPOA**, trial T377, pin `c2374593`) put through our own mixed NVFP4+FP8 recipe. Chosen because it beats the incumbent on **both** axes at once: author refusals 2/101 vs 12/100, first-token KL 0.0759 vs 0.1191. **Gate (probe :8017, pinned nightly, seat-matched flags): MTP 47.2% (inc. 48.2%), decode 103.5 tok/s (96.4), prefill 6618/5403 @6.7k/27k (6334/5085), PPL 6.910 (7.059 — 2.1% BETTER), surface 6/6, abliteration 4/4, and 0/55 refusals on our battery-instruct arm with ZERO EMPTY (no catatonia).** ⚠ speed deltas are **image-confounded** (probe on the pinned nightly, incumbent numbers from an earlier image) — read as "not worse", not a clean win. All 7 LiteLLM aliases verified end-to-end; GPU0 at 91.3/97.9 GB with meromero healthy (more headroom than the old build's 96.8). ⚠ **RC1, 2 days old, ~348 downloads.** **Operator-confirmed "working very well" in real use 2026-08-17**, same evening as the cutover — the signal the synthetic gates structurally cannot give (multi-turn degeneration is stochastic; four synthetic tests once validated three non-fixes). Not yet the 60k-token bar the prior seat cleared, so **keep watching and do NOT delete the rollback weights yet**. **ROLLBACK:** `sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-20260817 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen`; incumbent weights UNTOUCHED at `qwen38-27b-uncensored-nvfp4-mixed` — **do NOT delete** until this holds. Runbook `services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md`. @@ -154,10 +160,22 @@ _As of 2026-08-19 — **ESH fiber (Cityside 2Gb symmetric) is LIVE and the site - **althing monitor** ARMED (handle `infra-ops`). ⚠️ Re-arm ONLY after a real FIRE (rc0), never after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, never chained with `&` (orphans it — hit this twice 2026-08-17, `stop-monitor` reclaims). -- **eshpfi push state:** operator pushes **manually**; last push `50d13f5..8be8a51`. ⚠ **7 commits unpushed as of 2026-08-19** (`707a8cb`..this snapshot) — all docs/memory from the v6 investigation. Push over the INTERNAL gitea route: `git push ssh://git@10.250.50.70:222/vh/esh-pfi-infrastructure.git main:main` (`origin` resolves the public edge which fail2bans fleet-host egress, and stays stale locally as a result). `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED. +- **eshpfi push state:** operator pushes **manually**; last push `50d13f5..8be8a51`. ⚠ **17 commits unpushed as of 2026-08-19** (`707a8cb`..`b8003c7`, plus this snapshot commit) — nothing is on the remote from this session. ## Recent decisions +- `[2026-08-19]` **Fleet `.internal` DNS built and live — git-sourced, agent-managed, three resolvers.** Zone-scoped authority (ESH's hand-made `esteban.net` rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md` + +- `[2026-08-19]` **waterland studio containerised on irv-ml1 — three landmines, all measured.** cupy needs CUDA *headers* the host had by accident; `uv run` re-syncs and prunes cupy at RUNTIME; the A6000 is container-index 0, not the host's 1. → `persistent-memory.d/2026-08-19-waterland-studio-containerised.md` + +- `[2026-08-19]` **Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background.** Includes the hour lost to a self-healing tab-bar red herring, and the CSS-iteration loop that prevents it recurring. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md` + +- `[2026-08-19]` **Four unmanaged stacks found on live hosts — two quietly broken.** A dashboard card is a cheap census of what is actually running; check whether the stack is even in `stacks/` before debugging the symptom. → `persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md` + +- `[2026-08-19]` **`claude-bot` granted read on `vh/waterland`** (operator-empowered, verified `admin:false push:false pull:true`) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit `8189076`. + +- `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now"). + - `[2026-08-18]` **esh-pve-nas migration STAGED — and staging is where three landmines surfaced, none of which the plan predicted.** (1) The runbook's `/boot` LV had **nowhere to live**: VG `pve` had 4 MB free and mounted ext4 cannot shrink, so the space came from the 768 MB swap LV (operator's call: shrink to 256 MB, not drop). (2) The runbook's `zpool set cachefile=… nvme` would have **broken the NAS** — populating a cache flips the host to import-by-cache, and a one-pool cache leaves `ssd`+`tank` unimported under CT 103's twelve bind mounts. (3) **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`**, because GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` and the probe failure is swallowed. All three were caught by *verify steps that asserted effective state*, not by reading the plan. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md` - `[2026-08-17]` **esh-pve-nas PVE root is on a USB DOM — mitigated, and the migration replanned to split boot from root.** Operator's design beats my reinstall plan; wear was never the issue, blocked patching is. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md` @@ -271,15 +289,11 @@ _As of 2026-08-19 — **ESH fiber (Cityside 2Gb symmetric) is LIVE and the site - `[2026-07-31]` **worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred.** memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on `vh/worldtree-sdk` (source pulled via the **Gitea API archive** — git-HTTP 403s on that repo); publishing to the vh USER namespace **can't be delegated** (401 `reqPackageAccess` even with `write:package`) so it needs an owner token — operator saved a **FULL vh site-admin token at `~/.config/gitea/vh-token` (0600)** for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory `reference_infra_ops_vh_gitea_token_and_sdk_publish`. -- `[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap.** heid's cross-frontier panel was silently degraded (empty content, `finish_reason: stop`). dvalin+bil researched (docs said deprecated-max_tokens); heid's live data refuted that (completion hit 18455) → it's a reasoning ceiling. **Proven on the wire against heid's real 500KB bundle:** `reasoning_effort: low` drops reasoning under the ceiling → content returns, on BOTH coding + general endpoints. Fix is CALLER-side (no gateway change): send `reasoning_effort` via **`extra_body`** (LiteLLM `drop_params: true` strips the top-level param — why heid's earlier attempt no-op'd). Relayed to heid to validate; backstop = `allowed_openai_params` on the route. → `persistent-memory.d/2026-07-31-kimi-k3-reasoning-cap.md` -- `[2026-07-25]` **bil-smithy-dev wired as an althing zellij-window-ping (pane route).** She's a `driver: human` dwarf peer (pane `bil-smithy` already live alongside eitri/dvalin/regin-smithy in the `Claude` zellij session) but had no delivery route → smoke messages posted to the bus but never reached her window. **Mechanism (reusable for any pane-route handle):** `~/.althing/config.yaml` → `zellij_sessions.Claude.agents[]` maps `handle` → `target` (a zellij pane **TITLE**, matched via `list-panes -j` in `althing/zellij.py:resolve_pane_id`) → `command` (herald `write-chars` + CR into that pane). The **herald loads config ONCE at startup** (`herald.py main()`), so **`systemctl --user restart althing-herald.service`** after editing. Added bil (`target: bil-smithy`), restarted, verified: herald delivered the pending smoke `01KYD7W7CF…` (available→attempted→**delivered**). ⚠️ Noticed pre-existing pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range", empty msg_ids — likely `render_command messages[0]` on an empty list; NOT caused by this change, bil works) — worth a herald look. - - -_204 older entries archived to archival-memory.md._ +_206 older entries archived to archival-memory.md._ ## Tried and abandoned @@ -291,8 +305,6 @@ _204 older entries archived to archival-memory.md._ - `[2026-08-03]` **corviduo-dev shared containerd: a concurrent-pull race fails ONE instance's deploy; DON'T "prune to fix" — the image is in-use by the instance that won the race.** b169 personal deploy failed at `docker compose pull` (`Lchown … no such file or directory` on the big torch layer → looked like disk pressure / corrupt snapshot). ACTUAL: NOT disk (56G free, inodes 7%). demo + personal + pinned share ONE `/var/lib/containerd` on corviduo-dev; demo (from main) and personal (from staging tag) extracted b169's shared torch layer simultaneously → personal's hit a partial snapshot mid-race and aborted while demo's completed. The image `6e34a87` was FULLY VALID — demo was RUNNING it healthy. Fix = just re-run the failed deploy (image already materialized; compose pull finds it present). **NEAR-MISS:** worldtree-dev's suggested "prune unused images/snapshots" would have rmi'd `6e34a87` = the image the running demo depends on → demo outage. **Lesson: before any prune/rmi "cleanup," `docker ps` the running images — an "unused" image may be a co-tenant's live one; and verify the failure's REAL cause (disk? inode? in-use? race?) before applying the suggested remedy.** (Pipeline fix, deferred: serialize demo-from-main + personal-from-staging, or a per-image pull lock, to avoid the shared-layer extraction race.) -- `[2026-08-02]` **`docker exec` into worldtree containers defaults to ROOT — root writes contaminate the uid-1000 (vh) KB tree.** My `sudo docker exec … --reindex` on personal ran as ROOT (muninn app = uid 1000); its wing git-commit + atomic note-swap left root-owned files in the `worldtree-personal_worldtree-kb` volume: a root-owned `.old-` backup dir (blocked the uid-1000 retry's `rmtree` → Errno 13, because unlink needs write on the DIR and it was root:root 755) AND **60 root-owned loose git objects** in `.git/objects/`. Fix (host-side, corviduo-dev): `sudo rm -rf` the superseded `.old-` dir (tar'd aside to /tmp first) + `sudo find … -user 0 -exec chown 1000:1000` the objects (ownership-only, git-content-safe; the `.git/objects/XX/` dirs were vh-owned so these weren't a hard blocker, but violated "clean tree"). **RUNBOOK RULE (worldtree-dev, ADOPTED):** any `docker exec` into worldtree containers that WRITES pipeline state runs **`-u 1000`**, never default-root — same genus as the mv footgun (acting without matching the target's constraints; 3rd such slip in one session). **GOTCHA that hid the scope:** `find … -user 0 | head -20` TRUNCATED (the `.old-` dir alone had 153 files, so the first page was all `.old-`) → I "verified clean" off a partial list. Never `head` a scope-defining find; count first (`| wc -l`). **Related blind-spot (muninn-dev):** a root-owned job SUBDIR passes every requeue guard (job_row/dispatch/list_jobs render fine) AND `/health` (contract's `os.access(ingestion_root, W_OK)` tests only the ROOT dir, so a foreign-owned subdir under `pending/` still reports `ingestion_root_writable: true`) — then the uid-1000 gate can't write into it. "Clean board + green /health + failure at next mutation." muninn-dev added an OWNERSHIP column to the standing post-move check to catch it; two green signals both miss a foreign-owned subdir otherwise. - - `[2026-08-02]` **`mv complete/ → failed/` RENAMED the job to `failed` because failed/ didn't exist.** worldtree-dev's round-2 unblock command (`mv /data/state/ingestion/complete/ /data/state/ingestion/failed/`) assumed `failed/` existed; on PERSONAL muninn it did NOT (fresh instance — root was `active/ complete/ pending/ sources/`, no `failed/`). `mv src nonexistent/` **renames** src→nonexistent, so job1 became the `failed` dir and job2 nested inside it. Caught on post-move `ls` (failed/ held job *contents*, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt `failed/` (worldtree:worldtree 755) — NO data loss. **Lessons:** (1) before `mv X into-dir/`, verify the dir EXISTS (`[ -d dir ]`) — an empty `ls dir/ 2>/dev/null` is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is **`mv -t `** (`--target-directory`): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a **trailing slash does NOT protect** — `mv src failed/` with `failed/` missing STILL silently renames to `failed` (rc=0); "just add the slash" is a false guard. (`mkdir -p failed/` first also works, but `mv -t` inverts the failure from silent-wrong to loud-safe in one flag.) Container `sh` is dash — no `(` in echo strings. **SILENT failure mode (muninn-dev carry-forward):** a misplaced ingestion-state move doesn't crash anything — `list_jobs()` stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (`job_row`→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (`job_row` found + guards pass), don't trust mv exit codes — and confirm `job.dispatch.json` survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. **DON'T TIDY (round-2 pending):** both DCC + P&P jobs currently REST in personal `failed/` with manifests reading `state: complete` until round-2 requeue runs — deliberate + load-bearing (`requeue` keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. **Round-2 sequencing:** the requeue is **mimir-dev's** browser flow (pending their operator's board-vs-API ruling); **muninn-dev** is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); **infra-ops** = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z. **ARC COMPLETE (2026-08-03 ~05:49):** both books terminal — DCC `mimir-6351554e8e8f` 705 concepts + P&P `mimir-f3887c9b97b7` 667, extracted AND indexed, 5/5 phases, 0 failures/truncations (validates the #385 budget fix vs April's 785 control); **#381 restart-after-ingest FIRED** (personal api, healthz/readyz 200 ~25s), retrieval-visibility confirmed (search_library returns DCC+P&P from fiction post-restart); handed ratatoskr-verify go to worldtree-dev. **Delete-sweep precondition NOW MET** — the stale DCC rows in `main` are genuine duplicates of live `fiction` rows, so worldtree-dev's supervised sweep of the ~785 April orphans is unblocked (still comes to me supervised: snapshot + operator-in-loop). @@ -305,8 +317,6 @@ _204 older entries archived to archival-memory.md._ -- `[2026-07-25]` **Chaining the althing wake-listener arm orphans it.** `reply && althing-wake-listener &` (or spawning `althing-wake-listener` with `&` *inside* a `run_in_background` task) → the `&`-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a *plain operator turn* (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn `althing-wake-listener` as its OWN `run_in_background` task, and re-arm ONLY after a real fire (` completed rc0`). Reclaim an orphan with `althing-cli stop-monitor` then re-arm. - -_139 older entries archived to archival-memory.md._ +_141 older entries archived to archival-memory.md._