memory: snapshot — 2026-06-25 (althing v0.17 / nh3-extdev model-B mesh + zellij web pilot + Worldtree #314/#322/#317 config arc; rewrite stale althing Tools-row to v0.17; archive 10 pre-session 2026-06-20 entries)
This commit is contained in:
@@ -585,6 +585,21 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
||||
- `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (successor to nh3-ansible); hosts `/opt/externs` pi-on-GLM-5.2 client workspaces. (a841eab)
|
||||
_Archived 2026-06-21._
|
||||
|
||||
- `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-` prefix (operator) — reuse the existing capability convention.** gen/gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge (passthrough)/gen-frontier/-reasoning. The **swap-transparency rule**: embedding stays model-named (`qwen3-embedding`) because vectors aren't swap-transparent; everything else can swap behind a stable alias. (d372103, 7819f96)
|
||||
_Archived 2026-06-25._
|
||||
|
||||
- `[2026-06-20]` **Removed the litellm `*` wildcard → llama-swap** (operator; llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now 404 cleanly instead of a misleading "Connection error" — closes the long-standing footgun. (d0eb09c)
|
||||
_Archived 2026-06-25._
|
||||
|
||||
- `[2026-06-20]` **Langfuse retired** (operator) — redundant with LiteLLM native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse spew). Callbacks dropped, stack torn down. (f8eda1c)
|
||||
_Archived 2026-06-25._
|
||||
|
||||
- `[2026-06-20]` **OmniVoice streaming `/tts` — vendor chatterbox-fast's scheduler, not its whole package** (operator-approved vendor-copy). Diffusion TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox-fast remains the lowest-latency front-end. (288d085, cd92b85)
|
||||
_Archived 2026-06-25._
|
||||
|
||||
- `[2026-06-20]` **Daily image+buildx prune cron on ana-docker** (the missing automation behind the recurring image bloat) — `lkraven` crontab, prune unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context.
|
||||
_Archived 2026-06-25._
|
||||
|
||||
## Tried and abandoned (archived)
|
||||
|
||||
- `[2026-04-30]` task-board workflow with
|
||||
@@ -1137,3 +1152,18 @@ _76 older entries archived to archival-memory.md._
|
||||
|
||||
- `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — no HF `Mistral4` backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first.
|
||||
_Archived 2026-06-21._
|
||||
|
||||
- `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree (worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea` on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur).
|
||||
_Archived 2026-06-25._
|
||||
|
||||
- `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell (vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3` arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1 fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager` (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 (non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` + `playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending operator OK to free.
|
||||
_Archived 2026-06-25._
|
||||
|
||||
- `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere.** No per-stack `logging:` limit and no docker-daemon default → ClickHouse's verbose stdout grew unbounded to ~94 GB and filled ana-docker's `/` (healthchecks then failed fleet-wide on `runc /tmp: no space left`). Lesson: on a full disk, the symptom (all-unhealthy) is the FORK-can't-write healthcheck, not the containers; find the runaway via per-container-dir `du`, not `docker system df`. The systemic fix is a daemon-default `log-opts` (`max-size`/`max-file`) — NOT just per-service patching.
|
||||
_Archived 2026-06-25._
|
||||
|
||||
- `[2026-06-20]` **OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk streaming is a net loss.** Measured ~2.8× (32-step) / 5.6× (16-step) on the 3090; every chunk costs ~the same regardless of length, so naive per-sentence streaming triples total gen time + starves. Use few/aggressive chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end.
|
||||
_Archived 2026-06-25._
|
||||
|
||||
- `[2026-06-20]` **Named buildx builders accumulate cache the standard prunes MISS.** `docker image prune -a` + `docker builder prune` don't touch `buildx_buildkit_<name>-builder0_state` volumes (per-builder buildkit state). Prune them with `docker buildx prune --builder <name> -af`, or `docker volume rm` orphaned ones (builder already removed).
|
||||
_Archived 2026-06-25._
|
||||
|
||||
+93
-137
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-06-21_
|
||||
_Last updated: 2026-06-25_
|
||||
|
||||
## Repo purpose
|
||||
|
||||
@@ -22,7 +22,7 @@ Sister repos (separate gitea repos, deployed by playbooks here):
|
||||
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
|
||||
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
|
||||
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
|
||||
| `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) |
|
||||
| `vh/althing` | Lean trusted inter-agent message bus — **v0.17 multi-machine** (re-architected 2026-06): each box runs a local-SQLite bus + a courier/receiver for P2P over the 10.x net. The v0.15 lean-bus cut RIPPED moderation / chamber (UI 7881) / forseti-daemon / agent-runner / redis-valkey. | per-box `uv tool install` (NOT CI-deploy); **nh3-extdev** added as a mesh peer 2026-06-25 (model B: althing-svc svc account + shared `/srv/althing`) |
|
||||
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
|
||||
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
|
||||
| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker** (the image-bloat source); claude-bot now ADMIN collaborator (2026-06-20) | push-to-main → CI build-and-deploy (runner on ana-docker) |
|
||||
@@ -41,8 +41,7 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
through TWO rest-servers** (`rest-server-ana` @ ana-docker:8000 →
|
||||
ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; `rest-server-nh3` @
|
||||
nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS
|
||||
export of `/mnt/backup`. **As of 2026-06-20 `rest-server-ana` is DOWN
|
||||
on a failed ana-nas NFS mount — see Current state.**
|
||||
export of `/mnt/backup`. (rest-server-ana recovered 2026-06-20.)
|
||||
|
||||
- **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
|
||||
model/dataset onto ana-ml2's shared cache at
|
||||
@@ -105,51 +104,86 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
2026-06-20). **→ For any sudo op on ana-docker (mount, root-owned files,
|
||||
service control), use `ssh infra-ops@ana-docker`, NOT the default session.**
|
||||
lkraven-owned files (litellm config, most stack compose/conf) still take
|
||||
plain `cp`/edit under either identity.
|
||||
plain `cp`/edit under either identity. **NEW (2026-06-25):** `ssh infra-ops@10.100.10.50`
|
||||
(nh3-dev) ALSO has NOPASSWD sudo; on **nh3-extdev** infra-ops is sudo-LESS by
|
||||
design and **`ssh lkraven@10.100.50.42` is the NOPASSWD path** there.
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-06-21:_
|
||||
_As of 2026-06-25:_
|
||||
|
||||
- **Backups — RECOVERED + hardened (2026-06-20).** rest-server-ana is back (the
|
||||
~6.5-wk silent ana-side restic outage is closed); ana-ml2 + ana-docker
|
||||
re-snapshotted; daily freshness alert live (nh3-dev systemd timer →
|
||||
`scripts/check-backup-freshness.sh` → althing on any stale/down layer); both
|
||||
ana-docker NFS mounts moved to self-healing automount. Topology + 2-min check:
|
||||
`docs/runbooks/backups.md`. STILL OPEN: rotate the 5 disclosed rest-server creds
|
||||
(operator, offline); confirm esh-vm-db's resticprofile includes DB dumps
|
||||
(esh-pve-nas has NO PBS namespace → it is restic-only). Detail in Recent decisions.
|
||||
- **Worldtree persona-render config arc COMPLETE (demo + personal).** Pre-synced +
|
||||
deployed three bind-mount config deltas on corviduo-dev demo+personal: #314
|
||||
relational-stance (relational_extractor role + relational_stance block + stance/ dir),
|
||||
#322 affect/mood gate (affect-render-baseline-allow policy rule + mood_tier_deployment_cap),
|
||||
#317 valence teardown (REMOVED relational_valence — a **boot-blocking removal**). All
|
||||
deployed green; the full two-layer persona render (relational + mood) is live on
|
||||
demo+personal; Worldtree at v0.37.38. Recipe + the additions-vs-removals lesson in
|
||||
auto-memory `reference_corviduo_dev_emergency_ops`.
|
||||
|
||||
- **Worldtree — demo + personal GREEN (v0.37.7/.9).** Capability-gateway migration
|
||||
live; ratatoskr readonly-admin tier staged (cb2a79a). worldtree-dev fully
|
||||
self-serve: admin keys (demo d113207c / personal f4f75adb) + one consolidated
|
||||
claude-bot Gitea token (Actions+deploys+issues). Nothing pending. Detail in
|
||||
- **nh3-extdev (10.100.50.42) stood up as an althing v0.17.1 mesh PEER — MODEL B.** Full
|
||||
provision in one session: system zellij + althing (`/usr/local/bin`, both users),
|
||||
CLAUDE.md, and the mesh (handles `mailman`+`ldp-dev`, ↔nh3-dev). MODEL B (operator's
|
||||
multi-user choice): receiver runs as the **`althing-svc`** service account + a
|
||||
group-shared **`/srv/althing`** root (group `althing`; infra-ops+lkraven members;
|
||||
`ALTHING_ROOT=/srv/althing` via `/etc/profile.d/althing.sh`). Live + verified (two OS
|
||||
users sharing ONE config/DB). Layout + access facts in auto-memory
|
||||
`reference_nh3_extdev_althing_mesh`. Open: forseti's multi-user ADR.
|
||||
|
||||
- **zellij native web client piloted on nh3-dev** (`zellij-web.service` :8443, native TLS)
|
||||
ALONGSIDE the ttyd seats (ttyd KEPT as the no-auth fallback). Verdict: wins on
|
||||
clipboard/auth/TLS/simplicity, loses device-independent fonts; bind left `0.0.0.0` per
|
||||
operator. auto-memory `reference_zellij_web_seat`.
|
||||
|
||||
- **Backups — recovered + hardened (2026-06-20), STILL OPEN:** rotate the 5 disclosed
|
||||
rest-server creds (operator, offline); confirm esh-vm-db's resticprofile includes DB
|
||||
dumps (esh-pve-nas restic-only). Topology + 2-min check: `docs/runbooks/backups.md`.
|
||||
|
||||
- **R22 (brokkr/dwarves) — CONCLUDED, gateway-only.** Phase B (internal-model hooks)
|
||||
CANCELLED (Worldtree model-agnostic → no deploy path). Whole footprint = gateway calls
|
||||
to free `qwen3.5-122-a10b` via a PERSISTENT full-access key at
|
||||
`/home/lkraven/.r22-gateway-key` (mode 600, carries paid GLM, do NOT delete). Detail in
|
||||
Recent decisions.
|
||||
|
||||
- **R22 (brokkr/dwarves) — CONCLUDED, gateway-only.** Phase B (internal-model
|
||||
hooks) CANCELLED (Worldtree model-agnostic → no deploy path). R22 whole footprint
|
||||
= gateway calls to free `qwen3.5-122-a10b` via a PERSISTENT full-access key at
|
||||
`/home/lkraven/.r22-gateway-key` (mode 600, carries paid GLM, do NOT delete).
|
||||
Detail in Recent decisions.
|
||||
|
||||
- **Open follow-ups (low-priority):** clean phantom `qwen3.6-35b-a3b` off the
|
||||
gateway (lists in /v1/models but 400s — backend displaced); docker-daemon default
|
||||
log-cap on ana-docker (root → infra-ops; systemic fix behind the 94 GB disk
|
||||
incident).
|
||||
- **Open follow-ups (low-priority):** clean phantom `qwen3.6-35b-a3b` off the gateway
|
||||
(lists in /v1/models but 400s — backend displaced); docker-daemon default log-cap on
|
||||
ana-docker (systemic fix behind the 94 GB disk incident).
|
||||
|
||||
- **Standing / parked (from prior):** Mac Pro migration (`migration-plan.md`,
|
||||
hardware-gated); R17 v2 corpus push HELD (distribution barred); disclosed-keys
|
||||
rotation queue; clean legacy `news-digest` on ana-docker.
|
||||
hardware-gated); R17 v2 corpus push HELD (distribution barred); disclosed-keys rotation
|
||||
queue; clean legacy `news-digest` on ana-docker.
|
||||
|
||||
- **Worldtree config-propagation (reference):** demo+personal bind-mount
|
||||
`providers.yaml`/`model_roles.yaml`/`policies.yaml` from
|
||||
`/opt/worldtree{,-personal}/config` (infra-ops-deployable, byte-identical from
|
||||
worldtree-dev canonical sha); pinned is IN-IMAGE. Reload via `docker restart
|
||||
<container>`, NEVER `compose up` (stale-`:latest` footgun); WT PDP is rule-based
|
||||
(a tier needs allow RULES, not just scopes).
|
||||
- **Worldtree config-propagation (reference):** demo+personal bind-mount config
|
||||
(`providers.yaml`/`model_roles.yaml`/`policies.yaml`/`defaults.yaml`, now incl. the
|
||||
#314/#322/#317 additions) from `/opt/worldtree{,-personal}/config`
|
||||
(infra-ops-deployable, byte-identical from worldtree-dev canonical sha); pinned is
|
||||
IN-IMAGE. Reload via `docker restart <container>`, NEVER `compose up` (stale-`:latest`
|
||||
footgun); WT PDP is rule-based (a tier needs allow RULES, not just scopes). **#317
|
||||
lesson: config REMOVALS are NOT backward-compatible with the still-running image — push
|
||||
promptly, don't bounce the instance in the window.**
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-06-25]` **althing re-architected to the v0.17 lean multi-machine bus; nh3-extdev
|
||||
stood up as a mesh peer (MODEL B).** v0.15.0's lean-bus cut RIPPED
|
||||
moderation/chamber/forseti-daemon/agent-runner/redis-valkey; **v0.17 = per-box
|
||||
local-SQLite bus + a courier/receiver for P2P over the 10.x net** (installed per-box via
|
||||
`uv tool install`, NOT CI). Migrated nh3-dev's client v0.15→v0.17. nh3-extdev
|
||||
provisioned as a full mesh peer with the operator's **MODEL B** (dedicated `althing-svc`
|
||||
service account + group-shared `/srv/althing` root, so multiple OS users share one
|
||||
config/DB; `ALTHING_ROOT` via /etc/profile.d). Verified multi-user concurrent rw.
|
||||
Supersedes the old althing Tools-row (was chamber/daemons/valkey). auto-memory
|
||||
`reference_nh3_extdev_althing_mesh`.
|
||||
|
||||
- `[2026-06-23]` **zellij native web client piloted on nh3-dev** (`zellij-web.service`
|
||||
:8443, native TLS), system-installed for all users, ALONGSIDE ttyd (kept as the no-auth
|
||||
fallback). Bind left `0.0.0.0` per operator. auto-memory `reference_zellij_web_seat`.
|
||||
|
||||
- `[2026-06-22]` **Worldtree persona-render config arc (#314 relational-stance / #322
|
||||
affect-mood gate / #317 valence teardown) pre-synced + deployed green on demo+personal**
|
||||
— #317 a boot-blocking config REMOVAL (relational_valence). Config-sync recipe +
|
||||
additions-vs-removals lesson in auto-memory `reference_corviduo_dev_emergency_ops`.
|
||||
|
||||
- `[2026-06-20]` **R22 (brokkr/eitri/dwarves) stood down to gateway; full-access R22
|
||||
key minted; Phase B parked.** Operator caught the raw-weights/GPU ask as premature —
|
||||
Phase A (P00 + NL-rendering) is PROMPT-LEVEL on the LiteLLM gateway, not raw weights
|
||||
@@ -173,15 +207,10 @@ _As of 2026-06-21:_
|
||||
approach has no deploy path (the pragmatic-outcome gate firing). R22's ENTIRE compute
|
||||
is now the gateway (free qwen3.5-122-a10b MUT); NO raw-weights/GPU ever; the A6000
|
||||
slice reservation is released (was never provisioned). The full-scope R22 key stays
|
||||
(pinned to the free qwen-122, GLM untouched). **FOLLOW-UP: clean phantom
|
||||
`qwen3.6-35b-a3b` off the gateway** (shows in /v1/models but 400s "invalid model" —
|
||||
backend displaced 2026-06-19; the no-false-alias footgun). **OPERATOR STEER
|
||||
(2026-06-20): R22 is research for a PRAGMATIC/deployable outcome, NOT advancing-the-art
|
||||
("another day's work").** Before the dwarves invest in Phase B, establish a deploy path
|
||||
(distilled steering-vector / fine-tune applied to a model we already serve, e.g.
|
||||
Worldtree's) — don't research an avenue we won't implement. Phase A/P00 (the cheap
|
||||
prompt-level gate) runs regardless (it just tells us if the affect signal exists).
|
||||
Operator is taking the dwarf discussion directly.
|
||||
(pinned to the free qwen-122, GLM untouched). **OPERATOR STEER (2026-06-20): R22 is
|
||||
research for a PRAGMATIC/deployable outcome, NOT advancing-the-art.** Before the dwarves
|
||||
invest in Phase B, establish a deploy path (distilled steering-vector / fine-tune
|
||||
applied to a model we already serve). Operator is taking the dwarf discussion directly.
|
||||
|
||||
- `[2026-06-20]` **claude-bot issue-scope token minted for worldtree-dev self-serve**
|
||||
(closes their last `tea`-as-vh fallback — issues; they already self-serve Actions +
|
||||
@@ -189,105 +218,38 @@ _As of 2026-06-21:_
|
||||
`worldtree-dev-actions-issues-20260621` (id 16) with `write:repository`+`write:issue`
|
||||
via basic-auth (`claude-bot:gitea-password`, POST `/users/claude-bot/tokens`); dropped
|
||||
mode-600 `~/.claude-bot-token-issues` → worldtree-dev swaps. **Gitea COLLAPSES
|
||||
`read:issue` into `write:issue`** (write implies read) so the granted set shows
|
||||
write-only but read IS satisfied (GET issues → 200). Old token (id 15,
|
||||
worldtree-dev-actions-20260620) REVOKED (DELETE 204) after worldtree-dev verified
|
||||
read+write end-to-end via `core.issues` (show/list/comment on #311) → claude-bot now
|
||||
carries ONE consolidated worldtree-dev token (id 16) + arbo-ci (id 14), no stale dups.
|
||||
Advances the credential-migration directive (worldtree-dev fully self-serve: Actions +
|
||||
deploys + issues on one token). ALSO this session: re-staged cb2a79a `policies.yaml` (readonly-admin allow
|
||||
rules) to demo+personal (see Current-state ratatoskr entry).
|
||||
`read:issue` into `write:issue`** (write implies read). Old token (id 15) REVOKED after
|
||||
verification → claude-bot carries ONE consolidated worldtree-dev token (id 16) + arbo-ci
|
||||
(id 14). Advances the credential-migration directive.
|
||||
|
||||
- `[2026-06-20]` **rest-server-ana recovered + backup prevention shipped + worldtree-dev
|
||||
admin keys provisioned** (operator-directed). rest-server-ana fixed (mount-a +
|
||||
force-recreate via `infra-ops@ana-docker`); freshness alert + fstab hardening landed
|
||||
(see Current-state BACKUP entry + `docs/runbooks/backups.md`). Minted worldtree-dev
|
||||
**admin-tier Heimdall keys** on demo (key_id d113207c) + personal (f4f75adb) via
|
||||
`POST /admin/keys {tier:admin}` + `POST /admin/users/worldtree-dev/tier`, verified
|
||||
read+write, dropped mode-600 to `~/.wt-admin-{demo,personal}` on nh3-dev → worldtree-dev
|
||||
now self-serves key minting (cuts infra-ops out of the per-key loop, like the
|
||||
claude-bot gitea token). Cred rotation (5 rest-server pw) BELAYED per operator.
|
||||
|
||||
- `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-`
|
||||
prefix (operator) — reuse the existing capability convention.** gen/
|
||||
gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge
|
||||
(passthrough)/gen-frontier/-reasoning. The **swap-transparency rule**:
|
||||
embedding stays model-named (`qwen3-embedding`) because vectors aren't
|
||||
swap-transparent; everything else can swap behind a stable alias. (d372103,
|
||||
7819f96)
|
||||
|
||||
- `[2026-06-20]` **Removed the litellm `*` wildcard → llama-swap** (operator;
|
||||
llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now
|
||||
404 cleanly instead of a misleading "Connection error" — closes the
|
||||
long-standing footgun. (d0eb09c)
|
||||
|
||||
- `[2026-06-20]` **Langfuse retired** (operator) — redundant with LiteLLM
|
||||
native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse
|
||||
spew). Callbacks dropped, stack torn down. (f8eda1c)
|
||||
|
||||
- `[2026-06-20]` **OmniVoice streaming `/tts` — vendor chatterbox-fast's
|
||||
scheduler, not its whole package** (operator-approved vendor-copy). Diffusion
|
||||
TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox-
|
||||
fast remains the lowest-latency front-end. (288d085, cd92b85)
|
||||
(see `docs/runbooks/backups.md`). Minted worldtree-dev **admin-tier Heimdall keys** on
|
||||
demo (key_id d113207c) + personal (f4f75adb), dropped mode-600 to
|
||||
`~/.wt-admin-{demo,personal}` on nh3-dev → worldtree-dev now self-serves key minting.
|
||||
Cred rotation (5 rest-server pw) BELAYED per operator.
|
||||
|
||||
- `[2026-06-20]` **claude-bot → ADMIN on vh/Worldtree** (operator-authorized;
|
||||
one-time use of operator vh-admin to enable the migration) — claude-bot
|
||||
self-serves WT deploys/tokens henceforth, reducing personal-cred fallbacks.
|
||||
|
||||
- `[2026-06-20]` **Daily image+buildx prune cron on ana-docker** (the missing
|
||||
automation behind the recurring image bloat) — `lkraven` crontab, prune
|
||||
unused images >48 h + per-builder buildx cache. The real systemic fix
|
||||
(daemon log cap) is deferred to clean context.
|
||||
|
||||
- `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.** Stand up service accounts (claude-bot Gitea user + scoped tokens, distinct litellm admin key); re-mint consumer creds under them; flag personal-cred fallbacks until done. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
|
||||
|
||||
_113 older entries archived to archival-memory.md._
|
||||
_118 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it
|
||||
STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After
|
||||
the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push
|
||||
(push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while
|
||||
the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea
|
||||
errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree
|
||||
(worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea`
|
||||
on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively
|
||||
fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred
|
||||
docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur).
|
||||
|
||||
- `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 —
|
||||
ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check
|
||||
displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell
|
||||
(vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow"
|
||||
suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format
|
||||
NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3`
|
||||
arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1
|
||||
fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the
|
||||
official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in
|
||||
CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager`
|
||||
(~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4
|
||||
(non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` +
|
||||
`playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on
|
||||
ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending
|
||||
operator OK to free.
|
||||
|
||||
- `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the
|
||||
REAL disk-filler (not image bloat) — there was NO log rotation anywhere.**
|
||||
No per-stack `logging:` limit and no docker-daemon default → ClickHouse's
|
||||
verbose stdout grew unbounded to ~94 GB and filled ana-docker's `/`
|
||||
(healthchecks then failed fleet-wide on `runc /tmp: no space left`). Lesson:
|
||||
on a full disk, the symptom (all-unhealthy) is the FORK-can't-write
|
||||
healthcheck, not the containers; find the runaway via per-container-dir `du`,
|
||||
not `docker system df`. The systemic fix is a daemon-default `log-opts`
|
||||
(`max-size`/`max-file`) — NOT just per-service patching.
|
||||
|
||||
- `[2026-06-20]` **OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold —
|
||||
diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk
|
||||
streaming is a net loss.** Measured ~2.8× (32-step) / 5.6× (16-step) on the
|
||||
3090; every chunk costs ~the same regardless of length, so naive
|
||||
per-sentence streaming triples total gen time + starves. Use few/aggressive
|
||||
chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end.
|
||||
- `[2026-06-25]` **althing "unreachable: <machine> — retry later" can MASK an app-level
|
||||
500.** A multi-hour "intermittent :8087 / firewall flap" hunt was a red herring: raw
|
||||
network was always clean (curl POST to the receiver `:8087` worked; a connect probe = 0
|
||||
fails). Root cause = the receiver's DB agents-table wasn't synced with the config roster,
|
||||
so delivery 500'd `"unknown to: <handle>"`, which the courier MAPPED to "unreachable"
|
||||
(looks like network/DNS). **Diagnose: raw curl to :8087 + a connect-probe pass ⇒ NOT
|
||||
network — look up-stack.** Fixed in althing **v0.17.1** (receiver auto-ensures the
|
||||
recipient on delivery). Stop-gap on older builds: `ALTHING_HANDLE=<h> althing-cli inbox`
|
||||
as the receiver user syncs config→DB; re-run after any roster change. auto-memory
|
||||
`reference_nh3_extdev_althing_mesh`.
|
||||
|
||||
- `[2026-06-20]` **rest-server `.htpasswd: permission denied` = the ana-nas
|
||||
NFS mount FAILED (ghost file on the local mount point), NOT a decommission.**
|
||||
@@ -297,12 +259,6 @@ _113 older entries archived to archival-memory.md._
|
||||
a crash-looping legacy container until you've checked fstab + the docs — it
|
||||
was the live ana-side restic target.
|
||||
|
||||
- `[2026-06-20]` **Named buildx builders accumulate cache the standard prunes
|
||||
MISS.** `docker image prune -a` + `docker builder prune` don't touch
|
||||
`buildx_buildkit_<name>-builder0_state` volumes (per-builder buildkit
|
||||
state). Prune them with `docker buildx prune --builder <name> -af`, or
|
||||
`docker volume rm` orphaned ones (builder already removed).
|
||||
|
||||
- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) —
|
||||
but `ssh infra-ops@ana-docker` HAS NOPASSWD root** (corrected later same day).
|
||||
Early on a `sudo cp` as lkraven silently failed (password prompt) → one
|
||||
@@ -311,4 +267,4 @@ _113 older entries archived to archival-memory.md._
|
||||
reach for `infra-ops@ana-docker` for sudo ops; lkraven-owned files (litellm
|
||||
config, most stack compose/conf) take plain `cp` under either identity.
|
||||
|
||||
_93 older entries archived to archival-memory.md._
|
||||
_98 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user