memory: snapshot — mimir-inbox/#377 read-path arc (deploy + #380/#381/#382 + donut voice); archived 11 old entries
This commit is contained in:
+166
-1
@@ -1316,4 +1316,169 @@ _76 older entries archived to archival-memory.md._
|
|||||||
- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — but `ssh
|
- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — but `ssh
|
||||||
infra-ops@ana-docker` HAS NOPASSWD root.** A `sudo cp` as lkraven silently failed → nearly punted
|
infra-ops@ana-docker` HAS NOPASSWD root.** A `sudo cp` as lkraven silently failed → nearly punted
|
||||||
the rest-server recovery. Reach for `infra-ops@ana-docker` for sudo ops.
|
the rest-server recovery. Reach for `infra-ops@ana-docker` for sudo ops.
|
||||||
_Archived 2026-07-08._
|
_Archived 2026-07-08._
|
||||||
|
## Archived 2026-08-02 — Recent decisions (archived)
|
||||||
|
|
||||||
|
### 2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session
|
||||||
|
|
||||||
|
- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` +
|
||||||
|
`envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal),
|
||||||
|
YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my
|
||||||
|
parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.)
|
||||||
|
|
||||||
|
_Archived 2026-08-02._
|
||||||
|
|
||||||
|
### 2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b
|
||||||
|
|
||||||
|
- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker
|
||||||
|
docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB
|
||||||
|
registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the
|
||||||
|
2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to
|
||||||
|
remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because
|
||||||
|
daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never
|
||||||
|
reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd
|
||||||
|
opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`,
|
||||||
|
`rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the
|
||||||
|
frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime
|
||||||
|
(zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares
|
||||||
|
`live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start,
|
||||||
|
never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers
|
||||||
|
once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon;
|
||||||
|
log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running
|
||||||
|
containers).
|
||||||
|
|
||||||
|
_Archived 2026-08-02._
|
||||||
|
|
||||||
|
### 2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped
|
||||||
|
|
||||||
|
- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into
|
||||||
|
gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17,
|
||||||
|
vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack
|
||||||
|
(01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829);
|
||||||
|
max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and
|
||||||
|
`repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture
|
||||||
|
no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone
|
||||||
|
voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514).
|
||||||
|
**Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not
|
||||||
|
A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s).
|
||||||
|
|
||||||
|
_Archived 2026-08-02._
|
||||||
|
|
||||||
|
### 2026-07-09-granite-gen-memory-extractor-bind-host-synced-on
|
||||||
|
|
||||||
|
- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed,
|
||||||
|
#335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides
|
||||||
|
intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on
|
||||||
|
`summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit
|
||||||
|
both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen**
|
||||||
|
(b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted
|
||||||
|
`worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL
|
||||||
|
NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1`
|
||||||
|
after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config
|
||||||
|
INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven
|
||||||
|
safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed).
|
||||||
|
StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON:
|
||||||
|
the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never
|
||||||
|
updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) —
|
||||||
|
pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass`
|
||||||
|
parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider
|
||||||
|
description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix
|
||||||
|
(operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING
|
||||||
|
defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs
|
||||||
|
when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway
|
||||||
|
config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD;
|
||||||
|
live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09):
|
||||||
|
character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving
|
||||||
|
character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping
|
||||||
|
principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the
|
||||||
|
dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent
|
||||||
|
but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44,
|
||||||
|
UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly
|
||||||
|
from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) +
|
||||||
|
mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's
|
||||||
|
`load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character +
|
||||||
|
character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window
|
||||||
|
VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev
|
||||||
|
to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked
|
||||||
|
config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly
|
||||||
|
re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard
|
||||||
|
+ character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning
|
||||||
|
downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying
|
||||||
|
(both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37**
|
||||||
|
(char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09):
|
||||||
|
mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx
|
||||||
|
131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45
|
||||||
|
canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push
|
||||||
|
of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity
|
||||||
|
block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev),
|
||||||
|
monitor armed. See [[reference_corviduo_dev_emergency_ops]].
|
||||||
|
|
||||||
|
_Archived 2026-08-02._
|
||||||
|
|
||||||
|
### 2026-07-09-granite-gen-memory-extractor-bind-green-lit-for
|
||||||
|
|
||||||
|
- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).**
|
||||||
|
Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty
|
||||||
|
idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching).
|
||||||
|
Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08).
|
||||||
|
**This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the
|
||||||
|
operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread
|
||||||
|
01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]].
|
||||||
|
|
||||||
|
_Archived 2026-08-02._
|
||||||
|
|
||||||
|
## Archived 2026-08-02 — Tried and abandoned (archived)
|
||||||
|
|
||||||
|
### 2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s
|
||||||
|
|
||||||
|
- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized"
|
||||||
|
is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but
|
||||||
|
fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push
|
||||||
|
finishes. FIX: re-run once built, or gate on `workflow_run: completed`.
|
||||||
|
|
||||||
|
_Archived 2026-08-02._
|
||||||
|
|
||||||
|
### 2026-07-04-on-prem-t1-train-that-keeps-any-ana
|
||||||
|
|
||||||
|
- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe
|
||||||
|
ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast
|
||||||
|
on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the
|
||||||
|
566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`.
|
||||||
|
|
||||||
|
_Archived 2026-08-02._
|
||||||
|
|
||||||
|
### 2026-07-04-a-systemd-user-daemon-that-shells-out-to
|
||||||
|
|
||||||
|
- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools
|
||||||
|
needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The
|
||||||
|
althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO
|
||||||
|
routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`.
|
||||||
|
|
||||||
|
_Archived 2026-08-02._
|
||||||
|
|
||||||
|
### 2026-07-09-fp8-breaks-morpheus-audio-token-generation
|
||||||
|
|
||||||
|
- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC
|
||||||
|
frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks-
|
||||||
|
TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the
|
||||||
|
bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real.
|
||||||
|
|
||||||
|
_Archived 2026-08-02._
|
||||||
|
|
||||||
|
### 2026-07-09-sentence-chunking-tts-loses-prosody
|
||||||
|
|
||||||
|
- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that
|
||||||
|
spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning
|
||||||
|
(penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path).
|
||||||
|
|
||||||
|
_Archived 2026-08-02._
|
||||||
|
|
||||||
|
### 2026-07-09-hf-whisper-datasets-aren-t-actually-whispered
|
||||||
|
|
||||||
|
- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8
|
||||||
|
(not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the
|
||||||
|
cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) —
|
||||||
|
that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting.
|
||||||
|
|
||||||
|
_Archived 2026-08-02._
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +0,0 @@
|
|||||||
- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized"
|
|
||||||
is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but
|
|
||||||
fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push
|
|
||||||
finishes. FIX: re-run once built, or gate on `workflow_run: completed`.
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools
|
|
||||||
needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The
|
|
||||||
althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO
|
|
||||||
routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`.
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe
|
|
||||||
ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast
|
|
||||||
on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the
|
|
||||||
566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`.
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` +
|
|
||||||
`envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal),
|
|
||||||
YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my
|
|
||||||
parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.)
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC
|
|
||||||
frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks-
|
|
||||||
TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the
|
|
||||||
bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real.
|
|
||||||
@@ -1,7 +0,0 @@
|
|||||||
- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).**
|
|
||||||
Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty
|
|
||||||
idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching).
|
|
||||||
Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08).
|
|
||||||
**This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the
|
|
||||||
operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread
|
|
||||||
01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]].
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed,
|
|
||||||
#335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides
|
|
||||||
intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on
|
|
||||||
`summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit
|
|
||||||
both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen**
|
|
||||||
(b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted
|
|
||||||
`worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL
|
|
||||||
NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1`
|
|
||||||
after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config
|
|
||||||
INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven
|
|
||||||
safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed).
|
|
||||||
StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON:
|
|
||||||
the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never
|
|
||||||
updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) —
|
|
||||||
pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass`
|
|
||||||
parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider
|
|
||||||
description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix
|
|
||||||
(operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING
|
|
||||||
defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs
|
|
||||||
when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway
|
|
||||||
config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD;
|
|
||||||
live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09):
|
|
||||||
character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving
|
|
||||||
character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping
|
|
||||||
principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the
|
|
||||||
dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent
|
|
||||||
but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44,
|
|
||||||
UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly
|
|
||||||
from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) +
|
|
||||||
mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's
|
|
||||||
`load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character +
|
|
||||||
character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window
|
|
||||||
VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev
|
|
||||||
to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked
|
|
||||||
config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly
|
|
||||||
re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard
|
|
||||||
+ character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning
|
|
||||||
downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying
|
|
||||||
(both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37**
|
|
||||||
(char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09):
|
|
||||||
mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx
|
|
||||||
131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45
|
|
||||||
canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push
|
|
||||||
of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity
|
|
||||||
block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev),
|
|
||||||
monitor armed. See [[reference_corviduo_dev_emergency_ops]].
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8
|
|
||||||
(not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the
|
|
||||||
cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) —
|
|
||||||
that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting.
|
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into
|
|
||||||
gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17,
|
|
||||||
vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack
|
|
||||||
(01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829);
|
|
||||||
max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and
|
|
||||||
`repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture
|
|
||||||
no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone
|
|
||||||
voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514).
|
|
||||||
**Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not
|
|
||||||
A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s).
|
|
||||||
@@ -1,3 +0,0 @@
|
|||||||
- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that
|
|
||||||
spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning
|
|
||||||
(penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path).
|
|
||||||
@@ -1,16 +0,0 @@
|
|||||||
- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker
|
|
||||||
docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB
|
|
||||||
registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the
|
|
||||||
2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to
|
|
||||||
remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because
|
|
||||||
daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never
|
|
||||||
reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd
|
|
||||||
opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`,
|
|
||||||
`rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the
|
|
||||||
frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime
|
|
||||||
(zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares
|
|
||||||
`live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start,
|
|
||||||
never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers
|
|
||||||
once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon;
|
|
||||||
log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running
|
|
||||||
containers).
|
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
`[2026-08-02]` **The mimir-inbox / #377-read-path arc — deploy, four bugs found+fixed+verified, a cloned voice, all in one long session (2026-08-01→02).**
|
||||||
|
|
||||||
|
The browser-facing half of the #377 Muninn ingestion arc, end to end: mimir-inbox stood up, the write path proven, the read path chased through four defects to a verified-working state, and a character voice cloned into the TTS zoo. Peers: mimir-dev (the app), muninn-dev (gate/watcher spec), worldtree-dev (Worldtree app layer + the #380/#381/#382/#383 fixes), ratatoskr-dev (a consumer + the rigorous verifier).
|
||||||
|
|
||||||
|
## mimir-inbox deployed (#377)
|
||||||
|
- **New infra-ops stack, canonical eshpfi `stacks/mimir-inbox/`; live corviduo-dev `10.250.50.152:8091`** (co-located w/ muninn-gate :8090 + the worldtree-personal muninn watcher). Full deploy detail + procedures → auto-memory `reference_mimir_inbox_deploy`.
|
||||||
|
- **Placement decision (operator, reversed):** 7-31 he ruled mimir-inbox stays OFF corviduo-dev (shared/NFS mount); 8-01 he REVERSED to CO-LOCATE. Trigger: muninn-dev's code-check showed staging is NOT same-fs-constrained (gate reads staging metadata + passes path strings; `os.replace` is inside `ingestion_root`) — staging's real constraint is **path-identity across writer/gate/watcher**, which co-location buys outright while dodging NFS failure modes. I HELD the reversal for the operator's direct word (data/hosting on a team-managed box, reversing his own ruling) even against 3 peer relays — vindicated as the right instinct; muninn-dev agreed.
|
||||||
|
- Build: **`uv sync --no-dev --frozen`, SINGLE-STAGE** (project installs editable-linked to `src/`, so src/ MUST stay beside .venv — a multi-stage "copy only .venv" dies at import/404s assets). uid 1000, host-net bind 10.250.50.152:8091, TCP-liveness healthcheck (deliberately NOT gate-coupled). Redeploy = refresh build context (**preserve the on-server `.env`!**) → `docker build -t mimir-inbox:0.0.1 -t mimir-inbox:<sha> .` → `compose up -d`. Version stays 0.0.1 across dev commits → tag the image w/ the source SHA too. Live commit progression `0478452`→`c8ab38f`→`2dcc77e`→**`8ece117`** (3 redeploys).
|
||||||
|
- mimir-inbox key on the gate bumped [read,submit]→**[read,submit,control]** (cancel/retry); brokered via a 0600 drop on nh3-dev (never on the althing bus).
|
||||||
|
|
||||||
|
## The read-path bug chain (worldtree-dev's, all found via this arc)
|
||||||
|
- **#380 wing-blind indexing:** the book-ingest path upserted concepts into a hardcoded `main` Chroma collection while wing search reads the `fiction` collection → P&P written to disk but `search_library` returned total 0. A silent-success defect ("complete/69 indexed" was right about the WRITE, wrong drawer). Root-caused off MY physical evidence (files on disk + search empty). Fixed b164 + a one-shot `--reindex <job_id>` (re-upsert into the right wing collection + delete stray `main` rows).
|
||||||
|
- **#381 stale Chroma client:** the personal api opens its Chroma client before the watcher's cross-process writes → **a freshly-ingested/re-indexed book is NOT queryable until the api is restarted.** Proven by my restart-diagnostic (pre-restart total 0 → post-restart hits, same index). Workaround until fixed: `docker restart worldtree-personal-worldtree-api-1` after any ingest/re-index. Filed as #381.
|
||||||
|
- **#382 unreliable Mimir grounding (the subtle one):** post-#380-fix the index was correct, but Mimir's grounding was INTERMITTENT — some sessions navigated the opaque job-hash dir (`mimir-f3887c9b97b7`) to the content, others distrusted the correct vector hits and **silently answered from training knowledge** (worst of the looks-fine-isn't family). ratatoskr-dev caught it; I'd been over-confident ("Mimir read Austen back to you") having verified the INDEX, not the GROUNDING. Fixed b166 with BOTH shapes: a self-describing `_index.md` per wing job-dir (resolves the hash dir to its title) + a Mimir prompt rule (wing-scoped hits ARE library content, never discard on a name mismatch, never substitute training). **Verified: ratatoskr-dev re-ran 3× fresh sessions → 3/3 grounded**, citations in note-extracted language not raw Austen. #382 CLOSED.
|
||||||
|
- **DCC (Dungeon Crawler Carl, job `b59c147c5ce0`) backfill:** `--reindex` FAILED ("job not found in any state dir" — predates state-tracking). SETTLED = **no re-file** (the b166 prompt rule already grounds it even without an `_index.md`; ratatoskr confirmed incidentally); an `_index.md` rides whenever DCC is next re-ingested.
|
||||||
|
- **#377 mimir-inbox banner bug (mimir-dev's, `8ece117`):** `/health-banner` misattributed an unwritable `ingestion_root` to the WORKER, rendering "The worker is not running." for a running worker — a false lead pointed at infra-ops's half of #377. Fixed (guard split into two banners); I confirmed from the DEPLOYED handler (not just the test) that `ingestion_root_writable:False` now renders "The ingestion root is not writable."
|
||||||
|
|
||||||
|
## muninn-gate → muninn-dispatch 0.1.5
|
||||||
|
Rebuilt `muninn-gate` off `vh/muninn-gate` main `bc04c4c` (dispatch 0.1.4→0.1.5) so the gate serves the new `concept_schema`/`concept_schema_source` row fields (computed gate-side). Gate version unchanged 0.0.14 (dual-tag the SHA). Build needs the vh gitea token as a BuildKit secret (`--secret id=gitea_pw`, UV_INDEX_GITEA_USERNAME=vh, drop+shred). Recreate with `compose up -d` (NOT bare restart — needs the new image). Verified: P&P job serves `concept_schema='fiction'`, `concept_schema_source=null` (null correct — pre-b164 job). Registry tags by commit SHA — `v1.0.0bNNN` docker tags don't resolve; use the deployed SHA (confirm `--reindex` present before using an image for a data-op).
|
||||||
|
|
||||||
|
## donut voice (65-frost → Zonos gateway)
|
||||||
|
Operator: "pick up 65-frost, use that bundle as a voice for a character named donut." 65-frost = a **Booth id** (`~/booth-data/65-frost/`) holding a curated yt-voice-clipper dataset (`dataset-…-curated.zip`: 4 clips + manifest, all SPEAKER_02 = Princess Donut). **Zonos gateway voice registry = a filesystem drop:** `<Name>.wav` in the voices dir (44.1kHz mono s16 PCM) auto-registers as `voice:"<name>"` on **startup** (needs a restart). The LIVE dir is the bind mount `/opt/docker/compose/zonos-gateway/voices/` (lkraven-writable), NOT the working tree. Built `Donut.wav` from seg000 (best clip), dropped it, restarted → `voice:"donut"` live in the gateway AND the Asset Engine's make form. Also copied to the build-source tree `~/zonos-gateway/voices/` for rebuild-durability (true canonical = the gitea repo, not yet CI-wired). Auditioned in booth `donut-voice`.
|
||||||
|
|
||||||
|
## Zonos streaming (no gateway change needed)
|
||||||
|
ratatoskr wanted play-as-it-arrives. `/v1/audio/speech` ALREADY streams — chunked `StreamingResponse`, opens native `/tts/generate` with `stream=True`, wraps as a streaming int16 WAV with `0xFFFFFFFF` placeholder sizes (meant for progressive `<audio>`). Verified TTFB 0.44s vs 6.84s total, `transfer-encoding: chunked`, dials preserved. ratatoskr's proxy was rewriting the placeholder header → forced buffering. Fix was theirs (pass chunks through); shipped + confirmed (TTFB 0.46s progressive). The Asset Engine (ana-docker:8200) IS the fleet "TTS zoo" (~20 audio svcs w/ irv-ml1 endpoints); zonos-gateway registered there, state=ready.
|
||||||
|
|
||||||
|
## Lessons (also in Tried-and-abandoned)
|
||||||
|
- **Verifying the INDEX (search returns hits) is NOT verifying GROUNDING** (does the agent trust+use them vs. silently answer from training). Check that citations are note-extracted, not model-knowledge. ratatoskr caught this after my over-confident "it works."
|
||||||
|
- **Reading the DEPLOYED artifact > trusting the test** for "is the fix live" — the test proves the source is right; reading the running code proves the artifact is, which is what an on-call actually meets.
|
||||||
|
- Held a boundary-box/data reversal for the operator's DIRECT word against 3 peer relays — the right call (peer relay ≠ operator consent; the placement guard was vindicated).
|
||||||
|
|
||||||
|
See also: [[2026-07-31-muninn-gate-deploy]]. auto-memory: `reference_mimir_inbox_deploy`, `reference_muninn_gate_deploy`, `reference_muninn_gate_staging_path`, `reference_zonos_tts_stack`, `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
|
||||||
+15
-33
@@ -1,6 +1,6 @@
|
|||||||
# Persistent memory — eshpfi-management
|
# Persistent memory — eshpfi-management
|
||||||
|
|
||||||
_Last updated: 2026-07-31_
|
_Last updated: 2026-08-02_
|
||||||
|
|
||||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||||
@@ -109,24 +109,24 @@ no longer deployed sidecars here. See Recent decisions.)
|
|||||||
|
|
||||||
## Current state / in-flight
|
## Current state / in-flight
|
||||||
|
|
||||||
_As of 2026-07-31 — **session at a natural close; nothing infra-ops-side actively blocked.** A long single-session run 2026-07-30→31 (model pulls, TTS eval, a gateway root-cause, an SDK publish, and the muninn-gate deploy). All shipped/committed; `main == origin/main` after the muninn-gate commit `786462a`. Each item has a Recent-decisions entry:_
|
_As of 2026-08-02 — **session at a natural close; nothing infra-ops-side blocked.** A very long run 2026-08-01→02: the mimir-inbox / #377-read-path arc end to end (deploy + 4 bugs found/fixed/verified + a cloned voice). Full arc → `persistent-memory.d/2026-08-02-mimir-inbox-arc.md`; each sub-item also has a Recent-decisions entry. Headlines:_
|
||||||
- **muninn-gate DEPLOYED + healthy on corviduo-dev `10.250.50.152:8090`** (#377 ingestion front door). `/ping` + `/health` (watcher:running:true ⇒ ingestion_root byte-identity proven) verified first boot. **SUBMIT path DEFERRED** until mimir-inbox (pre-implementation) + the real shared NFS mount exist → then repoint the staging bind (**+restart**, config is read-once), run muninn-dev's one-file path-agreement probe, then the acceptance run; **hand mimir-dev the `mimir-inbox` bearer key** (in the on-server `/opt/docker/conf/muninn-gate/muninn-gate.yaml`, 0600) when they implement. auto-memory `reference_muninn_gate_deploy`.
|
- **mimir-inbox DEPLOYED + verified on corviduo-dev `10.250.50.152:8091`** (#377 browser-facing half; write+read proven end to end). Live commit `8ece117` (3 redeploys); co-located per the operator's reversed-to-CO-LOCATE ruling. auto-memory `reference_mimir_inbox_deploy`.
|
||||||
- **kimi-k3 output cap root-caused; fix RELAYED to heid, NOT applied gateway-side.** It's a ~16384 **reasoning**-token ceiling (completion is uncapped); fix = `reasoning_effort: low` via **`extra_body`** (LiteLLM `drop_params` strips the top-level param). Proven on the wire against heid's real bundle (reasoning drops under the ceiling → content returns, both coding + general endpoints). heid to validate on a real round; **backstop if `extra_body` doesn't survive LiteLLM = add `allowed_openai_params: ["reasoning_effort"]` to the kimi-k3 route** (gateway change + restart).
|
- **#377 read path fully working** — P&P ingested + queryable via Mimir on personal :8081. Chased through worldtree-dev bugs **#380** (wing-blind index → concepts in wrong collection; fixed b164 + one-shot `--reindex <job_id>`), **#381** (stale Chroma client → **restart `worldtree-personal-worldtree-api-1` after any ingest/re-index** until their fix), **#382** (intermittent Mimir grounding / silent training-substitution; fixed b166 = `_index.md` per wing job-dir + a prompt rule; **verified 3/3** by ratatoskr-dev). DCC re-file SETTLED = **no** (prompt rule grounds it; `_index.md` rides next re-ingest).
|
||||||
- **worldtree-sdk 1.1.0 (Python) PUBLISHED** to the vh Gitea PyPI (memory_context pass-through; unblocked wyrd-dev). claude-bot added as **write collaborator** on `vh/worldtree-sdk` (source via API archive — git-HTTP 403s on that repo). **Operator-directed cred: a FULL vh site-admin token is saved at `~/.config/gitea/vh-token` (0600)** for infra-ops SDK publishing — ⚠️ high blast radius; operator chose to keep it over a scoped one. auto-memory `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
|
- **muninn-gate → muninn-dispatch 0.1.5** (rebuilt off `vh/muninn-gate` `bc04c4c`, image 0.0.14; serves `concept_schema`/`concept_schema_source`). BuildKit gitea secret; recreate w/ `compose up -d` not bare restart.
|
||||||
- **magpie TTS evaluated → REJECTED** (Zonos wins expressive English; multilingual not needed). Not served; `magpie-nemo` torn down. **KEEP `/mnt/smithy/models/nvidia/magpie_tts_multilingual_357m/`** — brokkr's fine-tuning base. auto-memory `project_magpie_tts_eval_rejected`.
|
- **donut voice** cloned from the **65-frost Booth bundle** → registered in the Zonos gateway (`voice:"donut"`, live in the Asset Engine TTS-zoo make form; auditioned in booth `donut-voice`). Zonos `/v1/audio/speech` already streams (chunked, TTFB ~0.44s) — ratatoskr shipped the client-side chunk-passthrough for play-as-it-arrives; **no gateway change was needed**.
|
||||||
|
|
||||||
**Open follow-ups (non-blocking — pick one up or not):**
|
**Open follow-ups (non-blocking — pick one up or not):**
|
||||||
- **Zonos emotion:** sad axes/text pass on the 3 calibrated voices (only named-sad, untested); emotion-congruent-text pass (validates intensity, may rescue sad id); clone-char (Emmie/Penny/Natalie/Miranda) emotion rows use the mid-region fallback until measured. Presets are **provisional** (neutral-text ear-check was inconclusive). Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio,assemble_voice}.py` (run ON irv-ml1; dial-in studio = nohup :8898 on nh3-dev). dvalin thread at rest (`01KXT12FN0AS…`). → `persistent-memory.d/2026-07-18-zonos-gateway-0.2.1-emotion-presets.md`
|
- **Zonos emotion:** sad axes/text pass on the 3 calibrated voices (only named-sad, untested); emotion-congruent-text pass (validates intensity, may rescue sad id); clone-char (Emmie/Penny/Natalie/Miranda) emotion rows use the mid-region fallback until measured. Presets are **provisional** (neutral-text ear-check was inconclusive). Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio,assemble_voice}.py` (run ON irv-ml1; dial-in studio = nohup :8898 on nh3-dev). dvalin thread at rest (`01KXT12FN0AS…`). → `persistent-memory.d/2026-07-18-zonos-gateway-0.2.1-emotion-presets.md`
|
||||||
- **zonos-gateway CI-wire:** deployed irv-ml1 tree `/opt/docker/compose/zonos-gateway` is still NON-git (hand-updated build context) — git-connect + build-on-push like the other sisters. (Same pattern soong-lab now has.)
|
- **zonos-gateway CI-wire:** deployed irv-ml1 tree `/opt/docker/compose/zonos-gateway` is still NON-git (hand-updated build context) — git-connect + build-on-push like the other sisters. (Same pattern soong-lab now has.)
|
||||||
- **soong-lab:** cutover DONE + **auto-redeploy DONE + validated 2026-07-18** (CI-deploy step live; dispatch run #5 recreated the live container ...541f7730 → ...07526a08, health-gated green). Deploy dir now **/opt/soong-lab** (deploy-owned, mirrors /opt/worldtree); old `/home/infra-ops/soong-lab-deploy` retired (`.retired-20260718`). Dedicated soong-only ed25519 deploy key on `deploy`'s authorized_keys (fp SHA256:MG7M3Ri…). → `persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md`.
|
- **soong-lab:** cutover DONE + **auto-redeploy DONE + validated 2026-07-18** (CI-deploy step live; dispatch run #5 recreated the live container ...541f7730 → ...07526a08, health-gated green). Deploy dir now **/opt/soong-lab** (deploy-owned, mirrors /opt/worldtree); old `/home/infra-ops/soong-lab-deploy` retired (`.retired-20260718`). Dedicated soong-only ed25519 deploy key on `deploy`'s authorized_keys (fp SHA256:MG7M3Ri…). → `persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md`.
|
||||||
|
|
||||||
**Zonos voice stack (LIVE, unchanged):** 8 voices in `zonos-gateway` (`:8890` irv-ml1) — defaults AmericanFemale/Male/BritishFemale/Cora + 4 clones Emmie/Penny/Natalie/Miranda; add a voice = drop `<Name>.wav` in `/opt/docker/compose/zonos-gateway/voices/` + `docker compose restart` (host-managed bind-mount, NO rebuild). Clone pipeline: `/mnt/smithy/voice_clones/<name>.zip` → `assemble_voice.py` → drop. Dial-in studio http://10.100.10.50:8898/ (nohup on nh3-dev, relaunch `nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &`).
|
**Zonos voice stack (LIVE):** **9 voices** in `zonos-gateway` (`:8890` irv-ml1) — defaults AmericanFemale/Male/BritishFemale/Cora + 4 clones Emmie/Penny/Natalie/Miranda + **donut** (2026-08-02, from the 65-frost bundle); add a voice = drop `<Name>.wav` (44.1kHz mono s16 PCM) in `/opt/docker/compose/zonos-gateway/voices/` + `docker compose restart` (host-managed bind-mount, NO rebuild; registry scans at startup). Also mirror into the build-source tree `~/zonos-gateway/voices/` for rebuild-durability. Clone pipeline: `/mnt/smithy/voice_clones/<name>.zip` → `assemble_voice.py` → drop. Dial-in studio http://10.100.10.50:8898/ (nohup on nh3-dev, relaunch `nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &`).
|
||||||
|
|
||||||
**althing monitor** ARMED (handle `infra-ops`, wake-listener `b1fbu3tcb`; herald up). ⚠️ Re-arm ONLY after an actual FIRE (`<task-notification> completed rc0`), never after a plain operator turn (bounces rc3). Spawn `althing-wake-listener` as its OWN `run_in_background` task — NEVER chain with `&`/`&&` (orphans it → untracked → mail unwatched; I fumbled this ~4× this session, each time = stop-monitor + re-arm clean). **Open watches:** worldtree-dev may ping when their `/embed` code fix reaches PERSONAL (still 503 on b146 until a staging promotion — operator's call); #363 research-wing ingest PARKED.
|
**althing monitor** ARMED (handle `infra-ops`; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after an actual FIRE (`<task-notification> completed rc0`), never after a plain operator turn (bounces rc3). Spawn `althing-wake-listener` as its OWN `run_in_background` task — NEVER chain with `&`/`&&` (orphans it → untracked → mail unwatched; I fumbled this ~4× this session, each time = stop-monitor + re-arm clean). **Open watches:** worldtree-dev may ping when their `/embed` code fix reaches PERSONAL (still 503 on b146 until a staging promotion — operator's call); #363 research-wing ingest PARKED.
|
||||||
|
|
||||||
**Two small pending items (operator's call, non-urgent):** (1) bless/reshape the `env.public` non-secret-env-overlay mechanism in the config repo; (2) the pre-existing herald pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range" — likely `render_command messages[0]` on empty list; NOT infra-ops's, rec = flag to althing-dev).
|
**Two small pending items (operator's call, non-urgent):** (1) bless/reshape the `env.public` non-secret-env-overlay mechanism in the config repo; (2) the pre-existing herald pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range" — likely `render_command messages[0]` on empty list; NOT infra-ops's, rec = flag to althing-dev).
|
||||||
|
|
||||||
**eshpfi is PUSHED** — `main == origin/main`. `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's); `graphify-out/GRAPH_REPORT.md` = graphify-hook artifact (churns on every commit, ignore).
|
**eshpfi has UNPUSHED local commits** — `main` is ~8 ahead of `origin/main` (tonight's mimir-inbox + muninn-gate doc-syncs + Audio8/#383 notes + this snapshot, and the 7-31 muninn-gate/snapshot). **Push is the operator's call.** `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's); `graphify-out/GRAPH_REPORT.md` = graphify-hook artifact (churns on every commit, ignore).
|
||||||
|
|
||||||
**PARKED (grok-code/Codex):** operator asked about fronting grok-code / Codex behind the LiteLLM gateway. Rec (given): raw models behind the gateway → **API keys** (native `xai/` + `openai/` providers, the GLM-passthrough pattern); fleet *consults* → the **Heid/Eitri peer-CLI** pattern (Codex already wired). Do NOT reverse-proxy the subscription CLIs (grok CLI / Codex CLI, OAuth-auth) into the gateway — ToS + account-ban risk + brittle. Untracked by operator choice; no decision made.
|
**PARKED (grok-code/Codex):** operator asked about fronting grok-code / Codex behind the LiteLLM gateway. Rec (given): raw models behind the gateway → **API keys** (native `xai/` + `openai/` providers, the GLM-passthrough pattern); fleet *consults* → the **Heid/Eitri peer-CLI** pattern (Codex already wired). Do NOT reverse-proxy the subscription CLIs (grok CLI / Codex CLI, OAuth-auth) into the gateway — ToS + account-ban risk + brittle. Untracked by operator choice; no decision made.
|
||||||
|
|
||||||
@@ -134,6 +134,8 @@ _As of 2026-07-31 — **session at a natural close; nothing infra-ops-side activ
|
|||||||
|
|
||||||
## Recent decisions
|
## Recent decisions
|
||||||
|
|
||||||
|
- `[2026-08-02]` **mimir-inbox / #377-read-path arc — deployed + 4 bugs found/fixed/verified + a cloned voice.** mimir-inbox live on corviduo-dev:8091 (#377 write+read proven, live `8ece117`); worldtree-dev #380 (wing-blind index) + #381 (stale-client restart) + #382 (intermittent Mimir grounding) chased and **verified 3/3** by ratatoskr-dev; muninn-gate → dispatch 0.1.5; **donut** voice cloned from the 65-frost Booth bundle into the Zonos gateway; Zonos streaming confirmed already-working. Full arc, procedures, and lessons → `persistent-memory.d/2026-08-02-mimir-inbox-arc.md`
|
||||||
|
|
||||||
- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md`
|
- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md`
|
||||||
|
|
||||||
- `[2026-07-31]` **worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred.** memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on `vh/worldtree-sdk` (source pulled via the **Gitea API archive** — git-HTTP 403s on that repo); publishing to the vh USER namespace **can't be delegated** (401 `reqPackageAccess` even with `write:package`) so it needs an owner token — operator saved a **FULL vh site-admin token at `~/.config/gitea/vh-token` (0600)** for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
|
- `[2026-07-31]` **worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred.** memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on `vh/worldtree-sdk` (source pulled via the **Gitea API archive** — git-HTTP 403s on that repo); publishing to the vh USER namespace **can't be delegated** (401 `reqPackageAccess` even with `write:package`) so it needs an owner token — operator saved a **FULL vh site-admin token at `~/.config/gitea/vh-token` (0600)** for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
|
||||||
@@ -246,18 +248,8 @@ _As of 2026-07-31 — **session at a natural close; nothing infra-ops-side activ
|
|||||||
|
|
||||||
- `[2026-07-10]` Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized) → `persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md`
|
- `[2026-07-10]` Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized) → `persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md`
|
||||||
|
|
||||||
- `[2026-07-09]` Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker docker log-cap SOLVED no-bounce → `persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md`
|
|
||||||
|
|
||||||
- `[2026-07-09]` granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed, #335 Slice-4) → `persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md`
|
|
||||||
|
|
||||||
- `[2026-07-09]` mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into gateway-chat → `persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md`
|
|
||||||
|
|
||||||
- `[2026-07-09]` granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4) → `persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md`
|
|
||||||
|
|
||||||
- `[2026-07-08]` RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (3… → `persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md`
|
- `[2026-07-08]` RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (3… → `persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md`
|
||||||
|
|
||||||
- `[2026-07-08]` worldtree Mimir deploy-blocker resolved (mid-session): → `persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md`
|
|
||||||
|
|
||||||
- `[2026-07-08]` OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training → `persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md`
|
- `[2026-07-08]` OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training → `persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md`
|
||||||
|
|
||||||
- `[2026-07-08]` DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call) → `persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md`
|
- `[2026-07-08]` DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call) → `persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md`
|
||||||
@@ -266,10 +258,12 @@ _As of 2026-07-31 — **session at a natural close; nothing infra-ops-side activ
|
|||||||
|
|
||||||
- `[2026-07-08]` T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe → `persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md`
|
- `[2026-07-08]` T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe → `persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md`
|
||||||
|
|
||||||
_142 older entries archived to archival-memory.md._
|
_147 older entries archived to archival-memory.md._
|
||||||
|
|
||||||
## Tried and abandoned
|
## Tried and abandoned
|
||||||
|
|
||||||
|
- `[2026-08-02]` **Verifying the INDEX is not verifying GROUNDING** (#382). A `search_library` returning wing=fiction hits proves the content is *retrievable*; it does NOT prove the agent (Mimir) *trusts and uses* those hits vs. silently answering from training. I reported "Mimir read Austen back to you" off a grounded-*looking* answer; ratatoskr-dev caught that grounding was intermittent (some sessions discarded the correct hits and substituted training knowledge). Test the harder claim — are the citations note-extracted or model-knowledge? — and reading the DEPLOYED artifact beats trusting the test for "is the fix live."
|
||||||
|
|
||||||
- `[2026-07-30]` **brokkr's WebSearch "verification" CONFIRMED a hallucination — 3 phantom `microsoft/Mage-Flow-{Base,Turbo,Edit}` repo IDs.** brokkr-smithy-dev handed 3 gated-looking repo IDs for an operator-directed model pull; they don't exist (its own web-search fabricated an arXiv ID + project page, twice). Lesson: the HF **registry API is ground truth** — an unauth 401 ≠ exists (`{"error":"Invalid username or password"}` masks private/gated/nonexistent alike), an authed 404 = phantom, and `author=X&search=Y` refutes existence. API-verify every repo ID before a pull; LLM-summarized web fetches confabulate. auto-memory `reference_verify_hf_repo_ids_before_pull`.
|
- `[2026-07-30]` **brokkr's WebSearch "verification" CONFIRMED a hallucination — 3 phantom `microsoft/Mage-Flow-{Base,Turbo,Edit}` repo IDs.** brokkr-smithy-dev handed 3 gated-looking repo IDs for an operator-directed model pull; they don't exist (its own web-search fabricated an arXiv ID + project page, twice). Lesson: the HF **registry API is ground truth** — an unauth 401 ≠ exists (`{"error":"Invalid username or password"}` masks private/gated/nonexistent alike), an authed 404 = phantom, and `author=X&search=Y` refutes existence. API-verify every repo ID before a pull; LLM-summarized web fetches confabulate. auto-memory `reference_verify_hf_repo_ids_before_pull`.
|
||||||
|
|
||||||
- `[2026-07-30]` **magpie TTS serving — evaluated, ABANDONED.** Pulled `magpie_tts_multilingual_357m` (the one real repo of brokkr's batch) to NFS, stood it up on irv-ml1 (ephemeral NeMo-Speech-`main` container — stock PyPI/NGC NeMo can't load v2607), A/B'd vs Zonos → Zonos wins expressive English decisively, multilingual not needed. Not served; `magpie-nemo` torn down. `.nemo` KEPT on NFS as brokkr's fine-tuning base. auto-memory `project_magpie_tts_eval_rejected`.
|
- `[2026-07-30]` **magpie TTS serving — evaluated, ABANDONED.** Pulled `magpie_tts_multilingual_357m` (the one real repo of brokkr's batch) to NFS, stood it up on irv-ml1 (ephemeral NeMo-Speech-`main` container — stock PyPI/NGC NeMo can't load v2607), A/B'd vs Zonos → Zonos wins expressive English decisively, multilingual not needed. Not served; `magpie-nemo` torn down. `.nemo` KEPT on NFS as brokkr's fine-tuning base. auto-memory `project_magpie_tts_eval_rejected`.
|
||||||
@@ -306,8 +300,6 @@ _142 older entries archived to archival-memory.md._
|
|||||||
|
|
||||||
- `[2026-07-13]` `althing-cli reply <THREAD_id>` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to <peer>`. Bit me several times this session.
|
- `[2026-07-13]` `althing-cli reply <THREAD_id>` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to <peer>`. Bit me several times this session.
|
||||||
|
|
||||||
- `[2026-07-09]` FP8 breaks mOrpheus audio-token generation → `persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md`
|
|
||||||
|
|
||||||
- `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter,
|
- `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter,
|
||||||
`has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs.
|
`has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs.
|
||||||
(`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.)
|
(`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.)
|
||||||
@@ -316,10 +308,6 @@ _142 older entries archived to archival-memory.md._
|
|||||||
→ RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx]
|
→ RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx]
|
||||||
window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).
|
window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).
|
||||||
|
|
||||||
- `[2026-07-09]` Sentence-chunking TTS loses prosody → `persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md`
|
|
||||||
|
|
||||||
- `[2026-07-09]` HF whisper datasets aren't actually whispered → `persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md`
|
|
||||||
|
|
||||||
- `[2026-07-08]` Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE → `persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md`
|
- `[2026-07-08]` Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE → `persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md`
|
||||||
|
|
||||||
- `[2026-07-08]` Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24) → `persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md`
|
- `[2026-07-08]` Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24) → `persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md`
|
||||||
@@ -338,12 +326,6 @@ _142 older entries archived to archival-memory.md._
|
|||||||
|
|
||||||
- `[2026-07-04]` LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge → `persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md`
|
- `[2026-07-04]` LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge → `persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md`
|
||||||
|
|
||||||
- `[2026-07-04]` A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools needs an explicit `Environment=PATH` → `persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md`
|
|
||||||
|
|
||||||
- `[2026-07-04]` On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS → `persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md`
|
|
||||||
|
|
||||||
- `[2026-07-01]` A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" is usually the pull-only-vs-build RACE, not registry-auth → `persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md`
|
|
||||||
|
|
||||||
- `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS
|
- `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS
|
||||||
moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%,
|
moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%,
|
||||||
N=4 −20%). Reserve for dedicated/interactive deployments.
|
N=4 −20%). Reserve for dedicated/interactive deployments.
|
||||||
@@ -352,4 +334,4 @@ _142 older entries archived to archival-memory.md._
|
|||||||
needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA
|
needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA
|
||||||
ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-<uuid>`.
|
ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-<uuid>`.
|
||||||
|
|
||||||
_101 older entries archived to archival-memory.md._
|
_107 older entries archived to archival-memory.md._
|
||||||
|
|||||||
Reference in New Issue
Block a user