The :8033 seat now serves h2oai/h2o-lightning-4b @ b43f759b on vLLM 0.30.0 with
byte-pinned KV (4.48 GiB, 135,558 tokens) behind services/h2o-decision-front:
32k refuse-not-trim admission, the seat's bearer token, and /decide +
/decide/shared translated onto /v1/systemone for albok-nemi's judge. URL, port,
token and stack name are unchanged. INC down 13:08:18, h2o healthy 13:11:27.
Post-checks on the live seat: /health at the pinned revision; JevBench public
205/231 (= bench); 422 over the cap; 16q at the cap warm 2.0-2.16 s; seat peak
16,164 MiB under 4 concurrent 16q x 32k calls against a 16,847 budget.
compose.inc.yaml + rollback-to-inc.sh restore Intern-Decision-4B in one step.
The temporary GPU 3 eval seat (stacks/h2o-eval) is down.
Prime's call via brokkr: h2o-lightning-4b replaces the intern-decision seat once
albok-dev re-validates Nemi's sensitivity judge on it. services/h2o-decision-front
builds on infra-hermes's admission proxy (30f36f2): 32,000-token refuse-not-trim,
the seat's bearer token, and intern-decision-serve's /decide/shared mapping onto
/v1/systemone so the judge runs unchanged. stacks/h2o-eval serves it on GPU 3
:8034 exactly as benched (vLLM 0.30.0 + unmodified vendor shim; weights hashed
against HF b43f759b). INC keeps serving production on GPU 1.
Pinned to gitea.phasefinal.com/pfi/albok-service:0.1.17@sha256:b2920e22d112631da82cb747d152c5d600cc50134f905c5afeae8bc26d580630,
built from a clean archive of vh/albok 2b190e2 and deployed by albok-dev at 2233 (healthy in under 10 s;
compose.yaml.pre-0.1.17 beside it on the host).
Prime, 2026-10-09: "vh gets albok-everything". playbooks/nh3-albok-vh-viewer.yaml
joins vh to albok-read, albok-personal and albok-feedback on nh3-albok and
authorizes his four Macs' public keys for vh; verified at a fresh login that
every wing's .browse/ lists and reads.
The withdrawal harness gains a macOS driver (run-mac.sh: every mountable FUSE
backend, a read-only smoke of the real wings, results copied back), env
overrides for a non-container client, and summarize.py. Sanity-checked as vh
from the Linux container before staging.
Replica of a browse view on nh3-albok, Linux sshfs 3.7.3 client in an ephemeral
container on nh3-dev, N=5 per condition. Fresh open of a never-opened child
through a cached listing: ENOENT 5/5 with sshfs defaults and 5/5 with hour-long
client caches. Null control (no rename) opens 5/5; held fd and failed-rename
positive controls read 5/5. Replica removed afterwards.
Also closes the post-clear checks in persistent-memory: fv-ml1 GPU 0/1 plateau,
nh3-nas usage, and the mislabelled sfsrv-ana pre-pbs4 item.
Pinned to gitea.phasefinal.com/pfi/albok-service:0.1.16@sha256:b28559ec782f1ce93e89c5eb55c7418561667dde38ff819bb18bb53e2d1f2ddf,
deployed by albok-dev at 1943 (healthy in 20 s; compose.yaml.pre-0.1.16 beside it on the host).
Verified 2026-10-06 for comfy-dev: the gateway keeps no request or response
content. A canary sent through 'gen' appears in none of SpendLogs' text columns,
LiteLLM's container log, any ana-docker container log, or flash-next's log
(positive controls hit on the same stores). In 24h, 0 of 281,432 spend rows
carried messages, response or proxy_server_request. Langfuse is retired, there
are no DB-stored callbacks, and LiteLLM_ErrorLogs is empty. The per-request
no-log flag skips third-party callbacks but not the proxy's own spend-log
callback (v1.97 source), so the guarantee is this setting. Comment-only change;
the live file was updated in place with no restart.
HF_HOME, TORCH_HOME and NLTK_DATA now point under /app/whisperx-env (the
/tank bind mount). Before, ~6 GB of models sat in the container layer and
every recreate threw them away. The cache was copied across (98/2/78 files,
apparent sizes identical).
HF_TOKEN is now a placeholder. Measured: with an invalid token, cached gated
pyannote models still load (huggingface_hub falls back to the cache on the 401),
and public models still download (HTTP 200). Scriberr only checks that the token
is non-empty. Re-verified on GPU 3: pyannote community-1 and WhisperX+3.1
diarization both exit 0 from the /tank cache with no downloads.
Scriberr: pyannote's diarization models are gated, and Scriberr had no Hugging
Face token, so every diarized job died at the diarize step (WhisperX after a full
CPU transcription). HF_TOKEN now passes through compose; the value is only in the
host .env (vault fv-ml1/hf-token). The separate pyannote env had torchcodec 0.16
(needs torch >=2.11/CUDA 13) beside torch 2.8, so pyannote 4 had no
AudioDecoder. Pinned to 0.7 by hand on the live env; patch 0003 makes it
durable at the next rebuild. WhisperX's adapter defaults to cpu/small/float32,
now documented. Both paths verified on GPU 3 with a real multi-speaker clip.
pbs-nh3: the backups-mirror prune job now keeps the primary's history
(last 3, daily 14, weekly 8, monthly 12, yearly 5). Dry-run: keep 738,
remove 3,194 across 32 groups; first run at the job's 00:00.
Image built from a clean archive of vh/albok 751a729 (tag albok-service/v0.1.12),
pushed as gitea.phasefinal.com/pfi/albok-service:0.1.12@sha256:a29a7e46...,
pinned by digest. Recreated between Nemi walks at 1254; healthy at 1254:48,
all four wings ready, canaries alive, write embedder agrees. Config unchanged.
Rollback: compose.yaml.pre-0.1.12 (0.1.11).
Image built from a clean archive of vh/albok ffc6d4e (tag albok-service/v0.1.11),
pushed as gitea.phasefinal.com/pfi/albok-service:0.1.11@sha256:4b641c3e...,
pinned by digest. Recreated between Nemi walks at 0202; healthy at 0202:23, all
four wings ready, canaries alive, write embedder agrees. Config unchanged.
Rollback is compose.yaml.pre-0.1.11, valid only until a wing's first 0.1.11 reindex.
Layout: GPU 0 = cyberprev (KV 16 -> 15 GiB, 1.29x), erp-seat (moved from
GPU 1, KV 8.49 -> 5.5 GiB, 1.32x), voices (ctx 8,192 -> 40,960, its native
maximum, KV 6 GiB, 1.07x), coder (moved from GPU 1, ctx 16,384 -> 32,768,
the card maximum, 2.35x), parakeet-nemo. GPU 1 = gen-small (moved from
GPU 0, 5.5 GiB, 1.76x), meromero-rp (22 GiB, 1.32x), intern-decision.
Sizing rule: KV >= 1.25x max context rounded up to 0.5 GiB, checked against
7 days of LiteLLM traffic and the engines' own KV-usage logs. Each card keeps
spare memory for its bursty tenant (parakeet's cap on GPU 0, intern-decision's
32k call on GPU 1). YaRN extensions are not enabled: they do not fit, and the
Qwen cards warn static YaRN degrades short prompts.
gen-small gets --override-generation-config max_new_tokens 32768. It applies
only to requests that omit max_tokens (vLLM get_max_tokens) and answers 9
runaway ~260k-token replies in 7 days; the longest normal reply was 6,357.
Applied with playbooks/fv-ml1-rebalance-20261005.yaml (boot-check guard and
health/KV verification per seat); all nine seats healthy, gateway smoke
tests pass.
meromero-rp sized its KV from --gpu-memory-utilization, and vLLM's profile
counts co-booting neighbours as its own non-torch memory. Three boots gave
20.37 / 26.74 / 29.04 GiB of KV, and the 2026-10-05 Docker-move restart cut
GPU 1 free memory to 3,981 MiB, below intern-decision's 32k-token budget.
Coder read negative KV on every co-boot. Both are now pinned, so util only
gates the boot check (meromero util 0.52 -> 0.47, just above its footprint).
GPU 0 plateaus at ~60 MiB free 1-2 days after a boot (Beszel), so
gen-small's KV drops to 5.5 GiB (461,215 tokens, 1.76x) and parakeet-nemo
stays on GPU 0 with room for its 3,840 MiB cap. GPU 3 stays clear.
Applied with playbooks/fv-ml1-kv-pin-20261005.yaml: GPU 1 free 3,981 ->
11,801 MiB, GPU 0 495 -> 5,039 MiB right after boot; all nine seats healthy.
The db/ and runs/ bind mounts are root:docker 2775 since the 2026-09-14 tree
normalisation; the app runs as 1000:1000, so SQLite could not create its journal
and every write returned 500 (found at v0.40.2).
HunyuanVideo-Foley, MMAudio and ControlFoley load their Hugging Face dependencies
from pinned snapshots staged on the model store. Without this they look in the
container's ephemeral cache and download again on every recreate. Requested by
comfy-dev; HF_HUB_OFFLINE deliberately not set.
Since skaldsong v0.39.0 the structured pipeline passes call the LiteLLM
gateway directly; with no key every story generation fails 401 at its first
pass. Operator ruling 2026-10-05: the shared all-agents key until Skaldsong
v1.0 final, then a project-scoped key. The stack passes an explicit
environment list, so the .env line alone does not reach the container.
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.
Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).
albok-service leaves nh3-docker before unit 4 puts the personal vault and
licensed books into restricted wings. On nh3-docker, portainer, portainer_agent,
dockge (socket rw) and dockerd on plaintext tcp/2375 were each root over
/srv/albok. The new host runs only albok-service: no socket containers and no
TCP Docker API, so its Homepage tile is a static services.yaml entry.
- nh3-pve-2: vmbr0 made VLAN-aware (applied by reboot, no guests running);
pbs-ana storage in the new namespace nh3-pve-2, plus a 21:30 vzdump job
(pbs-nh3's recursive sync mirrors it)
- VM 110 nh3-albok, 10.100.50.44 on VLAN 50, Debian 13 cloud-init, 4 vCPU
x86-64-v3 / 12 GiB / 100 GB; built to fleet conventions (vh 1000,
infra-ops 850, docker 851)
- compose: mem_limit 4g -> 8g, homepage labels removed; host port 8392 kept
- cutover by tar stream, verified by digest (contents plus owner, group, mode,
mtime and type identical); albok.nh3.internal now points at nh3-albok
- old /srv/albok stays on nh3-docker (container removed, compose renamed)
until acceptance and the operator's approval to delete it
- sudo-io archive hosts and ops-log STACK_HOSTS include nh3-albok
albok-service 0.1.0 (vh/albok 0b37431), image pfi/albok-service pinned by
digest, published on host port 8392 because 8390 is the post office.
Host prep playbook creates the fixed ids (albok 1500, albok-read 1510,
albok-personal 1511) and the local store/private roots; the container
gets a mounted /etc/group and group_add so the service can resolve and
chgrp its wing dirs. The config carries a LiteLLM key scoped to
qwen3-embedding and lives outside the deploy-synced conf dir. DNS name
albok.nh3.internal.
vh