Image built from a clean archive of vh/albok ffc6d4e (tag albok-service/v0.1.11),
pushed as gitea.phasefinal.com/pfi/albok-service:0.1.11@sha256:4b641c3e...,
pinned by digest. Recreated between Nemi walks at 0202; healthy at 0202:23, all
four wings ready, canaries alive, write embedder agrees. Config unchanged.
Rollback is compose.yaml.pre-0.1.11, valid only until a wing's first 0.1.11 reindex.
Layout: GPU 0 = cyberprev (KV 16 -> 15 GiB, 1.29x), erp-seat (moved from
GPU 1, KV 8.49 -> 5.5 GiB, 1.32x), voices (ctx 8,192 -> 40,960, its native
maximum, KV 6 GiB, 1.07x), coder (moved from GPU 1, ctx 16,384 -> 32,768,
the card maximum, 2.35x), parakeet-nemo. GPU 1 = gen-small (moved from
GPU 0, 5.5 GiB, 1.76x), meromero-rp (22 GiB, 1.32x), intern-decision.
Sizing rule: KV >= 1.25x max context rounded up to 0.5 GiB, checked against
7 days of LiteLLM traffic and the engines' own KV-usage logs. Each card keeps
spare memory for its bursty tenant (parakeet's cap on GPU 0, intern-decision's
32k call on GPU 1). YaRN extensions are not enabled: they do not fit, and the
Qwen cards warn static YaRN degrades short prompts.
gen-small gets --override-generation-config max_new_tokens 32768. It applies
only to requests that omit max_tokens (vLLM get_max_tokens) and answers 9
runaway ~260k-token replies in 7 days; the longest normal reply was 6,357.
Applied with playbooks/fv-ml1-rebalance-20261005.yaml (boot-check guard and
health/KV verification per seat); all nine seats healthy, gateway smoke
tests pass.
meromero-rp sized its KV from --gpu-memory-utilization, and vLLM's profile
counts co-booting neighbours as its own non-torch memory. Three boots gave
20.37 / 26.74 / 29.04 GiB of KV, and the 2026-10-05 Docker-move restart cut
GPU 1 free memory to 3,981 MiB, below intern-decision's 32k-token budget.
Coder read negative KV on every co-boot. Both are now pinned, so util only
gates the boot check (meromero util 0.52 -> 0.47, just above its footprint).
GPU 0 plateaus at ~60 MiB free 1-2 days after a boot (Beszel), so
gen-small's KV drops to 5.5 GiB (461,215 tokens, 1.76x) and parakeet-nemo
stays on GPU 0 with room for its 3,840 MiB cap. GPU 3 stays clear.
Applied with playbooks/fv-ml1-kv-pin-20261005.yaml: GPU 1 free 3,981 ->
11,801 MiB, GPU 0 495 -> 5,039 MiB right after boot; all nine seats healthy.
Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4).
svos-dev: the service worker fetches its hash-pinned INVENTORY with credentials omitted. Exactly those files plus /sw.js are now served before basic_auth (GET/HEAD, exact files). The list is generated from the release's sw.js by svos-front-inventory, which hash-checks each file and validates before reload. Gated static routes follow the web/README hosting contract: missing assets 404 instead of index.html, other document requests get index.html, everything else 404, unversioned files no-cache. acceptance-matrix.sh: 35/35 live; 16 FAIL against the previous config (positive control). git added to the nh3-svos playbook (kernel's remote check). Live backup drill: quiesced backup ~1-2 s outage; a refused freeze does NOT abort vzdump (unfrozen backup, job reported successful); both images restore-mounted read-only and verified.
Prime's ruling: basic auth saved in the browser, mesh-only access. basic_auth argon2id with the account hash in a root:caddy file outside git (plaintext vaulted). After auth, Tailscale-User-Login is set from the authenticated user and Authorization is deleted, since the kernel's human path refuses requests that carry it. Removed svos-whois-auth (every mesh node is login pfi). Verified from a mesh peer against a temporary header echo: 401 without/with wrong credentials, 200 with exactly one Tailscale-User-Login: Vuong and no forged or Authorization headers upstream.
svos-dev review: the Caddyfile deleted named headers only. Prefix deletes cover the families; the named strays (Tailscale-Headers-Info, X-Real-IP, Forwarded, X-SVOS-Peer) stay. Wildcard semantics checked on a scratch Caddy (case-insensitive; look-alikes outside the family survive). Deployed, validated via systemd-run with EnvironmentFile, reloaded; deny tests re-run (403). README: safe validate recipe, pending acceptance list.
The policy section said WHICH guests are uncovered but not WHY, so every
reader had to reconstruct the ruling from memory or the bus. A missing
reason now prints as 'NO DOCUMENTED REASON', which turns an undocumented
new exclusion into a visible flag instead of a silent line.
The db/ and runs/ bind mounts are root:docker 2775 since the 2026-09-14 tree
normalisation; the app runs as 1000:1000, so SQLite could not create its journal
and every write returned 500 (found at v0.40.2).
HunyuanVideo-Foley, MMAudio and ControlFoley load their Hugging Face dependencies
from pinned snapshots staged on the model store. Without this they look in the
container's ephemeral cache and download again on every recreate. Requested by
comfy-dev; HF_HUB_OFFLINE deliberately not set.
Since skaldsong v0.39.0 the structured pipeline passes call the LiteLLM
gateway directly; with no key every story generation fails 401 at its first
pass. Operator ruling 2026-10-05: the shared all-agents key until Skaldsong
v1.0 final, then a project-scoped key. The stack passes an explicit
environment list, so the .env line alone does not reach the container.
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.
Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).
albok-service leaves nh3-docker before unit 4 puts the personal vault and
licensed books into restricted wings. On nh3-docker, portainer, portainer_agent,
dockge (socket rw) and dockerd on plaintext tcp/2375 were each root over
/srv/albok. The new host runs only albok-service: no socket containers and no
TCP Docker API, so its Homepage tile is a static services.yaml entry.
- nh3-pve-2: vmbr0 made VLAN-aware (applied by reboot, no guests running);
pbs-ana storage in the new namespace nh3-pve-2, plus a 21:30 vzdump job
(pbs-nh3's recursive sync mirrors it)
- VM 110 nh3-albok, 10.100.50.44 on VLAN 50, Debian 13 cloud-init, 4 vCPU
x86-64-v3 / 12 GiB / 100 GB; built to fleet conventions (vh 1000,
infra-ops 850, docker 851)
- compose: mem_limit 4g -> 8g, homepage labels removed; host port 8392 kept
- cutover by tar stream, verified by digest (contents plus owner, group, mode,
mtime and type identical); albok.nh3.internal now points at nh3-albok
- old /srv/albok stays on nh3-docker (container removed, compose renamed)
until acceptance and the operator's approval to delete it
- sudo-io archive hosts and ops-log STACK_HOSTS include nh3-albok
- Fit gap restated on nvidia-smi accounting (~17 MiB rest / ~130 MiB peak
over, vs the live seat's 15,220/15,437 headroom); the 1.2 GiB figure
conflated the torch pool fraction with nvidia-smi totals
- Footnote: baseline peak row is suite-only, not shape-matched to the
candidate gate peaks
- namedef cell labeled as the description-reversal stand-in, not the
loaded-name variant of arXiv 2609.26758; columns are overlaps
Vendor-path benches on fv-ml1 GPU 3 at the 14.4 GiB / 32k cap. Neither
candidate replaces Intern-Decision-4B: H2O reproduces its vendor figure
(205x3) and has the best calibration, but rests 1 GiB over the cap and
is 2.2x slower at the 21-criteria shape; Clef loses on accuracy (191-192),
hard tier (72-73/111), and ships a silent 16,384-token input slice.
Adds the namedef cell and JB_AUTH wiring to the 09-30 harness client.
vh