Prime ruled the cutover ("do it", 16:40, via albok-dev) after the judge's
note question was re-fitted (albok-nemi 0.1.18). switch-to-h2o.sh ran 17:02:23,
the GPU 1 gate passed, h2o healthy 17:04:54. Checks on :8033: /health at the
pin, /decide/shared 200, 422 over the cap, inline image answers, image URL
refused by the shim, vLLM media-domain backstop refuses (400). compose.yaml is
h2o again; rollback-to-inc.sh stays the one-step way back.
Prime (via albok-dev): understand why h2o failed and re-fit the judge. One
vLLM on GPU 3 behind two fronts: :8034 with the shim's yes/no floor at 0.801
exactly as the cutover ran (same judge stamp), :8035 with SHIM_NOUL_FLOOR=0 and
a distinct stamp (h2o-lightning-4b+noul-floor-0). Both refuse image URLs; the
vLLM --allowed-media-domains backstop is now verified (400, no fetch).
albok-dev withdrew the judge sign-off on fleet-scale evidence and brokkr
concurred: over 10,841 texts judged by both models, h2o flagged 242 adult at
0.5 against INC's 88, and no threshold reproduces INC's set. rollback-to-inc.sh
ran 14:03:28, INC healthy 14:03:56, /decide/shared smoke 200 x3.
compose.yaml is INC again; the h2o compose is kept as compose.h2o.yaml with
switch-to-h2o.sh (GPU 1 free-memory gate, falls back to INC). It now carries
Prime's inline-images-only setting: LIGHTNING_IMAGE_URLS=off on the front
(verified live: an https image URL is refused before any fetch, inline images
still answer) and vLLM --allowed-media-domains media.invalid as a backstop.
rollback-to-inc.sh no longer leaves timestamped compose copies behind.
The :8033 seat now serves h2oai/h2o-lightning-4b @ b43f759b on vLLM 0.30.0 with
byte-pinned KV (4.48 GiB, 135,558 tokens) behind services/h2o-decision-front:
32k refuse-not-trim admission, the seat's bearer token, and /decide +
/decide/shared translated onto /v1/systemone for albok-nemi's judge. URL, port,
token and stack name are unchanged. INC down 13:08:18, h2o healthy 13:11:27.
Post-checks on the live seat: /health at the pinned revision; JevBench public
205/231 (= bench); 422 over the cap; 16q at the cap warm 2.0-2.16 s; seat peak
16,164 MiB under 4 concurrent 16q x 32k calls against a 16,847 budget.
compose.inc.yaml + rollback-to-inc.sh restore Intern-Decision-4B in one step.
The temporary GPU 3 eval seat (stacks/h2o-eval) is down.
Prime's call via brokkr: h2o-lightning-4b replaces the intern-decision seat once
albok-dev re-validates Nemi's sensitivity judge on it. services/h2o-decision-front
builds on infra-hermes's admission proxy (30f36f2): 32,000-token refuse-not-trim,
the seat's bearer token, and intern-decision-serve's /decide/shared mapping onto
/v1/systemone so the judge runs unchanged. stacks/h2o-eval serves it on GPU 3
:8034 exactly as benched (vLLM 0.30.0 + unmodified vendor shim; weights hashed
against HF b43f759b). INC keeps serving production on GPU 1.
The vendor shim silently trims state at 32,000 tokens; the admission proxy is
what turns the seat contract into a loud refusal, so it must front the shim in
any deployment. Committing it with the Amendment 1 confirmation driver so
executing the swap does not depend on a bench scratch dir surviving.
Admission via thin wrapper (vendor shim unmodified; no shim config knob exists).
Gate re-run clean at every depth with usage honest; 16q32k warm 2,033 ms;
4x concurrent 16q32k peak 15,566 MiB vs 16,847 budget; boot matrix replaces the
unmeasured 280 s figure (cold median 153.6 s, warm-cache saves nothing, eager
slower). Wyrd itemised: 8 diffs, h2o net -2. h2o passes every Supersede clause
under the accepted 32,000-token refusing contract; seat swap is the operator's
call.
First bench-first run (prereg brokkr-smithy 2e0539f). H2O-lightning-4b is a
short-route candidate: public 205, battery CI lower bound +0.0023, fits the
GPU 1 budget at every shape — but its shipped shim silently trims state at
~32k and fails the long-input gate by the letter. Clef-flash lateral in both
precisions (Wyrd -0.0333 in both; bf16 does not fit at rest). RSI-Jev v6.1
loses on decision quality (public -23, Wyrd -0.0548). INC holds the seat.
Torn down; weights kept (model-runtime 116 GB).
Measured: bf16 ladder across six sizes, calibrated 32k shape cells with
O(q*state) wire accounting, bs=1 memory/latency curve, Engine-vs-vendor
fidelity rows, Stage 4 control quant trio (battery monotone in weight error),
flash calibrated-NVFP4 with survive rules applied by the letter, and the
Amendment 2 GPU 1 budget (16,851 MiB) re-read against every fit clause.
Halted before the INC bench-copy arms and mini/27B/small-size NVFP4 arms;
stated plainly. Weights kept pending operator ruling (80 GB).
- configs/svos-apps/nh3-dev/: user units for svos-terminal (running),
svos-postoffice, svos-miranda, svos-agent (installed, not yet started).
- configs/svos-front/Caddyfile: svos-terminal.phasefinal.com upstreams to
https://svos-seathost.nh3.phasefinal.com now that app-terminal runs on nh3-dev.
- configs/fv-ml1-sol/: an nh3-dev user unit holding fv-ml1's BMC SOL session
into the journal; fv-ml1's kernel console goes to ttyS1 (the SOL port,
marker-tested) from its next boot (ZFSBootMenu commandline on zroot/ROOT).
Pinned to gitea.phasefinal.com/pfi/albok-service:0.1.17@sha256:b2920e22d112631da82cb747d152c5d600cc50134f905c5afeae8bc26d580630,
built from a clean archive of vh/albok 2b190e2 and deployed by albok-dev at 2233 (healthy in under 10 s;
compose.yaml.pre-0.1.17 beside it on the host).
Per svoperatingsystem dc241f8 u13-svos-edit.md items 1, 4, 5 and 6 (Prime's
go stands): the kernel origin passes exactly the method-pinned list (apps and
Miranda's agent connector; 15 pairs reach the kernel, the rest are Caddy 404);
nh3-dev's apps env is rebuilt as dc241f8 with svos-miranda 0.0.3 and
svos-agent 0.0.3 and gains the svos-miranda origin (nh3-svos only); three
enrollment secrets minted into the vault; app-high-seat declared with one
reload. Also fixes nh3-dev's fleet Caddy sharing admin localhost:2019 with
ttyd-caddy via SO_REUSEPORT (reloads could land on either): now :2021.
Prime approved svos-dev's U13 pre-edit ask (2026-10-10 1639).
- B1: release afab123 on nh3-svos (kernel 0.0.9, migration 11->12) after a
vzdump and an offline svos.db copy; app-terminal and app-postoffice held on
b588510 by drop-ins until the edit.
- B2: --app-listen 127.0.0.1:8410 (drop-in) fronted by Caddy as
svos-kernel.phasefinal.com on the LAN, allowlisted to nh3-dev; the path list
adds /auth/kernel-key, which every connector client needs. DNS rewrite on
the three internal AdGuards.
- B3/B4: svos-seathost (-> :8404) and svos-highseat (-> :8773) on nh3-dev's
fleet Caddy, allowlisted to nh3-svos; /opt/svos apps env on nh3-dev with
svos-postoffice[push] (althing-core 3.7.0 from althing 7a2a1c1) and
svos-terminal. The fleet Caddyfile is now tracked in configs/.
Prime, 2026-10-09: "vh gets albok-everything". playbooks/nh3-albok-vh-viewer.yaml
joins vh to albok-read, albok-personal and albok-feedback on nh3-albok and
authorizes his four Macs' public keys for vh; verified at a fresh login that
every wing's .browse/ lists and reads.
The withdrawal harness gains a macOS driver (run-mac.sh: every mountable FUSE
backend, a read-only smoke of the real wings, results copied back), env
overrides for a non-container client, and summarize.py. Sanity-checked as vh
from the Linux container before staging.
Replica of a browse view on nh3-albok, Linux sshfs 3.7.3 client in an ephemeral
container on nh3-dev, N=5 per condition. Fresh open of a never-opened child
through a cached listing: ENOENT 5/5 with sshfs defaults and 5/5 with hour-long
client caches. Null control (no rename) opens 5/5; held fd and failed-rename
positive controls read 5/5. Replica removed afterwards.
Also closes the post-clear checks in persistent-memory: fv-ml1 GPU 0/1 plateau,
nh3-nas usage, and the mislabelled sfsrv-ana pre-pbs4 item.
Pinned to gitea.phasefinal.com/pfi/albok-service:0.1.16@sha256:b28559ec782f1ce93e89c5eb55c7418561667dde38ff819bb18bb53e2d1f2ddf,
deployed by albok-dev at 1943 (healthy in 20 s; compose.yaml.pre-0.1.16 beside it on the host).
comfy-dev's ask for Prime's arbo 'incognito' runs: incognito uploads and chained
images are written to /worktank/comfyui/basedir/input/incognito-* and swept after
a TTL. input/ is a backup source, so one alive at 01:00 would be kept for up to
3 years. Verified on irv-ml1: resticprofile show lists the exclude, and a restic
dry-run of input/ skips an incognito-* canary while keeping a control canary.
Verified 2026-10-06 for comfy-dev: the gateway keeps no request or response
content. A canary sent through 'gen' appears in none of SpendLogs' text columns,
LiteLLM's container log, any ana-docker container log, or flash-next's log
(positive controls hit on the same stores). In 24h, 0 of 281,432 spend rows
carried messages, response or proxy_server_request. Langfuse is retired, there
are no DB-stored callbacks, and LiteLLM_ErrorLogs is empty. The per-request
no-log flag skips third-party callbacks but not the proxy's own spend-log
callback (v1.97 source), so the guarantee is this setting. Comment-only change;
the live file was updated in place with no restart.
HF_HOME, TORCH_HOME and NLTK_DATA now point under /app/whisperx-env (the
/tank bind mount). Before, ~6 GB of models sat in the container layer and
every recreate threw them away. The cache was copied across (98/2/78 files,
apparent sizes identical).
HF_TOKEN is now a placeholder. Measured: with an invalid token, cached gated
pyannote models still load (huggingface_hub falls back to the cache on the 401),
and public models still download (HTTP 200). Scriberr only checks that the token
is non-empty. Re-verified on GPU 3: pyannote community-1 and WhisperX+3.1
diarization both exit 0 from the /tank cache with no downloads.
Scriberr: pyannote's diarization models are gated, and Scriberr had no Hugging
Face token, so every diarized job died at the diarize step (WhisperX after a full
CPU transcription). HF_TOKEN now passes through compose; the value is only in the
host .env (vault fv-ml1/hf-token). The separate pyannote env had torchcodec 0.16
(needs torch >=2.11/CUDA 13) beside torch 2.8, so pyannote 4 had no
AudioDecoder. Pinned to 0.7 by hand on the live env; patch 0003 makes it
durable at the next rebuild. WhisperX's adapter defaults to cpu/small/float32,
now documented. Both paths verified on GPU 3 with a real multi-speaker clip.
pbs-nh3: the backups-mirror prune job now keeps the primary's history
(last 3, daily 14, weekly 8, monthly 12, yearly 5). Dry-run: keep 738,
remove 3,194 across 32 groups; first run at the job's 00:00.
Image built from a clean archive of vh/albok 751a729 (tag albok-service/v0.1.12),
pushed as gitea.phasefinal.com/pfi/albok-service:0.1.12@sha256:a29a7e46...,
pinned by digest. Recreated between Nemi walks at 1254; healthy at 1254:48,
all four wings ready, canaries alive, write embedder agrees. Config unchanged.
Rollback: compose.yaml.pre-0.1.12 (0.1.11).
Three defects in one loop in ninety minutes, none visible by reading the
file, all caught by running it against known truth. The runbook's
tested-note now says what is actually tested (one positive control) and
what that implies for future edits.
PBS rewrites index.json.blob inside a snapshot directory at every verify
(the next 02:30, then every 30 days per verification.cfg outdated-after 30)
and on notes/protect edits, which moves the directory mtime forward.
Measured: ana-pve/vm/101's 2026-10-04T10:00:04Z snapshot read as 2026-10-06.
A covered guest whose backups stopped would have alarmed up to a day late,
then gone quiet for ~2 days after each monthly re-verify.
The age now comes from the snapshot name (its backup start time).
Verified: the default run exits 0 with 28 fresh PBS groups and vm/101 dated
2026-10-04. Positive control BACKUP_MAX_AGE_HOURS=5 flags 10 groups by name
age, exit 1.
A guest destroyed with 'qm destroy --purge' also drops out of the vzdump
job's exclude list, so its last snapshot silently ages into a false
🔴 once it passes the threshold (ana-pve/vm/110, destroyed 10-04, fired
this morning). Before reporting coverage, consult the owning cluster's
pvesh inventory: a guest the cluster no longer lists is reported in the
policy section as residue, never as stale. Inventory unreadable fails
loud (falls through to normal alarm).
Two traps pinned the hard way during verification:
- 'guest' is namespaced (ana-pve/vm/110); the case must match the
segment (*/vm/*) and pre-set pveid so a foreign path shape fails the
match, not the script, under set -u (infra-ops caught the first cut).
- ssh inside the while-read loop MUST take -n; without it ssh swallows
the loop's remaining stdin and the entire PBS layer silently vanishes
from the report while exit code and verdict stay green.
Image built from a clean archive of vh/albok ffc6d4e (tag albok-service/v0.1.11),
pushed as gitea.phasefinal.com/pfi/albok-service:0.1.11@sha256:4b641c3e...,
pinned by digest. Recreated between Nemi walks at 0202; healthy at 0202:23, all
four wings ready, canaries alive, write embedder agrees. Config unchanged.
Rollback is compose.yaml.pre-0.1.11, valid only until a wing's first 0.1.11 reindex.
Layout: GPU 0 = cyberprev (KV 16 -> 15 GiB, 1.29x), erp-seat (moved from
GPU 1, KV 8.49 -> 5.5 GiB, 1.32x), voices (ctx 8,192 -> 40,960, its native
maximum, KV 6 GiB, 1.07x), coder (moved from GPU 1, ctx 16,384 -> 32,768,
the card maximum, 2.35x), parakeet-nemo. GPU 1 = gen-small (moved from
GPU 0, 5.5 GiB, 1.76x), meromero-rp (22 GiB, 1.32x), intern-decision.
Sizing rule: KV >= 1.25x max context rounded up to 0.5 GiB, checked against
7 days of LiteLLM traffic and the engines' own KV-usage logs. Each card keeps
spare memory for its bursty tenant (parakeet's cap on GPU 0, intern-decision's
32k call on GPU 1). YaRN extensions are not enabled: they do not fit, and the
Qwen cards warn static YaRN degrades short prompts.
gen-small gets --override-generation-config max_new_tokens 32768. It applies
only to requests that omit max_tokens (vLLM get_max_tokens) and answers 9
runaway ~260k-token replies in 7 days; the longest normal reply was 6,357.
Applied with playbooks/fv-ml1-rebalance-20261005.yaml (boot-check guard and
health/KV verification per seat); all nine seats healthy, gateway smoke
tests pass.
meromero-rp sized its KV from --gpu-memory-utilization, and vLLM's profile
counts co-booting neighbours as its own non-torch memory. Three boots gave
20.37 / 26.74 / 29.04 GiB of KV, and the 2026-10-05 Docker-move restart cut
GPU 1 free memory to 3,981 MiB, below intern-decision's 32k-token budget.
Coder read negative KV on every co-boot. Both are now pinned, so util only
gates the boot check (meromero util 0.52 -> 0.47, just above its footprint).
GPU 0 plateaus at ~60 MiB free 1-2 days after a boot (Beszel), so
gen-small's KV drops to 5.5 GiB (461,215 tokens, 1.76x) and parakeet-nemo
stays on GPU 0 with room for its 3,840 MiB cap. GPU 3 stays clear.
Applied with playbooks/fv-ml1-kv-pin-20261005.yaml: GPU 1 free 3,981 ->
11,801 MiB, GPU 0 495 -> 5,039 MiB right after boot; all nine seats healthy.
Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4).
vh