Prime (18:05, via albok-dev): go to h2o and fix forward, do not fall back to
INC. albok-dev removed the chapter margin first (the two published nonfiction
titles declared visibility: public). switch-to-h2o.sh ran 18:05:13, h2o healthy
18:07:40; /health at the pin, /decide/shared 200, 422 over the cap, inline image
answers, image URL refused by the shim and by vLLM's media-domain backstop.
rollback-to-inc.sh stays on the host as an artifact only.
The v2 walk on h2o (17:05-17:45) held notes at 0 adult but moved two margins
past albok-dev's pre-named rollback line: library/nonfiction adult 0 -> 79 (one
textbook chapter at 0.531 on the book profile, INC 0.438, with 77 concepts
inheriting) and fleet/memory personal 6 -> 18. rollback-to-inc.sh ran 17:47:46,
INC healthy 17:48:11. compose.yaml is INC again.
Prime's CI fix A. ana-docker's runner sits on HDD mirrors with no SLOG; synced
8 KiB writes measured ~34 ms there vs ~2.7 ms on nh3-docker (same probe, one run,
n=3 each). Repos opt in with runs-on: nh3-docker; the runner carries no
pfi-fleet label. Its config is tighter because nh3-docker also runs NH3 DNS and
the post office: docker_host '-' (verified absent in the job, present on the
ana-docker baseline), capacity 1, jobs capped at 4 GiB / 6 CPUs, a network per
job. The deploy playbook takes runner_config; each runner's homepage name is
its runner name.
Prime ruled the cutover ("do it", 16:40, via albok-dev) after the judge's
note question was re-fitted (albok-nemi 0.1.18). switch-to-h2o.sh ran 17:02:23,
the GPU 1 gate passed, h2o healthy 17:04:54. Checks on :8033: /health at the
pin, /decide/shared 200, 422 over the cap, inline image answers, image URL
refused by the shim, vLLM media-domain backstop refuses (400). compose.yaml is
h2o again; rollback-to-inc.sh stays the one-step way back.
Prime (via albok-dev): understand why h2o failed and re-fit the judge. One
vLLM on GPU 3 behind two fronts: :8034 with the shim's yes/no floor at 0.801
exactly as the cutover ran (same judge stamp), :8035 with SHIM_NOUL_FLOOR=0 and
a distinct stamp (h2o-lightning-4b+noul-floor-0). Both refuse image URLs; the
vLLM --allowed-media-domains backstop is now verified (400, no fetch).
albok-dev withdrew the judge sign-off on fleet-scale evidence and brokkr
concurred: over 10,841 texts judged by both models, h2o flagged 242 adult at
0.5 against INC's 88, and no threshold reproduces INC's set. rollback-to-inc.sh
ran 14:03:28, INC healthy 14:03:56, /decide/shared smoke 200 x3.
compose.yaml is INC again; the h2o compose is kept as compose.h2o.yaml with
switch-to-h2o.sh (GPU 1 free-memory gate, falls back to INC). It now carries
Prime's inline-images-only setting: LIGHTNING_IMAGE_URLS=off on the front
(verified live: an https image URL is refused before any fetch, inline images
still answer) and vLLM --allowed-media-domains media.invalid as a backstop.
rollback-to-inc.sh no longer leaves timestamped compose copies behind.
The :8033 seat now serves h2oai/h2o-lightning-4b @ b43f759b on vLLM 0.30.0 with
byte-pinned KV (4.48 GiB, 135,558 tokens) behind services/h2o-decision-front:
32k refuse-not-trim admission, the seat's bearer token, and /decide +
/decide/shared translated onto /v1/systemone for albok-nemi's judge. URL, port,
token and stack name are unchanged. INC down 13:08:18, h2o healthy 13:11:27.
Post-checks on the live seat: /health at the pinned revision; JevBench public
205/231 (= bench); 422 over the cap; 16q at the cap warm 2.0-2.16 s; seat peak
16,164 MiB under 4 concurrent 16q x 32k calls against a 16,847 budget.
compose.inc.yaml + rollback-to-inc.sh restore Intern-Decision-4B in one step.
The temporary GPU 3 eval seat (stacks/h2o-eval) is down.
Prime's call via brokkr: h2o-lightning-4b replaces the intern-decision seat once
albok-dev re-validates Nemi's sensitivity judge on it. services/h2o-decision-front
builds on infra-hermes's admission proxy (30f36f2): 32,000-token refuse-not-trim,
the seat's bearer token, and intern-decision-serve's /decide/shared mapping onto
/v1/systemone so the judge runs unchanged. stacks/h2o-eval serves it on GPU 3
:8034 exactly as benched (vLLM 0.30.0 + unmodified vendor shim; weights hashed
against HF b43f759b). INC keeps serving production on GPU 1.
The vendor shim silently trims state at 32,000 tokens; the admission proxy is
what turns the seat contract into a loud refusal, so it must front the shim in
any deployment. Committing it with the Amendment 1 confirmation driver so
executing the swap does not depend on a bench scratch dir surviving.
Admission via thin wrapper (vendor shim unmodified; no shim config knob exists).
Gate re-run clean at every depth with usage honest; 16q32k warm 2,033 ms;
4x concurrent 16q32k peak 15,566 MiB vs 16,847 budget; boot matrix replaces the
unmeasured 280 s figure (cold median 153.6 s, warm-cache saves nothing, eager
slower). Wyrd itemised: 8 diffs, h2o net -2. h2o passes every Supersede clause
under the accepted 32,000-token refusing contract; seat swap is the operator's
call.
First bench-first run (prereg brokkr-smithy 2e0539f). H2O-lightning-4b is a
short-route candidate: public 205, battery CI lower bound +0.0023, fits the
GPU 1 budget at every shape — but its shipped shim silently trims state at
~32k and fails the long-input gate by the letter. Clef-flash lateral in both
precisions (Wyrd -0.0333 in both; bf16 does not fit at rest). RSI-Jev v6.1
loses on decision quality (public -23, Wyrd -0.0548). INC holds the seat.
Torn down; weights kept (model-runtime 116 GB).
Measured: bf16 ladder across six sizes, calibrated 32k shape cells with
O(q*state) wire accounting, bs=1 memory/latency curve, Engine-vs-vendor
fidelity rows, Stage 4 control quant trio (battery monotone in weight error),
flash calibrated-NVFP4 with survive rules applied by the letter, and the
Amendment 2 GPU 1 budget (16,851 MiB) re-read against every fit clause.
Halted before the INC bench-copy arms and mini/27B/small-size NVFP4 arms;
stated plainly. Weights kept pending operator ruling (80 GB).
- configs/svos-apps/nh3-dev/: user units for svos-terminal (running),
svos-postoffice, svos-miranda, svos-agent (installed, not yet started).
- configs/svos-front/Caddyfile: svos-terminal.phasefinal.com upstreams to
https://svos-seathost.nh3.phasefinal.com now that app-terminal runs on nh3-dev.
- configs/fv-ml1-sol/: an nh3-dev user unit holding fv-ml1's BMC SOL session
into the journal; fv-ml1's kernel console goes to ttyS1 (the SOL port,
marker-tested) from its next boot (ZFSBootMenu commandline on zroot/ROOT).
Pinned to gitea.phasefinal.com/pfi/albok-service:0.1.17@sha256:b2920e22d112631da82cb747d152c5d600cc50134f905c5afeae8bc26d580630,
built from a clean archive of vh/albok 2b190e2 and deployed by albok-dev at 2233 (healthy in under 10 s;
compose.yaml.pre-0.1.17 beside it on the host).
Per svoperatingsystem dc241f8 u13-svos-edit.md items 1, 4, 5 and 6 (Prime's
go stands): the kernel origin passes exactly the method-pinned list (apps and
Miranda's agent connector; 15 pairs reach the kernel, the rest are Caddy 404);
nh3-dev's apps env is rebuilt as dc241f8 with svos-miranda 0.0.3 and
svos-agent 0.0.3 and gains the svos-miranda origin (nh3-svos only); three
enrollment secrets minted into the vault; app-high-seat declared with one
reload. Also fixes nh3-dev's fleet Caddy sharing admin localhost:2019 with
ttyd-caddy via SO_REUSEPORT (reloads could land on either): now :2021.
Prime approved svos-dev's U13 pre-edit ask (2026-10-10 1639).
- B1: release afab123 on nh3-svos (kernel 0.0.9, migration 11->12) after a
vzdump and an offline svos.db copy; app-terminal and app-postoffice held on
b588510 by drop-ins until the edit.
- B2: --app-listen 127.0.0.1:8410 (drop-in) fronted by Caddy as
svos-kernel.phasefinal.com on the LAN, allowlisted to nh3-dev; the path list
adds /auth/kernel-key, which every connector client needs. DNS rewrite on
the three internal AdGuards.
- B3/B4: svos-seathost (-> :8404) and svos-highseat (-> :8773) on nh3-dev's
fleet Caddy, allowlisted to nh3-svos; /opt/svos apps env on nh3-dev with
svos-postoffice[push] (althing-core 3.7.0 from althing 7a2a1c1) and
svos-terminal. The fleet Caddyfile is now tracked in configs/.
Prime, 2026-10-09: "vh gets albok-everything". playbooks/nh3-albok-vh-viewer.yaml
joins vh to albok-read, albok-personal and albok-feedback on nh3-albok and
authorizes his four Macs' public keys for vh; verified at a fresh login that
every wing's .browse/ lists and reads.
The withdrawal harness gains a macOS driver (run-mac.sh: every mountable FUSE
backend, a read-only smoke of the real wings, results copied back), env
overrides for a non-container client, and summarize.py. Sanity-checked as vh
from the Linux container before staging.
Replica of a browse view on nh3-albok, Linux sshfs 3.7.3 client in an ephemeral
container on nh3-dev, N=5 per condition. Fresh open of a never-opened child
through a cached listing: ENOENT 5/5 with sshfs defaults and 5/5 with hour-long
client caches. Null control (no rename) opens 5/5; held fd and failed-rename
positive controls read 5/5. Replica removed afterwards.
Also closes the post-clear checks in persistent-memory: fv-ml1 GPU 0/1 plateau,
nh3-nas usage, and the mislabelled sfsrv-ana pre-pbs4 item.
Pinned to gitea.phasefinal.com/pfi/albok-service:0.1.16@sha256:b28559ec782f1ce93e89c5eb55c7418561667dde38ff819bb18bb53e2d1f2ddf,
deployed by albok-dev at 1943 (healthy in 20 s; compose.yaml.pre-0.1.16 beside it on the host).
comfy-dev's ask for Prime's arbo 'incognito' runs: incognito uploads and chained
images are written to /worktank/comfyui/basedir/input/incognito-* and swept after
a TTL. input/ is a backup source, so one alive at 01:00 would be kept for up to
3 years. Verified on irv-ml1: resticprofile show lists the exclude, and a restic
dry-run of input/ skips an incognito-* canary while keeping a control canary.
Verified 2026-10-06 for comfy-dev: the gateway keeps no request or response
content. A canary sent through 'gen' appears in none of SpendLogs' text columns,
LiteLLM's container log, any ana-docker container log, or flash-next's log
(positive controls hit on the same stores). In 24h, 0 of 281,432 spend rows
carried messages, response or proxy_server_request. Langfuse is retired, there
are no DB-stored callbacks, and LiteLLM_ErrorLogs is empty. The per-request
no-log flag skips third-party callbacks but not the proxy's own spend-log
callback (v1.97 source), so the guarantee is this setting. Comment-only change;
the live file was updated in place with no restart.
HF_HOME, TORCH_HOME and NLTK_DATA now point under /app/whisperx-env (the
/tank bind mount). Before, ~6 GB of models sat in the container layer and
every recreate threw them away. The cache was copied across (98/2/78 files,
apparent sizes identical).
HF_TOKEN is now a placeholder. Measured: with an invalid token, cached gated
pyannote models still load (huggingface_hub falls back to the cache on the 401),
and public models still download (HTTP 200). Scriberr only checks that the token
is non-empty. Re-verified on GPU 3: pyannote community-1 and WhisperX+3.1
diarization both exit 0 from the /tank cache with no downloads.
Scriberr: pyannote's diarization models are gated, and Scriberr had no Hugging
Face token, so every diarized job died at the diarize step (WhisperX after a full
CPU transcription). HF_TOKEN now passes through compose; the value is only in the
host .env (vault fv-ml1/hf-token). The separate pyannote env had torchcodec 0.16
(needs torch >=2.11/CUDA 13) beside torch 2.8, so pyannote 4 had no
AudioDecoder. Pinned to 0.7 by hand on the live env; patch 0003 makes it
durable at the next rebuild. WhisperX's adapter defaults to cpu/small/float32,
now documented. Both paths verified on GPU 3 with a real multi-speaker clip.
pbs-nh3: the backups-mirror prune job now keeps the primary's history
(last 3, daily 14, weekly 8, monthly 12, yearly 5). Dry-run: keep 738,
remove 3,194 across 32 groups; first run at the job's 00:00.
Image built from a clean archive of vh/albok 751a729 (tag albok-service/v0.1.12),
pushed as gitea.phasefinal.com/pfi/albok-service:0.1.12@sha256:a29a7e46...,
pinned by digest. Recreated between Nemi walks at 1254; healthy at 1254:48,
all four wings ready, canaries alive, write embedder agrees. Config unchanged.
Rollback: compose.yaml.pre-0.1.12 (0.1.11).
Three defects in one loop in ninety minutes, none visible by reading the
file, all caught by running it against known truth. The runbook's
tested-note now says what is actually tested (one positive control) and
what that implies for future edits.
PBS rewrites index.json.blob inside a snapshot directory at every verify
(the next 02:30, then every 30 days per verification.cfg outdated-after 30)
and on notes/protect edits, which moves the directory mtime forward.
Measured: ana-pve/vm/101's 2026-10-04T10:00:04Z snapshot read as 2026-10-06.
A covered guest whose backups stopped would have alarmed up to a day late,
then gone quiet for ~2 days after each monthly re-verify.
The age now comes from the snapshot name (its backup start time).
Verified: the default run exits 0 with 28 fresh PBS groups and vm/101 dated
2026-10-04. Positive control BACKUP_MAX_AGE_HOURS=5 flags 10 groups by name
age, exit 1.
A guest destroyed with 'qm destroy --purge' also drops out of the vzdump
job's exclude list, so its last snapshot silently ages into a false
🔴 once it passes the threshold (ana-pve/vm/110, destroyed 10-04, fired
this morning). Before reporting coverage, consult the owning cluster's
pvesh inventory: a guest the cluster no longer lists is reported in the
policy section as residue, never as stale. Inventory unreadable fails
loud (falls through to normal alarm).
Two traps pinned the hard way during verification:
- 'guest' is namespaced (ana-pve/vm/110); the case must match the
segment (*/vm/*) and pre-set pveid so a foreign path shape fails the
match, not the script, under set -u (infra-ops caught the first cut).
- ssh inside the while-read loop MUST take -n; without it ssh swallows
the loop's remaining stdin and the entire PBS layer silently vanishes
from the report while exit code and verdict stay green.
Image built from a clean archive of vh/albok ffc6d4e (tag albok-service/v0.1.11),
pushed as gitea.phasefinal.com/pfi/albok-service:0.1.11@sha256:4b641c3e...,
pinned by digest. Recreated between Nemi walks at 0202; healthy at 0202:23, all
four wings ready, canaries alive, write embedder agrees. Config unchanged.
Rollback is compose.yaml.pre-0.1.11, valid only until a wing's first 0.1.11 reindex.
vh