Replica of a browse view on nh3-albok, Linux sshfs 3.7.3 client in an ephemeral
container on nh3-dev, N=5 per condition. Fresh open of a never-opened child
through a cached listing: ENOENT 5/5 with sshfs defaults and 5/5 with hour-long
client caches. Null control (no rename) opens 5/5; held fd and failed-rename
positive controls read 5/5. Replica removed afterwards.
Also closes the post-clear checks in persistent-memory: fv-ml1 GPU 0/1 plateau,
nh3-nas usage, and the mislabelled sfsrv-ana pre-pbs4 item.
Pinned to gitea.phasefinal.com/pfi/albok-service:0.1.16@sha256:b28559ec782f1ce93e89c5eb55c7418561667dde38ff819bb18bb53e2d1f2ddf,
deployed by albok-dev at 1943 (healthy in 20 s; compose.yaml.pre-0.1.16 beside it on the host).
comfy-dev's ask for Prime's arbo 'incognito' runs: incognito uploads and chained
images are written to /worktank/comfyui/basedir/input/incognito-* and swept after
a TTL. input/ is a backup source, so one alive at 01:00 would be kept for up to
3 years. Verified on irv-ml1: resticprofile show lists the exclude, and a restic
dry-run of input/ skips an incognito-* canary while keeping a control canary.
Verified 2026-10-06 for comfy-dev: the gateway keeps no request or response
content. A canary sent through 'gen' appears in none of SpendLogs' text columns,
LiteLLM's container log, any ana-docker container log, or flash-next's log
(positive controls hit on the same stores). In 24h, 0 of 281,432 spend rows
carried messages, response or proxy_server_request. Langfuse is retired, there
are no DB-stored callbacks, and LiteLLM_ErrorLogs is empty. The per-request
no-log flag skips third-party callbacks but not the proxy's own spend-log
callback (v1.97 source), so the guarantee is this setting. Comment-only change;
the live file was updated in place with no restart.
HF_HOME, TORCH_HOME and NLTK_DATA now point under /app/whisperx-env (the
/tank bind mount). Before, ~6 GB of models sat in the container layer and
every recreate threw them away. The cache was copied across (98/2/78 files,
apparent sizes identical).
HF_TOKEN is now a placeholder. Measured: with an invalid token, cached gated
pyannote models still load (huggingface_hub falls back to the cache on the 401),
and public models still download (HTTP 200). Scriberr only checks that the token
is non-empty. Re-verified on GPU 3: pyannote community-1 and WhisperX+3.1
diarization both exit 0 from the /tank cache with no downloads.
Scriberr: pyannote's diarization models are gated, and Scriberr had no Hugging
Face token, so every diarized job died at the diarize step (WhisperX after a full
CPU transcription). HF_TOKEN now passes through compose; the value is only in the
host .env (vault fv-ml1/hf-token). The separate pyannote env had torchcodec 0.16
(needs torch >=2.11/CUDA 13) beside torch 2.8, so pyannote 4 had no
AudioDecoder. Pinned to 0.7 by hand on the live env; patch 0003 makes it
durable at the next rebuild. WhisperX's adapter defaults to cpu/small/float32,
now documented. Both paths verified on GPU 3 with a real multi-speaker clip.
pbs-nh3: the backups-mirror prune job now keeps the primary's history
(last 3, daily 14, weekly 8, monthly 12, yearly 5). Dry-run: keep 738,
remove 3,194 across 32 groups; first run at the job's 00:00.
Image built from a clean archive of vh/albok 751a729 (tag albok-service/v0.1.12),
pushed as gitea.phasefinal.com/pfi/albok-service:0.1.12@sha256:a29a7e46...,
pinned by digest. Recreated between Nemi walks at 1254; healthy at 1254:48,
all four wings ready, canaries alive, write embedder agrees. Config unchanged.
Rollback: compose.yaml.pre-0.1.12 (0.1.11).
Three defects in one loop in ninety minutes, none visible by reading the
file, all caught by running it against known truth. The runbook's
tested-note now says what is actually tested (one positive control) and
what that implies for future edits.
PBS rewrites index.json.blob inside a snapshot directory at every verify
(the next 02:30, then every 30 days per verification.cfg outdated-after 30)
and on notes/protect edits, which moves the directory mtime forward.
Measured: ana-pve/vm/101's 2026-10-04T10:00:04Z snapshot read as 2026-10-06.
A covered guest whose backups stopped would have alarmed up to a day late,
then gone quiet for ~2 days after each monthly re-verify.
The age now comes from the snapshot name (its backup start time).
Verified: the default run exits 0 with 28 fresh PBS groups and vm/101 dated
2026-10-04. Positive control BACKUP_MAX_AGE_HOURS=5 flags 10 groups by name
age, exit 1.
A guest destroyed with 'qm destroy --purge' also drops out of the vzdump
job's exclude list, so its last snapshot silently ages into a false
🔴 once it passes the threshold (ana-pve/vm/110, destroyed 10-04, fired
this morning). Before reporting coverage, consult the owning cluster's
pvesh inventory: a guest the cluster no longer lists is reported in the
policy section as residue, never as stale. Inventory unreadable fails
loud (falls through to normal alarm).
Two traps pinned the hard way during verification:
- 'guest' is namespaced (ana-pve/vm/110); the case must match the
segment (*/vm/*) and pre-set pveid so a foreign path shape fails the
match, not the script, under set -u (infra-ops caught the first cut).
- ssh inside the while-read loop MUST take -n; without it ssh swallows
the loop's remaining stdin and the entire PBS layer silently vanishes
from the report while exit code and verdict stay green.
vh