ops(albok): vh joins every albok read group; macOS withdrawal-test run staged
Prime, 2026-10-09: "vh gets albok-everything". playbooks/nh3-albok-vh-viewer.yaml joins vh to albok-read, albok-personal and albok-feedback on nh3-albok and authorizes his four Macs' public keys for vh; verified at a fresh login that every wing's .browse/ lists and reads. The withdrawal harness gains a macOS driver (run-mac.sh: every mountable FUSE backend, a read-only smoke of the real wings, results copied back), env overrides for a non-container client, and summarize.py. Sanity-checked as vh from the Linux container before staging.
This commit is contained in:
@@ -125,7 +125,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca
|
||||
- **Watching: Worldtree memory gate.** 10-09 FAILED on filing/user (9, 11, 8 of 12; median 0.75 < 0.80), the first fail that is not #416. Bootstrap from 27 prior runs puts P(median ≤ 9) ≈ 0.06 per batch, and the misses are the habitual fixtures, so it is not called a regression yet. But per-run means drifted 10.6 → 10.0 → 9.7 → 9.3 since the 10-08 fv-ml1 patch (writer + judge = flash-next on GPU 2). If the 10-10 batch also fails filing, infra-hermes frames it as a possible regression after that patch and sends it to me; then investigate flash-next. ⚠ worldtree-dev has not read mail in 6+ days, so the reports dead-end there (Prime's call, below).
|
||||
- **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable.
|
||||
- **Waiting on Prime:**
|
||||
- albok `.browse/` share: the sshfs withdrawal test PASSED (Linux client). Before a share: which wings his nh3-albok user (`vh`) gets (`albok-read` = fleet + library, `albok-personal` = notes; `albok-feedback` stays default-deny), and, if the workstation is macOS, a re-run of `scripts/oneoff/albok-browse-withdrawal/` on it.
|
||||
- albok `.browse/` share: vh holds every albok read group since 10-09 2244 (Prime: "vh gets albok-everything"; four Mac keys authorized for vh). The macOS run of the withdrawal test is staged on nh3-albok and waits on Prime running the one-liner in `scripts/oneoff/albok-browse-withdrawal/README.md`; then read `~vh/albok-browse-test/results/`, report to albok-dev, remove the replica.
|
||||
- Who sees Worldtree gate failures while worldtree-dev is not reading mail (Prime, Miranda, or keep waiting).
|
||||
- Rotate the fv-ml1 BMC password? It was printed into a local session transcript on 10-08 (never left nh3-dev); infra-ops leans no.
|
||||
- albok's 200-object load check, 2.01× against a 2× bar (albok-dev: the ruling is the operator's).
|
||||
@@ -186,6 +186,8 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-10-09]` **vh gets every albok read group (Prime 2244: "vh gets albok-everything")**: albok-read, albok-personal and albok-feedback (the restricted wing's default-deny ends for vh only), plus his four Macs' keys on vh (`playbooks/nh3-albok-vh-viewer.yaml`). No net new root path: those Macs already reach root there via lkraven@nh3-dev and the infra-ops key. Later restricted groups (1513+) are NOT included.
|
||||
- `[2026-10-09]` **nh3-dev: orphaned svos pytest killed 2242** (14.4 GB RSS, +2.3 GB/h, swap 100%, Beszel 90% alert; SIGTERM ignored, SIGKILL; used 25.4 → 11.3 GB). highseat-dev's retry-loop test with an instant fake sleep; they added pytest-timeout 300 s and a 50-attempt bound.
|
||||
- `[2026-10-09]` **Post-clear checks closed (2015–2025 PT):** albok `.browse/` sshfs pathname-withdrawal test PASSED (replica on nh3-albok, Linux sshfs 3.7.3 client in a container; fresh open ENOENT 5/5 default and 5/5 hour-long caches; null + positive controls ok; failed rename = path served; reported to albok-dev) → `scripts/oneoff/albok-browse-withdrawal/README.md`. fv-ml1 plateau (Beszel 120m, 42 h after the 0034 10-08 restart): GPU 0 flat at 5,606 MiB free (~3.9 GB at parakeet's cap), GPU 1 median 9,459 free (~3.3 GB at intern-decision's 32k peak), no creep; GPU 2 643 MiB free = flash-next util 0.96, single tenant, by design. nh3-nas `/volume1` 71% (79% on 10-07). The "sfsrv-ana pre-pbs4" item was a mislabel: the pre-pbs4 snapshots were pfi-pve VM 100 and nh3-pve VM 105, dropped by infra-hermes 10-05 and read gone 10-09.
|
||||
- `[2026-10-09]` **albok `library` building (fiction, nonfiction) live for unit 4.5; Nemi + albok-dev re-minted with library grants (token grants are immutable), old tokens revoked 1555 by albok-dev.** → `stacks/albok-service/README.md`
|
||||
- `[2026-10-09]` **Homepage: The High Seat's description no longer says "SVOS" (Prime: author it yourself); SVOS's now says "Internal network + mesh".**
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# vh on nh3-albok as the operator's albok viewer (Prime, 2026-10-09: "vh gets albok-everything").
|
||||
# scripts/elway infra-ops@10.100.50.44 --playbook playbooks/nh3-albok-vh-viewer.yaml --var keys_file=<local .pub list>
|
||||
# Idempotent. Joins vh to every albok read group that exists (albok-read 1510 = fleet + library,
|
||||
# albok-personal 1511 = personal/notes, albok-feedback 1512 = personal/agent-feedback, which had been
|
||||
# default-deny until this ruling) and authorizes the operator's workstation public keys for vh, so a
|
||||
# workstation can sshfs-mount a wing's .browse/ (see stacks/albok-service/README.md).
|
||||
# A later restricted group (1513+, e.g. the vault wing) is NOT included: add it here deliberately.
|
||||
# Group membership is read at login, so it applies to new SSH sessions only.
|
||||
vars:
|
||||
keys_file: ""
|
||||
groups: albok-read,albok-personal,albok-feedback
|
||||
|
||||
steps:
|
||||
- name: vh joins {{ groups }}
|
||||
sudo: true
|
||||
shell: usermod -aG {{ groups }} vh
|
||||
when: "! (for g in $(echo {{ groups }} | tr , ' '); do id -nG vh | tr ' ' '\\n' | grep -qx \"$g\" || exit 1; done)"
|
||||
|
||||
- name: stage the workstation public keys
|
||||
upload:
|
||||
src: "{{ keys_file }}"
|
||||
dest: /tmp/vh-workstation-keys.pub
|
||||
mode: "0644"
|
||||
|
||||
- name: append each key to vh's authorized_keys if absent
|
||||
sudo: true
|
||||
shell: |
|
||||
set -e
|
||||
added=0
|
||||
while IFS= read -r k; do
|
||||
[ -n "$k" ] || continue
|
||||
grep -qxF "$k" /home/vh/.ssh/authorized_keys || { printf '%s\n' "$k" >> /home/vh/.ssh/authorized_keys; added=$((added+1)); }
|
||||
done < /tmp/vh-workstation-keys.pub
|
||||
rm -f /tmp/vh-workstation-keys.pub
|
||||
echo "added $added"
|
||||
|
||||
verify:
|
||||
- name: vh holds every group
|
||||
shell: |
|
||||
for g in $(echo {{ groups }} | tr , ' '); do
|
||||
id -nG vh | tr ' ' '\n' | grep -qx "$g" || { echo "missing $g"; exit 1; }
|
||||
done
|
||||
changed_when: "false"
|
||||
- name: authorized_keys still vh-owned and 600
|
||||
sudo: true
|
||||
shell: test "$(stat -c '%U %a' /home/vh/.ssh/authorized_keys)" = "vh 600"
|
||||
changed_when: "false"
|
||||
@@ -50,6 +50,16 @@ the server in the clean state (`.browse/` present, no `.browse.withheld`).
|
||||
- **Failed rename:** the path remains and is served. The service cannot deny opens it could not
|
||||
withdraw; its `BROWSE_WITHHELD` marker only stops publication.
|
||||
|
||||
## macOS run (staged 2026-10-09 2245, not yet run)
|
||||
|
||||
`run-mac.sh` runs the same trials from the operator's Mac, as `vh`, against a vh-owned replica at
|
||||
`~vh/albok-browse-test/wing` (`manifest-mac.json`). It tries every FUSE backend the Mac can mount with
|
||||
(the default, plus `backend=smb` / `backend=fskit` where fuse-t or macFUSE offers them), smoke-tests every
|
||||
real wing's view as vh (counts only), and copies its results to `~vh/albok-browse-test/results/`. The vh
|
||||
path was sanity-checked first from the Linux container (N=1 per condition, all four as expected).
|
||||
|
||||
mkdir -p ~/albok-browse-test && cd ~/albok-browse-test && scp 'vh@10.100.50.44:albok-browse-test/harness/*' . && zsh run-mac.sh
|
||||
|
||||
## Limits
|
||||
|
||||
- **Proven for the Linux kernel FUSE client only.** A macOS workstation runs a different FUSE
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,79 @@
|
||||
#!/bin/zsh
|
||||
# macOS client run of the albok browse-view withdrawal test (unit 4.3 exposure gate).
|
||||
# Staged on nh3-albok under ~vh/albok-browse-test/harness/ with a vh-owned replica beside it.
|
||||
# On the Mac:
|
||||
# mkdir -p ~/albok-browse-test && cd ~/albok-browse-test && scp 'vh@10.100.50.44:albok-browse-test/harness/*' . && zsh run-mac.sh
|
||||
# It tests every FUSE backend this Mac can mount with (the backend decides whether a rename
|
||||
# withdraws access), smoke-tests the real wings as vh, and copies the results back to nh3-albok.
|
||||
set -u
|
||||
cd "${0:A:h}"
|
||||
SRV=vh@10.100.50.44
|
||||
MNT=$PWD/mnt
|
||||
N=${N:-5}
|
||||
stamp=$(date +%Y%m%d-%H%M)
|
||||
mkdir -p "$MNT" results
|
||||
|
||||
unmount() { umount "$MNT" 2>/dev/null || diskutil unmount force "$MNT" >/dev/null 2>&1; }
|
||||
|
||||
{
|
||||
echo "date: $(date '+%F %H%M %Z') host: $(scutil --get LocalHostName 2>/dev/null)"
|
||||
sw_vers
|
||||
echo "arch: $(uname -m)"
|
||||
echo "sshfs: $(command -v sshfs || echo MISSING)"
|
||||
command -v sshfs >/dev/null && { sshfs -V 2>&1 | head -5; otool -L "$(command -v sshfs)" 2>/dev/null | grep -i fuse; }
|
||||
[[ -d /Library/Filesystems/macfuse.fs ]] && echo "macFUSE: $(defaults read /Library/Filesystems/macfuse.fs/Contents/Info CFBundleShortVersionString 2>/dev/null)"
|
||||
pkgutil --pkgs 2>/dev/null | grep -i fuse
|
||||
} 2>&1 | tee "results/client-$stamp.txt"
|
||||
|
||||
if ! command -v sshfs >/dev/null; then
|
||||
print "\nNo sshfs on this Mac. Install fuse-t (no kernel extension), then re-run this script:"
|
||||
print " brew install macos-fuse-t/homebrew-cask/fuse-t macos-fuse-t/homebrew-cask/fuse-t-sshfs"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Candidate backends: "" = whatever this sshfs mounts with by default.
|
||||
candidates=("")
|
||||
if pkgutil --pkgs 2>/dev/null | grep -qi fuse-t || [[ -e /usr/local/lib/libfuse-t.dylib ]]; then
|
||||
candidates+=("backend=smb" "backend=fskit")
|
||||
fi
|
||||
[[ -d /Library/Filesystems/macfuse.fs ]] && candidates+=("backend=fskit")
|
||||
candidates=("${(@u)candidates}") # dedupe, keeping the empty "default" entry
|
||||
|
||||
working=()
|
||||
for c in "${candidates[@]}"; do
|
||||
label=${c:-default}; label=${label#backend=}
|
||||
o="ro,follow_symlinks${c:+,$c}"
|
||||
if sshfs "$SRV:albok-browse-test/wing/.browse" "$MNT" -o "$o" 2>>"results/client-$stamp.txt" \
|
||||
&& ls "$MNT/stash" >/dev/null 2>&1; then
|
||||
print "backend $label: mounts"; working+=("$c")
|
||||
else
|
||||
print "backend $label: does not mount here (skipped)"
|
||||
fi
|
||||
unmount
|
||||
done
|
||||
(( ${#working} )) || { print "No backend mounted; send results/client-$stamp.txt to infra-ops."; exit 3; }
|
||||
|
||||
for c in "${working[@]}"; do
|
||||
label=${c:-default}; label=${label#backend=}
|
||||
print "\n== trials, backend $label (N=$N per condition; ~3 min)"
|
||||
ALBOK_TEST_SRV=$SRV ALBOK_TEST_KEY= ALBOK_TEST_KNOWN_HOSTS= ALBOK_TEST_MNT=$MNT \
|
||||
ALBOK_TEST_EXTRA_OPTS=$c python3 run-trials.py manifest.json $N > "results/trials-$label-$stamp.json"
|
||||
python3 summarize.py "results/trials-$label-$stamp.json"
|
||||
done
|
||||
|
||||
# Smoke test of the REAL views as vh (read-only; prints counts, never content).
|
||||
first=${working[1]}
|
||||
print "\n== real wings as vh" | tee "results/smoke-$stamp.txt"
|
||||
for w in fleet/memory fleet/stash library/fiction library/nonfiction personal/notes personal/agent-feedback; do
|
||||
if sshfs "$SRV:/srv/albok/store/$w/.browse" "$MNT" -o "ro,follow_symlinks${first:+,$first}" 2>/dev/null; then
|
||||
n=$(find "$MNT" -type f 2>/dev/null | wc -l | tr -d ' ')
|
||||
f=$(find "$MNT" -type f 2>/dev/null | head -1)
|
||||
b=$([[ -n $f ]] && head -c 4096 "$f" | wc -c | tr -d ' ' || echo -)
|
||||
print "$w: mount ok, files=$n, first-file bytes read=$b"
|
||||
else
|
||||
print "$w: MOUNT FAILED"
|
||||
fi | tee -a "results/smoke-$stamp.txt"
|
||||
unmount
|
||||
done
|
||||
|
||||
scp -q results/*-$stamp.* "$SRV:albok-browse-test/results/" && print "\nResults copied to nh3-albok; tell infra-ops it is done."
|
||||
@@ -34,11 +34,16 @@ SRV = os.environ.get("ALBOK_TEST_SRV", "infra-ops@10.100.50.44")
|
||||
KEY = os.environ.get("ALBOK_TEST_KEY", "/keys/id")
|
||||
KH = os.environ.get("ALBOK_TEST_KNOWN_HOSTS", "/keys/known_hosts")
|
||||
MNT = os.environ.get("ALBOK_TEST_MNT", "/mnt/browse")
|
||||
SSH_OPTS = ["-i", KEY, "-o", "BatchMode=yes", "-o", f"UserKnownHostsFile={KH}",
|
||||
"-o", "StrictHostKeyChecking=yes", "-o", "ControlMaster=auto",
|
||||
"-o", "ControlPath=/tmp/cm-%r@%h", "-o", "ControlPersist=900"]
|
||||
SSHFS_BASE = ["ro", "follow_symlinks", f"IdentityFile={KEY}", f"UserKnownHostsFile={KH}",
|
||||
"StrictHostKeyChecking=yes", "BatchMode=yes"]
|
||||
# An empty KEY / KNOWN_HOSTS means "use the client's own ssh config" (a workstation run).
|
||||
_id = (["-i", KEY] if KEY else []) + (["-o", f"UserKnownHostsFile={KH}"] if KH else [])
|
||||
SSH_OPTS = _id + ["-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=yes",
|
||||
"-o", "ControlMaster=auto", "-o", "ControlPath=/tmp/cm-%r@%h",
|
||||
"-o", "ControlPersist=900"]
|
||||
SSHFS_BASE = (["ro", "follow_symlinks"] + ([f"IdentityFile={KEY}"] if KEY else [])
|
||||
+ ([f"UserKnownHostsFile={KH}"] if KH else [])
|
||||
+ ["StrictHostKeyChecking=yes", "BatchMode=yes"]
|
||||
# e.g. backend=fskit on macOS: the FUSE backend under test
|
||||
+ [o for o in os.environ.get("ALBOK_TEST_EXTRA_OPTS", "").split(",") if o])
|
||||
AGGRESSIVE = ["kernel_cache", "entry_timeout=3600", "attr_timeout=3600",
|
||||
"dir_cache=yes", "dcache_timeout=3600"]
|
||||
CONDITIONS = [("null", [], None), ("default", [], "ok"),
|
||||
@@ -157,7 +162,8 @@ def trial(cond, extra, rename, k, slot):
|
||||
finally:
|
||||
if fd_b is not None:
|
||||
os.close(fd_b)
|
||||
for um in (["fusermount3", "-u", MNT], ["fusermount", "-u", MNT], ["umount", MNT]):
|
||||
for um in (["fusermount3", "-u", MNT], ["fusermount", "-u", MNT], ["umount", MNT],
|
||||
["diskutil", "unmount", "force", MNT]):
|
||||
try:
|
||||
if subprocess.run(um, capture_output=True).returncode == 0:
|
||||
break
|
||||
|
||||
@@ -18,9 +18,11 @@ fetching a reference copy.
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
ROOT = "/home/infra-ops/albok-browse-test/wing"
|
||||
# Optional argv[1] places the replica elsewhere (python3 - <root> < setup-replica.py).
|
||||
ROOT = sys.argv[1] if len(sys.argv) > 1 else "/home/infra-ops/albok-browse-test/wing"
|
||||
COUNT = 60
|
||||
SIZE = 1 << 20 # 1 MiB: larger than FUSE readahead, so a late read must reach the server
|
||||
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Print one line per condition from a run-trials.py result file, and the gate verdict.
|
||||
|
||||
python3 summarize.py results-*.json
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from collections import Counter
|
||||
|
||||
|
||||
def c_outcome(t):
|
||||
c = t.get("post_open_C", {})
|
||||
if not c.get("opened"):
|
||||
return f"open {c.get('errno')}"
|
||||
if "read_errno" in c:
|
||||
return f"opened, read {c['read_errno']}"
|
||||
return "opened, bytes ok" if c.get("content_ok") else "opened, BYTES WRONG"
|
||||
|
||||
|
||||
for path in sys.argv[1:]:
|
||||
d = json.load(open(path))
|
||||
print(f"== {path} sshfs -o {','.join(o for o in d['sshfs_base'] if not o.startswith(('IdentityFile', 'UserKnownHosts')))}")
|
||||
by = {}
|
||||
for t in d["trials"]:
|
||||
by.setdefault(t["cond"], []).append(t)
|
||||
verdict = []
|
||||
for cond, ts in by.items():
|
||||
aborted = [t["aborted"] for t in ts if "aborted" in t]
|
||||
ok = [t for t in ts if "aborted" not in t]
|
||||
oc = Counter(c_outcome(t) for t in ok)
|
||||
held = Counter("ok" if t["post_held_B"].get("content_ok") else str(t["post_held_B"]) for t in ok)
|
||||
lst = Counter(str(t["post_list"].get("n", t["post_list"].get("errno"))) for t in ok)
|
||||
print(f" {cond:14} n={len(ok)} aborted={len(aborted)} open C: {dict(oc)} held B: {dict(held)} "
|
||||
f"cached list: {dict(lst)}" + (f" abort: {aborted[0][:80]}" if aborted else ""))
|
||||
if cond in ("default", "aggressive") and ok:
|
||||
denied = sum(1 for t in ok if not t["post_open_C"].get("opened")
|
||||
or "read_errno" in t["post_open_C"])
|
||||
verdict.append(f"{cond}: C denied {denied}/{len(ok)}")
|
||||
if cond == "null" and ok:
|
||||
verdict.append(f"null: C served {sum(1 for t in ok if c_outcome(t) == 'opened, bytes ok')}/{len(ok)}")
|
||||
print(" verdict:", "; ".join(verdict) or "no usable trials")
|
||||
@@ -26,9 +26,10 @@ The FastAPI process that owns Albok's buildings-and-wings store (`vh/albok`,
|
||||
**[2026-10-09] PASSED on sshfs `ro,follow_symlinks` with a Linux client** (SSHFS 3.7.3 / libfuse 3.14): fresh open
|
||||
ENOENT 5/5 with sshfs defaults and 5/5 with hour-long client caches, while the cached listing and stat still showed
|
||||
the child; null and positive controls behaved (`scripts/oneoff/albok-browse-withdrawal/`). Reported to albok-dev.
|
||||
Still before any share: the workstation's own client must re-run that harness if it is macOS (different FUSE
|
||||
backend), and no user is in `albok-read`/`albok-personal` yet, so the operator's user needs those groups (the
|
||||
operator's call; `albok-feedback` stays default-deny). Rollback: `compose.yaml.pre-0.1.14` (0.1.13) is free, because no
|
||||
**[2026-10-09 2244] vh holds every albok read group** (Prime: "vh gets albok-everything"; his four Macs' keys are
|
||||
authorized for vh; verified at a fresh login, every wing's view lists and reads). Still before he browses from a Mac:
|
||||
the macOS re-run of that harness (staged in `~vh/albok-browse-test/`, `run-mac.sh`), because the FUSE backend differs.
|
||||
Rollback: `compose.yaml.pre-0.1.14` (0.1.13) is free, because no
|
||||
sidecar shape changed.
|
||||
Prior: 0.1.13 @ `sha256:bea95617…` (2026-10-07 1135 – 1804) (healthy 1136:17,
|
||||
between walks). infra-ops deployed it at albok-dev's ask (althing 8413), from a clean archive of vh/albok `70b55ad7`
|
||||
@@ -107,8 +108,8 @@ The FastAPI process that owns Albok's buildings-and-wings store (`vh/albok`,
|
||||
|
||||
**Identities (fixed numeric ids, created by `playbooks/albok-service-host.yaml`):** user and group
|
||||
`albok` 1500:1500 (the image's user), read groups `albok-read` **1510** (building `fleet`) and
|
||||
`albok-personal` **1511** (building `personal`). **`albok-feedback` 1512** = restricted wing `personal/agent-feedback` (2026-10-03, default-deny: no viewer is
|
||||
in it). Reserve 1513+ for the vault wing and any later restricted wing; add each to the host, `conf/group` and `group_add`.
|
||||
`albok-personal` **1511** (building `personal`). **`albok-feedback` 1512** = restricted wing `personal/agent-feedback` (2026-10-03, default-deny: no viewer was
|
||||
in it until Prime's 2026-10-09 ruling "vh gets albok-everything": vh now holds 1510, 1511 and 1512, `playbooks/nh3-albok-vh-viewer.yaml`). Reserve 1513+ for the vault wing and any later restricted wing; add each to the host, `conf/group` and `group_add`.
|
||||
|
||||
**Why the group file and `group_add`:** the service resolves group NAMES with `grp.getgrnam()`
|
||||
inside the container and `chgrp`s wing dirs as uid 1500. Without the names in the container's
|
||||
|
||||
Reference in New Issue
Block a user