test(albok): sshfs pathname-withdrawal test for the .browse/ view passes (unit 4.3 exposure gate)

Replica of a browse view on nh3-albok, Linux sshfs 3.7.3 client in an ephemeral
container on nh3-dev, N=5 per condition. Fresh open of a never-opened child
through a cached listing: ENOENT 5/5 with sshfs defaults and 5/5 with hour-long
client caches. Null control (no rename) opens 5/5; held fd and failed-rename
positive controls read 5/5. Replica removed afterwards.

Also closes the post-clear checks in persistent-memory: fv-ml1 GPU 0/1 plateau,
nh3-nas usage, and the mislabelled sfsrv-ana pre-pbs4 item.
This commit is contained in:
vh
2026-10-09 20:23:30 -07:00
parent 9ba2ce476b
commit 172d893cdf
8 changed files with 1213 additions and 6 deletions
@@ -0,0 +1,65 @@
# albok browse view: sshfs pathname-withdrawal test (unit 4.3 exposure gate)
The albok unit 4.3 contract gates any workstation exposure of a wing's `.browse/` on infra-ops
demonstrating, on the chosen share surface, that renaming `.browse/` away on the server denies a
**fresh open of a never-opened child through a listing the client has cached**. The rename is what
albok-service does when it withholds a view (`.browse/` → `.browse.withheld/`).
**Result (2026-10-09 2019–2021 PT): PASS on sshfs `follow_symlinks`, Linux client.** Raw output:
`results-20261009-2019.json`; replica ids: `manifest.json`.
## Harness
- **Server:** a replica of a browse view on nh3-albok (`setup-replica.py`): 60 × 1 MiB objects at
`<wing>/<id[:2]>/<id>/<blob>`, `.browse/stash/<name>` → `../../<id[:2]>/<id>/<blob>`, the same shape
as the live view. Same root ext4 and the same OpenSSH `sftp-server` as `/srv/albok`. The live view
was NOT used: renaming it races albok-service's link/unlink and a failed mandatory unlink withholds
the wing. The replica was removed after the run (ops-log 2026-10-10T03:22:22Z).
- **Client:** ephemeral `debian:bookworm-slim` container on nh3-dev (`run.sh`): SSHFS 3.7.3,
libfuse 3.14.0, FUSE kernel interface 7.31, host kernel 6.1. Mount: `ro,follow_symlinks` plus SSH
options; the mount root is `.browse/` alone.
- **Per trial**, on a fresh mount: list `stash/` and stat every child (the client's caches now hold
the listing and every child's attributes); open+close child A; open child B and **hold** the fd;
rename on the server with `os.rename`; then immediately list again, stat C, open C (never opened
before), read B at offset 1 MiB − 8 KiB (past readahead, so it must reach the server), re-open A.
- **Conditions, N = 5 each**, distinct children every trial:
`null` (no rename), `default` (sshfs defaults), `aggressive` (`kernel_cache,entry_timeout=3600,
attr_timeout=3600,dir_cache=yes,dcache_timeout=3600`), `failed_rename` (target pre-created
non-empty, so `rename(2)` refuses with ENOTEMPTY).
## Results
| condition | rename | cached list after | stat C after | **open C (fresh)** | held B far read | re-open A | list at +25 s |
|---|---|---|---|---|---|---|---|
| null (null control) | none | 60 | ok | **opens, bytes ok** 5/5 | ok 5/5 | opens 5/5 | n/a |
| default | ok | 60 (stale) | ok (cached) | **ENOENT 5/5** | ok 5/5 | ENOENT 5/5 | ENOENT 5/5 |
| aggressive caches | ok | 60 (stale) | ok (cached) | **ENOENT 5/5** | ok 5/5 | ENOENT 5/5 | n/a |
| failed_rename | refused ENOTEMPTY | 60 | ok | **opens, bytes ok** 5/5 | ok 5/5 | opens 5/5 | n/a |
The fresh open was attempted 2–6 ms after the rename returned. Every trial started and ended with
the server in the clean state (`.browse/` present, no `.browse.withheld`).
- **The gate holds.** The client still listed the child and stat'ed it from cache, and the open
failed anyway: each sshfs open is an SFTP OPEN by path, resolved on the server, and hour-long
client caches do not change that.
- **Controls.** With no rename every open succeeds (the method does not manufacture failures). The
held fd keeps reading after the rename, and a refused rename leaves every path served, so the
instrument does register access that survives a rename (positive controls).
- **What a rename does NOT withdraw:** a file already open on the client keeps reading through its
SFTP handle. That is the contract's narrowed guarantee (new access by pathname only).
- **Failed rename:** the path remains and is served. The service cannot deny opens it could not
withdraw; its `BROWSE_WITHHELD` marker only stops publication.
## Limits
- **Proven for the Linux kernel FUSE client only.** A macOS workstation runs a different FUSE
backend (macFUSE kext, macFUSE FSKit, or fuse-t, which is NFS-backed and may cache handles the way
the contract warns NFS does). Re-run this harness on the real workstation before trusting that
surface: rebuild the replica, then run `run-trials.py` directly with `ALBOK_TEST_KEY`,
`ALBOK_TEST_KNOWN_HOSTS`, `ALBOK_TEST_MNT` (and `ALBOK_TEST_SRV`) set.
- **The mount root is a presentation choice, not a boundary.** The SSH user's authority is whatever
its groups can read: a member of the wing's read group can mount the wing root and see raw object
directories, sidecars with their sensitivity records, and git history. The contract already says
this ("the wing group's raw read is authority over files").
- The committed `run-trials.py` adds env overrides and a umount fallback after the run; the container
defaults it ran with are unchanged.
File diff suppressed because one or more lines are too long
@@ -0,0 +1,880 @@
{
"started_note": "times are monotonic seconds; see run.sh for wall clock",
"sshfs_base": [
"ro",
"follow_symlinks",
"IdentityFile=/keys/id",
"UserKnownHostsFile=/keys/known_hosts",
"StrictHostKeyChecking=yes",
"BatchMode=yes"
],
"aggressive": [
"kernel_cache",
"entry_timeout=3600",
"attr_timeout=3600",
"dir_cache=yes",
"dcache_timeout=3600"
],
"trials": [
{
"cond": "null",
"trial": 0,
"A": "doc00.md",
"B": "doc01.md",
"C": "doc02.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": true,
"read": 4096,
"content_ok": true,
"secs_after_rename": 0.002
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"server_after": "True False"
},
{
"cond": "null",
"trial": 1,
"A": "doc03.md",
"B": "doc04.md",
"C": "doc05.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": true,
"read": 4096,
"content_ok": true,
"secs_after_rename": 0.002
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"server_after": "True False"
},
{
"cond": "null",
"trial": 2,
"A": "doc06.md",
"B": "doc07.md",
"C": "doc08.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": true,
"read": 4096,
"content_ok": true,
"secs_after_rename": 0.002
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"server_after": "True False"
},
{
"cond": "null",
"trial": 3,
"A": "doc09.md",
"B": "doc10.md",
"C": "doc11.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": true,
"read": 4096,
"content_ok": true,
"secs_after_rename": 0.002
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"server_after": "True False"
},
{
"cond": "null",
"trial": 4,
"A": "doc12.md",
"B": "doc13.md",
"C": "doc14.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": true,
"read": 4096,
"content_ok": true,
"secs_after_rename": 0.002
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"server_after": "True False"
},
{
"cond": "default",
"trial": 0,
"A": "doc15.md",
"B": "doc16.md",
"C": "doc17.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 0,
"out": "renamed",
"err": "",
"secs": 0.018
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": false,
"errno": "ENOENT",
"secs_after_rename": 0.002
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": false,
"errno": "ENOENT"
},
"late_list": {
"errno": "ENOENT",
"secs_after_rename": 25.003
},
"restore": "renamed",
"server_after": "True False"
},
{
"cond": "default",
"trial": 1,
"A": "doc18.md",
"B": "doc19.md",
"C": "doc20.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 0,
"out": "renamed",
"err": "",
"secs": 0.017
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": false,
"errno": "ENOENT",
"secs_after_rename": 0.004
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": false,
"errno": "ENOENT"
},
"late_list": {
"errno": "ENOENT",
"secs_after_rename": 25.001
},
"restore": "renamed",
"server_after": "True False"
},
{
"cond": "default",
"trial": 2,
"A": "doc21.md",
"B": "doc22.md",
"C": "doc23.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 0,
"out": "renamed",
"err": "",
"secs": 0.014
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": false,
"errno": "ENOENT",
"secs_after_rename": 0.002
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": false,
"errno": "ENOENT"
},
"late_list": {
"errno": "ENOENT",
"secs_after_rename": 25.003
},
"restore": "renamed",
"server_after": "True False"
},
{
"cond": "default",
"trial": 3,
"A": "doc24.md",
"B": "doc25.md",
"C": "doc26.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 0,
"out": "renamed",
"err": "",
"secs": 0.019
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": false,
"errno": "ENOENT",
"secs_after_rename": 0.002
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": false,
"errno": "ENOENT"
},
"late_list": {
"errno": "ENOENT",
"secs_after_rename": 25.003
},
"restore": "renamed",
"server_after": "True False"
},
{
"cond": "default",
"trial": 4,
"A": "doc27.md",
"B": "doc28.md",
"C": "doc29.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 0,
"out": "renamed",
"err": "",
"secs": 0.018
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": false,
"errno": "ENOENT",
"secs_after_rename": 0.006
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": false,
"errno": "ENOENT"
},
"late_list": {
"errno": "ENOENT",
"secs_after_rename": 25.002
},
"restore": "renamed",
"server_after": "True False"
},
{
"cond": "aggressive",
"trial": 0,
"A": "doc30.md",
"B": "doc31.md",
"C": "doc32.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 0,
"out": "renamed",
"err": "",
"secs": 0.019
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": false,
"errno": "ENOENT",
"secs_after_rename": 0.003
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": false,
"errno": "ENOENT"
},
"restore": "renamed",
"server_after": "True False"
},
{
"cond": "aggressive",
"trial": 1,
"A": "doc33.md",
"B": "doc34.md",
"C": "doc35.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 0,
"out": "renamed",
"err": "",
"secs": 0.016
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": false,
"errno": "ENOENT",
"secs_after_rename": 0.002
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": false,
"errno": "ENOENT"
},
"restore": "renamed",
"server_after": "True False"
},
{
"cond": "aggressive",
"trial": 2,
"A": "doc36.md",
"B": "doc37.md",
"C": "doc38.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 0,
"out": "renamed",
"err": "",
"secs": 0.017
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": false,
"errno": "ENOENT",
"secs_after_rename": 0.002
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": false,
"errno": "ENOENT"
},
"restore": "renamed",
"server_after": "True False"
},
{
"cond": "aggressive",
"trial": 3,
"A": "doc39.md",
"B": "doc40.md",
"C": "doc41.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 0,
"out": "renamed",
"err": "",
"secs": 0.015
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": false,
"errno": "ENOENT",
"secs_after_rename": 0.002
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": false,
"errno": "ENOENT"
},
"restore": "renamed",
"server_after": "True False"
},
{
"cond": "aggressive",
"trial": 4,
"A": "doc42.md",
"B": "doc43.md",
"C": "doc44.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 0,
"out": "renamed",
"err": "",
"secs": 0.013
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": false,
"errno": "ENOENT",
"secs_after_rename": 0.003
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": false,
"errno": "ENOENT"
},
"restore": "renamed",
"server_after": "True False"
},
{
"cond": "failed_rename",
"trial": 0,
"A": "doc45.md",
"B": "doc46.md",
"C": "doc47.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 3,
"out": "refused ENOTEMPTY",
"err": "",
"secs": 0.018
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": true,
"read": 4096,
"content_ok": true,
"secs_after_rename": 0.003
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"server_after": "True False"
},
{
"cond": "failed_rename",
"trial": 1,
"A": "doc48.md",
"B": "doc49.md",
"C": "doc50.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 3,
"out": "refused ENOTEMPTY",
"err": "",
"secs": 0.016
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": true,
"read": 4096,
"content_ok": true,
"secs_after_rename": 0.003
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"server_after": "True False"
},
{
"cond": "failed_rename",
"trial": 2,
"A": "doc51.md",
"B": "doc52.md",
"C": "doc53.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 3,
"out": "refused ENOTEMPTY",
"err": "",
"secs": 0.018
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": true,
"read": 4096,
"content_ok": true,
"secs_after_rename": 0.003
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"server_after": "True False"
},
{
"cond": "failed_rename",
"trial": 3,
"A": "doc54.md",
"B": "doc55.md",
"C": "doc56.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 3,
"out": "refused ENOTEMPTY",
"err": "",
"secs": 0.017
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": true,
"read": 4096,
"content_ok": true,
"secs_after_rename": 0.002
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"server_after": "True False"
},
{
"cond": "failed_rename",
"trial": 4,
"A": "doc57.md",
"B": "doc58.md",
"C": "doc59.md",
"server_before": "True False",
"pre_list": 60,
"pre_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"pre_B_ok": true,
"rename": {
"rc": 3,
"out": "refused ENOTEMPTY",
"err": "",
"secs": 0.018
},
"post_list": {
"n": 60
},
"post_stat_C": {
"ok": true,
"size": 1048576
},
"post_open_C": {
"opened": true,
"read": 4096,
"content_ok": true,
"secs_after_rename": 0.003
},
"post_held_B": {
"read": 4096,
"content_ok": true
},
"post_reopen_A": {
"opened": true,
"read": 4096,
"content_ok": true
},
"server_after": "True False"
}
]
}
+183
View File
@@ -0,0 +1,183 @@
#!/usr/bin/env python3
"""Pathname-withdrawal test for an albok browse view shared over sshfs.
Runs on the CLIENT (an ephemeral container, see run.sh) against the replica built
by setup-replica.py. The gate (albok unit 4.3, exposure ruling): rename `.browse/`
away on the server, then a FRESH open of a child that was never opened, through a
listing the client has cached, must FAIL.
Per trial (fresh mount each time), three distinct children:
A opened and closed before the rename -> re-opened after it
B opened before the rename, fd HELD -> read at a far offset after it
C listed and stat'ed, never opened -> opened after it (the gate)
Conditions (N trials each):
null no rename C must OPEN (null control)
default rename ok, sshfs defaults C must FAIL (the gate)
aggressive rename ok, hour-long caches C must FAIL (gate, worst-case client caching)
failed_rename rename refused (ENOTEMPTY) C OPENS: the path remains and is served
B is the positive control: a retained handle must still read after a rename, which
proves the instrument can see access that survives a rename.
Usage: run-trials.py <manifest.json> <trials-per-condition> (JSON to stdout)
"""
import errno
import json
import os
import shlex
import subprocess
import sys
import time
# Overridable so the same harness runs on a workstation (e.g. macOS) without the container.
SRV = os.environ.get("ALBOK_TEST_SRV", "infra-ops@10.100.50.44")
KEY = os.environ.get("ALBOK_TEST_KEY", "/keys/id")
KH = os.environ.get("ALBOK_TEST_KNOWN_HOSTS", "/keys/known_hosts")
MNT = os.environ.get("ALBOK_TEST_MNT", "/mnt/browse")
SSH_OPTS = ["-i", KEY, "-o", "BatchMode=yes", "-o", f"UserKnownHostsFile={KH}",
"-o", "StrictHostKeyChecking=yes", "-o", "ControlMaster=auto",
"-o", "ControlPath=/tmp/cm-%r@%h", "-o", "ControlPersist=900"]
SSHFS_BASE = ["ro", "follow_symlinks", f"IdentityFile={KEY}", f"UserKnownHostsFile={KH}",
"StrictHostKeyChecking=yes", "BatchMode=yes"]
AGGRESSIVE = ["kernel_cache", "entry_timeout=3600", "attr_timeout=3600",
"dir_cache=yes", "dcache_timeout=3600"]
CONDITIONS = [("null", [], None), ("default", [], "ok"),
("aggressive", AGGRESSIVE, "ok"), ("failed_rename", [], "fail")]
LATE_LIST_DELAY = 25 # past sshfs's default 20 s cache timeout
manifest = json.load(open(sys.argv[1]))
N = int(sys.argv[2])
ROOT = manifest["root"]
SIZE = manifest["size"]
OBJS = manifest["objects"]
VIEW, WITHHELD = f"{ROOT}/.browse", f"{ROOT}/.browse.withheld"
def ename(e: OSError) -> str:
return errno.errorcode.get(e.errno or 0, str(e.errno))
def content(o) -> bytes:
line = f"replica object {o['i']} {o['id']}\n".encode()
return (line * (SIZE // len(line) + 1))[:SIZE]
def remote_py(code: str, *args: str):
cmd = "python3 -c " + shlex.quote(code) + "".join(" " + shlex.quote(a) for a in args)
t0 = time.monotonic()
r = subprocess.run(["ssh", *SSH_OPTS, SRV, cmd], capture_output=True, text=True, timeout=60)
return {"rc": r.returncode, "out": r.stdout.strip(), "err": r.stderr.strip()[-300:],
"secs": round(time.monotonic() - t0, 3)}
RENAME = ("import os,sys,errno\n"
"try:\n os.rename(sys.argv[1], sys.argv[2]); print('renamed')\n"
"except OSError as e:\n print('refused', errno.errorcode.get(e.errno)); sys.exit(3)")
STATE = "import os,sys; print(os.path.isdir(sys.argv[1]), os.path.lexists(sys.argv[2]))"
def open_read(path, expect, offset=0, n=4096):
try:
fd = os.open(path, os.O_RDONLY)
except OSError as e:
return {"opened": False, "errno": ename(e)}
try:
os.lseek(fd, offset, os.SEEK_SET)
b = os.read(fd, n)
return {"opened": True, "read": len(b), "content_ok": b == expect[offset:offset + n]}
except OSError as e:
return {"opened": True, "read_errno": ename(e)}
finally:
os.close(fd)
def listing(d):
try:
return {"n": len(os.listdir(d))}
except OSError as e:
return {"errno": ename(e)}
def stat(p):
try:
st = os.stat(p)
return {"ok": True, "size": st.st_size}
except OSError as e:
return {"ok": False, "errno": ename(e)}
def trial(cond, extra, rename, k, slot):
A, B, C = OBJS[3 * slot], OBJS[3 * slot + 1], OBJS[3 * slot + 2]
rec = {"cond": cond, "trial": k, "A": A["name"], "B": B["name"], "C": C["name"]}
rec["server_before"] = remote_py(STATE, VIEW, WITHHELD)["out"]
if rec["server_before"] != "True False":
rec["aborted"] = "server not in the clean state"
return rec
if rename == "fail": # a non-empty target makes rename(2) refuse
remote_py("import os,sys; os.makedirs(sys.argv[1])", f"{WITHHELD}/blocker")
os.makedirs(MNT, exist_ok=True)
m = subprocess.run(["sshfs", f"{SRV}:{VIEW}", MNT, "-o", ",".join(SSHFS_BASE + extra)],
capture_output=True, text=True, timeout=60)
if m.returncode != 0:
rec["aborted"] = f"mount failed: {m.stderr.strip()}"
return rec
renamed = False
fd_b = None
try:
st_dir = f"{MNT}/stash"
names = os.listdir(st_dir)
rec["pre_list"] = len(names)
for nm in names: # cache every child's attrs: the client "knows" C exists
os.stat(f"{st_dir}/{nm}")
rec["pre_A"] = open_read(f"{st_dir}/{A['name']}", content(A))
fd_b = os.open(f"{st_dir}/{B['name']}", os.O_RDONLY)
rec["pre_B_ok"] = os.read(fd_b, 4096) == content(B)[:4096]
if rename:
rec["rename"] = remote_py(RENAME, VIEW, WITHHELD)
renamed = rec["rename"]["out"] == "renamed"
t0 = time.monotonic()
rec["post_list"] = listing(st_dir)
rec["post_stat_C"] = stat(f"{st_dir}/{C['name']}")
rec["post_open_C"] = open_read(f"{st_dir}/{C['name']}", content(C))
rec["post_open_C"]["secs_after_rename"] = round(time.monotonic() - t0, 3)
far = SIZE - 8192 # well past readahead: this read must go to the server
try:
os.lseek(fd_b, far, os.SEEK_SET)
b = os.read(fd_b, 4096)
rec["post_held_B"] = {"read": len(b), "content_ok": b == content(B)[far:far + 4096]}
except OSError as e:
rec["post_held_B"] = {"read_errno": ename(e)}
rec["post_reopen_A"] = open_read(f"{st_dir}/{A['name']}", content(A))
if cond == "default":
time.sleep(max(0.0, LATE_LIST_DELAY - (time.monotonic() - t0)))
rec["late_list"] = listing(st_dir)
rec["late_list"]["secs_after_rename"] = round(time.monotonic() - t0, 3)
finally:
if fd_b is not None:
os.close(fd_b)
for um in (["fusermount3", "-u", MNT], ["fusermount", "-u", MNT], ["umount", MNT]):
try:
if subprocess.run(um, capture_output=True).returncode == 0:
break
except FileNotFoundError:
continue
if renamed:
rec["restore"] = remote_py(RENAME, WITHHELD, VIEW)["out"]
if rename == "fail":
remote_py("import os,sys; os.rmdir(sys.argv[1]); os.rmdir(sys.argv[2])",
f"{WITHHELD}/blocker", WITHHELD)
rec["server_after"] = remote_py(STATE, VIEW, WITHHELD)["out"]
return rec
results = []
slot = 0
for cond, extra, rename in CONDITIONS:
for k in range(N):
results.append(trial(cond, extra, rename, k, slot))
slot += 1
json.dump({"started_note": "times are monotonic seconds; see run.sh for wall clock",
"sshfs_base": SSHFS_BASE, "aggressive": AGGRESSIVE, "trials": results},
sys.stdout, indent=1)
+24
View File
@@ -0,0 +1,24 @@
#!/usr/bin/env bash
# Pathname-withdrawal test for the albok browse view over sshfs (unit 4.3 exposure gate).
#
# 1. ssh infra-ops@10.100.50.44 python3 - < setup-replica.py > manifest.json
# 2. ./run.sh [trials-per-condition] # default 5; JSON lands in results-<stamp>.json
# 3. remove the replica on nh3-albok by its literal path when done
#
# The client is an ephemeral Debian 12 container on nh3-dev (sshfs 3.7.3 / libfuse 3), so
# nothing is installed on the host. It needs /dev/fuse and CAP_SYS_ADMIN to mount.
set -euo pipefail
here=$(cd "$(dirname "$0")" && pwd)
n=${1:-5}
stamp=$(date +%Y%m%d-%H%M)
out="$here/results-$stamp.json"
echo "start $(date '+%F %H%M %Z')" >&2
docker run --rm --device /dev/fuse --cap-add SYS_ADMIN --security-opt apparmor=unconfined \
-v "$HOME/.ssh/infra-ops_ed25519:/keys/id:ro" \
-v "$HOME/.ssh/known_hosts:/keys/known_hosts:ro" \
-v "$here:/work" \
debian:bookworm-slim bash -c "
apt-get -qq update >/dev/null && apt-get -qq install -y sshfs python3 openssh-client >/dev/null 2>&1
sshfs --version >&2
python3 /work/run-trials.py /work/manifest.json $n" > "$out"
echo "end $(date '+%F %H%M %Z') -> $out" >&2
+51
View File
@@ -0,0 +1,51 @@
#!/usr/bin/env python3
"""Build a replica of an albok browse view on the share host (nh3-albok).
Run ON the server as infra-ops, no sudo:
ssh infra-ops@10.100.50.44 python3 - < setup-replica.py > manifest.json
Why a replica and not the live view: the test renames `.browse/` away. On a live
wing that rename races albok-service's own link()/unlink() calls, and a failed
mandatory unlink withholds the wing (BROWSE_WITHHELD marker + a failed write
transaction). The replica has the same shape as the real view
(`<wing>/.browse/<locator>` -> relative symlink `../../<id[:2]>/<id>/<blob>`), sits
on the same filesystem (root ext4), and is served by the same sftp-server, which
is what the withdrawal property depends on.
Content is deterministic per object, so the client can verify bytes without
fetching a reference copy.
"""
import hashlib
import json
import os
import uuid
ROOT = "/home/infra-ops/albok-browse-test/wing"
COUNT = 60
SIZE = 1 << 20 # 1 MiB: larger than FUSE readahead, so a late read must reach the server
def content(i: int, oid: str) -> bytes:
line = f"replica object {i} {oid}\n".encode()
return (line * (SIZE // len(line) + 1))[:SIZE]
os.makedirs(ROOT, exist_ok=False) # refuse to run over an existing replica
link_dir = os.path.join(ROOT, ".browse", "stash")
os.makedirs(link_dir)
objs = []
for i in range(COUNT):
oid = uuid.uuid4().hex
obj_dir = os.path.join(ROOT, oid[:2], oid)
os.makedirs(obj_dir)
name = f"doc{i:02d}.md"
blob = os.path.join(obj_dir, name)
data = content(i, oid)
with open(blob, "wb") as f:
f.write(data)
os.chmod(blob, 0o640)
os.symlink(os.path.relpath(blob, link_dir), os.path.join(link_dir, name))
objs.append({"i": i, "id": oid, "name": name, "size": SIZE,
"sha256": hashlib.sha256(data).hexdigest()})
print(json.dumps({"root": ROOT, "size": SIZE, "objects": objs}))