vh
b6340519bc
Session captured for a context reset. Six new detail entries. THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~550 Mbit/s aggregate against a circuit measured at 2,153 Mbit/s. Not WireGuard (it is IPsec on ana-gw), not CPU (idle), not crypto exhaustion (NPU-offloaded), not the fibre. Both tunnels negotiate aes256-sha1; AES-GCM proposed. Operator signalled authorization; execution pending, untracked by operator choice. Also recorded: selene retired after losing a head-to-head on its own job with chat-judge moved to gen and the model name left to 404; the 7-alias collision on the gen seat that makes cross-alias corroboration an echo; hrafn adopted and its CI found to have been reporting green while deploying nothing for its whole life; all three Worldtree instances de-armed from a 69-day-stale :latest and the Matrix homeserver re-plumbed to personal; every secret-bearing .env on ana-docker tightened to 0600; the pfi org closing the repo-creation half of the credential-migration directive; booth kept-board deletion and link pruning. Two entries under Tried and abandoned: the CI checkout assertion that broke the pipeline twice and was removed, and my proposal to alias a retired model name at a different model, which the operator correctly overruled. Index 271 -> 282 lines, under the 300 cap, so no archival fired.
51 lines
2.2 KiB
Markdown
51 lines
2.2 KiB
Markdown
# [2026-08-23] `/mnt/smithy` mounted on ana-ml2 — read-only and SOFT, deliberately not matching nh3-dev
|
|
|
|
brokkr-smithy-dev asked for `10.100.50.50:/volume1/smithy` on ana-ml2 to run R47's
|
|
CPU-bound corpus pipeline on 96 idle EPYC cores instead of one nh3-dev vCPU. Granted,
|
|
with two deliberate deviations from what was requested.
|
|
|
|
```
|
|
sudo mount -t nfs4 -o ro,soft,timeo=30,retrans=3,proto=tcp,vers=4.1 \
|
|
10.100.50.50:/volume1/smithy /mnt/smithy
|
|
```
|
|
|
|
The export already permitted ana-ml2 — no DSM change needed. Write is genuinely
|
|
refused.
|
|
|
|
## Why soft, not hard
|
|
|
|
They asked to match nh3-dev's mount, which is `hard`. **nh3-dev is same-site as the
|
|
NAS; ana-ml2 is not** — this is cross-site NFS on the box running the fleet's
|
|
inference seats. A hard mount turns a link blip into unkillable D-state, and this
|
|
fleet has already lost a host that way (esh-docker-vm; only fix was a reboot). Soft
|
|
returns EIO, the batch job fails, you rerun it. The soft-mount corruption caveat is a
|
|
**write** hazard and this is read-only. Mirrors the existing ESH books mount.
|
|
|
|
## Why not in fstab
|
|
|
|
Manual only, matching irv-ml1's `/mnt/smithy` precedent. A cross-site NFS entry in
|
|
fstab can hang boot on a GPU host with 71 days uptime. **Needs remounting after a
|
|
reboot.**
|
|
|
|
## The performance reality, measured on the same file through the same mount
|
|
|
|
```
|
|
ana-ml2 (cross-site) nh3-dev (same-site)
|
|
sequential read 24.7 MB/s 98.3 MB/s
|
|
small-file rate 45.3 files/s 34.6 files/s
|
|
```
|
|
|
|
Two different stories, and file layout decides which you get:
|
|
|
|
- **Many small records -> ana-ml2 wins on BOTH axes.** That path is bound by per-file
|
|
round-trips and NAS overhead, not bandwidth, and ana-ml2 is an idle 96-core box
|
|
while nh3-dev is a loaded 16-vCPU VM.
|
|
- **Bulk sequential streaming -> the link eats the win.** 4x read penalty against a
|
|
6x CPU gain. `datasets/raw` is 126 GB, `datasets/derived` is 1.7 GB — which one the
|
|
pipeline traverses changes the answer by two orders of magnitude. Staging a subset
|
|
to ana-ml2 local disk (195 GB free) beats pulling it over the wire repeatedly.
|
|
|
|
The 24.7 MB/s is the Anaheim tunnel, not NFS and not the NAS — see
|
|
[[2026-08-23-anaheim-ipsec-tunnel-ceiling]]. No mount tuning will move it;
|
|
parallelism will.
|