vh
87d7555fcf
A zsh glob failure in the staging step of91bda3cmeant the five renamed playbooks went in as pure renames with their bodies still saying ana-ml2 and 10.250.50.54, and the homepage docker.yaml direct-connection edit from17586abdid not stage either. Same content, just the half that got dropped.
44 lines
2.2 KiB
YAML
44 lines
2.2 KiB
YAML
# Make vm.overcommit_memory=1 durable on fv-ml1 (GPU inference host).
|
|
#
|
|
# Why: fv-ml1 runs vm.overcommit_memory=0 (heuristic) with zero swap, so the
|
|
# CommitLimit is ~RAM/2 (~283 GB of 566 GB). The resident vLLM services already
|
|
# commit ~224 GB of address space, leaving < 60 GB of headroom. A large model-file
|
|
# mmap (e.g. the 50 GB NVFP4 shard during HF->native conversion, or a vLLM model
|
|
# load) then fails with ENOMEM despite ~393 GB of RAM actually being free — the
|
|
# kernel rejects the *commit*, not the allocation.
|
|
#
|
|
# overcommit_memory=1 (always overcommit) is the conventional setting for ML hosts
|
|
# that mmap large files: the real RAM is there to back the pages, and the heuristic
|
|
# accounting is the only thing in the way. Operator-directed permanent + durable
|
|
# (2026-06-17). A drop-in under /etc/sysctl.d/ applies at every boot.
|
|
#
|
|
# Run: scripts/elway infra-ops@fv-ml1 --playbook playbooks/fv-ml1-overcommit-memory.yaml
|
|
# Rerunnable: a second run shows the write step `skipped` (idempotent via when:).
|
|
|
|
vars:
|
|
dropin: /etc/sysctl.d/99-overcommit-memory.conf
|
|
setting: "vm.overcommit_memory = 1"
|
|
|
|
steps:
|
|
- name: Write durable overcommit sysctl drop-in
|
|
# elway runs steps as the SSH user, so a shell `>` redirect can't write a
|
|
# root-owned path — pipe through `sudo tee` (infra-ops has NOPASSWD sudo).
|
|
shell: |
|
|
printf '# GPU inference host: large model-file mmaps (NVFP4 native convert, vLLM loads)\n# exceed the heuristic CommitLimit (overcommit=0 + zero swap) despite ample free RAM.\n# Operator-directed permanent setting 2026-06-17.\n%s\n' '{{ setting }}' | sudo tee {{ dropin }} >/dev/null
|
|
# Skip the write if the drop-in already holds exactly this line.
|
|
when: "! grep -qxF '{{ setting }}' {{ dropin }} 2>/dev/null"
|
|
|
|
- name: Apply all sysctl drop-ins now
|
|
shell: sudo sysctl --system >/dev/null
|
|
# Applying is a no-op when the runtime value already matches.
|
|
changed_when: "false"
|
|
|
|
verify:
|
|
- name: Runtime vm.overcommit_memory is 1
|
|
shell: test "$(cat /proc/sys/vm/overcommit_memory)" = "1"
|
|
changed_when: "false"
|
|
|
|
- name: Drop-in file persists the setting (survives reboot)
|
|
shell: grep -qxF '{{ setting }}' {{ dropin }}
|
|
changed_when: "false"
|