Files

T

vh 7e7130172e vllm: rename stack from vllm-qwen3 → vllm + add Skywork reward classifier

Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.

**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).

**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.

**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.

**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.

**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.

**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
  with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
  (single-output regression-style reward score, expected shape for a
  reward model)

2026-05-13 22:00:26 -07:00

8.2 KiB

Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-05-08

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo	Role	CI status
`vh/task-board`	MCP + web dashboard for assistant task state (port 7878)	push-to-main → CI deploys (2026-04-29)
`vh/vor`	Inquisitor UI sidecar (port 7879)	push-to-main → CI deploys (2026-04-29)
`vh/nevermore`	Twice-daily LLM-curated briefing (port 8181, replaces news-digest)	push-to-main → CI deploys (2026-04-30)

Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix.

Current state / in-flight

As of 2026-05-08:

CI/CD pipe is live for three repos (task-board, vor, nevermore). Pattern: workflow checks out triggering repo + vh/esh-pfi-infrastructure (for elway), SSHes to ana-docker, runs an elway playbook pinned to the triggering commit SHA. User-scope secrets DEPLOY_SSH_KEY and MGMT_REPO_TOKEN cover all repos under vh/.
Open: rotate MINIFLUX_PASSWORD — leaked in a prior session's transcript while seeding nevermore's .env (full leak once, partial leak once).
Open: clean up old news-digest detritus on ana-docker — /opt/docker/compose/news-digest/, /opt/docker/data/news-digest/, image local/news-digest:v5. Migrate the historical edition-*.html archive files into /opt/docker/data/nevermore/ first.
Watch: nh3-nas /volume1 at 65%; plan retention review or capacity expansion before ~80%.
Possible next: aggregator service + homepage widget for live "fleet backup health" dashboard. Discussed but not committed; user weighing against a /schedule weekly digest as the lighter alternative.
Open mystery: docker push from outside the LAN to gitea.phasefinal.com has a ~60s client-side per-PATCH ceiling for chunked blob uploads. We confirmed the timer is client-side (Traefik logged 499 / 60012ms — client closed; Gitea logged the same as unexpected EOF). Where exactly the 60s lives in the docker daemon / containerd stack — and whether it's tunable — is unidentified. Worked around for now by shrinking images below the cliff; if it bites again, instrument with dockerd -D + strace on a fresh push to find the actual timer.

Recent decisions

[2026-05-08] Pin torch to CPU-only via --index-url https://download.pytorch.org/whl/cpu in Worldtree/Dockerfile before the rest of requirements.txt. sentence-transformers transitively pulls torch and grabs the CUDA flavor by default (~2.5GB of nvidia libs). Worldtree runs on ana-docker (no GPU reservation in compose.yaml) so CUDA is dead weight. Image: 5.62GB → 1.6GB. The failing push from off-LAN now succeeds.
[2026-05-08] Split Worldtree/requirements.txt into runtime + requirements-dev.txt (pytest, pytest-asyncio, ruff, pylint). ~50MB out of the runtime image; smaller attack surface.
[2026-05-01] nevermore replaces news-digest at port 8181 (not coexist on 8182). User chose one stack at a time over soak window. ~30s briefing gap accepted as the cost.
[2026-04-30] Bumped nevermore deploy playbook's first-render wait from 60s to 6 min — first deploy with empty /output requires a worker LLM cycle (2-4 min) before index.html exists.
[2026-04-29] Workflow drops explicit container: directive after runner re-registration with :docker://node:20-bookworm-slim schema labels — single source of truth for the build environment.
[2026-04-29] Deploy playbooks accept SHA refs in addition to branch names (git rev-parse <ref> then origin/<ref>^{commit} fallback). CI passes --var ref=${{ github.sha }}; manual runs pass main / v0.1.0. Same code path either way.
[2026-04-29] Single central Gitea Actions runner on ana-docker (f014d55). Parameterized playbooks/deploy-gitea-runner.yaml so site-local runners (nh3, esh) drop in via --var overrides — not copy-pasted playbooks.
[2026-04-29] nevermore carries forward hidden.json and historical MINIFLUX_PASSWORD from news-digest during cutover (preserves user state across stack rename).

Tried and abandoned

[2026-05-08] Filtering Traefik's UTC access log by Gitea-local-PDT timestamp substrings (grep "2026/05/08 15:1[2-7]") returned zero matches and led to a wrong "no /v2/ traffic in 12 days" conclusion. Gitea logs in PDT, Traefik logs in UTC — same host, different timezones. Always normalize timezones (UTC) when correlating logs across services on the same box. Cost: ~30 min in the wrong direction.
[2026-05-08] Validating a user-proposed Traefik/Gitea timeout bump for "Traefik is dropping connections on big docker pushes" without first verifying which component was actually in the failure path. Traefik turned out to be innocent (499 / 60012ms = client closed, Traefik never timed out), Gitea's PER_WRITE_TIMEOUT governs response writes (wrong direction), and the real ceiling was a ~60s client-side timer no compose change can reach. Lesson: validate the diagnostic premise — which component is actually in the failure path? — before refining the proposed fix.
[2026-05-08] Bumping Gitea PER_WRITE_TIMEOUT / PER_WRITE_PER_KB_TIMEOUT to address unexpected EOF on /v2/.../blobs/uploads/ PATCH — wrong direction. Both govern response writes, not request body reads. unexpected EOF from Go's HTTP server means the client closed mid-body-upload; not a knob Gitea exposes server-side.
[2026-04-30] task-board workflow with container: image: debian:bookworm-slim — fails: actions/checkout@v4 needs node at runtime, slim image lacks it. Switched to node:20-bookworm-slim (has node + apt) or runner-label default.
[2026-04-30] Dropping the container: directive before runner re-registration with docker-schema labels — runner silently falls back to host mode (jobs run inside the alpine act_runner container itself, no apt). The :host suffix in startup logs (labels updated to: [pfi-fleet:host ana-docker:host]) is the giveaway. Fix: register with pfi-fleet:docker://<image> schema labels.
[2026-04-30] Updating runner labels by editing .env and bouncing — doesn't take. The .runner registration cache pins labels at first registration; env-var updates are read each start but the stored token + UUID are tied to the original label set on the gitea side. Fix: stop runner, delete .runner, generate new admin registration token, redeploy.
[2026-04-30] git reset --hard origin/<sha> in deploy-task-board.yaml (and the in-repo nevermore playbook before fix) — invalid syntax: origin/ prefix only works for branch refs. SHAs need git reset --hard <sha> directly. Resolved with git rev-parse --verify --quiet "origin/{{ ref }}^{commit}" first, then bare "{{ ref }}^{commit}" fallback.
[2026-04-30] Assuming DEPLOY_SSH_KEY was at user scope after task-board wiring — it was actually only repo-scope on vh/task-board. vor's first CI run failed with empty SSH key (printf '%s\n' "" > ~/.ssh/id_ed25519). Fix: copy secret to user scope at gitea.phasefinal.com/user/settings/actions/secrets.
[2026-04-30] grep -vE "^(#|$)" to inspect .env for sanity — leaked the full MINIFLUX_PASSWORD line into the transcript. Then a follow-up redaction attempt with sed -E "s/=(.{4}).*$/=\1<redacted>/" still leaked the first 4 chars. Lesson: when probing secret-bearing files, use field-by-field SELECTIVE inspection (grep -E "^(KEY1|KEY2)=") rather than negative filters; for any password line, grep -c (existence) or test -n "$(...)" (non-empty), never cat or value-printing.

8.2 KiB Raw Blame History