vh
7e7130172e
Two related changes shipped together. The stack rename is independent but adding `vllm-reward` to the existing `vllm-qwen3` would have made that name actively misleading. **Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo references (README.md root, servers/ana-ml2/, stacks/llama-swap/, configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two intentional history mentions retained (servers/ana-ml2 + stacks/vllm README). **Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ on port 8003. The AWQ output is a locally-quantized model (not from HF), so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the shared HF cache. Model config.json declares LlamaForSequenceClassification which vLLM's pooling runner picks up automatically — produces a single reward score per input via /classify. **Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1 deprecated --task in favor of --runner pooling (model architecture in config.json drives the classification head). Compose uses --runner pooling with a comment explaining the substitution. **GPU memory:** no rebalance needed — production had already tuned EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB headroom on the 48 GB Ada. **Server-side:** brought existing vllm-qwen3 down, mv'd /opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed new compose via scripts/deploy-stack.sh, brought all 3 services up. **Smoke tests:** - /health on 8001/8002/8003 → 200 - /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ with max_model_len 16384 - /classify with a sample conversation → returns LABEL_0 with prob 0.9999 (single-output regression-style reward score, expected shape for a reward model)
156 lines
8.2 KiB
Markdown
156 lines
8.2 KiB
Markdown
# Persistent memory — eshpfi-management
|
|
|
|
_Last updated: 2026-05-08_
|
|
|
|
## Repo purpose
|
|
|
|
Reference workspace for PFI infrastructure: server inventory, canonical
|
|
Docker Compose stacks, ops playbooks, and conventions. Authoritative
|
|
copies of compose files live on the servers under
|
|
`/opt/docker/compose/<stack>/`; this repo mirrors them for version
|
|
control, editing, planning, and CI-driven deploys.
|
|
|
|
## Tools and conventions
|
|
|
|
Sister repos (separate gitea repos, deployed by playbooks here):
|
|
|
|
| Repo | Role | CI status |
|
|
|---|---|---|
|
|
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) |
|
|
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
|
|
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
|
|
|
|
- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
|
|
fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
|
|
VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
|
|
cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
|
|
for the blast-radius matrix.
|
|
|
|
## Current state / in-flight
|
|
|
|
_As of 2026-05-08:_
|
|
|
|
- **CI/CD pipe is live for three repos** (task-board, vor, nevermore).
|
|
Pattern: workflow checks out triggering repo + `vh/esh-pfi-infrastructure`
|
|
(for elway), SSHes to ana-docker, runs an elway playbook pinned to the
|
|
triggering commit SHA. User-scope secrets `DEPLOY_SSH_KEY` and
|
|
`MGMT_REPO_TOKEN` cover all repos under `vh/`.
|
|
- **Open**: rotate `MINIFLUX_PASSWORD` — leaked in a prior session's
|
|
transcript while seeding nevermore's `.env` (full leak once, partial
|
|
leak once).
|
|
- **Open**: clean up old `news-digest` detritus on ana-docker —
|
|
`/opt/docker/compose/news-digest/`, `/opt/docker/data/news-digest/`,
|
|
image `local/news-digest:v5`. Migrate the historical `edition-*.html`
|
|
archive files into `/opt/docker/data/nevermore/` first.
|
|
- **Watch**: nh3-nas `/volume1` at 65%; plan retention review or
|
|
capacity expansion before ~80%.
|
|
- **Possible next**: aggregator service + homepage widget for live
|
|
"fleet backup health" dashboard. Discussed but not committed; user
|
|
weighing against a `/schedule` weekly digest as the lighter
|
|
alternative.
|
|
- **Open mystery**: `docker push` from outside the LAN to
|
|
`gitea.phasefinal.com` has a **~60s client-side per-PATCH ceiling**
|
|
for chunked blob uploads. We confirmed the timer is client-side
|
|
(Traefik logged `499 / 60012ms` — client closed; Gitea logged the
|
|
same as `unexpected EOF`). Where exactly the 60s lives in the docker
|
|
daemon / containerd stack — and whether it's tunable — is
|
|
unidentified. Worked around for now by shrinking images below the
|
|
cliff; if it bites again, instrument with `dockerd -D` + strace on a
|
|
fresh push to find the actual timer.
|
|
|
|
## Recent decisions
|
|
|
|
- `[2026-05-08]` Pin `torch` to CPU-only via
|
|
`--index-url https://download.pytorch.org/whl/cpu` in
|
|
`Worldtree/Dockerfile` before the rest of `requirements.txt`.
|
|
`sentence-transformers` transitively pulls torch and grabs the CUDA
|
|
flavor by default (~2.5GB of nvidia libs). Worldtree runs on
|
|
`ana-docker` (no GPU reservation in `compose.yaml`) so CUDA is dead
|
|
weight. Image: 5.62GB → 1.6GB. The failing push from off-LAN now
|
|
succeeds.
|
|
- `[2026-05-08]` Split `Worldtree/requirements.txt` into runtime +
|
|
`requirements-dev.txt` (pytest, pytest-asyncio, ruff, pylint).
|
|
~50MB out of the runtime image; smaller attack surface.
|
|
- `[2026-05-01]` nevermore **replaces** news-digest at port 8181 (not
|
|
coexist on 8182). User chose one stack at a time over soak window.
|
|
~30s briefing gap accepted as the cost.
|
|
- `[2026-04-30]` Bumped nevermore deploy playbook's first-render wait
|
|
from 60s to 6 min — first deploy with empty `/output` requires a
|
|
worker LLM cycle (2-4 min) before `index.html` exists.
|
|
- `[2026-04-29]` Workflow drops explicit `container:` directive after
|
|
runner re-registration with `:docker://node:20-bookworm-slim` schema
|
|
labels — single source of truth for the build environment.
|
|
- `[2026-04-29]` Deploy playbooks accept SHA refs in addition to branch
|
|
names (`git rev-parse <ref>` then `origin/<ref>^{commit}` fallback).
|
|
CI passes `--var ref=${{ github.sha }}`; manual runs pass `main` /
|
|
`v0.1.0`. Same code path either way.
|
|
- `[2026-04-29]` Single central Gitea Actions runner on ana-docker
|
|
(`f014d55`). Parameterized `playbooks/deploy-gitea-runner.yaml` so
|
|
site-local runners (nh3, esh) drop in via `--var` overrides — not
|
|
copy-pasted playbooks.
|
|
- `[2026-04-29]` `nevermore` carries forward `hidden.json` and
|
|
historical `MINIFLUX_PASSWORD` from `news-digest` during cutover
|
|
(preserves user state across stack rename).
|
|
|
|
## Tried and abandoned
|
|
|
|
- `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT
|
|
timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero
|
|
matches and led to a wrong "no /v2/ traffic in 12 days" conclusion.
|
|
**Gitea logs in PDT, Traefik logs in UTC** — same host, different
|
|
timezones. Always normalize timezones (UTC) when correlating logs
|
|
across services on the same box. Cost: ~30 min in the wrong
|
|
direction.
|
|
- `[2026-05-08]` Validating a user-proposed Traefik/Gitea timeout bump
|
|
for "Traefik is dropping connections on big docker pushes" without
|
|
first verifying which component was actually in the failure path.
|
|
Traefik turned out to be innocent (`499 / 60012ms` = client closed,
|
|
Traefik never timed out), Gitea's `PER_WRITE_TIMEOUT` governs
|
|
response writes (wrong direction), and the real ceiling was a ~60s
|
|
client-side timer no compose change can reach. **Lesson: validate
|
|
the diagnostic premise — which component is actually in the failure
|
|
path? — before refining the proposed fix.**
|
|
- `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` /
|
|
`PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on
|
|
`/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern
|
|
**response writes**, not request body reads. `unexpected EOF` from
|
|
Go's HTTP server means the client closed mid-body-upload; not a
|
|
knob Gitea exposes server-side.
|
|
- `[2026-04-30]` task-board workflow with
|
|
`container: image: debian:bookworm-slim` — fails:
|
|
`actions/checkout@v4` needs `node` at runtime, slim image lacks it.
|
|
Switched to `node:20-bookworm-slim` (has node + apt) or runner-label
|
|
default.
|
|
- `[2026-04-30]` Dropping the `container:` directive before runner
|
|
re-registration with docker-schema labels — runner silently falls
|
|
back to **host mode** (jobs run inside the alpine `act_runner`
|
|
container itself, no apt). The `:host` suffix in startup logs
|
|
(`labels updated to: [pfi-fleet:host ana-docker:host]`) is the
|
|
giveaway. Fix: register with `pfi-fleet:docker://<image>` schema
|
|
labels.
|
|
- `[2026-04-30]` Updating runner labels by editing `.env` and bouncing
|
|
— doesn't take. The `.runner` registration cache pins labels at
|
|
first registration; env-var updates are read each start but the
|
|
stored token + UUID are tied to the original label set on the gitea
|
|
side. Fix: stop runner, delete `.runner`, generate new admin
|
|
registration token, redeploy.
|
|
- `[2026-04-30]` `git reset --hard origin/<sha>` in
|
|
`deploy-task-board.yaml` (and the in-repo nevermore playbook before
|
|
fix) — invalid syntax: `origin/` prefix only works for branch refs.
|
|
SHAs need `git reset --hard <sha>` directly. Resolved with
|
|
`git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first,
|
|
then bare `"{{ ref }}^{commit}"` fallback.
|
|
- `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after
|
|
task-board wiring — it was actually only repo-scope on
|
|
`vh/task-board`. vor's first CI run failed with empty SSH key
|
|
(`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user
|
|
scope at `gitea.phasefinal.com/user/settings/actions/secrets`.
|
|
- `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity —
|
|
leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a
|
|
follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1<redacted>/"`
|
|
still leaked the first 4 chars. Lesson: when probing secret-bearing
|
|
files, use field-by-field SELECTIVE inspection
|
|
(`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any
|
|
password line, `grep -c` (existence) or `test -n "$(...)"`
|
|
(non-empty), never `cat` or value-printing.
|