# Persistent memory — eshpfi-management _Last updated: 2026-05-08_ ## Repo purpose Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under `/opt/docker/compose//`; this repo mirrors them for version control, editing, planning, and CI-driven deploys. ## Tools and conventions Sister repos (separate gitea repos, deployed by playbooks here): | Repo | Role | CI status | |---|---|---| | `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) | | `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) | | `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) | - **Two-layer backups** — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see `docs/runbooks/disaster-recovery.md` for the blast-radius matrix. ## Current state / in-flight _As of 2026-05-08:_ - **CI/CD pipe is live for three repos** (task-board, vor, nevermore). Pattern: workflow checks out triggering repo + `vh/esh-pfi-infrastructure` (for elway), SSHes to ana-docker, runs an elway playbook pinned to the triggering commit SHA. User-scope secrets `DEPLOY_SSH_KEY` and `MGMT_REPO_TOKEN` cover all repos under `vh/`. - **Open**: rotate `MINIFLUX_PASSWORD` — leaked in a prior session's transcript while seeding nevermore's `.env` (full leak once, partial leak once). - **Open**: clean up old `news-digest` detritus on ana-docker — `/opt/docker/compose/news-digest/`, `/opt/docker/data/news-digest/`, image `local/news-digest:v5`. Migrate the historical `edition-*.html` archive files into `/opt/docker/data/nevermore/` first. - **Watch**: nh3-nas `/volume1` at 65%; plan retention review or capacity expansion before ~80%. - **Possible next**: aggregator service + homepage widget for live "fleet backup health" dashboard. Discussed but not committed; user weighing against a `/schedule` weekly digest as the lighter alternative. - **Open mystery**: `docker push` from outside the LAN to `gitea.phasefinal.com` has a **~60s client-side per-PATCH ceiling** for chunked blob uploads. We confirmed the timer is client-side (Traefik logged `499 / 60012ms` — client closed; Gitea logged the same as `unexpected EOF`). Where exactly the 60s lives in the docker daemon / containerd stack — and whether it's tunable — is unidentified. Worked around for now by shrinking images below the cliff; if it bites again, instrument with `dockerd -D` + strace on a fresh push to find the actual timer. ## Recent decisions - `[2026-05-08]` Pin `torch` to CPU-only via `--index-url https://download.pytorch.org/whl/cpu` in `Worldtree/Dockerfile` before the rest of `requirements.txt`. `sentence-transformers` transitively pulls torch and grabs the CUDA flavor by default (~2.5GB of nvidia libs). Worldtree runs on `ana-docker` (no GPU reservation in `compose.yaml`) so CUDA is dead weight. Image: 5.62GB → 1.6GB. The failing push from off-LAN now succeeds. - `[2026-05-08]` Split `Worldtree/requirements.txt` into runtime + `requirements-dev.txt` (pytest, pytest-asyncio, ruff, pylint). ~50MB out of the runtime image; smaller attack surface. - `[2026-05-01]` nevermore **replaces** news-digest at port 8181 (not coexist on 8182). User chose one stack at a time over soak window. ~30s briefing gap accepted as the cost. - `[2026-04-30]` Bumped nevermore deploy playbook's first-render wait from 60s to 6 min — first deploy with empty `/output` requires a worker LLM cycle (2-4 min) before `index.html` exists. - `[2026-04-29]` Workflow drops explicit `container:` directive after runner re-registration with `:docker://node:20-bookworm-slim` schema labels — single source of truth for the build environment. - `[2026-04-29]` Deploy playbooks accept SHA refs in addition to branch names (`git rev-parse ` then `origin/^{commit}` fallback). CI passes `--var ref=${{ github.sha }}`; manual runs pass `main` / `v0.1.0`. Same code path either way. - `[2026-04-29]` Single central Gitea Actions runner on ana-docker (`f014d55`). Parameterized `playbooks/deploy-gitea-runner.yaml` so site-local runners (nh3, esh) drop in via `--var` overrides — not copy-pasted playbooks. - `[2026-04-29]` `nevermore` carries forward `hidden.json` and historical `MINIFLUX_PASSWORD` from `news-digest` during cutover (preserves user state across stack rename). ## Tried and abandoned - `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero matches and led to a wrong "no /v2/ traffic in 12 days" conclusion. **Gitea logs in PDT, Traefik logs in UTC** — same host, different timezones. Always normalize timezones (UTC) when correlating logs across services on the same box. Cost: ~30 min in the wrong direction. - `[2026-05-08]` Validating a user-proposed Traefik/Gitea timeout bump for "Traefik is dropping connections on big docker pushes" without first verifying which component was actually in the failure path. Traefik turned out to be innocent (`499 / 60012ms` = client closed, Traefik never timed out), Gitea's `PER_WRITE_TIMEOUT` governs response writes (wrong direction), and the real ceiling was a ~60s client-side timer no compose change can reach. **Lesson: validate the diagnostic premise — which component is actually in the failure path? — before refining the proposed fix.** - `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` / `PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on `/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern **response writes**, not request body reads. `unexpected EOF` from Go's HTTP server means the client closed mid-body-upload; not a knob Gitea exposes server-side. - `[2026-04-30]` task-board workflow with `container: image: debian:bookworm-slim` — fails: `actions/checkout@v4` needs `node` at runtime, slim image lacks it. Switched to `node:20-bookworm-slim` (has node + apt) or runner-label default. - `[2026-04-30]` Dropping the `container:` directive before runner re-registration with docker-schema labels — runner silently falls back to **host mode** (jobs run inside the alpine `act_runner` container itself, no apt). The `:host` suffix in startup logs (`labels updated to: [pfi-fleet:host ana-docker:host]`) is the giveaway. Fix: register with `pfi-fleet:docker://` schema labels. - `[2026-04-30]` Updating runner labels by editing `.env` and bouncing — doesn't take. The `.runner` registration cache pins labels at first registration; env-var updates are read each start but the stored token + UUID are tied to the original label set on the gitea side. Fix: stop runner, delete `.runner`, generate new admin registration token, redeploy. - `[2026-04-30]` `git reset --hard origin/` in `deploy-task-board.yaml` (and the in-repo nevermore playbook before fix) — invalid syntax: `origin/` prefix only works for branch refs. SHAs need `git reset --hard ` directly. Resolved with `git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first, then bare `"{{ ref }}^{commit}"` fallback. - `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after task-board wiring — it was actually only repo-scope on `vh/task-board`. vor's first CI run failed with empty SSH key (`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user scope at `gitea.phasefinal.com/user/settings/actions/secrets`. - `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity — leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1/"` still leaked the first 4 chars. Lesson: when probing secret-bearing files, use field-by-field SELECTIVE inspection (`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any password line, `grep -c` (existence) or `test -n "$(...)"` (non-empty), never `cat` or value-printing.