feat(langfuse): stand up Langfuse v3 + wire the LiteLLM trace callback

LLM observability for the fleet — pretty trace UI over the gateway: prompts, completions, reasoning, latency, token counts. The pretty layer LiteLLM's spend_logs lacked. - stacks/langfuse: v3 self-host stack (web/worker/postgres/clickhouse/redis/ minio) on ana-docker, adapted from upstream. UI on :3001 (gitea owns :3000). Project + API keys auto-provisioned via LANGFUSE_INIT_*. HOSTNAME=0.0.0.0 on langfuse-web so it's reachable via the published port while also on tnet. - litellm: enabled success_callback/failure_callback: ["langfuse"] (the passthrough env was already wired); keys + host go in the litellm .env. Verified: stack healthy, project keys authenticate, and a real gateway call landed a litellm-acompletion trace in Langfuse within ~6s. Secrets live only in the server .env (never committed).
2026-06-05 11:35:01 -07:00
parent 34a43a0bc5
commit 9171e6a20f
4 changed files with 293 additions and 6 deletions
@@ -0,0 +1,43 @@
+# Langfuse
+
+LLM trace / observability for the fleet — the "pretty" layer over the LiteLLM
+gateway: full prompt + completion, reasoning content, latency, token counts
+(and tok/s), per-call trace explorer. Fronts LiteLLM via its native `langfuse`
+success/failure callback.
+
+**Server:** ana-docker (10.250.50.70) — 6 containers (langfuse-web, -worker,
+postgres, clickhouse, redis, minio). v3, adapted from the official self-host
+compose.
+**UI:** http://10.250.50.70:3001  (host 3000 is taken by gitea → `LANGFUSE_PORT=3001`)
+**Login:** `lkraven@lkraven.com` — password is in the server `.env`
+(`LANGFUSE_INIT_USER_PASSWORD`); auto-provisioned on first boot.
+
+## How it's wired
+- A project (`gateway` / org `pfi`) + its API keys are auto-provisioned via the
+  `LANGFUSE_INIT_*` vars in the server `.env`.
+- Those keys (`pk-lf-…` / `sk-lf-…`) are set in the **litellm** stack's `.env`
+  as `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY`, with
+  `LANGFUSE_HOST=http://10.250.50.70:3001`.
+- LiteLLM ships traces because `litellm_settings.success_callback: ["langfuse"]`
+  is enabled in `stacks/litellm/conf/config.yaml`. Every consumer of the gateway
+  is traced automatically — no consumer change.
+
+## Deploy / operate
+```bash
+scripts/deploy-stack.sh ana-docker langfuse        # push compose
+# real secrets live ONLY in /opt/docker/compose/langfuse/.env on the box
+# (generated with openssl rand; NEVER committed). See .env.example.
+ssh ana-docker 'cd /opt/docker/compose/langfuse && docker compose up -d'
+# health: curl http://10.250.50.70:3001/api/public/health  -> 200
+```
+
+## Notes / gotchas
+- **HOSTNAME=0.0.0.0** is set on langfuse-web — the Next.js standalone server
+  otherwise binds only the default-network IP and is unreachable via the
+  published port once the container is also on `tnet`.
+- The 3 `LANGFUSE_S3_*_SECRET_ACCESS_KEY` must equal `MINIO_ROOT_PASSWORD`
+  (MinIO is the S3 backend; access-key-id stays `minio`).
+- ClickHouse is the resource driver; ana-docker had ~77 GB of reclaimable docker
+  cruft pruned (2026-06-05) to make room. Watch disk/RAM as trace volume grows.
+- Internal services bind 127.0.0.1 only; just langfuse-web:3001 + minio:9090 are
+  host-reachable on the internal network.