# Langfuse

LLM trace / observability for the fleet — the "pretty" layer over the LiteLLM
gateway: full prompt + completion, reasoning content, latency, token counts
(and tok/s), per-call trace explorer. Fronts LiteLLM via its native `langfuse`
success/failure callback.

**Server:** ana-docker (10.250.50.70) — 6 containers (langfuse-web, -worker,
postgres, clickhouse, redis, minio). v3, adapted from the official self-host
compose.
**UI:** http://10.250.50.70:3001  (host 3000 is taken by gitea → `LANGFUSE_PORT=3001`)
**Login:** `lkraven@lkraven.com` — password is in the server `.env`
(`LANGFUSE_INIT_USER_PASSWORD`); auto-provisioned on first boot.

## How it's wired
- A project (`gateway` / org `pfi`) + its API keys are auto-provisioned via the
  `LANGFUSE_INIT_*` vars in the server `.env`.
- Those keys (`pk-lf-…` / `sk-lf-…`) are set in the **litellm** stack's `.env`
  as `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY`, with
  `LANGFUSE_HOST=http://10.250.50.70:3001`.
- LiteLLM ships traces because `litellm_settings.success_callback: ["langfuse"]`
  is enabled in `stacks/litellm/conf/config.yaml`. Every consumer of the gateway
  is traced automatically — no consumer change.

## Deploy / operate
```bash
scripts/deploy-stack.sh ana-docker langfuse        # push compose
# real secrets live ONLY in /opt/docker/compose/langfuse/.env on the box
# (generated with openssl rand; NEVER committed). See .env.example.
ssh ana-docker 'cd /opt/docker/compose/langfuse && docker compose up -d'
# health: curl http://10.250.50.70:3001/api/public/health  -> 200
```

## Notes / gotchas
- **HOSTNAME=0.0.0.0** is set on langfuse-web — the Next.js standalone server
  otherwise binds only the default-network IP and is unreachable via the
  published port once the container is also on `tnet`.
- The 3 `LANGFUSE_S3_*_SECRET_ACCESS_KEY` must equal `MINIO_ROOT_PASSWORD`
  (MinIO is the S3 backend; access-key-id stays `minio`).
- ClickHouse is the resource driver; ana-docker had ~77 GB of reclaimable docker
  cruft pruned (2026-06-05) to make room. Watch disk/RAM as trace volume grows.
- Internal services bind 127.0.0.1 only; just langfuse-web:3001 + minio:9090 are
  host-reachable on the internal network.