feat(langfuse): stand up Langfuse v3 + wire the LiteLLM trace callback

LLM observability for the fleet — pretty trace UI over the gateway: prompts, completions, reasoning, latency, token counts. The pretty layer LiteLLM's spend_logs lacked. - stacks/langfuse: v3 self-host stack (web/worker/postgres/clickhouse/redis/ minio) on ana-docker, adapted from upstream. UI on :3001 (gitea owns :3000). Project + API keys auto-provisioned via LANGFUSE_INIT_*. HOSTNAME=0.0.0.0 on langfuse-web so it's reachable via the published port while also on tnet. - litellm: enabled success_callback/failure_callback: ["langfuse"] (the passthrough env was already wired); keys + host go in the litellm .env. Verified: stack healthy, project keys authenticate, and a real gateway call landed a litellm-acompletion trace in Langfuse within ~6s. Secrets live only in the server .env (never committed).
2026-06-05 11:35:01 -07:00
parent 34a43a0bc5
commit 9171e6a20f
4 changed files with 293 additions and 6 deletions
@@ -0,0 +1,44 @@
+# Langfuse v3 — copy to `.env` on the server and fill REAL secrets, then
+# `docker compose up -d`. NEVER commit the real `.env`.
+#
+# Generate secrets (use hex to stay safe inside DATABASE_URL + .env parsing):
+#   openssl rand -hex 32   → ENCRYPTION_KEY (must be exactly 64 hex chars)
+#   openssl rand -hex 32   → NEXTAUTH_SECRET, SALT
+#   openssl rand -hex 16   → the passwords
+#   uuidgen                → the pk-lf-/sk-lf- INIT keys
+
+# --- Core ---
+# Host port for the web UI. 3000 is taken by gitea on ana-docker → use 3001.
+# NEXTAUTH_URL must match the URL you browse to (host:LANGFUSE_PORT).
+LANGFUSE_PORT=3001
+NEXTAUTH_URL=http://10.250.50.70:3001
+NEXTAUTH_SECRET=CHANGEME           # openssl rand -hex 32
+SALT=CHANGEME                      # openssl rand -hex 32
+ENCRYPTION_KEY=CHANGEME            # openssl rand -hex 32  (exactly 64 hex chars)
+
+# --- Postgres (password must match in DATABASE_URL) ---
+POSTGRES_PASSWORD=CHANGEME
+DATABASE_URL=postgresql://postgres:CHANGEME@postgres:5432/postgres
+
+# --- ClickHouse / Redis ---
+CLICKHOUSE_PASSWORD=CHANGEME
+REDIS_AUTH=CHANGEME
+
+# --- MinIO (S3 backend). The 3 S3 secret keys MUST equal MINIO_ROOT_PASSWORD;
+#     access-key-ids stay the compose default "minio" (= MINIO_ROOT_USER). ---
+MINIO_ROOT_PASSWORD=CHANGEME
+LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=CHANGEME
+LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY=CHANGEME
+LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY=CHANGEME
+
+# --- Auto-provision org/project/user + API keys on first boot ---
+# The PUBLIC/SECRET keys become LiteLLM's LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY.
+LANGFUSE_INIT_ORG_ID=pfi
+LANGFUSE_INIT_ORG_NAME=PFI
+LANGFUSE_INIT_PROJECT_ID=gateway
+LANGFUSE_INIT_PROJECT_NAME=LiteLLM Gateway
+LANGFUSE_INIT_PROJECT_PUBLIC_KEY=pk-lf-CHANGEME
+LANGFUSE_INIT_PROJECT_SECRET_KEY=sk-lf-CHANGEME
+LANGFUSE_INIT_USER_EMAIL=lkraven@lkraven.com
+LANGFUSE_INIT_USER_NAME=lkraven
+LANGFUSE_INIT_USER_PASSWORD=CHANGEME   # initial UI login password
@@ -0,0 +1,43 @@
+# Langfuse
+
+LLM trace / observability for the fleet — the "pretty" layer over the LiteLLM
+gateway: full prompt + completion, reasoning content, latency, token counts
+(and tok/s), per-call trace explorer. Fronts LiteLLM via its native `langfuse`
+success/failure callback.
+
+**Server:** ana-docker (10.250.50.70) — 6 containers (langfuse-web, -worker,
+postgres, clickhouse, redis, minio). v3, adapted from the official self-host
+compose.
+**UI:** http://10.250.50.70:3001  (host 3000 is taken by gitea → `LANGFUSE_PORT=3001`)
+**Login:** `lkraven@lkraven.com` — password is in the server `.env`
+(`LANGFUSE_INIT_USER_PASSWORD`); auto-provisioned on first boot.
+
+## How it's wired
+- A project (`gateway` / org `pfi`) + its API keys are auto-provisioned via the
+  `LANGFUSE_INIT_*` vars in the server `.env`.
+- Those keys (`pk-lf-…` / `sk-lf-…`) are set in the **litellm** stack's `.env`
+  as `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY`, with
+  `LANGFUSE_HOST=http://10.250.50.70:3001`.
+- LiteLLM ships traces because `litellm_settings.success_callback: ["langfuse"]`
+  is enabled in `stacks/litellm/conf/config.yaml`. Every consumer of the gateway
+  is traced automatically — no consumer change.
+
+## Deploy / operate
+```bash
+scripts/deploy-stack.sh ana-docker langfuse        # push compose
+# real secrets live ONLY in /opt/docker/compose/langfuse/.env on the box
+# (generated with openssl rand; NEVER committed). See .env.example.
+ssh ana-docker 'cd /opt/docker/compose/langfuse && docker compose up -d'
+# health: curl http://10.250.50.70:3001/api/public/health  -> 200
+```
+
+## Notes / gotchas
+- **HOSTNAME=0.0.0.0** is set on langfuse-web — the Next.js standalone server
+  otherwise binds only the default-network IP and is unreachable via the
+  published port once the container is also on `tnet`.
+- The 3 `LANGFUSE_S3_*_SECRET_ACCESS_KEY` must equal `MINIO_ROOT_PASSWORD`
+  (MinIO is the S3 backend; access-key-id stays `minio`).
+- ClickHouse is the resource driver; ana-docker had ~77 GB of reclaimable docker
+  cruft pruned (2026-06-05) to make room. Watch disk/RAM as trace volume grows.
+- Internal services bind 127.0.0.1 only; just langfuse-web:3001 + minio:9090 are
+  host-reachable on the internal network.
@@ -0,0 +1,200 @@
+# Langfuse v3 — LLM trace / observability (prompts, completions, reasoning, tok/s).
+# Fronts the LiteLLM gateway via its native `langfuse` success_callback.
+#
+# Adapted from the official self-host compose
+# (https://raw.githubusercontent.com/langfuse/langfuse/main/docker-compose.yml).
+# CHANGES from upstream:
+#   - langfuse-web joined to `traefik-net` (tnet) so the LiteLLM container can
+#     reach it by name (LANGFUSE_HOST=http://langfuse-web:3000) + homepage labels.
+#   - everything else verbatim; internal services stay on the default network and
+#     bind to 127.0.0.1 (only langfuse-web:3000 + minio:9090 are host-reachable).
+# All secrets live in `.env` on the server (NOT in .env.example). Generate with
+# `openssl rand -hex 32` (ENCRYPTION_KEY) / `openssl rand -base64 32` (others).
+services:
+  langfuse-worker:
+    image: docker.io/langfuse/langfuse-worker:3
+    restart: always
+    depends_on: &langfuse-depends-on
+      postgres:
+        condition: service_healthy
+      minio:
+        condition: service_healthy
+      redis:
+        condition: service_healthy
+      clickhouse:
+        condition: service_healthy
+    ports:
+      - 127.0.0.1:3030:3030
+    environment: &langfuse-worker-env
+      NEXTAUTH_URL: ${NEXTAUTH_URL:-http://localhost:3000}
+      DATABASE_URL: ${DATABASE_URL:-postgresql://postgres:postgres@postgres:5432/postgres} # CHANGEME
+      SALT: ${SALT:-mysalt} # CHANGEME
+      ENCRYPTION_KEY: ${ENCRYPTION_KEY:-0000000000000000000000000000000000000000000000000000000000000000} # CHANGEME: openssl rand -hex 32
+      TELEMETRY_ENABLED: ${TELEMETRY_ENABLED:-true}
+      LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES: ${LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES:-false}
+      CLICKHOUSE_MIGRATION_URL: ${CLICKHOUSE_MIGRATION_URL:-clickhouse://clickhouse:9000}
+      CLICKHOUSE_URL: ${CLICKHOUSE_URL:-http://clickhouse:8123}
+      CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse}
+      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse} # CHANGEME
+      CLICKHOUSE_CLUSTER_ENABLED: ${CLICKHOUSE_CLUSTER_ENABLED:-false}
+      LANGFUSE_USE_AZURE_BLOB: ${LANGFUSE_USE_AZURE_BLOB:-false}
+      LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE: ${LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE:-false}
+      LANGFUSE_S3_EVENT_UPLOAD_BUCKET: ${LANGFUSE_S3_EVENT_UPLOAD_BUCKET:-langfuse}
+      LANGFUSE_S3_EVENT_UPLOAD_REGION: ${LANGFUSE_S3_EVENT_UPLOAD_REGION:-auto}
+      LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID:-minio}
+      LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY:-miniosecret} # CHANGEME
+      LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: ${LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT:-http://minio:9000}
+      LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE: ${LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE:-true}
+      LANGFUSE_S3_EVENT_UPLOAD_PREFIX: ${LANGFUSE_S3_EVENT_UPLOAD_PREFIX:-events/}
+      LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: ${LANGFUSE_S3_MEDIA_UPLOAD_BUCKET:-langfuse}
+      LANGFUSE_S3_MEDIA_UPLOAD_REGION: ${LANGFUSE_S3_MEDIA_UPLOAD_REGION:-auto}
+      LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID:-minio}
+      LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY:-miniosecret} # CHANGEME
+      LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT: ${LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT:-http://minio:9000}
+      LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE: ${LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE:-true}
+      LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: ${LANGFUSE_S3_MEDIA_UPLOAD_PREFIX:-media/}
+      LANGFUSE_S3_BATCH_EXPORT_ENABLED: ${LANGFUSE_S3_BATCH_EXPORT_ENABLED:-false}
+      LANGFUSE_S3_BATCH_EXPORT_BUCKET: ${LANGFUSE_S3_BATCH_EXPORT_BUCKET:-langfuse}
+      LANGFUSE_S3_BATCH_EXPORT_PREFIX: ${LANGFUSE_S3_BATCH_EXPORT_PREFIX:-exports/}
+      LANGFUSE_S3_BATCH_EXPORT_REGION: ${LANGFUSE_S3_BATCH_EXPORT_REGION:-auto}
+      LANGFUSE_S3_BATCH_EXPORT_ENDPOINT: ${LANGFUSE_S3_BATCH_EXPORT_ENDPOINT:-http://minio:9000}
+      LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT: ${LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT:-http://localhost:9090}
+      LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID: ${LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID:-minio}
+      LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY: ${LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY:-miniosecret} # CHANGEME
+      LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE: ${LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE:-true}
+      LANGFUSE_INGESTION_QUEUE_DELAY_MS: ${LANGFUSE_INGESTION_QUEUE_DELAY_MS:-}
+      LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS: ${LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS:-}
+      REDIS_HOST: ${REDIS_HOST:-redis}
+      REDIS_PORT: ${REDIS_PORT:-6379}
+      REDIS_AUTH: ${REDIS_AUTH:-myredissecret} # CHANGEME
+      REDIS_TLS_ENABLED: ${REDIS_TLS_ENABLED:-false}
+      EMAIL_FROM_ADDRESS: ${EMAIL_FROM_ADDRESS:-}
+      SMTP_CONNECTION_URL: ${SMTP_CONNECTION_URL:-}
+
+  langfuse-web:
+    image: docker.io/langfuse/langfuse:3
+    restart: always
+    depends_on: *langfuse-depends-on
+    ports:
+      # host 3000 is taken by gitea on ana-docker → LANGFUSE_PORT=3001 in .env.
+      # Internal port stays 3000 (LiteLLM reaches http://langfuse-web:3000 via tnet).
+      - ${LANGFUSE_PORT:-3000}:3000
+    networks:
+      - default
+      - tnet
+    environment:
+      <<: *langfuse-worker-env
+      # Bind all interfaces — the Next.js standalone server otherwise binds only
+      # the container's default-network IP, unreachable via the published port
+      # once langfuse-web is also on tnet (two IPs). 0.0.0.0 fixes both.
+      HOSTNAME: "0.0.0.0"
+      NEXTAUTH_SECRET: ${NEXTAUTH_SECRET:-mysecret} # CHANGEME
+      LANGFUSE_INIT_ORG_ID: ${LANGFUSE_INIT_ORG_ID:-}
+      LANGFUSE_INIT_ORG_NAME: ${LANGFUSE_INIT_ORG_NAME:-}
+      LANGFUSE_INIT_PROJECT_ID: ${LANGFUSE_INIT_PROJECT_ID:-}
+      LANGFUSE_INIT_PROJECT_NAME: ${LANGFUSE_INIT_PROJECT_NAME:-}
+      LANGFUSE_INIT_PROJECT_PUBLIC_KEY: ${LANGFUSE_INIT_PROJECT_PUBLIC_KEY:-}
+      LANGFUSE_INIT_PROJECT_SECRET_KEY: ${LANGFUSE_INIT_PROJECT_SECRET_KEY:-}
+      LANGFUSE_INIT_USER_EMAIL: ${LANGFUSE_INIT_USER_EMAIL:-}
+      LANGFUSE_INIT_USER_NAME: ${LANGFUSE_INIT_USER_NAME:-}
+      LANGFUSE_INIT_USER_PASSWORD: ${LANGFUSE_INIT_USER_PASSWORD:-}
+    labels:
+      - homepage.group=AI Systems
+      - homepage.name=Langfuse
+      - homepage.icon=mdi-chart-timeline-variant
+      - homepage.description=LLM trace + observability (prompts, reasoning, tok/s)
+      - homepage.href=http://10.250.50.70:${LANGFUSE_PORT:-3000}
+
+  clickhouse:
+    image: docker.io/clickhouse/clickhouse-server
+    restart: always
+    user: "101:101"
+    environment:
+      CLICKHOUSE_DB: default
+      CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse}
+      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse} # CHANGEME
+    volumes:
+      - langfuse_clickhouse_data:/var/lib/clickhouse
+      - langfuse_clickhouse_logs:/var/log/clickhouse-server
+    ports:
+      - 127.0.0.1:8123:8123
+      - 127.0.0.1:9000:9000
+    healthcheck:
+      test: wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1
+      interval: 5s
+      timeout: 5s
+      retries: 10
+      start_period: 1s
+
+  minio:
+    image: cgr.dev/chainguard/minio
+    restart: always
+    entrypoint: sh
+    command: -c 'mkdir -p /data/langfuse && minio server --address ":9000" --console-address ":9001" /data'
+    environment:
+      MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minio}
+      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-miniosecret} # CHANGEME
+    ports:
+      - 127.0.0.1:9090:9000
+      - 127.0.0.1:9091:9001
+    volumes:
+      - langfuse_minio_data:/data
+    healthcheck:
+      test: ["CMD", "mc", "ready", "local"]
+      interval: 1s
+      timeout: 5s
+      retries: 5
+      start_period: 1s
+
+  redis:
+    image: docker.io/redis:7
+    restart: always
+    command: >
+      --requirepass ${REDIS_AUTH:-myredissecret}
+      --maxmemory-policy noeviction
+    ports:
+      - 127.0.0.1:6379:6379
+    volumes:
+      - langfuse_redis_data:/data
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 3s
+      timeout: 10s
+      retries: 10
+
+  postgres:
+    image: docker.io/postgres:${POSTGRES_VERSION:-17}
+    restart: always
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U postgres"]
+      interval: 3s
+      timeout: 3s
+      retries: 10
+    environment:
+      POSTGRES_USER: ${POSTGRES_USER:-postgres}
+      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-postgres} # CHANGEME
+      POSTGRES_DB: ${POSTGRES_DB:-postgres}
+      TZ: UTC
+      PGTZ: UTC
+    ports:
+      - 127.0.0.1:5432:5432
+    volumes:
+      - langfuse_postgres_data:/var/lib/postgresql/data
+
+networks:
+  default:
+  tnet:
+    name: traefik-net
+    external: true
+
+volumes:
+  langfuse_postgres_data:
+    driver: local
+  langfuse_clickhouse_data:
+    driver: local
+  langfuse_clickhouse_logs:
+    driver: local
+  langfuse_minio_data:
+    driver: local
+  langfuse_redis_data:
+    driver: local
@@ -104,9 +104,9 @@ litellm_settings:
  # vLLM rejects some OpenAI params other backends accept; drop silently
  # rather than 400 the caller.
  drop_params: true
-  # --- Langfuse-ready: uncomment to ship full traces to a Langfuse
-  #     instance (set LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY /
-  #     LANGFUSE_HOST in .env). This is the entire upgrade — the gateway
-  #     and every consumer stay pointed here. ---
-  # success_callback: ["langfuse"]
-  # failure_callback: ["langfuse"]
+  # --- Langfuse trace export (live 2026-06-05). Full prompt/completion +
+  #     reasoning + tok-derivable latency traces ship to the Langfuse stack on
+  #     ana-docker (project "gateway"). Keys + host in .env. The gateway and
+  #     every consumer stay pointed here — this callback is the whole upgrade. ---
+  success_callback: ["langfuse"]
+  failure_callback: ["langfuse"]