From 9171e6a20f3f77a659f424984219bb6fcdf999dc Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 5 Jun 2026 11:35:01 -0700 Subject: [PATCH] feat(langfuse): stand up Langfuse v3 + wire the LiteLLM trace callback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LLM observability for the fleet — pretty trace UI over the gateway: prompts, completions, reasoning, latency, token counts. The pretty layer LiteLLM's spend_logs lacked. - stacks/langfuse: v3 self-host stack (web/worker/postgres/clickhouse/redis/ minio) on ana-docker, adapted from upstream. UI on :3001 (gitea owns :3000). Project + API keys auto-provisioned via LANGFUSE_INIT_*. HOSTNAME=0.0.0.0 on langfuse-web so it's reachable via the published port while also on tnet. - litellm: enabled success_callback/failure_callback: ["langfuse"] (the passthrough env was already wired); keys + host go in the litellm .env. Verified: stack healthy, project keys authenticate, and a real gateway call landed a litellm-acompletion trace in Langfuse within ~6s. Secrets live only in the server .env (never committed). --- stacks/langfuse/.env.example | 44 +++++++ stacks/langfuse/README.md | 43 +++++++ stacks/langfuse/compose.yaml | 200 ++++++++++++++++++++++++++++++++ stacks/litellm/conf/config.yaml | 12 +- 4 files changed, 293 insertions(+), 6 deletions(-) create mode 100644 stacks/langfuse/.env.example create mode 100644 stacks/langfuse/README.md create mode 100644 stacks/langfuse/compose.yaml diff --git a/stacks/langfuse/.env.example b/stacks/langfuse/.env.example new file mode 100644 index 0000000..383b3d2 --- /dev/null +++ b/stacks/langfuse/.env.example @@ -0,0 +1,44 @@ +# Langfuse v3 — copy to `.env` on the server and fill REAL secrets, then +# `docker compose up -d`. NEVER commit the real `.env`. +# +# Generate secrets (use hex to stay safe inside DATABASE_URL + .env parsing): +# openssl rand -hex 32 → ENCRYPTION_KEY (must be exactly 64 hex chars) +# openssl rand -hex 32 → NEXTAUTH_SECRET, SALT +# openssl rand -hex 16 → the passwords +# uuidgen → the pk-lf-/sk-lf- INIT keys + +# --- Core --- +# Host port for the web UI. 3000 is taken by gitea on ana-docker → use 3001. +# NEXTAUTH_URL must match the URL you browse to (host:LANGFUSE_PORT). +LANGFUSE_PORT=3001 +NEXTAUTH_URL=http://10.250.50.70:3001 +NEXTAUTH_SECRET=CHANGEME # openssl rand -hex 32 +SALT=CHANGEME # openssl rand -hex 32 +ENCRYPTION_KEY=CHANGEME # openssl rand -hex 32 (exactly 64 hex chars) + +# --- Postgres (password must match in DATABASE_URL) --- +POSTGRES_PASSWORD=CHANGEME +DATABASE_URL=postgresql://postgres:CHANGEME@postgres:5432/postgres + +# --- ClickHouse / Redis --- +CLICKHOUSE_PASSWORD=CHANGEME +REDIS_AUTH=CHANGEME + +# --- MinIO (S3 backend). The 3 S3 secret keys MUST equal MINIO_ROOT_PASSWORD; +# access-key-ids stay the compose default "minio" (= MINIO_ROOT_USER). --- +MINIO_ROOT_PASSWORD=CHANGEME +LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=CHANGEME +LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY=CHANGEME +LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY=CHANGEME + +# --- Auto-provision org/project/user + API keys on first boot --- +# The PUBLIC/SECRET keys become LiteLLM's LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY. +LANGFUSE_INIT_ORG_ID=pfi +LANGFUSE_INIT_ORG_NAME=PFI +LANGFUSE_INIT_PROJECT_ID=gateway +LANGFUSE_INIT_PROJECT_NAME=LiteLLM Gateway +LANGFUSE_INIT_PROJECT_PUBLIC_KEY=pk-lf-CHANGEME +LANGFUSE_INIT_PROJECT_SECRET_KEY=sk-lf-CHANGEME +LANGFUSE_INIT_USER_EMAIL=lkraven@lkraven.com +LANGFUSE_INIT_USER_NAME=lkraven +LANGFUSE_INIT_USER_PASSWORD=CHANGEME # initial UI login password diff --git a/stacks/langfuse/README.md b/stacks/langfuse/README.md new file mode 100644 index 0000000..8918c44 --- /dev/null +++ b/stacks/langfuse/README.md @@ -0,0 +1,43 @@ +# Langfuse + +LLM trace / observability for the fleet — the "pretty" layer over the LiteLLM +gateway: full prompt + completion, reasoning content, latency, token counts +(and tok/s), per-call trace explorer. Fronts LiteLLM via its native `langfuse` +success/failure callback. + +**Server:** ana-docker (10.250.50.70) — 6 containers (langfuse-web, -worker, +postgres, clickhouse, redis, minio). v3, adapted from the official self-host +compose. +**UI:** http://10.250.50.70:3001 (host 3000 is taken by gitea → `LANGFUSE_PORT=3001`) +**Login:** `lkraven@lkraven.com` — password is in the server `.env` +(`LANGFUSE_INIT_USER_PASSWORD`); auto-provisioned on first boot. + +## How it's wired +- A project (`gateway` / org `pfi`) + its API keys are auto-provisioned via the + `LANGFUSE_INIT_*` vars in the server `.env`. +- Those keys (`pk-lf-…` / `sk-lf-…`) are set in the **litellm** stack's `.env` + as `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY`, with + `LANGFUSE_HOST=http://10.250.50.70:3001`. +- LiteLLM ships traces because `litellm_settings.success_callback: ["langfuse"]` + is enabled in `stacks/litellm/conf/config.yaml`. Every consumer of the gateway + is traced automatically — no consumer change. + +## Deploy / operate +```bash +scripts/deploy-stack.sh ana-docker langfuse # push compose +# real secrets live ONLY in /opt/docker/compose/langfuse/.env on the box +# (generated with openssl rand; NEVER committed). See .env.example. +ssh ana-docker 'cd /opt/docker/compose/langfuse && docker compose up -d' +# health: curl http://10.250.50.70:3001/api/public/health -> 200 +``` + +## Notes / gotchas +- **HOSTNAME=0.0.0.0** is set on langfuse-web — the Next.js standalone server + otherwise binds only the default-network IP and is unreachable via the + published port once the container is also on `tnet`. +- The 3 `LANGFUSE_S3_*_SECRET_ACCESS_KEY` must equal `MINIO_ROOT_PASSWORD` + (MinIO is the S3 backend; access-key-id stays `minio`). +- ClickHouse is the resource driver; ana-docker had ~77 GB of reclaimable docker + cruft pruned (2026-06-05) to make room. Watch disk/RAM as trace volume grows. +- Internal services bind 127.0.0.1 only; just langfuse-web:3001 + minio:9090 are + host-reachable on the internal network. diff --git a/stacks/langfuse/compose.yaml b/stacks/langfuse/compose.yaml new file mode 100644 index 0000000..a115658 --- /dev/null +++ b/stacks/langfuse/compose.yaml @@ -0,0 +1,200 @@ +# Langfuse v3 — LLM trace / observability (prompts, completions, reasoning, tok/s). +# Fronts the LiteLLM gateway via its native `langfuse` success_callback. +# +# Adapted from the official self-host compose +# (https://raw.githubusercontent.com/langfuse/langfuse/main/docker-compose.yml). +# CHANGES from upstream: +# - langfuse-web joined to `traefik-net` (tnet) so the LiteLLM container can +# reach it by name (LANGFUSE_HOST=http://langfuse-web:3000) + homepage labels. +# - everything else verbatim; internal services stay on the default network and +# bind to 127.0.0.1 (only langfuse-web:3000 + minio:9090 are host-reachable). +# All secrets live in `.env` on the server (NOT in .env.example). Generate with +# `openssl rand -hex 32` (ENCRYPTION_KEY) / `openssl rand -base64 32` (others). +services: + langfuse-worker: + image: docker.io/langfuse/langfuse-worker:3 + restart: always + depends_on: &langfuse-depends-on + postgres: + condition: service_healthy + minio: + condition: service_healthy + redis: + condition: service_healthy + clickhouse: + condition: service_healthy + ports: + - 127.0.0.1:3030:3030 + environment: &langfuse-worker-env + NEXTAUTH_URL: ${NEXTAUTH_URL:-http://localhost:3000} + DATABASE_URL: ${DATABASE_URL:-postgresql://postgres:postgres@postgres:5432/postgres} # CHANGEME + SALT: ${SALT:-mysalt} # CHANGEME + ENCRYPTION_KEY: ${ENCRYPTION_KEY:-0000000000000000000000000000000000000000000000000000000000000000} # CHANGEME: openssl rand -hex 32 + TELEMETRY_ENABLED: ${TELEMETRY_ENABLED:-true} + LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES: ${LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES:-false} + CLICKHOUSE_MIGRATION_URL: ${CLICKHOUSE_MIGRATION_URL:-clickhouse://clickhouse:9000} + CLICKHOUSE_URL: ${CLICKHOUSE_URL:-http://clickhouse:8123} + CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse} + CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse} # CHANGEME + CLICKHOUSE_CLUSTER_ENABLED: ${CLICKHOUSE_CLUSTER_ENABLED:-false} + LANGFUSE_USE_AZURE_BLOB: ${LANGFUSE_USE_AZURE_BLOB:-false} + LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE: ${LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE:-false} + LANGFUSE_S3_EVENT_UPLOAD_BUCKET: ${LANGFUSE_S3_EVENT_UPLOAD_BUCKET:-langfuse} + LANGFUSE_S3_EVENT_UPLOAD_REGION: ${LANGFUSE_S3_EVENT_UPLOAD_REGION:-auto} + LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID:-minio} + LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY:-miniosecret} # CHANGEME + LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: ${LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT:-http://minio:9000} + LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE: ${LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE:-true} + LANGFUSE_S3_EVENT_UPLOAD_PREFIX: ${LANGFUSE_S3_EVENT_UPLOAD_PREFIX:-events/} + LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: ${LANGFUSE_S3_MEDIA_UPLOAD_BUCKET:-langfuse} + LANGFUSE_S3_MEDIA_UPLOAD_REGION: ${LANGFUSE_S3_MEDIA_UPLOAD_REGION:-auto} + LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID:-minio} + LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY:-miniosecret} # CHANGEME + LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT: ${LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT:-http://minio:9000} + LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE: ${LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE:-true} + LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: ${LANGFUSE_S3_MEDIA_UPLOAD_PREFIX:-media/} + LANGFUSE_S3_BATCH_EXPORT_ENABLED: ${LANGFUSE_S3_BATCH_EXPORT_ENABLED:-false} + LANGFUSE_S3_BATCH_EXPORT_BUCKET: ${LANGFUSE_S3_BATCH_EXPORT_BUCKET:-langfuse} + LANGFUSE_S3_BATCH_EXPORT_PREFIX: ${LANGFUSE_S3_BATCH_EXPORT_PREFIX:-exports/} + LANGFUSE_S3_BATCH_EXPORT_REGION: ${LANGFUSE_S3_BATCH_EXPORT_REGION:-auto} + LANGFUSE_S3_BATCH_EXPORT_ENDPOINT: ${LANGFUSE_S3_BATCH_EXPORT_ENDPOINT:-http://minio:9000} + LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT: ${LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT:-http://localhost:9090} + LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID: ${LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID:-minio} + LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY: ${LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY:-miniosecret} # CHANGEME + LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE: ${LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE:-true} + LANGFUSE_INGESTION_QUEUE_DELAY_MS: ${LANGFUSE_INGESTION_QUEUE_DELAY_MS:-} + LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS: ${LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS:-} + REDIS_HOST: ${REDIS_HOST:-redis} + REDIS_PORT: ${REDIS_PORT:-6379} + REDIS_AUTH: ${REDIS_AUTH:-myredissecret} # CHANGEME + REDIS_TLS_ENABLED: ${REDIS_TLS_ENABLED:-false} + EMAIL_FROM_ADDRESS: ${EMAIL_FROM_ADDRESS:-} + SMTP_CONNECTION_URL: ${SMTP_CONNECTION_URL:-} + + langfuse-web: + image: docker.io/langfuse/langfuse:3 + restart: always + depends_on: *langfuse-depends-on + ports: + # host 3000 is taken by gitea on ana-docker → LANGFUSE_PORT=3001 in .env. + # Internal port stays 3000 (LiteLLM reaches http://langfuse-web:3000 via tnet). + - ${LANGFUSE_PORT:-3000}:3000 + networks: + - default + - tnet + environment: + <<: *langfuse-worker-env + # Bind all interfaces — the Next.js standalone server otherwise binds only + # the container's default-network IP, unreachable via the published port + # once langfuse-web is also on tnet (two IPs). 0.0.0.0 fixes both. + HOSTNAME: "0.0.0.0" + NEXTAUTH_SECRET: ${NEXTAUTH_SECRET:-mysecret} # CHANGEME + LANGFUSE_INIT_ORG_ID: ${LANGFUSE_INIT_ORG_ID:-} + LANGFUSE_INIT_ORG_NAME: ${LANGFUSE_INIT_ORG_NAME:-} + LANGFUSE_INIT_PROJECT_ID: ${LANGFUSE_INIT_PROJECT_ID:-} + LANGFUSE_INIT_PROJECT_NAME: ${LANGFUSE_INIT_PROJECT_NAME:-} + LANGFUSE_INIT_PROJECT_PUBLIC_KEY: ${LANGFUSE_INIT_PROJECT_PUBLIC_KEY:-} + LANGFUSE_INIT_PROJECT_SECRET_KEY: ${LANGFUSE_INIT_PROJECT_SECRET_KEY:-} + LANGFUSE_INIT_USER_EMAIL: ${LANGFUSE_INIT_USER_EMAIL:-} + LANGFUSE_INIT_USER_NAME: ${LANGFUSE_INIT_USER_NAME:-} + LANGFUSE_INIT_USER_PASSWORD: ${LANGFUSE_INIT_USER_PASSWORD:-} + labels: + - homepage.group=AI Systems + - homepage.name=Langfuse + - homepage.icon=mdi-chart-timeline-variant + - homepage.description=LLM trace + observability (prompts, reasoning, tok/s) + - homepage.href=http://10.250.50.70:${LANGFUSE_PORT:-3000} + + clickhouse: + image: docker.io/clickhouse/clickhouse-server + restart: always + user: "101:101" + environment: + CLICKHOUSE_DB: default + CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse} + CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse} # CHANGEME + volumes: + - langfuse_clickhouse_data:/var/lib/clickhouse + - langfuse_clickhouse_logs:/var/log/clickhouse-server + ports: + - 127.0.0.1:8123:8123 + - 127.0.0.1:9000:9000 + healthcheck: + test: wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1 + interval: 5s + timeout: 5s + retries: 10 + start_period: 1s + + minio: + image: cgr.dev/chainguard/minio + restart: always + entrypoint: sh + command: -c 'mkdir -p /data/langfuse && minio server --address ":9000" --console-address ":9001" /data' + environment: + MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minio} + MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-miniosecret} # CHANGEME + ports: + - 127.0.0.1:9090:9000 + - 127.0.0.1:9091:9001 + volumes: + - langfuse_minio_data:/data + healthcheck: + test: ["CMD", "mc", "ready", "local"] + interval: 1s + timeout: 5s + retries: 5 + start_period: 1s + + redis: + image: docker.io/redis:7 + restart: always + command: > + --requirepass ${REDIS_AUTH:-myredissecret} + --maxmemory-policy noeviction + ports: + - 127.0.0.1:6379:6379 + volumes: + - langfuse_redis_data:/data + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 3s + timeout: 10s + retries: 10 + + postgres: + image: docker.io/postgres:${POSTGRES_VERSION:-17} + restart: always + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres"] + interval: 3s + timeout: 3s + retries: 10 + environment: + POSTGRES_USER: ${POSTGRES_USER:-postgres} + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-postgres} # CHANGEME + POSTGRES_DB: ${POSTGRES_DB:-postgres} + TZ: UTC + PGTZ: UTC + ports: + - 127.0.0.1:5432:5432 + volumes: + - langfuse_postgres_data:/var/lib/postgresql/data + +networks: + default: + tnet: + name: traefik-net + external: true + +volumes: + langfuse_postgres_data: + driver: local + langfuse_clickhouse_data: + driver: local + langfuse_clickhouse_logs: + driver: local + langfuse_minio_data: + driver: local + langfuse_redis_data: + driver: local diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index dff73b9..9270706 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -104,9 +104,9 @@ litellm_settings: # vLLM rejects some OpenAI params other backends accept; drop silently # rather than 400 the caller. drop_params: true - # --- Langfuse-ready: uncomment to ship full traces to a Langfuse - # instance (set LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY / - # LANGFUSE_HOST in .env). This is the entire upgrade — the gateway - # and every consumer stay pointed here. --- - # success_callback: ["langfuse"] - # failure_callback: ["langfuse"] + # --- Langfuse trace export (live 2026-06-05). Full prompt/completion + + # reasoning + tok-derivable latency traces ship to the Langfuse stack on + # ana-docker (project "gateway"). Keys + host in .env. The gateway and + # every consumer stay pointed here — this callback is the whole upgrade. --- + success_callback: ["langfuse"] + failure_callback: ["langfuse"]