Observability

SurfSense uses OpenTelemetry for backend traces and metrics. Application logs include trace and span IDs so you can correlate logs with traces, but logs stay on the normal container stderr path.

Enable Locally

The development compose file reads backend settings from surfsense_backend/.env. Add these values there:

SURFSENSE_ENABLE_OTEL=true
SURFSENSE_ENV=dev
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-lgtm:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_RESOURCE_ATTRIBUTES=service.namespace=surfsense
OTEL_METRIC_EXPORT_INTERVAL=300000

Then start the development stack with the local LGTM backend:

docker compose -f docker/docker-compose.dev.yml up --build

Grafana is exposed on http://localhost:3001 by default.

Enable in Production Docker Compose

Production Docker Compose reads backend and collector settings from docker/.env. The API and Celery worker export telemetry to the bundled collector at otel-collector:4317; the collector is the only service that uses the Grafana Cloud credentials.

Add these values to docker/.env:

SURFSENSE_ENV=production
SURFSENSE_ENABLE_OTEL=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_RESOURCE_ATTRIBUTES=service.namespace=surfsense
OTEL_METRIC_EXPORT_INTERVAL=300000

GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-<region>.grafana.net/otlp
GRAFANA_CLOUD_INSTANCE_ID=<stack instance id>
GRAFANA_CLOUD_API_KEY=<cloud access policy token>

Then start the stack:

docker compose -f docker/docker-compose.yml --profile observability up -d

The collector receives OTLP on otel-collector:4317, scrubs sensitive span attributes, applies the configured tail-sampling policy, batches exports, retries failures, and forwards traces and metrics to Grafana Cloud over OTLP HTTP.

When deploying surfsense_backend/Dockerfile directly instead of production compose, use the same split: SurfSense containers export to a collector, and the collector owns the Grafana Cloud credentials.

Automatic Traces

When OpenTelemetry is enabled, the backend instruments:

• FastAPI inbound requests.
• SQLAlchemy queries from the main async engine and Celery task engine.
• Raw psycopg calls used by the LangGraph checkpointer.
• Redis commands.
• HTTPX outbound requests.
• Celery producer and worker execution.

Manual Spans

SurfSense keeps project-specific spans behind app.observability.otel:

• model.call
• tool.call
• chat.request
• kb.search
• kb.persist
• connector.sync
• subagent.invoke
• etl.extract
• etl.parse
• etl.ocr
• etl.picture.describe
• etl.picture.ocr
• compaction.run
• permission.asked
• interrupt.raised

Keep span names and attributes low-cardinality. Do not attach user content, prompts, document titles, file paths, user-specific URLs, secrets, or raw queries as span attributes.

Metrics

The OpenTelemetry instrumentors provide HTTP, HTTPX, and Celery runtime metrics. SurfSense adds these project metrics from app.observability.metrics:

• surfsense.model.call.duration
• gen_ai.client.token.usage
• surfsense.tool.call.duration
• surfsense.tool.call.errors
• surfsense.chat.request.duration
• surfsense.chat.request.outcome
• surfsense.kb.search.duration
• surfsense.compaction.runs
• surfsense.permission.asks
• surfsense.interrupt.raised
• surfsense.indexing.document.duration
• surfsense.indexing.document.outcome
• surfsense.connector.sync.duration
• surfsense.connector.sync.outcome
• surfsense.subagent.invoke.duration
• surfsense.subagent.invoke.outcome
• surfsense.etl.extract.duration
• surfsense.etl.extract.outcome
• surfsense.celery.heartbeat.refreshes
• surfsense.celery.heartbeat.failures
• surfsense.celery.queue.latency
• surfsense.auth.failures
• surfsense.rate_limit.rejections
• surfsense.perf.elapsed_ms

Runtime gauges include process RSS, CPU utilization, threads, open file descriptors, asyncio tasks, and CPython GC counters.

Logs

LoggingInstrumentor().instrument() injects otelTraceID and otelSpanID into standard Python LogRecords. The root log format writes them as trace_id=... span_id=....

SurfSense intentionally does not create an OpenTelemetry LoggerProvider, LoggingHandler, or OTLPLogExporter. Container stderr remains the log transport.

Verification

1. Hit a FastAPI endpoint and confirm an inbound server span appears in Grafana.
2. Run a chat request and confirm model.call and tool.call child spans.
3. Run a knowledge-base search and confirm kb.search spans and SQL child spans.
4. Run connector indexing and confirm Celery producer/worker spans share a trace ID and connector sync metrics increment.
5. Confirm gen_ai.client.token.usage, model/tool durations, request duration, Celery runtime, and runtime gauges appear within one export interval.
6. Confirm logs emitted inside a traced request show non-zero trace and span IDs.

Product Analytics (PostHog)

Separate from OpenTelemetry, the backend can emit server-side product events to PostHog. This is the authoritative source for outcome events (chats, document ingestion, connector indexing, billing, automations) because it captures traffic the browser never sees — MCP clients, personal-access-token scripts, and Celery background jobs. It is fully opt-in and mirrors the OTel contract: with POSTHOG_API_KEY unset, every capture is a silent no-op.

Use the same project key as the frontend's NEXT_PUBLIC_POSTHOG_KEY so server events merge onto the same PostHog persons the web app identifies by user id. Add these to surfsense_backend/.env (local) or docker/.env (production); they reach the API, Celery worker, and beat services via env_file, so no compose changes are needed:

POSTHOG_API_KEY=phc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
POSTHOG_HOST=https://us.i.posthog.com
POSTHOG_AI_PRIVACY_MODE=true

POSTHOG_AI_PRIVACY_MODE defaults to true; set it to false only if you want LLM prompt and completion bodies shipped to PostHog's AI observability views.

Every backend event is stamped source=backend (so it is distinguishable from frontend captures), carries auth_method / client for surface attribution, and sends disable_geoip=true so the server IP never overwrites a person's real location. LangGraph chat turns additionally emit $ai_generation / $ai_span traces via the PostHog LangChain callback handler, keyed by turn and chat id.

Keep event properties low-cardinality. Never attach user content — workspace names, connector titles, document titles, prompts, or raw queries — as event properties; they carry no aggregation value and are a privacy risk.

Out Of Scope

• Frontend/browser OpenTelemetry.
• OpenTelemetry log export.
• Profiling.
• Production backend selection.
• Tail-sampling collector configuration.
• Replacing LangSmith.
• Vendor SDKs.