LLM Monitoring & Observability 2026

Monitoring your LLM applications is standard practice in 2026. This guide compares the top tools.

Quick Comparison

Langfuse vs LangSmith

Langfuse

Pricing:

• Open Source: Free (self-hosted)
• Cloud Hobby: Free tier (50k units/mo)
• Cloud Pro: $29/Mo
• Cloud Team: $100+/Mo

Strengths:

• Open source (data ownership)
• Framework-agnostic (works with any framework)
• No per-seat charges
• Transparent pricing

Best For:

• EU-based teams (GDPR)
• Privacy-critical applications
• Non-LangChain frameworks

LangSmith

Pricing:

• Free: 1 seat, 5k traces/month
• Plus: $39/User/month
• Trace Overages: $0.50-5 per 1k traces

Strengths:

• Deepest LangChain integration
• Managed service (zero DevOps)
• Great for LangChain/LangGraph projects

Weaknesses:

• Per-user pricing gets expensive
• Not ideal if not using LangChain
• Proprietary (no data ownership)

Unit-Based vs Per-User Pricing Impact

Team Growing 2 → 10 people:

LANGFUSE:
- 2 people: $29/Mo (flat)
- 10 people: $29/Mo (same!)

LANGSMITH:
- 2 people: $39 × 2 = $78
- 10 people: $39 × 10 = $390

WINNER: Langfuse saves $361/Mo at 10 people!

Setup Example (Langfuse Cloud)

from langfuse.callback import CallbackHandler
from langchain.chat_models import ChatOpenAI

# Create handler
handler = CallbackHandler(
    public_key="pk_xxx",
    secret_key="sk_xxx"
)

# Use with LangChain
llm = ChatOpenAI(callbacks=[handler])
response = llm.predict("Your question")

Traces automatically appear in Langfuse dashboard!

Budget by Team Size

Langfuse is 13× cheaper at 10 people!

Advanced Monitoring Features

LangSmith: Deep LangChain Integration

Execution Traces:

from langsmith import traceable

@traceable(name="my_function")
def process_query(query: str) -> str:
    # Automatically traced in LangSmith
    return llm.invoke(query)

Trace Features:

• Automatic function call capture
• Token counting
• Latency per step
• Cost calculation per call

Best for: LangChain projects with deep instrumentation requirements

Langfuse: Framework-Agnostic Tracing

Works with any framework:

from langfuse import Langfuse

langfuse = Langfuse()

trace = langfuse.trace(name="query_processing")
# Any code inside this trace is logged
llm_response = trace.generation(
    name="llm_call",
    model="gpt-4",
    input="What is AI?"
)

Advantage: Works with OpenAI, Anthropic, local LLMs, custom code

Detailed Pricing Analysis

LangSmith Team Growth Scenario

Month 0:
- 2 engineers: $78 (2 × $39)
- 100k traces/month: Included

Month 3:
- 5 engineers: $195 (5 × $39)
- 500k traces/month: $0.50 × 4 = $2 extra

Month 6:
- 10 engineers: $390 (10 × $39)
- 2M traces/month: $0.50 × 16 = $8 extra
- Total: $398/month

TOTAL BY MONTH 6: $398 (grows linearly with headcount)

Langfuse Team Growth Scenario

Month 0:
- 2 engineers: $29/month (Pro plan, flat rate)
- 100k traces/month: Included

Month 3:
- 5 engineers: $29/month (SAME!)
- 500k traces/month: Included

Month 6:
- 10 engineers: $29/month (STILL SAME!)
- 2M traces/month: Included

TOTAL BY MONTH 6: $174 for 6 months (no increase!)

At 10 people, Langfuse saves $2,268/month ($27,216/year!)

Implementation Checklist

For Langfuse Self-Hosted

• PostgreSQL instance (managed cloud: Supabase $25/month)
• Langfuse container (0.2 CPU, 512MB RAM minimum)
• Environment variables: DATABASE_URL, NEXTAUTH_SECRET
• Expose via reverse proxy (Nginx, Caddy)
• Set up cron for data cleanup (optional)
• Configure backup (daily via pg_dump)

# Self-hosted via Docker
docker run -p 3000:3000 \
  -e DATABASE_URL="postgresql://..." \
  langfuse/langfuse:latest

For LangSmith Integration

• Install langsmith package
• Set environment: LANGCHAIN_API_KEY, LANGCHAIN_ENDPOINT
• Decorate functions with @traceable
• View traces in LangSmith dashboard
• Set up alerts for errors/latency

import os
from langsmith import Client

os.environ["LANGCHAIN_API_KEY"] = "your-key"
os.environ["LANGCHAIN_PROJECT"] = "my-project"

# Automatic trace capture with decorators

Real-World Use Cases

Case Study 1: RAG Application Optimization

Goal: Identify slow retrieval steps in RAG pipeline

Setup: Langfuse + LangChain

RAG Pipeline:
1. Query embedding: 0.2s
2. Vector search: 0.5s (SLOW!)
3. LLM generation: 2.1s

Finding: Qdrant vector search bottleneck
Solution: Added HNSW indexing → 0.15s
Savings: 350ms per query × 100k queries/day = 40 GPU hours saved

Cost before monitoring: $500/month (overcapacity) Cost after optimization: $300/month ROI: $200/month × 12 = $2,400/year (vs $29/month Langfuse)

Case Study 2: Error Tracking in Production

Goal: Identify why 2% of requests timeout

Setup: Langfuse with custom error handler

from langfuse import Langfuse

langfuse = Langfuse()

try:
    response = llm.invoke(query)
except TimeoutError as e:
    langfuse.trace(
        name="timeout_error",
        input=query,
        error=str(e),
        metadata={"retry_count": 3}
    )

Finding: Specific prompts consistently timeout (avg 8s, limit 5s) Solution: Reduce prompt complexity, use faster model for fallback Impact: Timeout rate dropped from 2% to 0.1%

Case Study 3: Cost Attribution by Feature

Goal: Which feature costs most to run?

Setup: Langfuse with cost per feature

langfuse.generation(
    name="feature_x_llm",
    model="gpt-4",
    input="...",
    metadata={"feature": "semantic_search"}  # Tag for grouping
)

Results:

• Semantic search: $1,200/month (40%)
• Summarization: $1,000/month (33%)
• Chatbot: $800/month (27%)

Action: Optimize semantic search, switch to Claude 3.5 Sonnet (25% cheaper)

Monitoring Metrics Explained

Latency

• Definition: Time from request to response
• Target: <2 seconds for user-facing, <5s for batch
• What to optimize: Batch requests, parallel processing

Throughput

• Definition: Requests per second
• Target: Match your SLA requirements
• What to monitor: Queue depth, concurrent requests

Token Efficiency

• Definition: Input tokens consumed vs output generated
• Target: Ratio <5:1 is good (input < output × 5)
• Action: Too many input tokens → need RAG/embeddings

Error Rate

• Definition: % of requests that fail
• Target: <0.1% for production, <1% for beta
• Action: Errors > 1% → investigate root cause immediately

Cost per Request

• Definition: Total API costs / total requests
• Target: <$0.01 per request (for typical LLM app)
• Optimization: Use smaller models, better prompts, caching

Advanced Monitoring Stack (Enterprise)

For companies with >$10k/month LLM spending:

Langfuse (Tracing)          → $100-500/month
+ Prometheus (Metrics)      → Self-hosted
+ Grafana (Visualization)   → Self-hosted
+ PagerDuty (Alerting)      → $100/month
+ PostHog (Analytics)       → $500-2000/month (optional)

Total: $1,000-3,000/month (for comprehensive monitoring)

Decision Matrix (Updated)

Advanced Resources

• Langfuse Pricing Calculator
• LangSmith Best Practices
• Helicone Advanced Features
• Prometheus Alerting Rules
• Grafana Dashboard Templates

Last Updated: 21.03.2026 | Total Lines: 450+

Sources and Links

• Grafana
• Prometheus
• OpenTelemetry