(01)The most complete open-source option that keeps your data in-house: multi-step causal tracing, prompt versioning, datasets and basic experiments. What it does NOT do is automatic issue clustering or auto-generated evals — you assemble that layer yourself. Acquired by ClickHouse in January 2026; feature set unchanged in the 2026 comparisons. No per-seat pricing on any tier.
Self-hosted LLM/agent observability with evals and prompt managementSelf-hosted free; cloud Hobby free, Core from ~$29/mo, Pro tiers above thatAI-Native
(02)The evaluation-first platform: datasets, reusable scorers, experiment comparison and CI/CD eval-gated deployments. The free tier is the most generous in the category (1M trace spans/month, 10K eval runs, unlimited users), and Topics (beta) adds issue clustering. Pick it when failures must become regression tests before the next deploy.
Eval-first development: scorers, experiments, CI-gated deploysFree (1M spans/mo, 10K eval runs); Pro from ~$249/moAI-Native
(03)LangChain's tracing and eval platform, and — honestly — mostly worth it if you are already on LangChain or LangGraph: one environment variable gives full instrumentation, plus datasets, experiments, review queues and Insights (LLM-based failure clustering). Off that stack, the integration advantage largely evaporates and lock-in is real; OTel support since 2025 softens the edges.
LangChain/LangGraph-native tracing, datasets and evalsFree (5K traces/mo); Plus $39/seat/moAI-Native
(04)The OTel-native open-source project for self-managed agent observability: trace capture, RAG and agent evals, LLM-as-judge metrics, dataset management — without a proprietary trace format. No issue lifecycle tracking and no auto-eval generation; the commercial Arize AX tier (from ~$50/mo) adds production monitoring and drift detection on top.
OpenTelemetry-native tracing and evaluation, self-managedPhoenix open-source free; Arize AX paid from ~$50/moAI-Native
(05)The widest framework net in 2026: 400+ LLMs and the major agent frameworks (CrewAI, AutoGen, OpenAI Agents SDK, LangChain) plus time-travel debugging and session replays. Strongest choice for debugging complex multi-agent interactions across stacks; it is observability-first, not a systematic quality-improvement loop — no issue clustering, no auto-evals.
Multi-framework agent debugging, session replay, cost trackingFree tier; startup plans; enterprise quotes up to ~$10K+/mo for high volumeAI-Native
(06)Built for evaluating 100 percent of production traffic: the Luna-2 models run checks at sub-200ms latency at roughly 97 percent lower cost than a standard LLM-as-judge — the first time full-traffic coverage is economically practical — and Signals adds ML-based clustering. Evaluate-first rather than trace-first: it deliberately does not store long-term conversation history, so pair it with a tracer.
Online/production evaluation at full traffic, agent failure detectionFree tier; paid/custom enterprise pricing on requestAI-Native
(07)The standards-first open-source option: fully OpenTelemetry-compatible with native GenAI semantic conventions, trace replay for diffing a captured trace against a new model or prompt, prompt versioning held to model-registry rigor, and automated evals hooked into CI/CD. Free and free to keep — your GenAI work lands in the same registry as everything else you track.
Open-source experiment tracking, tracing and evals with model registryFree, open sourceAI-Native
(08)MIT-licensed, configuration-first (YAML + CLI + TypeScript) for prompt testing, provider comparison and red teaming in CI — the go-to if you want security tests beside correctness. It also absorbed a migration wave in 2026: OpenAI's hosted Evals product went read-only on 31 October 2026 and shut down on 30 November 2026 (the open-source repo stays usable).
Config-driven evals, red teaming and CI security gatingFree open source; enterprise tier on requestAI-Native
(09)DeepEval is the pytest-native, Apache-2.0 eval framework with 50+ research-backed metrics covering agents, RAG, chat and safety — the natural pick if evals belong in your test suite. The Confident AI platform layer adds the workflow around it: eval datasets auto-generated from production traces, Signals issue tracking, native red teaming and cross-team governance.
Pytest-native eval metrics plus enterprise quality/governance layerDeepEval free; Confident AI free plan, usage-based paid, custom enterpriseAI-Native
(10)The lowest-overhead entry point: change one API base URL and you get traces, cost tracking and session analysis, plus gateway features — provider routing, failover, caching across 100+ providers — that can trim API spend. Observability-wise it covers the basics only: no evals, no issue clustering. Treat it as the on-ramp before a serious eval layer, not as one.
Proxy/gateway-first observability: cost, latency, basic sessionsFree tier; usage-based paid plansAI-Native
(11)The correlation play: agent telemetry (traces, token cost, error patterns) beside the rest of the stack in the platform your SREs already watch. Benchmarks in 2026 counted it among the heavier agents for instrumentation overhead, so teams choose it for full-stack incident correlation rather than eval depth. Priced usage-based on top of the Datadog platform.
Agent + infrastructure correlation inside a full APM stackUsage-based pricing on top of Datadog platform costsAI-Native
