AI Observability & Evaluation
10.5k
Stars
974
Forks
661
Open issues
30
Contributors
AI Analysis
Phoenix is an AI observability and evaluation platform built for monitoring, evaluating, and debugging LLM applications and AI agents. It serves data scientists, ML engineers, and LLMOps teams who need visibility into model behavior, prompt performance, and system reliability across frameworks like LangChain, LlamaIndex, and custom deployments. It is not a general-purpose monitoring tool—it targets the specific needs of production LLM systems.
Inferred from signals mentioned in the README (tests, CI, type safety) — not a review of the actual code.
AI's overall editorial judgment — not an average of the bars above, can weigh other factors too.
Arize Phoenix: open-source LLM observability platform with tracing, evals, and experiment tracking
Phoenix is an open-source AI observability platform from Arize AI that covers the full LLM application lifecycle: OpenTelemetry-based tracing, LLM-as-judge evaluation, versioned datasets, and experiment management. It targets ML engineers and AI product teams who need visibility into LLM and RAG pipeline behavior in development and production. Backed by Arize AI (a commercial MLOps company), it has reached 10K+ GitHub stars, is available on PyPI, conda-forge, Docker Hub, and Helm, and maintains a companion instrumentation library (OpenInference). The cloud-hosted version serves as a commercial funnel, while the self-hosted path is genuinely open.
Created in November 2022 as the LLM boom accelerated demand for observability tooling. Arize AI leveraged its existing ML monitoring expertise to build a purpose-built LLM observability layer, evolving from a local debugging notebook into a full server-based platform with tracing, evals, and now MCP integration.
Growth accelerated alongside the broader explosion in LLM application development from 2023 onward. Adoption of OpenTelemetry as the tracing backbone lowered integration friction, while the open-source model attracted developers who needed free self-hosted tooling. The companion OpenInference instrumentation library extended reach across frameworks. 97 stars in the past 7 days suggests continued, steady organic growth rather than a viral spike.
Phoenix is backed by Arize AI, a funded company with an existing enterprise customer base in ML monitoring, which likely drives some production adoption internally and through commercial users. PyPI download statistics are not cited in the README, but availability on conda-forge, Docker Hub with versioned images, and Helm charts strongly suggest production deployments exist. The 10K+ stars and 945 forks are consistent with a tool used beyond experimentation. Concrete production case studies are not surfaced in the README excerpt, so scale of production adoption is not fully verified.
Appears to follow a server-client architecture: a Python backend providing a REST/GraphQL API, with a web UI for trace visualization and experiment management. Tracing is built on OpenTelemetry, meaning instrumentation is likely handled via the separate OpenInference library. Likely uses a local SQLite or PostgreSQL backend for persistence. Docker and Helm chart availability suggest it is designed for both local dev and production deployment. MCP (Model Context Protocol) integration is present as a separate JS package.
Not documented in README
Last push on 2026-06-28 (same day as evaluation date), indicating very active, continuous development. The project has been maintained for over 3.5 years with no apparent gaps. Multiple distribution channels (PyPI, conda-forge, Docker, Helm) suggest mature release engineering. Slack community and social media presence indicate active maintainer engagement.
ADOPT IF: you are building LLM or RAG applications in Python and need a self-hosted, open-source observability stack that covers tracing, LLM-as-judge evaluation, and experiment tracking in a single tool. AVOID IF: your team is TypeScript-first, you need enterprise SLA guarantees on the open-source tier, or you are already deeply invested in a competing observability platform with equivalent LLM support. MONITOR IF: you are evaluating the LLM observability space but not yet in production — Phoenix is mature enough to adopt now, but the category is evolving fast and feature parity among competitors may shift.
Independent dimensions
Mainstream potential
7/10
Technical importance
8/10
Adoption evidence
6/10
- Commercial alignment risk: Arize AI controls the project roadmap, and open-source feature development may be deprioritized in favor of features that drive cloud upsell.
- Category commoditization: LLM observability tooling is converging rapidly; differentiators like Phoenix's eval framework may become table stakes across competitors within 12-18 months.
- OpenInference instrumentation dependency: reliance on a separate companion library (openinference) for framework integrations adds a second point of maintenance risk.
- Self-hosted operational complexity: running Phoenix in production (persistent storage, auth, scaling) adds infrastructure burden that smaller teams may underestimate based on the 'just run pip install' entry point.
- LLM-as-judge eval quality: the evaluation framework relies on LLMs to score outputs, which inherits the reliability and cost limitations of the underlying model — results may be inconsistent without careful calibration.
Phoenix is likely to maintain its position as one of the top two or three open-source LLM observability tools. The Arize commercial backing provides runway, and OpenTelemetry adoption as the tracing backbone is a durable architectural bet. Expect continued feature expansion into agent observability and automated red-teaming.
Newsletter
Get analyses like this every Monday
Free weekly digest of the most interesting open-source discoveries.
Languages
Information
- Website
- https://arize.com/docs/phoenix
- Language
- Python
- License
- NOASSERTION
- Last updated
- 8h ago
- Created
- 45mo ago
- Analyzed with
- anthropic/claude-haiku-4-5
Stars over time
Contributors over time
Top 100 contributors only — repos with more will plateau at 100.
Top contributors
Similar repos
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for observability, monitoring, and evaluation...
AgentOps-AI/agentops
AgentOps is a Python SDK for monitoring, observability, and cost tracking of AI...
| Repository | Stars | Week Δ | Language | Score | Updated |
|---|---|---|---|---|---|
|
Arize-ai/phoenix
this
|
10.5k | +105 | Python | 8/10 | 8h ago |
|
raga-ai-hub/RagaAI-Catalyst
|
16.1k | — | Python | 7/10 | 5mo ago |
 Agenta-AI/agenta
|
4.3k | — | TypeScript | 7/10 | 10h ago |
 Observal/Observal
|
2.2k | — | Python | 7/10 | 3d ago |
|
AgentOps-AI/agentops
|
5.7k | — | Python | 8/10 | 2w ago |
|
Arize-ai/openinference
|
1.1k | — | Python | 7/10 | 17h ago |
Langfuse is the most direct open-source competitor, also offering LLM tracing and evals with a self-host option. Langfuse has stronger community visibility in some circles and a TypeScript-first SDK alongside Python. Phoenix has broader framework instrumentation via OpenInference and tighter integration with the Arize commercial platform.
Logfire is a general-purpose observability tool from the Pydantic team with OpenTelemetry at its core. It covers LLM use cases but is not LLM-specific. Phoenix is more opinionated about LLM evaluation workflows, which may suit AI-focused teams better, while Logfire may suit teams needing unified app+LLM observability.
Laminar is a newer, TypeScript-native LLM observability tool with 3K stars. It appears to target similar use cases but has significantly less adoption and ecosystem depth. Phoenix's Python-first design and OpenTelemetry foundation give it broader compatibility in the ML/AI stack.
W&B Weave is a closed-source, hosted LLM observability layer built on top of the mature W&B platform. It has strong adoption among teams already using W&B for training. Phoenix's self-hostability and open-source nature are a key differentiator for teams with data residency or cost concerns.
Helicone is a proxy-based LLM observability tool focused primarily on cost and usage tracking for OpenAI-compatible APIs. Phoenix is architecturally different (SDK instrumentation, not proxy) and covers a broader surface area including evals and experiments, though Helicone's proxy approach requires zero code changes.
