Llm evaluation

iFixAi is a specialized diagnostic tool that runs 45 automated inspections against AI agents and LLMs to detect operational misalignment—gaps between...

Langfuse is an open-source AI engineering platform that provides LLM observability, evaluation, metrics, prompt management, and a playground...

AI-Infra-Guard is a comprehensive red teaming platform by Tencent's Zhuque Lab designed to identify security vulnerabilities in AI systems through...

Opik is an open-source observability and evaluation platform specifically designed for LLM applications, RAG systems, and agentic workflows. It...

This is the official GitHub repository for the LLM Engineer's Handbook, a practical guide covering the full lifecycle of building and deploying LLM...

garak is an LLM vulnerability scanner that probes large language models for security weaknesses including hallucination, data leakage, prompt...

Phoenix is an AI observability and evaluation platform built for monitoring, evaluating, and debugging LLM applications and AI agents. It serves data...

Promptfoo is a CLI and library for evaluating, testing, and red-teaming LLM applications, enabling users to compare model performance across multiple...

DeepEval is an open-source Python framework for evaluating large language model systems using metrics like G-Eval, hallucination detection, and...

Helicone is an open-source AI Gateway and LLM observability platform that enables developers to monitor, evaluate, and manage large language model...

Laminar is an open-source observability platform purpose-built for AI agents, offering tracing, signals, evaluations, dashboards, and data annotation...

Agenta is an open-source LLMOps platform designed for building and managing reliable LLM applications through prompt management, evaluation, and...

MLflow is a comprehensive open-source platform for managing the lifecycle of AI applications, specifically optimized for LLM agents, generative AI,...

UQLM is a specialized Python library for detecting hallucinations in Large Language Model outputs through uncertainty quantification techniques. It...

LMMs-Eval is a unified evaluation toolkit for multimodal large language models across text, image, video, and audio tasks. It serves researchers and...

Giskard is an open-source Python library for testing, evaluating, and red-teaming agentic AI systems. It specializes in dynamic multi-turn agent...

Judgeval is an open-source Python SDK for agent monitoring and continuous improvement that provides OpenTelemetry-based tracing, agent-judge...

FuzzyAI is a specialized security testing tool for identifying jailbreaks and vulnerabilities in LLM APIs through automated fuzzing. It targets...

AutoRAG is an open-source AutoML framework for automatically discovering and optimizing retrieval-augmented generation (RAG) pipelines. It...

FinSight is an open-source AI equity research agent that automates financial analysis by combining resilient workflow orchestration, vector RAG with...

Agentic Security is a specialized vulnerability scanner for LLM-based agent systems and large language models, designed to identify jailbreaks,...

Prompty is a markdown-based format (`.prompty`) for managing LLM prompts across Python, TypeScript, and VS Code environments. It targets developers...

TruLens is a specialized evaluation and tracking platform for LLM applications and AI agents, designed to systematically assess performance through...

ReLE is a comprehensive Chinese LLM evaluation benchmark that assesses 386+ large language models across 7 domains (~300 sub-dimensions) including...

AI Sheets is an open-source, no-code platform for building and transforming datasets using AI models from Hugging Face Hub or custom LLMs. It serves...