Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

AI Observability & Evaluation

🐢 Open-Source Evaluation & Testing library for LLM Agents

Evaluation and Tracking for LLM Experiments and AI Agents

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

A desktop MCP client designed as a tool unitary utility integration, accelerating AI adoption through the Model Context Protocol (MCP) and enabling cross-vendor LLM API orchestration.

Python SDK for running evaluations on LLM generated responses

Complete AI governance and LLM Evals platform with support for EU AI Act, ISO 42001, NIST AI RMF and 20+ more AI frameworks and regulations. Join our Discord channel: https://discord.com/invite/d3k3E4uEpR

A simple GPT-based evaluation tool for multi-aspect, interpretable assessment of LLMs.

An open-source benchmark for evaluating AI agents' ability to find backdoors hidden in compiled binaries.

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

llm-eval-simple is a simple LLM evaluation framework with intermediate actions and prompt pattern selection

First-of-its-kind AI benchmark for evaluating the protection capabilities of large language model (LLM) guard systems (guardrails and safeguards)

Бенчмарк сравнивает русские аналоги ChatGPT: Saiga, YandexGPT, Gigachat

Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs

🎯 Your free LLM evaluation toolkit helps you assess the accuracy of facts, how well it understands context, its tone, and more. This helps you see how good your LLM applications are.

Develop reliable AI apps

Valor is a lightweight, numpy-based library designed for fast and seamless evaluation of machine learning models.

An open source library for asynchronous querying of LLM endpoints

Structured output benchmarks comparing DSPy and BAML with different LLMs