ai-evaluation

Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

[JMLR 2026] "UQLM: A Python Package for Uncertainty Quantification in Large Language Models"

Hallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.

build and benchmark deep research

Ranking LLMs on agentic tasks

Agentic AI research papers, benchmarks, frameworks, and tools curated across 24 domains.

开源劳动者AI研究与评测项目：建设劳动知识、公开评测基准与共同治理机制｜Open worker-centered AI research and evaluation project.

Vivaria is METR's tool for running evaluations and conducting agent elicitation research.

one click to open multi AI sites ｜ 一键打开多个 AI 站点，查看 AI 结果

Statistical analysis for LLM evaluations, from model comparisons to inference resilient to LLM judge bias, optimized for small sample sizes. All defaults battle-tested in Monte Carlo simulations.

A benchmark for AI-driven CAD generation and editing

☑️ A curated list of tools, methods & platforms for evaluating AI reliability in real applications

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

AgentVitals Checkup (/checkup) — an AI agent skill that gives your agent a professional health checkup: dual-axis Stability + Welfare scoring, a personality-style title, and a public cross-platform leaderboard. One-line install for Claude Code / OpenClaw / Codex / Coze. Bilingual EN/ZH.

Code scanner to check for issues in prompts and LLM calls

Comprehensive AI Model Evaluation Framework with advanced techniques including Temperature-Controlled Verdict Aggregation via Generalized Power Mean. Support for multiple LLM providers and 15+ evaluation metrics for RAG systems and AI agents.

Make your OpenClaw agents better, cheaper, and faster.

Causal Judge Evaluation: calibrate LLM-as-judge scores against oracle labels with valid uncertainty.

Bring Galileo powered eval workflows into Claude Code and Codex.

Benchmark evaluating LLMs on their ability to create and resist disinformation. Includes comprehensive testing across major models (Claude, GPT-4, Gemini, Llama, etc.) with standardized evaluation metrics.