Evaluation Infrastructure for AI Agents
-
Updated
Feb 25, 2026 - TypeScript
#
ai-evals
Here are 25 public repositories matching this topic...
Learn to evaluate AI products for production — 21 hands-on lessons on evals, metrics, fairness, agents, red teaming, and release decisions for working PMs.
bootcamp red-teaming rag prompt-engineering llmops ai-product-management llm-evaluation claude-code ai-pm ai-evals
-
Updated
May 6, 2026
🔍 Benchmark jailbreak resilience in LLMs with JailBench for clear insights and improved model defenses against jailbreak attempts.
python flask analytics openai alignment model-evaluation ai-safety security-testing red-teaming model-robustness anthropic litellm content-safety llm-jailbreaks tool-calling llm-benchmark ai-evals textual-tui
-
Updated
May 9, 2026 - Python
Benchmark LLM jailbreak resilience across providers with standardized tests, adversarial mode, rich analytics, and a clean Web UI.
python flask analytics openai alignment model-evaluation ai-safety security-testing red-teaming model-robustness prompt-injection anthropic litellm content-safety llm-jailbreaks tool-calling llm-benchmark adversarial-testing ai-evals textual-tui
-
Updated
Aug 12, 2025 - Python
Unofficial TypeScript starter for deterministic local contract testing around Foundry-oriented workflows with Themis.
react typescript schema-validation themis contract-testing osdk developer-tooling agentic-workflows ai-evals foundry-workflows
-
Updated
Mar 28, 2026 - TypeScript
Free, local Langfuse OSS setup with Ollama for LLM evaluation, scoring, and datasets.
docker open-source self-hosted free no-cost local-llm ollama langfuse llm-evaluation prompt-evaluation offline-ai llm-as-judge llm-observability ai-evals
-
Updated
Apr 13, 2026 - TypeScript
End-to-end AI evals orchestration platform for comparing LLM outputs across providers with transcription, structured logging, human review, and Supabase-backed decision tracking.
gemini openai multi-model transcription human-in-the-loop model-comparison supabase anthropic llm-evaluation ai-evals evaluation-pipeline
-
Updated
Mar 10, 2026 - TypeScript
Evaluation prompts for AI systems against the EU AI Act.
-
Updated
May 6, 2026 - TypeScript
A lightweight workbench for dataset-driven agent and LLM evaluation.
-
Updated
May 1, 2026 - Python
Local web app that teaches PMs how to do open coding and error analysis on LLM traces. Keyboard-first labeling, native trace rendering, JSONL export.
-
Updated
May 8, 2026 - TypeScript
Experimentation framework for LLM systems using simulated users, conversational behavioral metrics, and causal inference to evaluate prompt strategies, temperature, and model scaling.
-
Updated
Mar 8, 2026 - Python
Hands-on Agentic AI learning project — ReAct agents, memory systems, evals, and multi-agent architecture. Built as a structured AI PM curriculum.
python product-management react-pattern ai-agent llm research-automation anthropic llm-as-judge agentic-ai ai-evals
-
Updated
Apr 25, 2026 - Python
Evaluation harness for domain-specific RAG and QA systems with benchmark datasets, scoring, and regression workflows.
benchmarking domain-qa retrieval-augmented-generation llm-evaluation rag-evaluation evaluation-harness ai-evals
-
Updated
Mar 8, 2026 - Python
Lightweight eval framework for LLMs & AI apps combining deterministic scoring, LLM-as-judge, and regression testing.
-
Updated
Apr 8, 2026 - Python
Multi-agent system orchestrating an AI-driven software team using the Claude Agents SDK. Agents take on defined roles and collaborate autonomously on software tasks.
jira ai orchestration multi-agent confluence atlassian llm generative-ai anthropic llm-agents agentic-ai mcp-server ai-evals claude-agent-sdk
-
Updated
Feb 4, 2026 - Python
LLM extraction pipeline for job postings; ~80% skill classification accuracy via chain-of-thought scaffolding.
-
Updated
Apr 9, 2026 - Python
Track AI evals, metrics, and tests with Git as the database.
-
Updated
May 5, 2026 - Go
A smart clinical AI Copilot for Diabetic Retinopathy that uses a multi-agent 'Maker-Checker' system to cross-reference medical scans, improving diagnostic accuracy while lowering running costs.
-
Updated
May 7, 2026 - Python
Portfolio project showing product strategy, evals, roadmap, and GTM for a GenAI coding assistant
developer-tools product-management portfolio-project product-strategy llm genai ai-coding-assistant ai-evals
-
Updated
Apr 2, 2026
Improve this page
Add a description, image, and links to the ai-evals topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the ai-evals topic, visit your repo's landing page and select "manage topics."