Test your prompts, agents, and RAGs. AI Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

LLM Reasoning and Generation Benchmark. Evaluate LLMs in complex scenarios systematically.

Prompture is an API-first library for requesting structured JSON output from LLMs (or any structure), validating it against a schema, and running comparative tests between models.

Test, compare, and optimize your AI prompts in minutes

The prompt engineering, prompt management, and prompt evaluation tool for TypeScript, JavaScript, and NodeJS.

EvalWise is a developer-friendly platform for LLM evaluation and red teaming that helps test AI models for safety, compliance, and performance issues

LLM Prompt Test helps you test Large Language Models (LLMs) prompts to ensure they consistently meet your expectations.

prompt-evaluator is an open-source toolkit for evaluating, testing, and comparing LLM prompts. It provides a GUI-driven workflow for running prompt tests, tracking token usage, visualizing results, and ensuring reliability across models like OpenAI, Claude, and Gemini.

Community Plugin for Genkit to use Promptfoo

An open-source AI prompt engineering playground with live code execution. Test OpenAI & Claude prompts, execute JavaScript, and iterate in real-time.

Sample project demonstrates how to use Promptfoo, a test framework for evaluating the output of generative AI models

A pytest-based framework for testing multi AI agents systems. It provides a flexible and extensible platform for complex multi-agent simulations. Supports many integrations like LiteLLM, CrewAI, LangChain etc.

prompt-testing

A collection of prompts that I use on a day-to-day basis for work and leisure.

Quickstart guide for using PromptFoo to evaluate LLM prompts via CLI or Colab.

AI RAG evaluation project using Ragas. Includes RAG metrics (precision, recall, faithfulness), retrieval diagnostics, and prompt testing examples for fintech/banking LLM systems. Designed as an AI QA Specialist portfolio project.

Visual prompt engineering platform for creating, testing, and versioning LLM prompts across multiple providers (OpenAI, Anthropic, Mistral, Gemini).

Sample implementation demonstrating how to use Firebase Genkit with Promptfoo

Testing framework for LLM prompts. Started as a weekend project after getting tired of manually testing prompts in ChatGPT. Async experiment runner with BLEU/ROUGE/BERTScore metrics.