Continuous evaluation system for local LLMs across multiple roles: coding, reasoning, planning, and tool use. Features automatic model discovery, job queue scheduling, GPU monitoring, and weighted leaderboards.

Benchmark UI	Description
	Model comparison leaderboard
	Past executions and evaluation results

All containers run with network_mode: host to access Ollama instances on the host.

┌─────────────────────────────────────────────────────────────────┐
│  Host Machine                                                   │
│                                                                 │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────────┐│
│  │ UI :8700 │  │ API :8701│  │ DB :5445 │  │ Metrics :8702    ││
│  │ (nginx)  │─▶│ (FastAPI)│─▶│ (Postgres)│  │ (Prometheus)     ││
│  └──────────┘  └────┬─────┘  └──────────┘  └──────────────────┘│
│                     │                                           │
│         ┌───────────┼───────────┐                               │
│         ▼           ▼           ▼                               │
│  ┌────────────┐ ┌────────┐ ┌────────┐                          │
│  │ Scheduler  │ │ Ollama │ │ Ollama │                          │
│  │ (APSched)  │ │ :11434 │ │ :11435 │                          │
│  │            │ │ (NVIDIA)│ │ (AMD)  │                          │
│  │ - Discovery│ └────────┘ └────────┘                          │
│  │ - Worker   │                                                 │
│  │ - GPU Mon  │ ┌────────────────────┐                          │
│  │ - Cleanup  │ │ GPUs               │                          │
│  └────────────┘ │ - NVIDIA RTX 3090  │                          │
│                 │ - AMD (ROCm)       │                          │
│                 └────────────────────┘                          │
└─────────────────────────────────────────────────────────────────┘

# Start all services
cd docker
docker compose up -d

# Open the UI
xdg-open http://localhost:8700

Models are discovered from Ollama endpoints but require manual assignment to a GPU instance before benchmarks run. The scheduler starts immediately and begins:

1. Discovery — polls Ollama endpoints every 5 minutes for new models (registered as pending)
2. Job Worker — picks up pending benchmark jobs every 15 seconds
3. GPU Monitor — collects GPU stats every 15 seconds
4. Cleanup — removes jobs older than 90 days (daily)

New models appear as pending in the Model Registry. To start benchmarking a model, assign it to an Ollama instance (NVIDIA or AMD) via the UI — this activates the model and creates benchmark jobs.

cd docker
docker compose down

Models are discovered from configured Ollama endpoints and registered as pending until manually assigned to a GPU instance. This is critical for shared model stores where both Ollama instances see the same models — you control which GPU runs each model.

Each model entry includes:

• Provider and runtime (ollama / ollama-amd)
• Quantization level (Q4_K_M, Q8_0, etc.)
• Parameter count (e.g., 8.2B, 70.6B)
• Model family (qwen3, llama, etc.)
• VRAM requirement
• Status (pending, active, broken)

Workflow:

1. Click Discover Models (or wait for auto-discovery every 5 minutes)
2. New models appear as pending with an Assign button
3. Select a GPU instance (NVIDIA :11434 or AMD :11435) and click Go
4. Model becomes active, benchmark jobs are created automatically
5. Active models can be Reassigned to a different GPU at any time

Only pending models that disappear from Ollama are auto-removed. Active models (user-assigned) are preserved. Manual registration via the API is also supported.

Benchmark jobs are managed through a priority queue:

• Jobs are created when a model is activated (assigned to a GPU instance)
• Create additional jobs for specific models and suites via UI or API
• Parameter matrix testing (sweep temperature, top_p, seed)
• Cancel pending jobs, track running/completed/failed

Real-time GPU stats via nvidia-smi and rocm-smi:

• GPU utilization percentage
• VRAM usage (absolute for NVIDIA, percentage for AMD)
• Temperature
• Exposed as Prometheus gauges

The benchmark runner automatically manages GPU memory between model switches:

1. Unloads the current model from Ollama
2. Waits up to 30 seconds for VRAM to be released
3. Preloads the target model before benchmarking

Ranked by quality (avg score across all datasets in a suite). Additional columns show speed (tok/s), latency, and run count. Scores can be cleared per-entry or in bulk via checkboxes to allow re-evaluation after scoring changes.

Exclude non-logic failures checkbox filters out results where the model failed due to code extraction errors, import failures, or other non-logic issues, showing rankings based only on genuine test execution results.

Per-dataset scoring adapts to validation confidence:

This means "all tests passed" (high confidence) → score 1.0, not penalized by irrelevant heuristics.

• Run History: View all benchmark runs with full details. Delete individual runs or select multiple with checkboxes and batch delete.
• Leaderboard: Clear scores for individual model/suite entries or select multiple and clear in bulk. Useful when evaluation criteria change and previous results need to be discarded.
• Model Registry: Remove individual models or use "Refresh Models" to sync with Ollama (adds new, removes stale).

Generate evaluation reports with per-model aggregates. Reports can be saved as JSON files for archival.

Models are discovered from Ollama endpoints configured via BENCH_DISCOVERY_ENDPOINTS. With a shared model store, both instances see all models — the UI lets you decide which GPU runs each one.

# Activate a model via API (assigns to NVIDIA and creates jobs)
curl -X POST http://localhost:8701/models/42/activate \
  -H "Content-Type: application/json" \
  -d '{"endpoint": "http://localhost:11434"}'

# Activate with specific suites only
curl -X POST http://localhost:8701/models/42/activate \
  -H "Content-Type: application/json" \
  -d '{"endpoint": "http://localhost:11435", "suites": ["coding_suite"]}'

# Reassign an active model to a different GPU
curl -X PATCH http://localhost:8701/models/42 \
  -H "Content-Type: application/json" \
  -d '{"endpoint": "http://localhost:11435", "provider": "ollama-amd"}'

Models are accessed via OpenAI-compatible /v1/chat/completions endpoints (Ollama, vLLM, etc.).

Defined in config/benchmark_suites.yaml. Five built-in suites:

coding_suite:
  name: Coding Benchmarks
  datasets:
    - coding/bug_fix_01
    - coding/refactor_01
    - coding/write_function_01
    - coding/multi_file_edit_01
    - coding/hard/lru_cache
    - coding/hard/merge_intervals_adv
    - coding/hard/moving_average_stream
    - coding/hard/lru_cache_ttl_batch
    - coding/hard/causal_kv_store_single

Five high-difficulty datasets are included in coding_suite to improve ranking separation at the top tier:

These datasets use sequence-based testing where the function receives an operation list and returns results, forcing models to implement correct state management rather than stateless transformations. They expose differences between top-performing models that cluster tightly on standard problems.

The system uses a hybrid deterministic evaluation engine. When a dataset includes a VALIDATION block, the appropriate deterministic validator is used. Otherwise, it falls back to keyword-overlap scoring.

The coding validator runs a multi-stage pipeline to differentiate strong vs weak models:

1. Base Tests (weight 0.5) — provided test cases from the dataset
2. Random Tests (weight 0.3) — auto-generated mutations of base inputs, validated against reference solution
3. Edge Cases (weight 0.2) — auto-generated boundary inputs (empty, single, large)
4. Hidden Tests — server-side tests not exposed in the dataset (penalty-only)
5. Anti-Hardcoding Detection — AST inspection (lookup tables, if/elif chains), literal matching, random test failure ratio
6. Code Quality — AST-based scoring: structure, naming, DRY, readability, documentation (0-1)
7. Performance — execution time compared to reference solution (0-1)

Composite score for coding validators: correctness×0.5 + robustness×0.2 + quality×0.15 + performance×0.15

This means a model that passes all tests but writes poor code scores lower than one that passes all tests with clean, well-documented code. Hardcoded solutions that pass base tests but fail random tests are penalized heavily.

Failure classification — non-logic failures are classified separately so they don't pollute model rankings:

The leaderboard's "Exclude non-logic failures" checkbox filters out all results with a non-null execution_error_type, computing rankings only from genuine test executions.

• High — deterministic validation (code execution, exact match)
• Medium — partial match or close numeric comparison
• Low — keyword overlap or heuristic scoring

Each scored result includes:

Extended fields (coding validators only):

Each dataset is a markdown file in datasets/<category>/<name>.md:

PROMPT:
Your prompt text here.

EXPECTED:
Expected output description.

SCORING:
correctness reasoning format

VALIDATION:
```yaml
type: coding
language: python
function_name: flatten_dict
tests:
  - input: '{"a": 1, "b": {"c": 2}}'
    expected: '{"a": 1, "b.c": 2}'
```​

Coding — enhanced multi-stage code evaluation:

type: coding
language: python
function_name: my_function
randomizable: true           # enables random test generation
reference_solution: |        # canonical solution for random tests + perf
  def my_function(x):
      return x * 2
base_tests:                  # required: provided test cases
  - input: '{"key": "value"}'
    expected: '{"result": true}'
edge_tests:                  # optional: additional edge cases
  - input: '""'
    expected: '""'

Coding Multi-File — multi-file project validation:

type: coding_multifile
files:                       # initial project files
  models/user.py: |
    class User: ...
modified_files:              # expected files to be modified
  - services/user_service.py
integration_tests:           # tests run against merged files
  - code: |
      from services.user_service import UserService
      svc = UserService(mock_db)
      assert svc.get_user(1).name == "Alice"
    expected: "pass"

Reasoning — pattern matching against expected answers:

type: reasoning
expected_answer: 42
extract_patterns:
  - "alice.*fish"
  - "bob.*dog"

Tool Use — validates function calls:

type: tool_use
available_functions: [get_weather, convert_temperature]
expected_calls:
  - function: get_weather
    arguments: {city: Tokyo}
  - function: convert_temperature
    arguments: {from_unit: celsius, to_unit: fahrenheit}

Planning — rubric-based keyword scoring:

type: planning
rubric:
  - criterion: "Identifies core components"
    keywords: ["api", "cache", "database"]
    weight: 0.3

1. Go to the Run Benchmark tab
2. Click to select one or more models (chip selector, "Select All" supported)
3. Choose benchmark suites (leave empty for all)
4. Set parameters: temperature, top_p, max_tokens, seed
5. Click "Create Jobs"

Jobs are queued and automatically picked up by the scheduler worker.

1. Go to the Run Benchmark tab, scroll to "Custom Benchmark"
2. Select one or more models
3. Enter your prompt in the text area
4. Optionally set expected output (used for correctness scoring)
5. Choose scoring criteria (correctness, reasoning, format)
6. Set generation parameters
7. Click "Run Custom Benchmark"

Custom benchmark results store the prompt, expected output, and criteria alongside the scores so runs are fully self-contained and reproducible.

# Create jobs for multiple models with custom parameters
curl -X POST http://localhost:8701/jobs/ \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["deepseek-r1-8b", "qwen3-coder"],
    "suites": ["coding_suite"],
    "parameters": {"temperature": 0.5, "top_p": 0.9, "max_tokens": 2048, "seed": 99}
  }'

# Create jobs with parameter matrix (tests all combinations)
curl -X POST http://localhost:8701/jobs/ \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["deepseek-r1-8b"],
    "suites": ["coding_suite"],
    "parameter_matrix": {
      "temperature": [0.0, 0.3, 0.7],
      "seed": [42, 123]
    }
  }'

# Run custom benchmark against multiple models
curl -X POST http://localhost:8701/runs/custom \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["deepseek-r1-8b", "qwen3-coder"],
    "prompt": "Write a Python function that returns the nth Fibonacci number using memoization.",
    "expected": "def fibonacci memoization cache dictionary recursive",
    "scoring_criteria": ["correctness", "reasoning", "format"],
    "temperature": 0.0,
    "max_tokens": 2048
  }'

# Direct run (bypasses job queue, single model)
curl -X POST http://localhost:8701/runs/ \
  -H "Content-Type: application/json" \
  -d '{"suite": "coding_suite", "model": "qwen3-coder", "temperature": 0, "seed": 42}'

python runner/run_benchmarks.py trigger coding_suite qwen3-coder

All configuration uses the BENCH_ prefix:

The API exposes metrics at port 8702. Add to your Prometheus config:

scrape_configs:
  - job_name: 'model-benchmarks'
    static_configs:
      - targets: ['localhost:8702']

Available metrics:

model_benchmarks/
├── api/
│   ├── main.py                    # FastAPI app with scheduler
│   ├── config.py                  # Pydantic Settings (BENCH_ prefix)
│   ├── db/
│   │   └── session.py             # Async SQLAlchemy session
│   ├── models/
│   │   ├── model_config.py        # Model registry table
│   │   ├── benchmark_run.py       # Benchmark runs table
│   │   ├── benchmark_job.py       # Job queue table
│   │   └── result.py              # Results & performance metrics
│   ├── routers/
│   │   ├── models.py              # /models/ endpoints
│   │   ├── benchmarks.py          # /benchmarks/ endpoints
│   │   ├── runs.py                # /runs/ + leaderboard
│   │   ├── jobs.py                # /jobs/ queue management
│   │   ├── discovery.py           # /discovery/ + GPU stats
│   │   └── reports.py             # /reports/ generation
│   ├── schemas/
│   │   └── run.py                 # Pydantic request/response models
│   ├── services/
│   │   ├── benchmark_runner.py    # Runs benchmarks against models
│   │   ├── model_clients.py       # Ollama/OpenAI client with VRAM mgmt
│   │   ├── scoring.py             # Hybrid scoring engine (dispatch)
│   │   ├── validators/
│   │   │   ├── base.py            # ValidationResult, Confidence enum
│   │   │   ├── coding.py          # Enhanced multi-stage code evaluation
│   │   │   ├── coding_multifile.py # Multi-file project validation
│   │   │   ├── hardcoding_detector.py # Anti-hardcoding heuristics
│   │   │   ├── quality_analyzer.py # AST-based code quality scoring
│   │   │   ├── test_generator.py  # Random test + edge case generation
│   │   │   ├── reasoning.py       # Answer extraction + pattern matching
│   │   │   └── tool_use.py        # Function call + schema validation
│   │   ├── discovery.py           # Ollama model discovery
│   │   ├── job_queue.py           # Job lifecycle management
│   │   ├── gpu_monitor.py         # nvidia-smi/rocm-smi stats
│   │   ├── scheduler.py           # APScheduler task definitions
│   │   └── reports.py             # Report generation
│   └── metrics/
│       └── prometheus.py          # Metric definitions
├── runner/
│   └── run_benchmarks.py          # Standalone runner script
├── ui/
│   ├── Dockerfile                 # nginx + Vite build
│   └── src/
│       ├── App.jsx                # Sidebar navigation, light/dark theme
│       ├── Leaderboard.jsx        # Sortable leaderboard, CSV export, score clearing, failure filtering
│       ├── RunHistory.jsx         # Split-pane history, bulk delete, validation details
│       ├── RunBenchmark.jsx       # Batch suite run + custom benchmark
│       ├── ModelRegistry.jsx      # Model discovery, GPU assignment, activate/reassign
│       ├── JobQueue.jsx           # Job monitoring + cancellation
│       ├── api.js                 # API client functions
│       └── index.css              # Design system (CSS custom properties, light/dark)

├── datasets/                      # Benchmark datasets (markdown)
│   ├── coding/
│   ├── reasoning/
│   ├── planning/
│   ├── tool_use/
│   └── hidden/                    # Hidden tests (not exposed via API)
│       └── coding/                # JSON files with secret test cases
├── config/
│   ├── models.yaml                # Model endpoint config
│   └── benchmark_suites.yaml      # Suite definitions
├── docker/
│   ├── docker-compose.yml
│   └── Dockerfile.api
└── requirements.txt

# Install API dependencies
pip install -r requirements.txt

# Run API locally
uvicorn api.main:app --reload --port 8701

# Run UI locally
cd ui && npm install && npm run dev

• Discovery + Manual Control: Models auto-discovered, but GPU assignment and activation are manual
• Deterministic: Default params temperature=0, top_p=1, seed=42
• Model-agnostic: Unified client for any OpenAI-compatible endpoint
• GPU-aware: Manual GPU assignment, VRAM management, and monitoring for multi-GPU setups
• Independent: No dependency on ai-gateway, openwebui, or qdrant
• On-demand: Start and stop as needed via docker compose