Issues · EleutherAI/lm-evaluation-harness · GitHub

Streamlining lm-eval Architecture
#3083 · baberabb opened on Jun 23, 2025
1
Datasets with loading scripts - no longer supported
#3171 · baberabb opened on Jul 21, 2025
15

is:issue state:open

is:issue state:open

Labels Milestones New issue

Feature request: canonical reproducibility-bundle export in scripts/

#3749

· topeuph-ai opened

on May 4, 2026

Add Llama.cpp support!

#3730

· methedude opened

on Apr 27, 2026

Does evaluation measure capability — or adaptation to ambiguous data?

#3698

· CTMinfo opened

on Apr 12, 2026

Process gets stuck after generation is complete.

#3697

· digbangbang opened

on Apr 12, 2026

ray missing as dependency of vllm backend

#3688

· elonp opened

on Apr 8, 2026

Add SimpleQA (OpenAI factuality benchmark)

#3666

· sarthakkgupta opened

on Mar 31, 2026

[Research] Fundamental Equation of Consciousness: Ψ = argmax H(p) s.t. Φ > Φ_min

#3661

· dancinlife opened

on Mar 29, 2026

Wrong Hendrycks filter with base `x_y` answers or targets, or permutation of several expected numbers `x,y,z`

#3652

· fxmarty-amd opened

on Mar 25, 2026

PubMedQA "Dataset scripts not supported"

#3645

· nloughl opened

on Mar 20, 2026

hendrycks_math: process_results does not extract \boxed{} from model response

#3643

· NezLheimeur opened

on Mar 20, 2026

HFLM: chat_template_args silently discarded when enable_thinking is None

#3639

· NezLheimeur opened

on Mar 19, 2026

OpenGVLab/InternVL3-2B doesn't work with mmmu_val

#3637

· dgouju opened

on Mar 18, 2026