Benchmark Hub

Summarizes precision-recall performance across thresholds and is especially informative for imbalanced data.

Summarizes ROC curve performance across thresholds by measuring ranking quality between positives and negatives.

Question-answering benchmark for detecting social bias through stereotype reliance under ambiguous context.

Contextual embedding-based similarity metric for scoring generated text against reference outputs.

Corpus-level n-gram overlap metric (MT-originated) used to score generated text against reference translations or summaries.

Proper scoring rule measuring mean squared error between predicted probabilities and observed binary outcomes used to assess calibration and reliability.

Reliability diagram showing how predicted probabilities align with observed outcome frequencies across bins.

Bias benchmark using minimal pairs to measure preference for stereotyped vs. anti-stereotyped sentences.

Clinical utility evaluation that compares net benefit across decision thresholds to determine whether a model improves outcomes versus treat-all or treat-none strategies.

Calibration metric that quantifies the discrepancy between predicted probabilities and observed accuracy across probability bins.

Balanced metric that summarizes precision and recall into one harmonic-mean score for classification performance.

Healthcare evaluation suite developed by OpenAI that assesses clinical, administrative, and patient-communication tasks using safety-aware scenarios and physician-authored, rubric-based scoring.

A comprehensive evaluation framework for language models that standardizes tasks, prompts, metrics, and reporting across diverse tasks, domains, and use cases.

Benchmark for measuring social bias across demographic attributes using templated prompts and model completions.

Aggregates probabilistic prediction errors by penalizing incorrect and overconfident predictions.

Correlation-based metric that accounts for true/false positives and negatives, robust to class imbalance.

A healthcare-focused evaluation suite that assesses large language models across 35 medical benchmarks covering clinical, biomedical, and healthcare-related tasks.

USMLE-style medical multiple-choice QA benchmark (~12k items) evaluating diagnostic reasoning, treatment selection, and contraindication assessment across major clinical domains.

Broad multi-domain benchmark with ~15k questions across 57 subjects that evaluates general knowledge and multiple-choice reasoning.

Multi-turn conversational benchmark evaluated using LLM-as-judge scoring to assess instruction adherence, coherence, and response quality across dialogue turns.

Positive predictive value (precision) and negative predictive value, measuring correctness for predicted positives and negatives.

Biomedical research QA benchmark with ~1k questions that evaluates evidence grounded answering using PubMed abstracts.

Recall-oriented overlap metrics (ROUGE-N, ROUGE-L, ROUGE-Lsum) for comparing generated summaries to reference summaries.

Companion metrics measuring true positive rate (sensitivity) and true negative rate (specificity).

Question-answering benchmark designed to detect model falsehoods and overconfident incorrect answers across adversarial prompts.

Benchmark suite of real-world datasets designed to evaluate model robustness under distribution shift, with explicit domain and subgroup metadata.