Benchmark Hub

Summarizes precision-recall performance across thresholds and is especially informative for imbalanced data.

Summarizes ROC curve performance across thresholds by measuring ranking quality between positives and negatives.

Question-answering benchmark for detecting social bias through stereotype reliance under ambiguous context.

Contextual embedding-based similarity metric for scoring generated text against reference outputs.

Corpus-level n-gram overlap metric (MT-originated) used to score generated text against reference translations or summaries.

Bias in Open-ended Language Generation benchmark with Wikipedia-derived prompts across profession, gender, race, religion, and political ideology.

Proper scoring rule measuring mean squared error between predicted probabilities and observed binary outcomes used to assess calibration and reliability.

Reliability diagram showing how predicted probabilities align with observed outcome frequencies across bins.

Healthcare safety benchmark with 18k+ prompts spanning harmfulness levels and jailbreak-style prompt variations to test unsafe compliance and over-refusal.

Bias benchmark using minimal pairs to measure preference for stereotyped vs. anti-stereotyped sentences.

Clinical utility evaluation that compares net benefit across decision thresholds to determine whether a model improves outcomes versus treat-all or treat-none strategies.

Calibration metric that quantifies the discrepancy between predicted probabilities and observed accuracy across probability bins.

Balanced metric that summarizes precision and recall into one harmonic-mean score for classification performance.

Healthcare evaluation suite developed by OpenAI that assesses model behavior in health conversations using physician-authored, conversation-specific rubric criteria.

A comprehensive evaluation framework for language models that standardizes tasks, prompts, metrics, and reporting across diverse tasks, domains, and use cases.

Benchmark for measuring social bias across demographic attributes using templated prompts and model completions.

Aggregates probabilistic prediction errors by penalizing incorrect and overconfident predictions.

Correlation-based metric that accounts for true/false positives and negatives, robust to class imbalance.

A healthcare-focused evaluation suite that assesses large language models across 35 medical benchmarks covering clinical, biomedical, and healthcare-related tasks.

Large-scale multiple-choice medical QA benchmark built from AIIMS/NEET-PG style exam questions, used to evaluate medical knowledge recall and question-level reasoning.

USMLE-style medical multiple-choice QA benchmark (~12k items) evaluating diagnostic reasoning, treatment selection, and contraindication assessment across major clinical domains.

Broad multi-domain benchmark with ~15k questions across 57 subjects that evaluates general knowledge and multiple-choice reasoning.

Multi-turn conversational benchmark evaluated using LLM-as-judge scoring to assess instruction adherence, coherence, and response quality across dialogue turns.

Positive predictive value (precision) and negative predictive value, measuring correctness for predicted positives and negatives.

Recall-oriented overlap metrics (ROUGE-N, ROUGE-L, ROUGE-Lsum) for comparing generated summaries to reference summaries.

Companion metrics measuring true positive rate (sensitivity) and true negative rate (specificity).

Question-answering benchmark designed to detect model falsehoods and overconfident incorrect answers across adversarial prompts.

Benchmark suite of real-world datasets designed to evaluate model robustness under distribution shift, with explicit domain and subgroup metadata.

Customer-service agent benchmark for τ-retail and τ-airline, where agents interact with simulated users, follow domain policies, and use API tools to read and update databases.