MMLU-Pro

Run with Quantiles

Use the following command to run MMLU-Pro in Quantiles using either the built-in demo model or your own hosted AI model:

qt run mmlu-pro

When using a coding agent, install the Quantiles agent skill first, then copy the prompt below into the agent.

Use the Quantiles eval skill to run the MMLU-Pro benchmark and summarize the results.

Overview

MMLU-Pro extends the original Massive Multitask Language Understanding benchmark with harder, more reasoning-focused questions. It removes many trivial and noisy MMLU items, expands the maximum number of answer choices from four to ten, and groups questions into 14 subject categories. These changes reduce the effect of random guessing and make the benchmark more discriminative for capable language models.

The model receives a question and its answer choices. Under the reference protocol, it is also given five category-specific chain-of-thought examples and generates reasoning followed by a final option label. The benchmark authors found that MMLU-Pro scores were more stable across prompt variations than MMLU scores and that chain-of-thought prompting improved performance, consistent with the benchmark containing more questions that benefit from multi-step reasoning.

Dataset Specification

Size

The official MMLU-Pro test split contains 12,032 scored questions. A separate validation split contains 70 examples, five per category, which the reference evaluation uses as few-shot demonstrations. The Quantiles built-in benchmark uses the test split and evaluates 1,000 samples by default.

Source

MMLU-Pro filters out easy, ambiguous, and incorrectly annotated questions from MMLU’s 57 subjects, reorganizes them into 14 categories, and adds harder problems from online STEM sources, TheoremQA, and SciBench. The categories are biology, business, chemistry, computer science, economics, engineering, health, history, law, mathematics, philosophy, physics, psychology, and other.

Input Format

• question: string
• options: list containing up to ten valid answer choices after any N/A placeholders are removed
• answer: correct option label in the dataset, withheld from the model

Example:

{
  "question": "[MMLU-Pro question]",
  "options": [
    "A",
    "B",
    "C",
    "D",
    "E",
    "F",
    "G",
    "H",
    "I",
    "J"
  ]
}

Output Format

The scored output is a single option label from A through J, depending on the choices available for the question. The raw model response may also contain reasoning, depending on the prompting protocol.

{
  "answer": "E"
}

Metrics

• Accuracy (primary): fraction of questions for which the model selects the ground-truth option.
  $$\text{Accuracy} = \frac{\text{correct predictions}}{\text{total scored questions}}$$
• Category accuracy: accuracy reported separately for each of the 14 subject categories.

Known Limitations

• Multiple-choice accuracy does not measure open-ended generation, tool use, or other behaviors that might emerge in production usage.
• Scores can be sensitive to prompt format, answer extraction, and other prompting techniques.
• Broad category averages can hide large differences between subjects and question difficulty.
• Public benchmark data can appear in training corpora, creating a risk that scores reflect contamination rather than generalization.
• Performance on academic multiple-choice questions does not establish real-world safety, professional competence, or deployment readiness.
• MMLU-Pro is an English-language, text-only benchmark and does not evaluate multilingual or multimodal capabilities.

Versioning and Provenance

Make sure to record the dataset, evaluation protocol, and runtime configuration for every run. Include details such as the exact dataset revision and split, sample-selection method, prompt and chain-of-thought settings, answer-extraction policy, model configuration, and Quantiles version. When you use the qt CLI, many of these values are saved for you. These values are important for reproducibililty because MMLU-Pro has received answer corrections and formatting updates since its initial release, and results are directly comparable only when these settings are aligned.

References

Wang et al., 2024. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark.

Paper: https://arxiv.org/abs/2406.01574

GitHub Repository: https://github.com/TIGER-AI-Lab/MMLU-Pro

Related Benchmarks