GPQA Diamond

Run with Quantiles

Use the following command to run GPQA Diamond in Quantiles using either the built-in demo model or your own hosted AI model:

qt run gpqa

When using a coding agent, install the Quantiles agent skill first, then copy the prompt below into the agent.

Use the Quantiles eval skill to run the full GPQA Diamond benchmark and summarize the results.

Overview

GPQA, or Graduate-Level Google-Proof Q&A, is a multiple-choice benchmark of difficult biology, physics, and chemistry questions written and validated by domain experts. GPQA Diamond is its 198-question, highest-quality subset, selected for strong expert agreement and difficulty for skilled non-experts. It evaluates advanced scientific question answering and supports research on scalable oversight, but it does not represent the full range of scientific practice or real-world research workflows.

On GPQA Extended, same-domain experts achieved approximately 65% accuracy, while highly skilled validators from other domains achieved 34% despite spending an average of 37 minutes per question with unrestricted internet access but no LLM assistance.

Dataset Specification

Size

The main GPQA set contains 448 questions, while GPQA Extended contains 546. GPQA Diamond contains 198 questions meeting stricter quality and difficulty criteria.

Source

Questions were written by experts who had or were pursuing PhDs in the relevant field. Each question was reviewed by a same-domain expert, revised by its author, and then evaluated by a second same-domain expert and three highly skilled validators from other domains.

Input Format

• Question: graduate-level science question
• Correct Answer: ground-truth answer, withheld from the model
• Incorrect Answers: three plausible but incorrect answer choices
• Record ID: unique identifier for the dataset record

Before evaluation, the correct answer and three incorrect answers are shuffled and mapped to labels A through D.

The following example uses synthetic placeholders because GPQA’s authors request that official questions not be shared online.

{
  "question": "[GPQA Diamond question]",
  "options": [
    "[Shuffled option A]",
    "[Shuffled option B]",
    "[Shuffled option C]",
    "[Shuffled option D]"
  ]
}

Output Format

The scored output is a single option label: A, B, C, or D. The raw model response may also contain reasoning, depending on the prompting protocol.

{
  "answer": "A"
}

Metrics

• Accuracy (primary): fraction of questions for which the model selects the ground-truth option. Uniform random guessing has an expected accuracy of 25%.
  $$\text{Accuracy} = \frac{\text{correct predictions}}{\text{total scored questions}}$$

Known Limitations

• GPQA Diamond has only 198 questions, so score uncertainty can be substantial and small differences between models may not be meaningful.
• Coverage is limited to biology, physics, and chemistry and should not be generalized to all graduate-level knowledge.
• Multiple-choice scoring does not evaluate experimental design, literature synthesis, long-form explanation, or tool use.
• Results depend on prompt style, reasoning instructions, answer extraction, and answer-choice shuffling.
• High benchmark accuracy does not establish scientific reliability or readiness for unsupervised research.
• Exposure of GPQA questions creates a contamination risk.
• GPQA is an English-language, text-only benchmark and does not evaluate multilingual or multimodal scientific reasoning.

Versioning and Provenance

Record the dataset revision, subset, and split, prompt and reasoning protocol, answer-extraction policy, choice-shuffling method and seed, model settings, and evaluation framework version for every run. GPQA results are directly comparable only when these settings are aligned, particularly because answer-choice order can affect accuracy.

References

Rein et al., 2024. GPQA: A Graduate-Level Google-Proof Q&A Benchmark.

Paper: https://arxiv.org/abs/2311.12022

GitHub Repository: https://github.com/idavidrein/gpqa

Related Benchmarks