Run and Customize Public AI Benchmarks

Quantiles Built-in benchmarks are ready-to-run evaluations with predefined datasets, prompts, and scoring methods. The Quantiles benchmark registry now includes SimpleQA Verified, GPQA Diamond, MedMCQA, MedQA, MMLU-Pro, and PubMedQA, with more benchmarks planned. You can run any benchmark with a single qt run command or add its definition to your configuration file for customization, such as adding an AI model to evaluate.

New built-in benchmarks added

Quantiles now includes five new built-in public benchmarks covering advanced scientific reasoning, broad academic knowledge, medical board-style questions, and question answering grounded in biomedical literature. These benchmarks are useful public baselines for model comparison, regression testing, and evaluation workflow validation.

The Quantiles Benchmark Hub includes additional information about each benchmark and its metrics. Below is the current list of benchmarks available to run through the hosted Quantiles benchmark registry.

Run a built-in benchmark

Built-in benchmarks run with a demo model by default unless another model is configured. This provides a credential-free way to validate installation, execution, scoring, and result inspection.

Get started with two commands:

• 1. Install the Quantiles CLI
  curl -fsSL https://cli.quantiles.io/install.sh | bash
• 2. Run the benchmark
  qt run simpleqa-verified

After the benchmark completes, use qt list to find its run ID, inspect its sample-level results with qt show <run_id>, and compare two runs with qt compare <run_id_a> <run_id_b>. That same command sequence works whether you're running a built-in benchmark, a custom configuration evaluation or a custom code evaluation.

See the Quickstart for more information about installation, configuration, and benchmark execution.

Customize a built-in benchmark

Built-in benchmark settings can be changed with one-time overrides or reusable configurations that apply to all future runs of that benchmark.

One-time override with --input

Use --input when you want to override your eval configuration once. You can use this flag, for example, to replace the demo model with a hosted AI model or change the number of samples to run while preserving the original benchmark configuration. It's often useful for doing one-off testing on a small sample size or trying your eval against a new AI model.

This example configures MMLU-Pro to use an OpenAI model and run only a subset of the benchmark.

qt run mmlu-pro --input '{"model":"openai:gpt-5.6-luna","limit":10}'

Configure persistent benchmark settings

To persistently customize a built-in benchmark, use qt add <benchmark-name> to add its configuration to quantiles.toml or.quantiles.toml file. The command updates an existing configuration file or creates one if neither file exists. You can then edit the local benchmark configuration, and Quantiles will use those settings for future runs of that benchmark.

qt add mmlu-pro

Once the benchmark is added to the quantiles.toml, you can configure:

• model: the model to evaluate
• samples: the maximum number of dataset rows to evaluate
• max_workers: the number of rows that can be evaluated concurrently. Lower max_workers if the model provider enforces stricter rate limits.

The following is an example MMLU-Pro configuration:

[benchmarks.mmlu-pro]
type = "custom_nocode"
dataset = "hf://quantiles/MMLU-Pro"

# Evaluate the first 50 rows rather than the full dataset.
samples = 50

# Use a hosted model instead of the demo model.
model = "openai:gpt-5.6-luna"

# Reduce concurrency if the provider throttles requests.
max_workers = 10

To preserve the default registry configuration, give the customized benchmark in quantiles.toml a different name, such as my-mmlu-pro. You can then run the registry version with qt run mmlu-pro and the customized version with qt run my-mmlu-pro. Distinct names make each configuration identifiable in run history and help prevent comparisons between runs that used different settings.

Run and configure built-in benchmarks with agents

Coding agents such as Codex and Claude Code can use the Quantiles agent skill to run, customize, analyze, and compare built-in benchmarks. The Agent Quickstart provides complete instructions, or use the steps below to get started quickly:

• 1. Install the Quantiles skill
  Please install the Quantiles skill at github.com/quantiles-evals/skill
• 2. Run a built-in benchmark
  Use the Quantiles skill to run the <benchmark-name> and summarize the results.

Documentation & Resources

• Built-in benchmarks: Learn how to run and customize benchmarks from the Quantiles benchmark registry.
• Quantiles quickstart: Install the CLI, run a benchmark, and inspect the resulting run record.
• Benchmark Hub: Explore available benchmarks, their datasets, and reported metrics.

Keep reading

View all

How to build Custom No-code AI Evaluations

ProductJuly 28, 2026

Quantiles: Open-Source, Local-First Infrastructure for AI Evaluation Workflows

ProductJune 30, 2026

Evaluating Emergency Recognition in Patient-Facing AI Systems

EvaluationsApril 23, 2026