Evaluate

Built-in Benchmarks

Built-in benchmarks are ready-to-run evaluations with predefined datasets, scoring methods, and metrics. Use them when you want a common reference point, a repeatable baseline, or a maintained implementation of a public benchmark.

Built-in benchmarks use the demo model by default to validate the installation and inspect evaluation workflow, execution steps, recorded inputs and outputs, scoring behavior, and reported metrics without calling an external model API or incurring usage charges.

Use the following command pattern to run a built-in benchmark:


qt run <benchmark-name>

Available Built-in Benchmarks

Browse the Benchmark Hub for the available benchmarks and their documentation.

Suggestions for new open source benchmarks are always welcome. To propose one for inclusion in the Benchmark Hub and support through the qt CLI, open a GitHub issue with its name, source dataset or repository, and any available reference implementation.

How it works

Running qt run <benchmark-name> resolves the benchmark in this order:

Quantiles first checks quantiles.toml or .quantiles.toml for a local definition matching the requested benchmark name. A matching local definition may be:

A built-in benchmark definition cached during a previous run.
A locally customized version of the built-in benchmark.

If no local definition exists, Quantiles queries the remote benchmark service, which requires internet access.

After resolving the built-in benchmark, qt run:

Selects the benchmark dataset and loads its examples into the local run.
Runs the benchmark using either the configured hosted model or the built-in demo model.
Scores each example with the benchmark’s evaluation metric.
Records per-example results and emits aggregate metrics.

Inspect and compare results

Every built-in benchmark run is recorded locally. The following operations use the saved inputs and outputs retrieved from local storage.

qt list to see a history of evaluation runs, including their run IDs.
qt show <run_id> to inspect sample-level results and metrics.
qt compare <run_id_a> <run_id_b> to compare two evaluation runs.

See Inspect Evaluation Results and Compare Evals for complete the workflows.

Apply one-time configuration overrides

Use --input to apply one-time overrides without changing the built-in benchmark definition or adding the benchmark to a quantiles.toml. The following example replaces the demo model with an OpenAI model and evaluates 10 samples instead of running the full benchmark:


qt run gpqa --input '{"model":"openai:gpt-5.6-luna","limit":10}'

These overrides apply only to the current run. For hosted models, use a small sample limit first to confirm that the evaluation works as expected before running the full benchmark.

Apply persistent configuration settings

Built-in benchmark settings can be customized in a quantiles.toml configuration file so they persist across future runs. Unlike --input, which applies only to the current run, this uses the same configuration workflow as a custom configuration evaluation.

A benchmark defined in quantiles.toml takes precedence over the definition in the Quantiles registry when both use the same name. To keep both versions available, assign the customized benchmark a different name. For example, the registry name for SimpleQA Verified is simpleqa-verified, so you could name the customized version my-simpleqa-verified. You may choose any name as long as it does not match the built-in benchmark name.

Run the following command to add a benchmark definition to your configuration. Quantiles updates the existing quantiles.toml or .quantiles.toml, or creates a configuration file if neither exists.


qt add <benchmark-name>

At minimum, a built-in benchmark configuration must define the type, dataset, and model. Quantiles applies default values for any optional fields that are omitted.

Field	Type	Required	Description	Default
`type`	string	yes	Selects the benchmark runner. Must be set to `"custom_nocode"`.	`"custom_nocode"`
`dataset`	string	yes	Hugging Face dataset source. Must begin with `hf://` or `huggingface://`.	Benchmark-specific dataset
`model`	string or table	yes	Model used to generate responses. See Model Configuration for supported providers, credentials, and examples.	Benchmark-specific demo model
`samples`	integer	no	Maximum number of dataset rows to evaluate, in their original order.	All samples available in the benchmark’s dataset
`max_workers`	integer	no	Maximum number of dataset rows evaluated concurrently.	25, unless `QUANTILES_MAX_WORKERS` is set

Example

The following example customizes the built-in simpleqa-verified benchmark in quantiles.toml:


[benchmarks.simpleqa-verified]
type = "custom_nocode"
 
# Built-in dataset sources require an explicit Hugging Face prefix.
dataset = "hf://quantiles/simpleqa-verified"
 
# Limit the benchmark to the first 50 samples. Omit this
# field to run the full benchmark.
samples = 50
 
# Use OpenAI's GPT 5.6 model instead of the built-in
# demo model. This requires an OPENAI_API_KEY environment
# variable, and OpenAI will charge you for usage.
model = "openai:gpt-5.6-luna"

Resume Interrupted Benchmark Runs

As described in the documentation about resuming runs , Quantiles can resume evaluation runs if they get prematurely interrupted by rate limits, timeouts, terminated processes, or anything else. If a partially-completed run is resumed, only its steps that were not successfully completed will be run. For built-in benchmarks loaded from the Quantiles registry originally, resuming requires internet access to retrieve the benchmark configuration.


qt resume <run_id>

See Resume Runs for more details.