Evaluation Quickstart

Quantiles lets teams run benchmarks and evaluations from the command line, inspect sample-level outputs, compare runs against baselines, and define custom evaluations with minimal setup.

Install the CLI

To install the Quantiles CLI, run the following command:

curl -fsSL https://cli.quantiles.io/install.sh | bash

Because Quantiles is local by default, your project directory holds the database for run history, metadata, samples, and metrics, all under .quantiles/.

Run a sample benchmark

Run a simpleqa-verified sample benchmark with the built-in demo model to validate the installation and inspect how Quantiles runs evaluations and records inputs, outputs, and results.

qt run simpleqa-verified

The SimpleQA-verified benchmark tests 1,000 short-form factuality prompts for testing parametric knowledge. Learn more about SimpleQA Verified.

The above command uses a demo model which generates random text, does not call an external model provider, and does not incur inference charges. Runs using the demo model are only useful for validating the workflow. See Built-in Benchmarks for more details.

Inspect the benchmark results

The first Quantiles run in a new workspace will have a run_id of 1. Use the run_id to inspect the complete run record, including workflow inputs, outputs, sample-level results, metrics, and execution metadata, in JSON format.

qt show 1 --json

You can omit the --json flag to see a human-readable summary of the run.

Use qt list to view run history and find a run_id to analyze

Run a benchmark against your own model

Use --input to run a built-in benchmark, such as SimpleQA Verified, with a model other than the default demo model. See Model Configuration for supported providers and model identifiers.

First, set the credential required by the provider. This example uses OpenAI:

export OPENAI_API_KEY="<your_openai_api_key>"

Then pass the model directly to the evaluation:

qt run simpleqa-verified --input '{"model":"openai:gpt-5.6-luna"}'

This model override applies only to the current evaluation run. To reuse a model across evaluations, configure it in a quantiles.toml configuration file.

Compare benchmark runs

Use the qt CLI to compare and analyze the differences between two benchmark runs:

qt compare <run_id_a> <run_id_b>

See Compare Evaluations for full documentation on run comparison.

Custom evaluations

Custom evaluations measure behavior specific to your product, workflow, prompt, dataset, rubric, or release process. You can define them in configuration without writing any logic in Python. Both run through the qt CLI and use the same recording, analysis, and comparison workflow.

Custom configuration evaluation

Quantiles currently supports several custom configuration evaluation types defined in a quantiles.toml or .quantiles.toml configuration file:

• Exact match: an evaluation that compares the model’s response with an expected answer.
• Multiple choice: an evaluation that extracts and scores the model’s selection from a configured set of answer choices.

Custom code evaluation

Use a custom code evaluation when the behavior cannot be captured by a custom configuration evaluation or an existing built-in benchmark. See the custom code evaluation documentation for information about configuring and running custom code evaluations, including how to implement them with the Python SDK.

Use Coding Agents to Run Evaluation Workflows

For an agent-assisted workflow, install the Quantiles skill and use your preferred coding agent to run evaluations. The skill works with agents such as Codex, Claude, and others.

See the following guides to get started with coding-agent workflows in Quantiles.

• Agent Quickstart

  Use coding agents to create, run, inspect, compare, and analyze evals

• Agent Overview

  Learn how agents fit into the Quantiles evaluation workflow

Useful Commands

See the CLI Reference for full command details and output examples.