Skip to Content
Core Concepts

The Quantiles configuration file

Quantiles uses a single quantiles.toml or .quantiles.toml file in the current working directory to configure built-in benchmarks and define custom configuration evaluations. The file specifies how evaluations are loaded and executed, including datasets, models, prompts, scoring methods, inputs, and runtime settings.

Only one of the two filenames can exist in the same directory. If both exist, the CLI will exit with an error.

When to use a configuration file

Create or configure a quantiles.toml or .quantiles.toml configuration file when you want to do any of the following:

Built-in benchmarks can run without a quantiles.toml configuration file using their default settings. See Built-in Benchmarks for the resolution order and network behavior.

File name and location

The qt CLI looks for either quantiles.toml or .quantiles.toml in the current working directory. If neither exists, it searches ancestor directories and uses the first matching file it finds. To configure Quantiles for your project, we recommend adding one of these files to the project’s working directory.

File structure

Every benchmark and evaluation definition lives under its own [benchmarks.<eval_name>] section. The section key is the evaluation name passed to qt run <eval_name>.

For example, to change the default model and sample count for all future runs of the built-in SimpleQA Verified benchmark, add the following section to your configuration file:

# Configure the sample limit and model for SimpleQA Verified.
[benchmarks.simpleqa-verified]
dataset = "hf://quantiles/simpleqa-verified"
 
# Limit the number of samples evaluated.
samples = 50
 
# Use OpenAI's GPT 5.6 Luna model instead of the built-in
# demo model. This requires an OPENAI_API_KEY environment
# variable, and OpenAI will charge you for usage.
model = "openai:gpt-5.6-luna"

Configuring built-in and custom evaluations

Built-in benchmarks and custom evaluations have different required and optional configuration fields. Quantiles uses the following defaults for these fields when they are not specified:

  • type="built-in"
  • model="random"
  • samples defaults to all samples

Refer to the documentation for each evaluation type to identify its required and optional configuration fields.

A benchmark defined in quantiles.toml takes precedence over the definition in the Quantiles registry when both use the same name. To keep both versions available, assign the customized benchmark a different name. For example, the registry name for SimpleQA Verified is simpleqa-verified, so you could name the customized version my-simpleqa-verified. You may choose any name as long as it does not match the built-in benchmark name.

CLI --input overrides

To apply a one-time configuration override, pass --input to qt run. These values are recorded in the local run history but are not written to the configuration file or applied to subsequent runs. For example, the following command overrides the model for one evaluation run:

qt run my-eval --input '{"model":"openai:gpt-5.6-sol"}'

In --json mode, a warning is included in the JSON output if there is an --input flag. For example:

Warning: --input overrides config input for keys: model

Configuration validation

The CLI validates benchmark and evaluation configuration fields before execution:

  • built-in benchmarks:

    • May not contain command or input fields.
    • Must include dataset with an hf:// or huggingface:// prefix.
    • May include fields such as samples, model, and max_workers.

  • custom_code evaluations:

    • Must have a non-empty command array.
    • May not contain built-in-only fields such as samples or model.

  • custom_configuration evaluations:

    • Must include dataset, style, and prompt_template_file.
    • prompt_template_file must point to an existing file.
    • Must set style.type to either exact_match or multiple_choice.
    • When style.type = "exact_match", a style.golden_column field must be present.
    • When style.type = "multiple_choice", a valid choice source, answer source, and a non-empty list of unique choice_labels must be present in the style dictionary.
    • May not contain command, input, or other unsupported fields.

Validation failures produce clear error messages before any run is created.

How qt resume uses the configuration file

When you run qt resume <run_id>, the CLI reuses the same configuration from that run, including input, from the database. It also re-reads the command from the configuration file. This setup means the following:

  • You do not need to re-submit input when resuming a previous run.
  • If you update the command between qt run and qt resume, the resumed run uses the updated command, but retains the input values used in the original run.
  • If a custom_code or custom_configuration configuration section is removed after a qt run, resuming that run will fail with a clear error.
  • For built-in benchmarks loaded from the Quantiles registry, resuming requires internet access to retrieve the benchmark configuration.

See Restart and Resume Runs for the full recovery workflow.

Last updated on