Model configuration

Quantiles can run evaluations with models from supported providers such as OpenAI and Anthropic. You can configure a provider model by doing the following two things:

• Setting the evaluation’s model field, in the configuration file or --input flag, to a provider-prefixed model identifier.
• Providing the required API credentials through environment variables.

Provider-backed evaluation runs make network requests, can send rendered prompts and dataset content to the selected provider, and may incur usage charges. Review the provider’s pricing and data-handling policies before running evaluations.

Built-in demo model

When no model is configured, evaluation runs use the built-in demo model by default. It provides a fast, credential-free way to test an evaluation end to end before connecting a hosted model.

Use the demo model to:

• Validate the complete evaluation workflow
• Inspect recorded inputs, outputs, scores, and metrics
• Exercise run inspection and comparison workflows
• Test coding-agent, local, or CI automation

The demo model produces random results that may vary between runs. For example, in a multiple-choice evaluation such as PubMedQA, it randomly selects yes, no, or maybe. Therefore, do not use demo results to:

• Measure or compare model quality
• Assess prompt changes
• Detect model regressions
• Estimate production performance

In built-in benchmarks and custom no-code benchmarks, the demo model is the default, so you can omit model from both the quantiles.toml configuration and the JSON passed through --input to use it. Custom code evaluations may override the model in code.

In code and configuration examples below and in other parts of Quantiles documentation, you may see a model = "random" value in TOML or a "model": "random" key/value pair in JSON to make the demo-model selection explicit for illustrative purposes.

Supported model providers

Quantiles supports the model providers listed below. Use the exact model identifier accepted by the provider API and available to the configured account. Quantiles does not maintain an internal model allowlist. Open a GitHub issue  in the Quantiles OS repository to request another provider integration or report an incompatible model.

The provider prefix and environment variable names are exact and case-sensitive.

Refer to the provider catalogs below for current model identifiers.

• OpenAI models 
• Anthropic models 
• Gemini models 
• Cloudflare Workers AI models 

Provider-specific configuration

OpenAI

First, set OPENAI_API_KEY:

export OPENAI_API_KEY="<your_openai_api_key>"

Then use the openai: prefix in your configuration file:

model = "openai:gpt-5.6-luna"

Your OpenAI project must have access to the configured model. See the OpenAI example in the Quantiles repository  for a complete built-in benchmark configuration.

Anthropic

First, set ANTHROPIC_API_KEY:

export ANTHROPIC_API_KEY="<your_anthropic_api_key>"

Then use the anthropic: prefix in your configuration file:

model = "anthropic:claude-fable-5"

See the Anthropic example in the Quantiles repository  for a complete built-in benchmark configuration.

Google Gemini API

First, set GEMINI_API_KEY:

export GEMINI_API_KEY="<your_gemini_api_key>"

Then use the gemini: prefix in your configuration file:

model = "gemini:gemini-3.5-flash-lite"

The gemini: provider uses the public Gemini API. See the Gemini example in the Quantiles repository  for a complete built-in benchmark configuration.

Cloudflare AI Gateway

Cloudflare’s integration routes requests to Workers AI  through AI Gateway . First, set the API key, account ID, and gateway ID:

export CLOUDFLARE_API_KEY="<your_cloudflare_api_key>"
export CLOUDFLARE_ACCOUNT_ID="<your_cloudflare_account_id>"
export CLOUDFLARE_GATEWAY_ID="<your_cloudflare_gateway_id>"

Then, in your configuration file, use the cloudflare_ai_gateway: prefix followed by a Workers AI model identifier:

model = "cloudflare_ai_gateway:@cf/meta/llama-3.3-70b-instruct-fp8-fast"

If you prefer, you can put your account ID and gateway ID values into your configuration file instead of storing them in environment variables:

model = { type = "cloudflare_ai_gateway:@cf/meta/llama-3.3-70b-instruct-fp8-fast", account_id = "<account_id>", gateway_id = "<gateway_id>" }

CLOUDFLARE_API_KEY must still always be set in the environment. See the Cloudflare example in the Quantiles repository  for a complete configuration example.

Rate limits and request concurrency

Quantiles processes samples in parallel and may send multiple requests to a model provider at the same time. For built-in benchmarks and custom no-code evaluations, max_workers controls the maximum number of dataset rows processed concurrently. When neither max_workers nor QUANTILES_MAX_WORKERS is set, Quantiles uses 25 workers by default, but max_workers can be increased for greater throughput or reduced to limit request concurrency.

For all types of benchmarks, the max_workers value can be set in quantiles.toml:

#  Configure concurrency for a specific model
[benchmarks.my-eval]
model = "openai:gpt-5.6-luna"
 
# Set this value to a positive integer. The default is 25.
# Higher values might trigger provider rate limits.
max_workers = 10

If you need to temporarily change the max_workers value for a built-in benchmark, you can include a max_workers value in the --input flag.

qt run builtin-benchmark --input '{"max_workers":5}'

When specifying max_workers, keep the following details in mind:

• If you specify --input for a built-in benchmark, you will override any of that benchmark’s pre-existing configuration in a quantiles.toml file.
• For custom benchmarks, you must specify max_workers only in the configuration file. You cannot pass it on the command line via the --input flag.
• max_workers is a concurrency limit only, not a requests-per-minute or tokens-per-minute throttle.
• Provider limits vary by model, project or account, and usage tier, and may apply to request rates, input or output tokens, daily usage, spend, or available capacity.
• A low max_workers value can reduce request bursts but does not guarantee that a run will remain within every provider limit.

Use the following guidance to tune concurrency and recover from provider limits:

Check the selected provider for the limits that apply to your account and model:

• OpenAI API rate limits 
• Anthropic API rate limits 
• Gemini API rate limits 
• Cloudflare Workers AI limits 

Configure and verify a model

The following example configures a custom evaluation called “my-eval” to use an OpenAI model.

1. Set the provider credential

Set the API key in the environment where you run qt. For example, to use an OpenAI model:

export OPENAI_API_KEY="<your_openai_api_key>"

Quantiles always reads provider API keys from the environment. You can use your shell environment, CI secret store, or preferred secret manager to provide them. Never include any API keys in any quantiles.toml file, commit them to source control, print them in logs, include them in agent prompts, or otherwise persist them in clear text anywhere outside your local machine.

2. Add the model

Model configuration can be persistent or run-specific. Define the model in the configuration file to use it as the default for subsequent runs of the evaluation, or apply a one-time override for an individual evaluation run using --input.

Add the model to a quantiles.toml configuration file

Use a configuration file for built-in benchmarks or custom no-code evaluations when you want Quantiles to reuse the same model across multiple evaluation runs. This is the recommended approach because it provides a shared, reviewable default for teammates, coding agents, and CI workflows.

# Example configuration for running my-eval with an OpenAI model
[benchmarks.my-eval]
 
# The string before the first colon selects the provider.
# The remaining string is the provider's model identifier.
model = "openai:gpt-5.6-luna"

Note that custom code evaluations do not currently accept this model field because the custom code is responsible for configuring and issuing RPCs to the model provider. If your custom code evaluation expects a model identifier, pass it through [benchmarks.<name>.input] and configure the provider client and credentials in Python. See the Python SDK reference for more details.

Pass the model with --input

For custom no-code evaluations, values passed through --input, such as model and limit, apply only to the current evaluation run. All remaining evaluation settings continue to come from quantiles.toml, making this approach useful for smoke tests, model comparisons, and temporary experiments.

# Example input for running my-eval with an OpenAI model
qt run my-eval --input '{"model":"openai:gpt-5.6-luna"}'

For built-in benchmarks, all configuration specified in the quantiles.toml file will be ignored when an --input flag is passed.

Configure with a coding agent

Use the Quantiles skill  to have a coding agent update the model configuration. Replace the placeholders with the provider, model ID, and evaluation name you want to configure:

Add <model provider and model_id> to the existing custom no-code evaluation <eval_name>. Preserve the other evaluation settings. Confirm that the required provider credentials are available without showing their values; if they are not, stop and tell me what is missing. When finished, summarize what you changed. Do not run the evaluation until I confirm.

Modify the above to reference if your <eval_name> identifies a built-in benchmark.

See Agent Prompts for more coding-agent workflows.

3. Run and inspect the evaluation

If you configured the model in quantiles.toml, run the evaluation with the standard qt run command below, optionally appending a --json flag if you want to parse the output:

qt run my-eval

Cost and data handling

Provider-backed evaluations send rendered prompts to an external model API and may incur usage costs or expose evaluation data to the selected provider. Before starting a full run, use a small subset of samples to validate that the run works, and also confirm the provider’s access and data-handling policies.

The following tips can help control costs and protect evaluation data:

• Run a small smoke test, such as 10 samples, to verify credentials, model access, prompt rendering, response parsing, and scoring.
• Estimate usage before a full run based on the sample count, prompt size, expected response length, model pricing, and any additional model-based scoring.

Quantiles stores run history and evaluation artifacts locally, but requests to provider-backed models require network access and are processed outside the local environment. See Local-first and Offline and Security and Privacy for details.

Current limitations

The built-in provider configuration currently has these limitations:

• Quantiles uses standard, public APIs and fixed API-key auth for the OpenAI, Anthropic, and Gemini platforms.
• Custom base URLs, OpenAI-compatible proxies, local gateways, and per-benchmark API keys are not currently supported.
• The gemini: provider does not currently support Google Cloud Vertex AI or Application Default Credentials (ADC).

If you or your agent want to look at the code the qt client uses to issue calls to provider APIs, please see the following code in the Quantiles GitHub repository :

• The model configuration parser 
• Provider implementations 

Troubleshooting

Provider-backed evaluation runs can fail before or during execution because of missing credentials, model-access restrictions, provider limits, or unsupported endpoint configurations. Use the table below to identify the likely cause and the appropriate next step before retrying the evaluation.

Resume an interrupted or failed evaluation run

If a provider-backed evaluation is interrupted by throttling, a rate limit, a timeout, a network error, or another transient failure, first inspect the run using its run_id:

qt show <run_id>

Resolve the error using the troubleshooting guidance above. For example, wait for a provider limit to reset or correct a temporary network issue. Then resume the same run:

qt resume <run_id>

After resuming, inspect the run again to verify its final status and review the results:

qt show <run_id>

Append --json to these commands for agent- or machine-readable output.

The qt resume command will reuse the stored output of each completed step, and incomplete steps will execute again. Built-in benchmarks and custom no-code evaluations both wrap their provider calls in steps, so this means successfully completed provider calls won’t be repeated.

Resume a run only when recovering the same evaluation. If you want to intentionally change the model, prompt, dataset, rubric, or another behavior-changing input, remember to start a new run instead.

See Workflows and Steps for more information about step caching and Restart and Resume Runs for resume eligibility and recovery guidance.