CLI Reference

The qt CLI runs local evaluation workflows. It stores run metadata in the local workspace, starts a local HTTP server when needed, records workflow steps and metrics, and compares runs from the command line.

Commands

Include --json at the end of any command when you need structured output with detailed run information, including sample-level results. JSON output is recommended for agents and scripts that need to parse evaluation results.

qt init

Initialize a local Quantiles workspace explicitly:

qt init

This creates a .quantiles/ directory under the current project directory, including:

• .quantiles/quantiles.sqlite for workflow metadata
• .quantiles/metrics/ for metrics and benchmark analytics data

qt run creates the workspace automatically when needed, so you’ll rarely need to run qt init.

Example output:

Initialized Quantiles workspace at .quantiles/quantiles.sqlite

qt run

Run a benchmark or evaluation:

qt run <eval_name>

To run a custom_code evaluation, qt run does the following:

1. Starts up the local Quantiles REST API server.
2. Runs the configured command with Quantiles environment variables.
3. Records step inputs, outputs, and metrics.
4. Saves the workflow output.
5. Stops the temporary server.
6. Records whether the workflow succeeded or failed.

For custom_nocode evaluations, qt run simply runs the evaluation internally.

Structured input

Pass JSON input with --input:

qt run eval-name --input '{"model":"gpt-5-nano", "limit":10}'

SDK workflows receive this JSON through the environment. In the current implementation, qt run injects:

The default local server URL is http://127.0.0.1:8765. You can override it by setting QUANTILES_BASE_URL.

JSON output

Use --json when calling qt run from scripts:

qt run eval-name --json

In JSON mode, qt run prints machine-readable output.

For built-in benchmarks, the output includes:

• run_id
• aggregate_metrics
• remote_hash
• warning, when applicable

JSON output for a built-in benchmark includes its resolved version and manifest hash. Because the manifest hash is not currently stored in the run record, record it separately when exact benchmark provenance is required.

For custom code evaluations, child process stdout and stderr are captured and returned inside one machine-readable JSON object. The output includes:

Run metadata

• run_id
• workflow_name
• input
• command
• base_url

Execution state

• server_started_by_us
• status
• success
• exit_code
• duration_seconds

Process output

• stdout
• stderr
• error

Quantiles marks a run as completed when the child process exits successfully and failed when it does not. qt run records the child process result, but intentionally does not propagate it as the CLI exit code.

For automation, inspect success, exit_code, or the saved run status.

See Built-in Benchmarks for details on running or configuring built-in benchmarks, Custom configuration evaluations to learn more about building custom evaluations without writing or maintaining code, and Custom code evaluations for guidance on building highly specialized custom evaluations with the Quantiles Python SDK.

Resume an interrupted or failed run

Use qt resume to continue an interrupted workflow, restore completed steps from cache, and rerun only failed or incomplete steps.

qt resume <run_id>

You cannot resume a run if it’s marked as status: completed in qt list or qt show output. To re-run it, do another qt run.

See Resume Runs for details on recovering interrupted or failed runs.

qt list

List workflow runs in reverse chronological order:

qt list

The output includes run ID, eval name, status, sample count, creation time, and duration.

Example output:

ID  EVAL            STATUS     SAMPLES    CREATED                      DURATION
2   support-triage  completed  1000       2026-06-19T18:30:00.000000Z  3.000s
1   support-triage  failed     642        2026-06-19T18:15:00.000000Z  1.000s

Use --json for machine-readable output:

qt list --json

qt show

Inspect one workflow run:

qt show <run_id>

By default, qt show prints run metadata and metrics:

• Eval name
• Status
• Creation time and duration
• Input
• Output
• Error
• Metrics

Machine-readable output

Use --json to inspect structured run details from scripts or agents:

qt show <run_id> --json

JSON output includes the run metadata, input, output, metrics, and sample-level results in a machine-readable format.

See Evaluation Results for details on inspecting evaluation results.

qt compare

Compare two workflow runs with IDs <run_id_a> and <run_id_b>:

qt compare <run_id_a> <run_id_b>

qt compare checks:

• Workflow input
• Final workflow output
• Step presence
• Step input hash changes
• Step status changes
• Step output changes
• Emitted metrics

If you try to compare a run against itself, the qt CLI will output an error and exit with code 1.

The qt CLI displays a warning when comparing runs where the evaluation / benchmark names differ.

Exit codes

qt compare exits with:

• 0 when the compared runs are identical
• 1 when the compared runs differ

See Compare Evals for details on comparing evaluation runs.

qt add

Import a built-in benchmark configuration and prompt from the hosted Quantiles benchmark registry into quantiles.toml:

qt add <benchmark_name>

This command:

• Appends the benchmark configuration to a quantiles.toml or .quantiles.toml file.
  • Creates quantiles.toml if neither configuration file exists.
• Creates <benchmark_name>-prompt/prompt.txt and configures the benchmark to use that local prompt.
• Preserves the existing configuration content when appending the new benchmark.

The command exits with an error if both quantiles.toml and .quantiles.toml exist, the requested benchmark is already configured locally, the benchmark is not found in the registry, or there is no internet connection.

For example, add MMLU-Pro with the following command:

qt add mmlu-pro

See Built-in Benchmarks for configuration details and the Quantiles Benchmark Hub for the available built-in benchmarks.

qt serve

Start the local HTTP server manually. The server binds to 127.0.0.1:8765 by default:

qt serve

Most workflows do not need qt serve directly because qt run starts a temporary server when one is not already reachable. Use qt serve when you want a persistent local server for multiple shells, SDK experiments, or direct REST API calls.

Use another address

Use --addr to choose a different address:

qt serve --addr 127.0.0.1:9000

Common evaluation workflows

Run an evaluation and show the results

# Run the evaluation, show a summary of results, and get the associated
# `run_id`
qt run <eval_name>
 
# Use the `run_id` from above to show the same summary of results
qt show <run_id>
 
# Show results summary and sample-level results, all in machine-readable JSON
qt show <run_id> --json

Compare two evaluations

# Run the first evaluation to create a new run record and `run_id_a`
qt run <eval_name>
 
# Run the second evaluation to create a new run record and `run_id_b`
qt run <eval_name>
 
# Use the corresponding `run_id`s to compare the evaluations
qt compare <run_id_a> <run_id_b> --json

Resume an interrupted for failed evaluation

qt resume <run_id>