Get Started with Quantiles

Quantiles is a local-first CLI and SDK for running AI evaluation workflows with fast, continuous feedback. Teams can iterate on model behavior, prompts, agent workflows, and debugging scripts while preserving the metrics and run histories needed to understand what improved, what regressed, and why.

• Evaluation Quickstart

  Run an evaluation with a single command

• Agent Quickstart

  Use coding agents to run, inspect, compare, and summarize evals

With Quantiles, teams can rely on built-in infrastructure for creating, running, recording, analyzing, and comparing AI evaluations:

• Local execution and open storage through the qt CLI, with evaluation execution and run data kept locally by default.
• Automatic run recording for steps, metrics, events, inputs, outputs, errors, timing, and evaluation configuration.
• Sample-level inspection and comparison using traces, intermediate outputs, metric differences, changed outputs, failed steps, and regression analysis.
• Resilient execution with caching, durable step reuse, and resumable runs.
• Efficient dataset handling for loading, limiting, slicing, and iterating over evaluation datasets.
• Model sampling infrastructure for demo and provider-backed models.
• Scoring and grading with deterministic metrics, reference checks, rubrics, model judges, and custom evaluators.
• Coding-agent integration with reusable instructions for creating, configuring, running, inspecting, comparing, debugging, resuming, and summarizing evaluations.

This architecture lets teams run O(10,000)-sample evaluations immediately, without maintaining and deploying infrastructure or managing complex cloud environments.

Build and run evaluations

Quantiles provides two ways to build and run evaluations. Built-in benchmarks make widely used evaluations available out of the box, while custom evaluations support requirements specific to a product, dataset, model, prompt, or agent. Custom evaluations can be defined through configuration or implemented in Python for more specialized logic.

Built-in benchmarks

Built-in benchmarks are ready-to-run evaluations with predefined datasets, scoring methods, and metrics. Use them when you want a common reference point, a repeatable baseline, or a maintained implementation of a public benchmark.

• Built-in Benchmarks

  Run ready-to-use benchmarks from the qt CLI

• Benchmark Hub

  Explore benchmark datasets, evaluation methods, and metrics

Custom evaluations

Custom evaluations measure behavior specific to your product, workflow, prompt, dataset, rubric, or release process. Define a custom configuration evaluation in a quantiles.toml configuration file, or use Python when you need specialized evaluation logic.

• Custom Configuration Evaluations

  Define evaluations in configuration without writing Python

• Custom Code Evaluations

  Build specialized evaluation logic with Python

Use coding agents throughout the evaluation workflow

Quantiles gives coding agents a native CLI workflow for creating and running evaluations, inspecting inputs and outputs, comparing results, debugging failures, resuming interrupted runs, and summarizing regressions. The Quantiles agent skill provides reusable instructions that help agents carry out each step consistently from the development environment.

• Agent Overview

  Learn how coding agents fit into the Quantiles evaluation workflow

• SKILL.md Repository

  View the reusable instructions coding agents use with Quantiles