Run lightweight checks or full-scale evaluations locally with the open-source Quantiles CLI, manually and with coding agents.
Written by
Today we're releasing the open-source Quantiles CLI and Python SDK, a local-first infrastructure for running AI evaluation workflows. It's built for the development loop where evaluations are run, compared, inspected, and analyzed for applied AI systems.
Developers and coding agents can get started by installing the CLI:
Evaluation workflows quickly outgrow one-off scripts once teams need reproducible evals, durable execution, dataset and configuration tracking, baseline comparison, sample-level failure analysis, and metric analysis across smoke tests, ablations, and full-scale eval runs. Quantiles provides those primitives as local-first infrastructure, letting teams write standard Python to run evaluation workflows from the CLI. Each run automatically captures metrics, events, inputs, outputs, and execution history in the local workspace. Teams can inspect individual samples, resume interrupted or failed eval runs with cached work, and compare results directly from the CLI.
Quantiles includes built-in benchmarks, with two examples below so you can try the workflow without configuring a model or dataset. These examples use a built-in demo model so there are no inference costs. They show how Quantiles runs evaluations, summarizes aggregate metrics, compares evals, and displays sample-level results.
The Quickstart walks through the full workflow whether you want to run a built-in benchmark and run your own custom evaluation.
Quantiles is designed to work natively with coding agents such as Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and other agentic development tools.
The Agent Quickstart documentation provides details on installing the skill, common agentic workflows, and advanced agent integration. The two prompts below are a quick way to get started.
Use the prompt below to set up your coding agent with the Quantiles CLI and agent skill:
After your agent completes the install, have it run its first benchmark using the following prompt:
This prompt uses a local demo model to validate the evaluation workflow and generates random text without hosted provider calls or inference costs.
Quantiles includes a Python SDK that provides a Pythonic API for using Quantiles built-in workflow primitives like durable steps, structured inputs/outputs, and metrics emission. These primitives help you write custom evals that are more reliable, efficient, reproducible, faster to build, and easier for agents to build. The SDK integrates tightly with theqt CLI for running, recording, analyzing custom evaluations.
Install the Python SDK with your preferred dependency management system. We recommend using the uv tool:
For custom evaluations, the Python SDK lets you load datasets, validate rows with Pydantic, call a model or agent, score each sample, and emit metrics. Each operation can be wrapped in a durable step, so interrupted eval runs can resume without repeating completed model calls. For larger datasets, helpers like the dataset and map_dataset functions provide typed row loading, batching, and bounded concurrency, all while preserving the run history that qt show and qt compare inspect later.
Combined with the Quantiles skill, this infrastructure makes it easier for coding agents to create custom evaluations and analyze results using the qt workflow.
Quantiles is designed around a local-first, offline toolchain that lets teams run evaluations with local compute and storage without needing to deploy (and pay for) hosted infrastructure or manage complex cloud environments.
The qt CLI and Python SDK run evaluation code on your machine, store metadata in a SQLite database, and keep metrics and evaluation outputs in open analytics formats on disk. Commands like qt show and qt compare read from those local artifacts, so iterating on and analyzing evals is fast and reproducible. Network access is only required when the workflow itself depends on a remote service, such as an LLM provider or an external dataset source.
The qt CLI and the rest of the Quantiles stack move the evaluation loop as close to you and your agent as possible. Use the Quickstart for a minimal manual setup, or the Agent Quickstart to delegate setup to a coding agent. CLI commands are consistent across both paths, so commands for running, comparing, and analyzing evals work whether you run everything yourself, choose an agent-first workflow, or a combination of both.
We’re excited to share Quantiles as open-source evaluation infrastructure for measuring AI systems behavior and making evidence-based decisions as models, prompts, and workflows evolve.
We’ll be sharing ongoing updates on GitHub, and we'd welcome your contributions.
Common questions this article helps answer
qt run, qt show, and qt compare make each run resilient to failure, efficient, inspectable and comparable. The CLI-first workflow is also designed to work natively with coding agents, which can run evals, inspect outputs, and repeat the same steps consistently..quantiles/ directory, while metrics are stored in local Parquet files that qt show and qt compare query locally.openai:, anthropic:, gemini:, and cloudflare_ai_gateway: and reads sensitive API keys and other credentials from the environment. Custom Python evals can also call a local model server, internal endpoint, hosted provider, or custom agent harness, but still store all eval data and observe/compare evals with the same tooling.