Local-first and Offline Toolchain

Quantiles is built as a local-first system that keeps evaluation execution, metadata, metrics, and analysis on your computer by default. Evaluations run fully offline when all required inputs, outputs, and dependencies are stored or cached locally. Any step that retrieves a benchmark configuration or dataset, calls a hosted model or judge, or invokes an external tool might require network access.

Local-first

Most modern development machines have enough computing and storage resources for many benchmark workloads, but that capacity is often underused by most tools.

The Quantiles toolchain, including the qt CLI, Python SDK, on-disk data formats, and REST API, is optimized to use the local computing power by default instead of relying on cloud or other non-local resources.

Quantiles uses several high-performance storage and execution technologies to support local benchmark execution:

• SQLite  - a widely-used database  for local storage of benchmark metadata
• Several tools for ingesting, storing, processing and querying benchmark metrics:
  • Apache Arrow  - a columnar format and memory layout optimized for data analytics and OLAP workloads
  • Apache Parquet  - a columnar file format built atop Apache Arrow, optimized for storing and querying analytics data
  • Apache DataFusion  - a query engine built atop Apache Arrow and optimized for highly parallel, vectorized, local-only OLAP workloads, including fast queries from Parquet files
• Rust  - a memory-safe, runtime- and garbage-collector-free language designed for efficiency and reliability. The qt CLI, including its analytics engine, is built completely with Rust.

Offline

The CLI and Python SDK support offline evaluation workflows, including the following local execution and analysis features:

• Benchmark code runs locally on your machine
• Metadata is recorded to a local, on-disk database
• Metrics and evaluation outputs are computed and recorded to local, on-disk files
• qt show and qt compare commands access only local metadata and analytics databases

Certain evaluation-related tasks require an internet connection:

• Retrieving configuration settings for built-in benchmarks that are not available locally
• Downloading datasets that are not already cached on your disk
• Accessing hosted AI models, model-based judges, or other network-dependent tools used by the evaluation workflow
• Fetching data from the internet during evaluation execution

Generally, Quantiles documentation will specify if any Quantiles operations rely on non-local resources to fetch data, store data, or do any computation.

Built-in benchmark configuration

Built-in benchmark configurations are fetched from the Quantiles hosted benchmark registry and loaded into memory for the current run, but are not cached on disk. To make the benchmark configuration available offline, add it to your local configuration file. This also allows you to customize its settings.

Learn how to save and customize built-in benchmarks in your configuration file.