Skip to Content
Get Started

Local-first and Offline Toolchain

Quantiles is built as a local-first, offline system that keeps benchmark execution, metadata, metrics, and analysis on your computer by default.

Local-first

Most modern development machines have enough computing and storage resources for many benchmark workloads, but that capacity is often underused by most tools.

The Quantiles toolchain, including the qt CLI, SDKs, on-disk data formats, and REST API, is optimized to use the local computing power by default instead of relying on cloud or other non-local resources.

Quantiles uses several high-performance storage and execution technologies to support local benchmark execution:

  • SQLite  - a widely-used database  for local storage of benchmark metadata
  • Several tools for ingesting, storing, processing and querying benchmark metrics:
    • Apache Arrow  - a columnar format and memory layout optimized for data analytics and OLAP workloads
    • Apache Parquet  - a columnar file format built atop Apache Arrow, optimized for storing and querying analytics data
    • Apache DataFusion  - a query engine built atop Apache Arrow and optimized for highly parallel, vectorized, local-only OLAP workloads, including fast queries from Parquet files
  • Rust  - a memory-safe, runtime- and garbage-collector-free language designed for efficiency and reliability. The qt CLI, including its analytics engine, is built completely with Rust.

Offline

The CLI and Python SDK support offline benchmark workflows, including the following local execution and analysis features:

  • Benchmark code runs locally on your machine
  • Measurements are computed locally, except for remote model calls, hosted judges, external tools, and LLM-as-judge evaluations that may call remote providers (e.g. OpenAI, Anthropic, cloud providers, etc.)
  • Metadata is recorded to a local, on-disk database
  • Metrics and evaluation outputs are recorded to local, on-disk files
  • qt show and qt compare commands access only local metadata and analytics databases

Certain benchmark-related tasks, such as downloading external data sources or calling remote models and tools, require an internet connection:

  • Downloading datasets that are not already cached on your disk
  • Fetching data from the internet during benchmark execution

Generally, Quantiles documentation will specify if any Quantiles operations rely on non-local resources to fetch data, store data, or do any computation.

Last updated on