Security and Privacy

Quantiles runs workflow code locally by default and stores Quantiles-managed run metadata, samples, metrics, and outputs on disk in your project workspace. HuggingFace dataset batches fetched through the built-in dataset loader are cached separately under ~/.cache/quantiles/datasets. Workflows may still use the network when they call remote models, hosted judges, external tools, or uncached dataset sources.

Securing your data

Benchmarks can be run against one or more public or private datasets using the built-in dataset API. The built-in CLI Dataset REST API currently supports HuggingFace datasets through huggingface:// and hf:// sources, including private HuggingFace datasets when HF_TOKEN is configured. Other private sources, such as S3, Azure Blob Storage, Google Cloud Storage, Cloudflare R2, private databases, internal APIs, or local files, remain your responsibility to secure and should be accessed from custom workflow code or a custom Python DatasetSource.

Below are recommended practices for securing private datasets.

Storage

Store private datasets in a controlled storage location, either on local disk or in secure cloud storage such as Amazon S3, Azure Blob Storage, Google Cloud Storage, Cloudflare R2, or a similar service.

• Require authentication for cloud storage. Buckets and storage containers should not be publicly accessible.
• Disable public network access for stronger isolation. Use this when datasets require tighter access controls.
• Choose local storage when direct control matters most. Local disk can reduce external exposure, but you are responsible for durability, backups, and recovery.
• Choose cloud storage when reliability matters most. Cloud storage generally provides stronger durability, but requires careful access control.
• Treat datasets as immutable. Avoid modifying dataset contents, creating unnecessary copies, or sending data across the network.

The Quantiles SDKs are designed to minimize unnecessary data movement. HuggingFace dataset loading still creates a controlled local cache by writing fetched row batches as Parquet files under ~/.cache/quantiles/datasets. Custom dataset sources run in your workflow process, so your source implementation controls whether data is streamed, cached, copied, or rewritten.

We may add managed private dataset capabilities in the future to support secure handling of private benchmark data. If this would be useful for your workflows, please get in touch with us.

Cloud storage authentication

If datasets are stored in cloud storage, apply the principle of least privilege : grant read access only to the smallest set of clients that need it, and do not grant write, delete, or mutation permissions to dataset contents.

In most workflows, read access is limited to these principals:

Avoid granting any client permission to write, delete, or otherwise modify any of your datasets in cloud storage.

Securing your code

Some benchmark workflows use public code and datasets, while others involve private benchmark logic and private data. When working with private datasets, keep benchmark code in private repositories or environments whenever possible.

Since Quantiles runs your code locally by default, your code is usually secured by using the same practices you already use for your private repositories. If your code should stay private, ensure your hosting provider (e.g., GitHub , GitLab , AWS CodeCommit , Azure DevOps , etc.) prohibits unauthenticated access and limits access to authorized users, teams, and service accounts.

Coding agents and private data

When using coding agents with private code, datasets, or evaluation results, give the agent explicit security constraints before it runs commands or summarizes results:

• Do not print API keys, .env values, secrets, or private credentials.
• Do not paste private samples, traces, or dataset rows into external chat tools unless explicitly approved.
• Do not upload .quantiles/ databases, cached dataset files, metrics, or artifacts unless explicitly approved.
• Do not modify access controls, repository visibility, bucket policies, or service account permissions unless explicitly asked.
• Report commands, run IDs, metrics, and failure summaries without exposing private sample contents when possible.

For repeatable agent behavior, install the Quantiles skill before running qt commands.

Securing samples and metrics

When you run benchmark code over a dataset, Quantiles stores run metadata, steps, inputs, outputs, and errors in a local SQLite database under .quantiles/. Metrics are stored as local Parquet files and queried with DataFusion. These local files can contain valuable information about your dataset and AI system, including the following:

• The composition of your dataset, even if it’s private
• The methods used to send samples to your AI system and evaluate its outputs
• Your AI system’s performance on a specific dataset
• Changes to your AI system that affect benchmark performance

Because substantial information can be inferred from these files, they should not be copied or moved to locations on your disk or the network that may expose them to unauthorized clients. Quantiles-managed storage stays local by default, but your workflow code, model providers, judges, agents, export commands, or manual uploads can still transmit samples, metrics, or artifacts. To reduce the risk of unauthorized access or corruption, avoid manually copying or moving the .quantiles/ directory, metrics files, or cached dataset batches.

Securing the local server

The local qt HTTP server binds to 127.0.0.1:8765 by default. This loopback address is intended for local SDK and CLI communication. If you start the server manually with qt serve --addr, do not bind it to a public interface such as 0.0.0.0:8765 unless you intentionally want other machines to reach it and have protected that environment with appropriate network controls.