Agent Prompts

After installing the skill, use the following prompt templates for common agent-driven Quantiles evaluation workflows. Use them as written, adapt them to your benchmarks, or use their structure to create your own prompts.

Each prompt tells the agent what to run, what to inspect, and what to report so evaluation results are consistent and reviewable. The Quantiles skill instructs the agent to use --json for structured run output.

Prompt Templates

Use these prompts as starting points for common Quantiles agent workflows. Replace the placeholders with your evaluation name, run IDs, model configuration, and evaluation goals before passing the prompt to your coding agent.

Run your first agent-driven evaluation

Use the built-in demo model to run a built-in benchmark locally without external model calls, provider API keys, or inference costs.

Pass the following prompt to your coding agent:

Run a SimpleQA Verified benchmark and summarize the results.

If your coding agent does not detect the Quantiles skill automatically, add the $quantiles prefix to your prompt.

Run a subset of samples

Use a sample limit to run a subset of any built-in benchmarks, custom configuration evaluations, or custom code evaluations. When no sample limit is specified in the command or configuration file, the full evaluation runs by default.

Run 10 samples of <eval_name> as a smoke test. Summarize the results and tell me if it is safe to continue to a full run.

Built-in benchmarks you can try include gpqa, medmcqa, medqa, mmlu-pro, pubmedqa, and simpleqa-verified.

Customize a benchmark

To customize a single run of a built-in benchmark such as simpleqa-verified, use the following prompt to configure a selected hosted LLM and run a limited sample.

Use --input overrides to run <number of samples> samples of the simpleqa-verified benchmark with the <your model here> model. Inspect the completed run and summarize the results.

See configuration documentation for more details.

Add a model to an existing evaluation

Add a hosted model to an evaluation. Replace the placeholders with the provider, model ID, and evaluation name you want to configure:

Add <model provider and model_id> to the existing <built-in benchmark or custom configuration evaluation>. Preserve the other evaluation settings. Confirm that the required provider credentials are available without showing their values; if they are not, stop and tell me what is missing. When finished, summarize what you changed. Do not run the evaluation until I confirm.

See Model configuration for supported providers, credentials, and configuration examples.

Create a custom configuration evaluation

Choose the prompt for the custom configuration evaluation style you want to create, then replace its placeholders with values from your dataset and Jinja prompt template.

Exact-match agent prompt

Create and run a custom configuration exact-match evaluation named <evaluation_name> using the Hugging Face dataset <dataset_id>. Use <prompt_column> as the prompt column, <answer_column> as the golden-answer column, and the following Jinja prompt template: <jinja_prompt_template>. Inspect the run and summarize the results.

Multiple-choice agent prompt

Create and run a custom configuration multiple-choice evaluation named <evaluation_name> using the Hugging Face dataset <dataset_id>. Use <prompt_column> as the prompt column, <choice_source> as the choice source, <choice_labels> as the choice labels, <answer_source> as the correct-answer source, and the following Jinja prompt template: <jinja_prompt_template>. Inspect the run and summarize the results.

See Custom configuration evaluations for more details on these evaluations and their required and optional configurations.

See a list of all evaluation runs

Your agent can list all of your evaluation runs, including each run’s ID, status, creation time, and other summary metadata.

Give me a list of all my evaluation runs.

Compare two evaluation runs

Use this prompt to have your coding agent locate the two most recent runs for an evaluation, compare their aggregate metrics, and summarize sample-level regressions, improvements, and output differences.

Compare the two most recent runs for <eval_name>. Summarize the aggregate metrics, sample-level results, failures, and any notable errors. Identify the highest-impact issues to review first, and recommend specific next steps.

Inspect a failed or interrupted evaluation

Use this prompt when you want the agent to diagnose failed or interrupted runs before deciding whether to resume, rerun, or change the evaluation configuration. The agent should identify what failed, why it likely failed, which samples were affected, and what to fix next.

List recent evaluation runs, identify the most recent failed or interrupted run, and inspect its run record. Summarize the failure reason, affected samples or steps, likely causes, and recommended fixes. Do not resume or rerun the evaluation unless explicitly asked.

Resume run <run_id>. Inspect the original failure, verify whether the resumed run completed successfully, summarize aggregate metrics and sample-level results, and recommend specific next actions.

Create a custom code evaluation

For custom code evaluations that use your own datasets, models, and measurement techniques, you can build evaluations with the Quantiles Python SDK.

To have your coding agent build and run a custom evaluation, customize the below prompt template to your needs:

Write a Quantiles custom code evaluation using the Python SDK that uses the <your dataset> dataset, run samples through the <your model> model, and measures the output using the following metrics:

<list your metrics here>.

Call the evaluation <name>, and make sure to include it in the `quantiles.toml` config file. When you're done, run the new eval and summarize the results.

See custom evaluations documentation for details on how to write custom evaluations with your own code, using the Quantiles SDKs and tooling.