Custom no-code evaluations

Custom no-code evaluations let you evaluate a specific dataset with an exact-match or multiple-choice scoring style without writing or maintaining custom evaluation code. Define the dataset, prompt template, scoring style, and optional model declaratively in the configuration file.

For highly specialized evaluations such as multi-step workflows, LLM judge-based evaluations, or agent evaluations, use a custom code evaluation instead.

Configuration reference

You can configure your custom_nocode evaluation with the following fields in your quantiles.toml or .quantiles.toml configuration file:

Styles

Custom no-code evaluations have two different styles, defined in the style.type field:

• exact_match: an evaluation that compares the model’s response with an expected answer.
• multiple_choice: an evaluation that extract and score the model’s selection from a pre-configured (in the dataset) set of choices.

The following is an example of an exact-match custom no-code evaluation:

# Example of a custom no-code evaluation called "my_custom_nocode_eval"
[benchmarks.my_custom_nocode_eval]
 
# Type must be set to "custom_nocode" or it will default to "built-in"
type = "custom_nocode"
 
# Style sets how responses are parsed and scored
style = { type = "exact_match", golden_column = "answer" }
 
# Identifies the Hugging Face dataset to evaluate
dataset = { name = "quantiles/my_custom_nocode_eval_data" }
 
# This example uses the built-in demo model.
# Specify a provider backed model (such as OpenAI or Anthropic)
# if you want to use a different model. This will require
# an API key OPENAI_API_KEY environment variable.
model = "random"
 
# Path to an existing Jinja prompt template.
prompt_template_file = "prompts/qa.txt"

Run it with:

qt run my_custom_nocode_eval

For additional examples, see the sample custom_nocode configuration .

Note: The demo model is intended only to validate the evaluation workflow. When using model="random", style.type = "exact_match" generates random text and will typically produce very low accuracy. With style.type = "multiple_choice", it samples uniformly from style.choice_labels, so accuracy will usually be higher.

Prompt template

The prompt template defines the full prompt sent to the model for each dataset row. Set the prompt_template_file value to the path of a Jinja -formatted template file. At runtime, Quantiles exposes the complete dataset row to the template as row and, for multiple-choice evaluations, exposes the normalized choices as choices.

Dataset

Choose or publish a Hugging Face dataset whose rows contain the fields needed to render your prompt and score the model’s response. The evaluation style determines which additional fields are required:

• style = { type = "exact_match", ... } evaluations need a golden-answer column
• style = { type = "multiple_choice", ... } evaluations need fields describing the choices and the correct answer

Prompt fields

Quantiles exposes the complete dataset row to the Jinja template as row, so the template can reference any field needed to build the prompt. For example, Question: {{ row.question }} reads the question field. You can also use bracket notation, such as {{ row["Question"] }}, for field names that cannot be accessed cleanly with dot notation.

Exact-match answer column

For style.type = "exact_match" evaluations, the style.golden_column field identifies the dataset field containing the expected response. The field must contain a string, number, or boolean, which Quantiles converts to text. Quantiles trims leading and trailing whitespace from the model response and expected response before comparing them; the comparison is otherwise case-sensitive.

Multiple-choice columns

For style.type = "multiple_choice", choices can come from one dataset field containing an array or label-keyed object, configured with style.choices.column, or from several scalar fields, configured with style.choices.columns. Quantiles assigns the configured labels to these choices and exposes the normalized { label, text } entries to the prompt template as choices.

Multiple-choice answer column

Multiple-choice evaluations can identify the correct choice through style.answer.label_column, which contains a configured choice label; style.answer.index_column, which contains a numeric index and supports an optional index_base; or style.answer.correct_choice_column, which names the choice field known to contain the correct answer.

Shuffle seed column

When choice shuffling is enabled, style.shuffle.seed_column identifies a dataset field containing a stable string, number, or boolean for each row. Quantiles uses this value to shuffle the row’s choices deterministically.

Prompt template

The prompt template defines the full prompt sent to the model for each dataset row. Set the prompt_template_file value to the path of a Jinja -formatted template file. At runtime, Quantiles exposes the complete dataset row to the template as row and, for multiple-choice evaluations, exposes the normalized choices as choices.

{{ row.question }}
 
{% for choice in choices %}
{{ choice.label }}. {{ choice.text }}
{% endfor %}

Dataset

Choose or publish a Hugging Face dataset whose rows contain the fields needed to render your prompt and score the model’s response. The evaluation style determines which additional fields are required:

• style = { type = "exact_match", ... } evaluations need a golden-answer column
• style = { type = "multiple_choice", ... } evaluations need fields describing the choices and the correct answer

Prompt fields

Quantiles exposes the complete dataset row to the Jinja template as row, so the template can reference any field needed to build the prompt. For example, Question: {{ row.question }} reads the question field. You can also use bracket notation, such as {{ row["Question"] }}, for field names that cannot be accessed cleanly with dot notation.

Exact-match answer column

For style.type = "exact_match" evaluations, the style.golden_column field identifies the dataset field containing the expected response. The field must contain a string, number, or boolean, which Quantiles converts to text. Quantiles trims leading and trailing whitespace from the model response and expected response before comparing them; the comparison is otherwise case-sensitive.

Multiple-choice columns

For style.type = "multiple_choice", choices can come from one dataset field containing an array or label-keyed object, configured with style.choices.column, or from several scalar fields, configured with style.choices.columns. Quantiles assigns the configured labels to these choices and exposes the normalized { label, text } entries to the prompt template as choices.

Multiple-choice answer column

Multiple-choice evaluations can identify the correct choice through style.answer.label_column, which contains a configured choice label; style.answer.index_column, which contains a numeric index and supports an optional index_base; or style.answer.correct_choice_column, which names the choice field known to contain the correct answer.

Shuffle seed column

When choice shuffling is enabled, style.shuffle.seed_column identifies a dataset field containing a stable string, number, or boolean for each row. Quantiles uses this value to shuffle the row’s choices deterministically.

Metrics and results

Each custom_nocode sample emits the following metrics:

• is_correct indicates whether the parsed response matches the expected answer.
• response_parsed indicates whether Quantiles could parse the response. Exact-match responses are always considered parsed; for multiple-choice evaluations, this value is 0 when the response cannot be parsed as a configured choice label.
• latency_ms records the sample’s execution latency in milliseconds.

While each run emits these aggregate metrics:

• accuracy, correct_count, incorrect_count, and total_count
• parsed_response_count, unparsed_response_count, and parse_rate
• mean_latency_ms, median_latency_ms, p95_latency_ms, p99_latency_ms, min_latency_ms, and max_latency_ms

Opt-in aggregates for multiple_choice evaluations

Multiple-choice runs can also emit more aggregate metrics, if you opt-in to them. Use the metrics array in your evaluation’s configuration to do so.

For example, to enable F1 and related aggregates, specify the following:

metrics = ["f1"]

This opt-in metric enables the following aggregates:

• macro_precision, macro_recall, and macro_f1 - the arithmetic means of the raw, un-weighted precision, recall and F1 metrics for each label.
• weighted_precision, weighted_recall, and weighted_f1 - the weighted average of the precision, recall and F1 metrics for each label. To compute the weight of each label, that label’s support (e.g. the number of samples whose expected answer is that label) is used.
• precision_label_N, recall_label_N, and f1_label_N - the evaluation of the label N as the positive class against all other labels.
• support_label_N - the number of samples whose expected answer is that label, and N is the label’s index in style.choice_labels.

Precision, recall, and F1 are reported as 0 when their denominator is zero. The confusion matrix uses expected labels as rows and predicted labels, plus the unparsed bucket, as columns.

To enable a confusion matrix, specify the following:

metrics = ["confusion"]

This opt-in metric shows a representation of a confusion matrix, with the following outputs:

• confusion_matrix_G_P - the number of samples whose expected label has index G and whose parsed prediction has index P. These cells form the parsed-label columns of the run’s confusion matrix.
• confusion_matrix_G_unparsed - the number of samples whose expected label has index G but whose response could not be parsed as a configured label. These cells form the confusion matrix’s additional unparsed-prediction column.

The --json flag and opt-in aggregates

By default, if you specify a metric type like metrics = ["confusion"], that metric will only be computed and displayed if you pass the --json to your command (e.g. qt show --json). To compute and show the metric in all cases, regardless of whether the --json flag is passed, specify the metrics like the below:

metrics = [{name = "confusion", show = "all"}]

Use coding agents to create custom no-code evaluations

Use the Quantiles agent skill to use coding agents to build, run and analyze custom no-code evaluations with the qt CLI. Before proceeding, install the skill. Then, customize the the below prompt to create a custom no-code evaluation:

Create a custom no-code evaluation in `quantiles.toml` using the <your HuggingFace dataset> dataset, the <prompt column> prompt column, the <answer column> golden answer column, and a prompt template file. Run 10 samples and summarize the results.

See Agents Overview for more detail on using agents with Quantiles.