← Back to Benchmark Hub

Metrics

Accuracy

Fraction of evaluated examples assigned the correct class label.

Overview

Accuracy is the proportion of examples for which the predicted label matches the reference label. It supports single-label binary and multiclass classification and weights each example equally unless sample weights are provided.

Probabilities, logits, or scores must be converted into labels using a fixed rule, such as a binary threshold or multiclass argmax. Label mappings, parsing, decision rules, thresholds, and weights must remain consistent across compared runs.

Input Format

  • predictions: discrete class labels produced by the evaluated system
  • references: ground-truth labels in the same label space
  • sample_weights: optional nonnegative weights controlling each example's contribution

Example:

{
  "predictions": ["positive", "negative", "positive", "positive"],
  "references": ["positive", "negative", "negative", "positive"]
}

Output Format

Report accuracy as a score between 0 and 1 or as a percentage between 0% and 100%, and state the scale explicitly so equivalent values such as 0.84 and 84% are interpreted consistently. Include the number of correct predictions, total evaluated examples, and, when relevant, per-class metrics and a confidence interval. 

{
  "accuracy": 0.75,
  "correct": 3,
  "incorrect": 1,
  "total": 4
}

Metrics

Unweighted accuracy is the mean of the per-example correctness indicator:

Accuracy=1N∑i=1N1[y^i=yi]\text{Accuracy} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}[\hat{y}_i = y_i]

Here, N is the number of evaluated examples, ŷᵢ is the predicted label, and yᵢ is the reference label. For binary classification, the equivalent confusion-matrix calculation is:

Accuracy=TP+TNTP+TN+FP+FN\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}

When sample weights are used:

Weighted Accuracy=∑i=1Nwi1[y^i=yi]∑i=1Nwi\text{Weighted Accuracy} = \frac{\sum_{i=1}^{N} w_i \mathbb{1}[\hat{y}_i = y_i]}{\sum_{i=1}^{N} w_i}

Evaluation Rules

  • Record the label mapping and the decision rule used to convert model scores into labels. Top-k accuracy is a separate metric.
  • Keep missing, invalid, or unparseable predictions in the denominator and count them as incorrect unless another policy is explicitly documented. Report these failures separately so they are not indistinguishable from valid but incorrect predictions.
  • Define how abstentions are scored. They may count as incorrect, represent a valid class, or be reported separately with coverage.
  • For multilabel tasks, name the calculation explicitly. Subset accuracy requires the complete predicted label set to equal the reference label set and is stricter than label-wise accuracy.

Known Limitations

  • Accuracy can appear high on imbalanced datasets when a model mostly predicts the majority class.
  • Every error receives the same penalty even when some mistakes have greater operational consequences.
  • A single aggregate score can hide class-specific, subgroup, and data-slice failures.
  • The score depends on the threshold or decision rule used to create discrete predictions.
  • Accuracy does not measure probability calibration, confidence quality, or ranking performance.
  • Small evaluation sets and noisy reference labels can make results unstable or misleading.

Versioning and Provenance

For reproducibility, record the dataset version and split, evaluated and excluded example counts, label definitions and mappings, reference-label distribution, output parser, decision rule, thresholds, invalid-output and abstention policies, sample weights, metric implementation, and score scale.

References

Google for Developers: Classification accuracy, precision, and recall

scikit-learn: accuracy_score documentation

Related Metrics

F1 Score

Discrete Decision Performance

Balanced metric that summarizes precision and recall into one harmonic-mean score for classification performance.

Discrete predictions + labelsBinary and multiclass classification (aggregated)F1 score

MCC

Agreement & Robustness

Correlation-based metric that accounts for true/false positives and negatives, robust to class imbalance.

Discrete predictions + ground-truth labelsBinary classification tasksMCC

Sensitivity & Specificity

Discrete Decision Performance

Companion metrics measuring true positive rate (sensitivity) and true negative rate (specificity).

Binary class predictions + ground-truth labelsBinary classification tasksSensitivity · Specificity