Part 2: How to Analzye and Interpret Evaluations in Healthcare AI

In high-risk domains like healthcare, benchmark analysis is not a reporting exercise, it's risk analysis. This article builds on Part I: Foundations of Healthcare AI Evaluation to examine how benchmark results should be interpreted in healthcare AI, clarifying what different metrics actually measure, where they fail, how to assess statistical validity, and how evaluation signals translate into production readiness. Our goal is to turn benchmark outputs into interpretable, comparable, and operationally meaningful evidence that supports real-world clinical use by AI researchers, healthcare practitioners, and clinical leaders.

The shift from benchmark reporting to risk-based interpretation is a best practice and is now reflected in formal guidance such as NIST's AI Risk Management Framework.

Deterministic Benchmarks as the First Layer of Healthcare AI Evaluation

Deterministic or discrete evaluation forms the first layer of benchmarking in healthcare AI, applying when model outputs can be categorized as correct or incorrect relative to a reference standard under a fixed decision rule. These evaluations remain foundational for tasks such as classification, coding, and retrieval.

In deterministic benchmark evaluations, model outputs are first categorized into outcome classes - true positives, false positives, true negatives, and false negatives - which serve as the building blocks for higher-level metrics.

In semantic entailment benchmarks like OpenAI's HealthBench (which is also discussed later alongside NLI benchmarks), these outcome categories can be formalized probabilistically. This framing matters because hallucination is treated not as a binary event, but as a graded risk signal.

For a input-output pair $(x, y_k)$, an NLI model estimates the following:

Using these calculations, we can then define a false/true outcome as the following:

• True Positive (TP) if$P_e \ge \tau$
• False Positive (FP) if$P_c \ge c \;\text{or}\; P_n \ge n$

where $\tau$, $c$, and $n$are task-specific risk thresholds for entailment, contradiction, and neutrality (respectively).

Interpreting Classification Behavior from Deterministic Benchmarks

Once deterministic outcomes have been computed, the central challenge shifts from measurement to interpretation. Precision, recall, ROC curves, calibration, and agreement metrics all describe different facets of the same underlying errors. Interpreting classification behavior in healthcare AI requires understanding how these signals interact, revealing trade-offs between false positives and false negatives, uncertainty in probability estimates, and alignment with human clinical judgment.

Point Metrics: Precision, Recall, and F1 Score

Although mathematically simple and used in ML science long before modern LLM/Transformer-based systems were popular, these metrics are critical to healthcare and must be interpreted with attention to safety, workflow, and clinical risk.

• Precision
  Precision measures how often a positive prediction is correct. In other words, when the model signals a problem, how often is that problem real? High precision is especially important when interventions are costly, invasive, or disruptive, or when human attention is the limiting resource.
  
  $$\text{Precision} = \frac{TP}{TP + FP}$$
• Recall
  Recall measures the proportion of true cases the model successfully identifies. It indicates how often clinically meaningful events are missed. High recall is particularly important in screening and surveillance, where false negatives carry significant risk.
  
  $$\text{Recall} = \frac{TP}{TP + FN}$$
• F1 Score
  The F1 score summarizes precision and recall into a single number by taking their harmonic mean. It is useful as a compact comparative metric, especially when class distributions are imbalanced. However, in healthcare settings, F1 should be interpreted with caution. Two models can achieve the same F1 score while exhibiting very different error profiles with one favoring recall and the other precision, resulting in radically different clinical implications.
  $$\text{F1} = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

Precision and recall are almost always in tension. Improving one typically degrades the other, and the “right” balance depends on the clinical context.

• High recall, low precision systems cast a wide net. They are useful when the goal is to surface all possible risk but can strain workflows if downstream review capacity is limited.
• High precision, low recall systems are conservative. They produce fewer alerts but may miss edge cases, rare presentations, or atypical patients.

Unfortunately, there is no universally optimal balance. The acceptable tradeoff depends on:

• Clinical severity
• Actionability of the alert
• Availability of human review
• Regulatory and liability considerations

This is why reporting a single metric without context is often misleading.

Threshold analysis with ROC curves and AUC, Confusion matrices, and Calibration curves

Threshold analysis examines how a model behaves as its decision cutoff changes, helping teams choose how the model should actually be used in practice. Rather than relying on a single score, these tools reveal trade-offs between sensitivity, error types, and confidence, factors that are especially important in healthcare settings.

Measuring agreement with Cohen's $\kappa$ and Krippendorff's $\alpha$

Metrics such as Cohen's $\kappa$ and Krippendorff's $\alpha$ measure how closely a model's outputs align with human annotations after accounting for agreement that could occur by chance. Rather than asking whether a prediction is strictly “correct,” these metrics assess whether the model behaves consistently with human judgment.

Importantly, agreement metrics assess consistency with human judgment, not clinical correctness, and are therefore especially important in healthcare AI when:

• Ground truth (the best answer) is inherently subjective
• Multiple clinicians would reasonably disagree on labels
• Annotations reflect clinical interpretation rather than objective fact

The role of sequence and generation metrics

The classification metrics discussed above assume that model outputs can be deterministically categorized relative to a reference, enabling explicit TP, FP, FN, and TN outcomes. Generative models - like modern LLMs - can break this assumption.

Tasks such as clinical summarization, explanation, and report drafting produce free-form text, where correctness is no longer a binary decision and failure modes can be harder to detect and localize. As a result, evaluation shifts from discrete outcome analysis to proxy measures of textual similarity and semantic overlap, introducing both flexibility and new risks.

These are the most commonly used sequence and generation metrics in practice, along with an explanation of what they measure and why they are often misinterpreted in clinical settings.

• BLEU
  BLEU evaluates generated text by measuring n-gram overlap between a model’s output and one or more reference texts. It computes a weighted geometric mean of modified n-gram precision scores, with an additional penalty to discourage overly short outputs:
  
  $$\text{BLEU} = BP \cdot \exp\left(\sum_{n=1}^N w_n \log p_n\right)$$

  Where:

  • $p_n$ is the modified precision for n-grams of length 𝑛, clipped to the maximum count observed in the reference text
  • $w_n$ are weights over n-gram lengths (typically uniform)
  • $BP$ is the brevity penalty, which reduces the score for outputs that are shorter than the reference

  In practice, BLEU measures surface-level overlap with a reference, not factual correctness or source grounding. As a result, it can score fluent but incorrect or hallucinated clinical text highly, making it a weak signal for safety-critical healthcare applications.

• ROUGE

  ROUGE-L evaluates generated text by measuring the longest common subsequence (LCS) between a model’s output and a reference text. Unlike n-gram–based metrics, LCS does not require contiguous matches, allowing it to capture in-order overlap even when words are separated.

  $$\text{ROUGE-L} = \frac{(1 + \beta^{2}) \cdot R_{\text{LCS}} \cdot P_{\text{LCS}}}{R_{\text{LCS}} + \beta^{2} P_{\text{LCS}}}$$

  Where:

  • $P_{\text{LCS}}$ measures how much of the generated text is covered by the LCS
  • $R_{\text{LCS}}$ measures how much of the reference text is captured
  • $\beta$ controls the relative weight of recall versus precision (often favoring recall)

  In practice, ROUGE-L emphasizes coverage (how much of the reference content appears in the output) making it popular for summarization. However, like BLEU, it measures surface overlap rather than factual correctness and can score clinically incorrect or hallucinated text highly if it resembles the reference, limiting its usefulness for evaluating clinical faithfulness and safety.

• METEOR

  METEOR evaluates generated text by aligning words in the model output with words in a reference text using exact matches, stem matches, and synonym matches. It then computes a harmonic mean of unigram precision and recall, with an additional penalty for fragmented or disordered matches.

  $$\text{METEOR} = F_{mean} \cdot (1 - Penalty)$$
  • $F_{mean}$ is the harmonic mean of unigram precision and recall (often weighted toward recall)
  • $Penalty$ increases when matched words are spread across many disjoint segments, discouraging overly fragmented outputs

  METEOR was designed to improve on BLEU by handling paraphrasing and word variation more gracefully. However, despite these improvements, METEOR still operates at the level of surface text alignment. It doesn't verify factual correctness, logical consistency, or source grounding, and can therefore assign high scores to fluent but clinically incorrect or hallucinated outputs.

Sequence and generation metrics like BLEU, ROUGE, and METEOR measure textual similarity, not clinical correctness, making them insufficient alone for evaluating the safety and faithfulness of generative healthcare AI.

Across BLEU, ROUGE, and METEOR, the core limitation is the same: surface-level similarity can assign high scores to fluent but clinically-incorrect text, making these metrics insufficient for safety-critical evaluation. Because of this, deterministic benchmark signals, where outputs can be explicitly categorized and audited, remain an essential foundation for healthcare AI evaluation, even as generative models become larger and more advanced. The limitations have motivated the development of semantic, inference-based, and source-grounded evaluation methods, which are discussed next.

From Similarity to Faithfulness

Sequence-based metrics measure textual similarity through surface-level token overlap, which makes them poor at capturing deep meaning or logical consistency. Embedding-based metrics address this by measuring semantic similarity through vector representations of text - using models like SBERT or ClinicalBERT - rather than exact token matches.

Underlying these methods lies a vector similarity metric. Vector similarity between two vectors $u$ (often, generated text) and $v$ (often reference or ground-truth text) is typically computed via cosine similarity (though other similarity metrics can also be used):

Using this fundamental metric, the above methods capture paraphrasing and semantic equivalence more effectively than n-gram-based metrics but they have limitations:

• They often fail to detect factual hallucinations
• They are sensitive to domain shift (where training data are categorically different from benchmark data)
• They lack grounding in medical knowledge

As a result, semantic similarity alone can't determine whether a generated statement is supported by the source input, making it insufficient for evaluating clinical faithfulness in safety-critical settings.

NLI-based metrics

Natural Language Inference (NLI) introduces an explicit logical constraint into evaluation by asking whether a model's output is entailed by, contradicted by, or unsupported by the source input. In healthcare, this distinction maps directly to faithfulness: is the output justified by the underlying clinical evidence?

Given input $x$ and output $y$, NLI models estimate the following properties of a generative model's output:

• $P_e$: Entailment
• $P_c$: Contradiction
• $P_n$: Neutral

Using these metrics for a given $(x, y)$ pair, hallucination risk can be defined as:

Embedding-based metrics capture semantic similarity, but only NLI-based methods impose the logical constraints required to evaluate faithfulness in healthcare AI.

From Metrics to Decisions

Once metrics are computed, the challenge shifts to decision-making under competing objectives, where no single score can capture safety, robustness, and clinical cost simultaneously.

Weighted composite scores

Organizations often compute weighted scores across dimensions including (but not limited to) the following:

• Safety
• Correctness
• Robustness
• Cost

While useful for executive summaries and high-level comparison, weighted scores encode value judgments. These weights should be explicitly documented and justified, as different stakeholders may prioritize dimensions differently.

• Pareto Front Analysis
  Pareto analysis identifies models that are non-dominated across dimensions, meaning no other model performs better on all criteria simultaneously. This approach preserves trade-off transparency and avoids collapsing complex performance profiles into a single number and it is used in various research and engineering organizations, including MITRE, when examining multi-criteria trade-offs.

Statistical significance

Observed performance differences between models are often small, making it essential to determine whether improvements reflect real signal or sampling noise.

• McNemar's Test
  Often used for paired binary outcomes (model A vs model B on same samples).
  
  $$\chi^2 = \frac{(|b - c| - 1)^2}{b + c}$$

  where $b = \frac{A_{wrong}}{B_{right}}$ and $c = \frac{A_{right}}{B_{wrong}}$.

• Paired Bootstrap Resampling
  Bootstrap resampling repeatedly samples datapoints with replacement to estimate confidence intervals for performance differences without assuming a particular distribution.
  

  These methods prevent over-interpreting marginal score differences and reduce the risk of deploying models based on spurious gains.

When is a model "production-ready"?

Benchmark performance alone does not and cannot be used to determine whether a healthcare AI system is ready for deployment. Production readiness requires explicit risk thresholds that translate evaluation signals (e.g., hallucination rates, consistency, and calibration) into operational decision criteria.

These thresholds align closely with regulatory expectations, including the FDA's Artificial Intelligence and Machine Learning in SaMD guidance, together with its Good Machine Learning Practice (GMLP) principles which emphasizes transparency, change management, and post-market monitoring, as well as the EU AI Act’s classification of medical AI systems as high-risk, requiring robustness testing and human oversight.

In the upcoming Part 3, we bring these ideas together by examining OpenAI's HealthBench , walking through how the benchmark is used and analyzed to evaluate faithfulness, error modes, and clinical risk.