Skip to content

Medical AI Robustness Audit

Understand How Your Medical AI Model Performs Beyond Its Baseline Dataset

Investigate model performance under controlled imaging variations, challenging cases and dataset conditions through reproducible technical evaluation.

Built with these teams in mind

  • CTO
  • Head of AI
  • ML Engineering Lead
  • Medical Imaging AI Research Lead
  • Computer Vision Lead

Technical evaluation for research and engineering. Not clinical validation, not a medical device and no regulatory claims.

Baseline
Noise
Low resolution

Typical challenges

Where baseline accuracy stops telling the story

  • Imaging noise

    Lower-dose or noisier acquisitions can hide faint findings.

  • Contrast variation

    Protocol and windowing differences change tissue boundaries.

  • Resolution variation

    Coarser slices blur edges and merge small structures.

  • Dataset shift

    Populations and case mixes differ between sources.

  • Acquisition differences

    Scanners and reconstruction settings vary across sites.

  • Small lesion sensitivity

    Small findings are where errors concentrate first.

  • Segmentation failures

    Boundary and missed-region errors hidden by averages.

  • Model generalisation uncertainty

    Good scores on one dataset may not transfer.

Proposed evaluation workflow

A reproducible audit, step by step

The proposed workflow. Each step's configuration is recorded so results can be re-run and compared.

  1. 01

    Model and Dataset Review

    Understand the task, model, data and how data may be accessed securely.

  2. 02

    Evaluation Protocol

    Agree metrics, test sets, conditions and success criteria in advance.

  3. 03

    Controlled Test Conditions

    Define transformations and challenging subsets, with recorded parameters.

  4. 04

    Model Execution

    Run the model under fixed settings in an agreed environment.

  5. 05

    Metric Analysis

    Compute metrics overall and per condition, against the baseline.

  6. 06

    Failure Investigation

    Group failures and inspect representative cases with your experts.

  7. 07

    Technical Evaluation Report

    Methodology, results, limitations and recommendations.

Data governance first

How data is accessed is agreed during scoping and must meet your governance requirements. This website never accepts datasets or patient information.

Research and engineering only

An audit is a technical evaluation. It is not clinical validation and supports no diagnostic, clinical or regulatory claim.

Technical evaluation

Measures an audit could use

Which metrics are appropriate depends on the clinical research task, the data and the cost of different errors. Technical metrics do not on their own establish clinical effectiveness.

Segmentation and detection

Dice Score2 |A ∩ B| / (|A| + |B|)
Overlap between predicted and reference masks.
IoU|A ∩ B| / |A ∪ B|
Stricter overlap measure related to Dice.
PrecisionTP / (TP + FP)
How many flagged findings were correct.
Recall / SensitivityTP / (TP + FN)
How many true findings were found — often by lesion size.

Robustness

Scenario-level differencesm_condition − m_baseline
How each metric changes under each controlled condition.
Failure-case distributionfailures by category
Which kinds of cases fail, and how often, per condition.

Suitability depends on context. TP, FP and FN are true positives, false positives and false negatives; A and B are predicted and reference regions. A metric that suits detection may mislead for segmentation, and class imbalance, label quality and the cost of each error type all change which measures matter.

Evaluation report preview

What an audit report could look like

An interactive, frontend-only example. All values are illustrative sample data — not results from any customer, model or study.

Robustness evaluation report

Lesion segmentation · sample audit

AUDIT-SAMPLE-001 · all values are illustrative

Illustrative Demonstration
Test condition

Added image noise

In this sample, faint findings are missed more often as noise increases.

Metrics vs baseline
MetricBaselineAdded image noiseΔ
Dice0.820.74−0.08
IoU0.700.60−0.10
Precision0.850.79−0.06
Recall0.800.71−0.09
Dice by condition
  • Baseline test set0.82
  • Added image noise0.74
  • Contrast variation0.77
  • Reduced resolution0.68
  • Small-lesion subset0.61
Failure cases by category
  • Missed small findings6 → 14
  • Boundary errors9 → 12
  • False positives5 → 8

Counts of sample failure cases (baseline → selected).

Example recommendations
  • Review the weakest condition with domain experts before drawing conclusions.
  • Report performance by lesion size, not only as an overall average.
  • Re-test on an independent, real-world dataset where governance permits.
Illustrative layout with sample values written for this page. Not results from any model, dataset or customer; not clinical validation. Nothing is processed and nothing can be downloaded.

The audit draws on the methodology described in our medical imaging research direction: patient-level splits, real-data evaluation, controlled conditions and explicit limitations. That research is a direction, not a clinically validated result.

Medical AI Robustness Audit

Explore a Robustness Evaluation for Your Medical AI Model

Tell us about your model, task and the conditions you are concerned about. No data is needed for a first conversation — please do not send patient data through the form.

Not a medical device. Nothing on this page is intended for diagnosis or clinical decision-making.