Skip to content

Type to search titles, section headings and summaries across the Developer Portal.

↑ ↓ to move · Enter to open · Esc to close · searched locally in your browser

Guide

Evaluating AI Models with Synthex

Intended model evaluation, benchmarking and robustness workflows: metrics by task, experiment configuration, report structure and reproducibility.

Page type: Explains a workflow step by step.

Overview

An aggregate score can hide the cases that matter most. A model with good average accuracy may fail at dusk, on small objects or on an unseen sensor. The Evaluation Engine is designed to test models against fixed protocols, report results per condition, and turn failures into specific next experiments.

Evaluation pipeline

Evaluation pipeline
Conceptual workflow
  1. 01Model and dataset selection
  2. 02Evaluation configuration
  3. 03Experiment execution
  4. 04Metric calculation
  5. 05Scenario-level analysis
  6. 06Failure investigation
  7. 07Evaluation report

Scenario refinement: failure categories become new scenario requirements

Evaluation metrics

Metrics depend on the task. Not every metric applies to every task, and technical metrics alone do not establish real-world fitness.

Classification

One label per input.

Accuracy
Share of correct predictions; misleading when classes are imbalanced.
Precision
Of the inputs predicted as a class, how many truly were (TP / (TP + FP)).
Recall
Of the inputs truly in a class, how many were found (TP / (TP + FN)).
F1 Score
Harmonic mean of precision and recall.

Object detection

Boxes matched to ground truth at an overlap threshold.

Precision
Share of predicted boxes that match a real object.
Recall
Share of real objects that were detected.
Mean Average Precision
Area under the precision–recall curve, averaged over classes (and often over IoU thresholds).

Segmentation

Per-pixel predictions compared with reference masks.

Dice Score
2|A ∩ B| / (|A| + |B|), the overlap between predicted and reference masks.
Intersection over Union
|A ∩ B| / |A ∪ B|, a stricter overlap measure related to Dice.

Robustness

Applies on top of any task metric.

Performance by condition
The task metric reported separately for each scenario group.
Performance degradation
The change from a reference condition to a harder one.
Failure distribution
How failures split across categories and conditions.
Scenario coverage
Which conditions the test set actually contains, and how many samples each has.

Experiment configuration

An illustrative schema. Fix it before looking at results, so the protocol cannot drift to fit the numbers.
experiments/EXP-SAMPLE-014.jsonIllustrative
{
  "experiment_id": "EXP-SAMPLE-014",
  "model_reference": {
    "name": "example-segmentation-model",
    "version": "0.3.0",
    "checkpoint_sha256": "<recorded at run time>"
  },
  "dataset_reference": "example-maize-weed-seg",
  "dataset_version": "1.0.0",
  "evaluation_protocol": {
    "split": "test",
    "held_out_real_data": true,
    "resize": [
      512,
      512
    ],
    "threshold": 0.5
  },
  "metrics": [
    "iou",
    "dice",
    "precision",
    "recall"
  ],
  "scenario_groups": [
    "clear-day",
    "overcast",
    "dusk",
    "dense-weeds"
  ],
  "random_seed": 1234
}
Experiment configuration fields
FieldTypeDescription
experiment_idstringUnique identifier for the evaluation run.
model_referenceobjectModel name, version and a checksum of the evaluated weights.
dataset_referencestringThe dataset_id from the dataset manifest.
dataset_versionstringExact dataset version evaluated; results are only comparable at the same version.
evaluation_protocolobjectSplit, preprocessing and thresholds, fixed before results are seen.
metricsstring[]Metrics appropriate to the task (see Evaluation metrics).
scenario_groupsstring[]Condition subsets reported separately, not only in aggregate.
random_seedintegerSeed for any stochastic step, so the run can be repeated.

Interactive evaluation example

Compare a baseline with an augmented dataset and with challenging conditions. Every number is sample data.

Offline evaluation example

Sample data
Comparison

IoU by scenario group (baseline → augmented) · sample data

  • Clear day0.76 → 0.77

  • Overcast0.70 → 0.73

  • Dusk0.58 → 0.66

  • Dense weeds0.55 → 0.61

Baseline Augmented Dataset

Example failure categories · sample data

CategoryBaselineCompared
Weed missed under crop canopy1813
Soil texture labelled as weed1114
Boundary error at row edges2420

Reading this example: In this sample, augmentation helps most where the baseline was weakest, but one failure type gets worse. Both effects must be checked on real data. IoU measures overlap between predicted and reference masks (1 is perfect); see Evaluation metrics above.

experiment metadataSample data
{
  "experiment_id": "EXP-SAMPLE-011",
  "training_data": "real + synthetic (1:1)",
  "test_split": "standard",
  "random_seed": 1234,
  "sample_data": true
}

Report structure

Evaluation report · EXP-SAMPLE-014

Crop and weed segmentation, scenario breakdown

Sample data

Experiment summary

Illustrative report layout. Values are sample data, not results.

Dataset reference

example-maize-weed-seg v1.0.0 · test split

Model reference

example-segmentation-model v0.3.0

Evaluation conditions

clear-day, overcast, dusk, dense-weeds

Metric definitions

IoU = |A ∩ B| / |A ∪ B|; Dice = 2|A ∩ B| / (|A| + |B|), computed per class and averaged.

Scenario analysis (sample data)

ConditionIoUDice
clear-day0.760.86
overcast0.720.84
dusk0.630.77
dense-weeds0.600.75

Failure categories (sample data)

  • weed missed under crop canopy14
  • soil texture labelled as weed9
  • boundary error at row edges21

Technical limitations

  • Sample values only.
  • Single seed; no confidence intervals.
  • No real-world field validation.

Next experiment recommendations

  • Add dusk and dense-weed scenarios to the training data.
  • Repeat with three seeds.
  • Validate on held-out real field images.
Sample report layout. Values are sample data written for documentation, not results.

Reproducibility

  • Dataset versioning: evaluate against an immutable dataset id and version.
  • Model versioning: record the model version and a checksum of the evaluated weights.
  • Fixed evaluation protocols: decide the split, preprocessing and thresholds before seeing results.
  • Random seeds: record seeds, and repeat with several seeds before trusting small differences.
  • Configuration tracking: store the experiment configuration next to its report.
  • Result provenance: every number in a report should trace back to a run, a configuration and a dataset version.