Evaluating AI Models with Synthex
Intended model evaluation, benchmarking and robustness workflows: metrics by task, experiment configuration, report structure and reproducibility.
Page type: Explains a workflow step by step.
Overview
An aggregate score can hide the cases that matter most. A model with good average accuracy may fail at dusk, on small objects or on an unseen sensor. The Evaluation Engine is designed to test models against fixed protocols, report results per condition, and turn failures into specific next experiments.
Evaluation pipeline
- 01Model and dataset selection
- 02Evaluation configuration
- 03Experiment execution
- 04Metric calculation
- 05Scenario-level analysis
- 06Failure investigation
- 07Evaluation report
Scenario refinement: failure categories become new scenario requirements
Evaluation metrics
Classification
One label per input.
- Accuracy
- Share of correct predictions; misleading when classes are imbalanced.
- Precision
- Of the inputs predicted as a class, how many truly were (TP / (TP + FP)).
- Recall
- Of the inputs truly in a class, how many were found (TP / (TP + FN)).
- F1 Score
- Harmonic mean of precision and recall.
Object detection
Boxes matched to ground truth at an overlap threshold.
- Precision
- Share of predicted boxes that match a real object.
- Recall
- Share of real objects that were detected.
- Mean Average Precision
- Area under the precision–recall curve, averaged over classes (and often over IoU thresholds).
Segmentation
Per-pixel predictions compared with reference masks.
- Dice Score
- 2|A ∩ B| / (|A| + |B|), the overlap between predicted and reference masks.
- Intersection over Union
- |A ∩ B| / |A ∪ B|, a stricter overlap measure related to Dice.
Robustness
Applies on top of any task metric.
- Performance by condition
- The task metric reported separately for each scenario group.
- Performance degradation
- The change from a reference condition to a harder one.
- Failure distribution
- How failures split across categories and conditions.
- Scenario coverage
- Which conditions the test set actually contains, and how many samples each has.
Experiment configuration
{
"experiment_id": "EXP-SAMPLE-014",
"model_reference": {
"name": "example-segmentation-model",
"version": "0.3.0",
"checkpoint_sha256": "<recorded at run time>"
},
"dataset_reference": "example-maize-weed-seg",
"dataset_version": "1.0.0",
"evaluation_protocol": {
"split": "test",
"held_out_real_data": true,
"resize": [
512,
512
],
"threshold": 0.5
},
"metrics": [
"iou",
"dice",
"precision",
"recall"
],
"scenario_groups": [
"clear-day",
"overcast",
"dusk",
"dense-weeds"
],
"random_seed": 1234
}// Illustrative TypeScript type for the example experiment configuration.
// Not part of a published Synthex package.
export type ExperimentConfig = {
experiment_id: string;
model_reference: { name: string; version: string; checkpoint_sha256: string };
dataset_reference: string;
dataset_version: string;
evaluation_protocol: {
split: "validation" | "test";
held_out_real_data: boolean;
resize?: [number, number];
threshold?: number;
};
metrics: Array<"accuracy" | "precision" | "recall" | "f1" | "map" | "iou" | "dice">;
scenario_groups: string[];
random_seed: number;
};
| Field | Type | Description |
|---|---|---|
| experiment_id | string | Unique identifier for the evaluation run. |
| model_reference | object | Model name, version and a checksum of the evaluated weights. |
| dataset_reference | string | The dataset_id from the dataset manifest. |
| dataset_version | string | Exact dataset version evaluated; results are only comparable at the same version. |
| evaluation_protocol | object | Split, preprocessing and thresholds, fixed before results are seen. |
| metrics | string[] | Metrics appropriate to the task (see Evaluation metrics). |
| scenario_groups | string[] | Condition subsets reported separately, not only in aggregate. |
| random_seed | integer | Seed for any stochastic step, so the run can be repeated. |
Interactive evaluation example
Offline evaluation example
Sample dataIoU by scenario group (baseline → augmented) · sample data
Clear day0.76 → 0.77
Overcast0.70 → 0.73
Dusk0.58 → 0.66
Dense weeds0.55 → 0.61
Baseline Augmented Dataset
Example failure categories · sample data
| Category | Baseline | Compared |
|---|---|---|
| Weed missed under crop canopy | 18 | 13 |
| Soil texture labelled as weed | 11 | 14 |
| Boundary error at row edges | 24 | 20 |
Reading this example: In this sample, augmentation helps most where the baseline was weakest, but one failure type gets worse. Both effects must be checked on real data. IoU measures overlap between predicted and reference masks (1 is perfect); see Evaluation metrics above.
{
"experiment_id": "EXP-SAMPLE-011",
"training_data": "real + synthetic (1:1)",
"test_split": "standard",
"random_seed": 1234,
"sample_data": true
}Report structure
Evaluation report · EXP-SAMPLE-014
Crop and weed segmentation, scenario breakdown
Experiment summary
Dataset reference
Model reference
Evaluation conditions
Metric definitions
Scenario analysis (sample data)
| Condition | IoU | Dice |
|---|---|---|
| clear-day | 0.76 | 0.86 |
| overcast | 0.72 | 0.84 |
| dusk | 0.63 | 0.77 |
| dense-weeds | 0.60 | 0.75 |
Failure categories (sample data)
- weed missed under crop canopy14
- soil texture labelled as weed9
- boundary error at row edges21
Technical limitations
- Sample values only.
- Single seed; no confidence intervals.
- No real-world field validation.
Next experiment recommendations
- Add dusk and dense-weed scenarios to the training data.
- Repeat with three seeds.
- Validate on held-out real field images.
Reproducibility
- Dataset versioning: evaluate against an immutable dataset id and version.
- Model versioning: record the model version and a checksum of the evaluated weights.
- Fixed evaluation protocols: decide the split, preprocessing and thresholds before seeing results.
- Random seeds: record seeds, and repeat with several seeds before trusting small differences.
- Configuration tracking: store the experiment configuration next to its report.
- Result provenance: every number in a report should trace back to a run, a configuration and a dataset version.