Model Benchmarking
Run models against fixed, versioned test sets with consistent metrics and settings.
Synthex Evaluation Engine
Compare model behaviour across datasets, conditions and difficult scenarios using repeatable evaluation workflows.
Evaluation capabilities
Planned capabilities for structured, repeatable evaluation of models and the datasets they are trained on.
Run models against fixed, versioned test sets with consistent metrics and settings.
Break results down by environment, condition and difficulty instead of one headline number.
Surface the rare situations where errors cluster, with examples to inspect.
Check coverage, balance and label consistency to judge whether a dataset is fit for purpose.
Measure how models trained with synthetic data perform on real-world held-out data.
Compare runs side by side and produce reproducible reports tied to exact data and model versions.
Metrics explainer
No single metric describes model quality. Which measures are suitable depends on the task, the dataset and the cost of different errors.
TP / (TP + FP)TP / (TP + FN)2 · P · R / (P + R)|A ∩ B| / |A ∪ B|2 |A ∩ B| / (|A| + |B|)metric | condition(m_ref − m_cond) / m_refΔ inputs, Δ confidencefailures / samplesSuitability depends on context. TP, FP and FN are true positives, false positives and false negatives; A and B are predicted and reference regions. A metric that suits detection may mislead for segmentation, and class imbalance, label quality and the cost of each error type all change which measures matter.
Interactive evaluation example
Switch between evaluation sets to see how a report could break results down. All values are sample data for demonstrating the interface — not experimental results and not Synthex research findings.
Sample data only
Sample data · not experimental results
Illustrative DemonstrationReading this sample: In this sample, the headline F1 looks healthy, but the scenario view shows a clear drop in fog and low light. That gap is what a single accuracy number hides.
| Measure | Baseline Dataset | Baseline Dataset |
|---|---|---|
| Precision | 0.84 | 0.84 |
| Recall | 0.78 | 0.78 |
| F1 Score | 0.81 | 0.81 |
| IoU | 0.69 | 0.69 |
| F1 · Clear | 0.86 | 0.86 |
| F1 · Rain | 0.71 | 0.71 |
| F1 · Fog | 0.62 | 0.62 |
| F1 · Low light | 0.66 | 0.66 |
| Failures · Missed small objects | 42 | 42 |
| Failures · Occlusion | 31 | 31 |
| Failures · Low contrast | 27 | 27 |
| Failures · False positives on reflections | 12 | 12 |
Evaluation workflow
Findings are only useful if they change what happens next. Failure analysis informs new scenarios in the Simulation Engine and new data in the Data Engine.
STEP 01
Register the model version and the dataset versions it will be tested on.
STEP 02
Choose metrics, scenario breakdowns, thresholds and comparison baselines.
STEP 03
Run inference and scoring under fixed, recorded settings.
STEP 04
Break results down by environment, condition and difficulty.
STEP 05
Group errors into failure categories and inspect representative examples.
STEP 06
Summarise findings in a reproducible report tied to exact versions.
STEP 07
Turn findings into scenario and dataset requirements for the next iteration.
Next cycle · returns to step 01
Report findings define new scenarios and dataset requirements for the Simulation and Data engines.
Report preview
An illustrative layout with sample values. It is not a live report, and no download or export is available.
Evaluation report
EXP-SAMPLE-001 · values are illustrative
| Metric | Baseline Dataset | Augmented Dataset | Δ |
|---|---|---|---|
| Precision | 0.84 | 0.85 | +0.01 |
| Recall | 0.78 | 0.82 | +0.04 |
| F1 Score | 0.81 | 0.83 | +0.02 |
| IoU | 0.69 | 0.71 | +0.02 |
Related engines
Next steps
We are interested in evaluation and robustness collaborations, from defining scenario suites to analysing failure modes.