Skip to content

Synthex Evaluation Engine

Go Beyond Accuracy. Understand Model Reliability.

Compare model behaviour across datasets, conditions and difficult scenarios using repeatable evaluation workflows.

Engine 03Platform VisionIn development — not generally available.
Evaluation overviewIllustrative sample data
Illustrative evaluation overview using sample data: a distribution of per-sample IoU, a scenario coverage grid showing fewer samples in fog and at night, failure categories led by small and occluded objects, and two sample models compared by F1 across conditions. Not real results.

Evaluation capabilities

Evidence about where models hold up — and where they do not

Planned capabilities for structured, repeatable evaluation of models and the datasets they are trained on.

  • Platform Vision

    Model Benchmarking

    Run models against fixed, versioned test sets with consistent metrics and settings.

  • Platform Vision

    Scenario-Based Testing

    Break results down by environment, condition and difficulty instead of one headline number.

  • Platform Vision

    Edge-Case Analysis

    Surface the rare situations where errors cluster, with examples to inspect.

  • Platform Vision

    Dataset Quality Evaluation

    Check coverage, balance and label consistency to judge whether a dataset is fit for purpose.

  • Research Direction

    Synthetic-to-Real Generalisation

    Measure how models trained with synthetic data perform on real-world held-out data.

  • Platform Vision

    Experiment Comparison and Reporting

    Compare runs side by side and produce reproducible reports tied to exact data and model versions.

How to read status labels

Platform Vision
A planned capability that defines where the platform is heading. Not yet available.
Research Direction
Active exploration of methods and questions. No prototype or results yet; not a product.
Illustrative Demonstration
A visual or interactive explanation using sample content. No live system or real results behind it.
Validated Implementation
Implemented and verified against defined criteria. No capability on this site is presented at this level yet.

Metrics explainer

Choosing the right measures

No single metric describes model quality. Which measures are suitable depends on the task, the dataset and the cost of different errors.

Computer vision

PrecisionTP / (TP + FP)
Of everything the model flagged, how much was correct. Matters when false alarms are costly.
RecallTP / (TP + FN)
Of everything that should be found, how much was found. Matters when misses are costly.
F1 Score2 · P · R / (P + R)
Harmonic mean of precision and recall; a single balance point between the two.
IoU|A ∩ B| / |A ∪ B|
Overlap between predicted and reference regions, used for boxes and masks.
Dice Score2 |A ∩ B| / (|A| + |B|)
Overlap measure common in medical segmentation; closely related to IoU.

Model robustness

Performance by scenariometric | condition
The same metric computed per environment, condition or difficulty level.
Performance degradation(m_ref − m_cond) / m_ref
Relative drop from a reference condition, showing how sharply performance falls.
Distribution shift indicatorsΔ inputs, Δ confidence
Signals that test data differs from training data, such as feature statistics or calibration changes.
Failure-case frequencyfailures / samples
How often defined failure types occur, overall and per scenario.

Suitability depends on context. TP, FP and FN are true positives, false positives and false negatives; A and B are predicted and reference regions. A metric that suits detection may mislead for segmentation, and class imbalance, label quality and the cost of each error type all change which measures matter.

Interactive evaluation example

Compare evaluation contexts

Switch between evaluation sets to see how a report could break results down. All values are sample data for demonstrating the interface — not experimental results and not Synthex research findings.

Sample data only

Values are invented for demonstration. They are not Synthex research findings or results from any model.

Sample data · not experimental results

Illustrative Demonstration
Overall metrics
  • Precision0.84
  • Recall0.78
  • F1 Score0.81
  • IoU0.69
F1 score by scenario
  • Clear0.86
  • Rain0.71
  • Fog0.62
  • Low light0.66
Failure cases by category (sample counts)
  • Missed small objects42
  • Occlusion31
  • Low contrast27
  • False positives on reflections12

Reading this sample: In this sample, the headline F1 looks healthy, but the scenario view shows a clear drop in fog and low light. That gap is what a single accuracy number hides.

View sample values as a table
Sample values for Baseline Dataset compared with Baseline Dataset
MeasureBaseline DatasetBaseline Dataset
Precision0.840.84
Recall0.780.78
F1 Score0.810.81
IoU0.690.69
F1 · Clear0.860.86
F1 · Rain0.710.71
F1 · Fog0.620.62
F1 · Low light0.660.66
Failures · Missed small objects4242
Failures · Occlusion3131
Failures · Low contrast2727
Failures · False positives on reflections1212

Evaluation workflow

Evaluation that feeds the next data cycle

Findings are only useful if they change what happens next. Failure analysis informs new scenarios in the Simulation Engine and new data in the Data Engine.

  1. 01

    Input Model and Dataset

    Register the model version and the dataset versions it will be tested on.

  2. 02

    Evaluation Configuration

    Choose metrics, scenario breakdowns, thresholds and comparison baselines.

  3. 03

    Benchmark Execution

    Run inference and scoring under fixed, recorded settings.

  4. 04

    Scenario-Level Analysis

    Break results down by environment, condition and difficulty.

  5. 05

    Failure Investigation

    Group errors into failure categories and inspect representative examples.

  6. 06

    Evaluation Report

    Summarise findings in a reproducible report tied to exact versions.

  7. 07

    Next Data Generation Cycle

    Turn findings into scenario and dataset requirements for the next iteration.

Next cycle · returns to step 01

Report findings define new scenarios and dataset requirements for the Simulation and Data engines.

Report preview

What an evaluation report could contain

An illustrative layout with sample values. It is not a live report, and no download or export is available.

Evaluation report

Adverse-weather robustness · sample experiment

EXP-SAMPLE-001 · values are illustrative

Illustrative Demonstration

Experiment summary

Objective
Assess detection under rain, fog and low light
Comparison
Augmented Dataset vs Baseline Dataset
Status
Illustrative sample

Dataset information

Test set
Standard split + scenario breakdown (sample)
Conditions
Clear, Rain, Fog, Low light
Version
dataset v0.1 (sample)

Model information

Model
Sample object detector
Training data
Baseline + synthetic adverse weather
Version
model v0.3 (sample)

Metric breakdown

MetricBaseline DatasetAugmented DatasetΔ
Precision0.840.85+0.01
Recall0.780.82+0.04
F1 Score0.810.83+0.02
IoU0.690.71+0.02

Scenario-level comparison · F1

  • Clear0.86 → 0.86
  • Rain0.71 → 0.77
  • Fog0.62 → 0.71
  • Low light0.66 → 0.74
Baseline Dataset Augmented Dataset

Identified failure categories

  • Missed small objects42 → 37
  • Occlusion31 → 28
  • Low contrast27 → 16
  • False positives on reflections12 → 15

Suggested next experiments

  • Investigate the rise in reflection false positives; add wet-road and glass scenarios.
  • Generate more small and occluded pedestrians in fog for the next dataset version.
  • Validate against held-out real-world imagery before drawing conclusions.
Illustrative layout with sample values. Not a live report, not experimental results, and no download is available.