Getting Started with Synthex
Core Synthex concepts, the intended dataset and evaluation workflow, an illustrative project structure and a first dataset manifest. No backend required.
Page type: Explains a workflow step by step.
Introduction
Synthex is developing infrastructure to generate, simulate, annotate and evaluate datasets for physical-world AI. It is organised around three proposed engines:
- Simulation Engine: turns scenario configurations into virtual environments and sensor data with ground truth.
- Data Engine: prepares annotations, checks quality, and versions and packages datasets.
- Evaluation Engine: tests models against fixed protocols, overall and per condition, and reports failures.
A typical workflow moves from a task and a scenario to a versioned dataset, then to training in your own environment, then to evaluation. Evaluation findings feed back into new scenarios, so dataset generation is driven by where models fail rather than by volume.
Platform concepts
- Scenario
- A structured description of the conditions data should cover or a model should face: environment, lighting, weather, objects, agents, sensors and a random seed.
- Simulation
- Running a scenario in a virtual environment to produce sensor data together with exact ground truth.
- Synthetic Dataset
- A dataset whose samples were generated rather than captured. It is only useful if it improves results on real data.
- Annotation
- The labels attached to a sample: a class, bounding boxes, polygons or a per-pixel mask, plus who or what produced them.
- Dataset Manifest
- A small JSON document that identifies a dataset version: its task, classes, format, splits, licence and provenance.
- Experiment
- A recorded attempt to answer one question, such as whether synthetic dusk images improve detection at dusk.
- Evaluation Run
- One execution of a model against a fixed dataset version and protocol, producing metrics and failure cases.
- Robustness Test
- An evaluation that compares performance across controlled conditions to see where and how much it degrades.
- Model Benchmark
- A repeatable evaluation used to compare models or versions on equal terms.
- Research Output
- A documented result (report, dataset card, figure) with its configuration, so others can check and reproduce it.
Understanding a synthetic data evaluation workflow
- 1
Define an AI task
State the task, the classes and what improvement would matter. Example: semantic segmentation of soil, crop and weed, where missing weeds is the costly error.
task.jsonIllustrativeJSON{ "task_type": "semantic-segmentation", "classes": [ "soil", "crop", "weed" ], "priority_metric": "weed recall" } - 2
Describe a scenario
List the conditions the data must cover, especially those your current data lacks, such as overcast light or dense weeds.
scenario.jsonIllustrativeJSON{ "environment": "agricultural", "lighting": [ "day", "overcast", "dusk" ], "weed_density": { "min": 0.05, "max": 0.25 }, "random_seed": 1234 } - 3
Specify required data representations
Decide what each sample contains: an RGB image, a per-pixel mask aligned to it, and per-sample metadata recording the scenario parameters.
- 4
Define annotation types
Choose the label format your tooling reads. For segmentation, indexed PNG masks whose pixel values index the class list are a simple, common choice.
- 5
Create a dataset manifest
Record the dataset id, version, task, format, classes, splits, licence and provenance, so the dataset can be referenced exactly. See the manifest below.
- 6
Choose model evaluation metrics
Pick metrics that match the task and report them per condition, not only overall. For segmentation: IoU and Dice per class, plus recall for the costly class.
experiment.jsonIllustrativeJSON{ "experiment_id": "EXP-SAMPLE-014", "model_reference": { "name": "example-segmentation-model", "version": "0.3.0", "checkpoint_sha256": "<recorded at run time>" }, "dataset_reference": "example-maize-weed-seg", "dataset_version": "1.0.0", "evaluation_protocol": { "split": "test", "held_out_real_data": true, "resize": [ 512, 512 ], "threshold": 0.5 }, "metrics": [ "iou", "dice", "precision", "recall" ], "scenario_groups": [ "clear-day", "overcast", "dusk", "dense-weeds" ], "random_seed": 1234 } - 7
Review an illustrative evaluation report
A report states what was run, results by condition, failure categories, limitations and the next experiment. The Evaluation Engine guide includes a sample report viewer.
Illustrative project structure
synthex-example/
├── scenarios/
│ └── maize-weed-overcast-v1.json
├── datasets/
│ ├── images/ # field_000001.png …
│ ├── annotations/ # indexed PNG masks, same file names
│ ├── metadata/ # one JSON file per sample
│ └── dataset_manifest.json
├── experiments/
│ └── EXP-SAMPLE-014.json
├── evaluation/
│ └── EXP-SAMPLE-014-report.json
└── README.mdYour first dataset manifest
{
"dataset_id": "example-maize-weed-seg",
"dataset_version": "1.0.0",
"created_at": "2026-01-15T09:00:00Z",
"license": "CC-BY-4.0",
"task_type": "semantic-segmentation",
"annotation_format": "png-indexed-masks",
"classes": [
"soil",
"crop",
"weed"
],
"scenario_reference": "scenarios/maize-weed-overcast-v1.json",
"sample_count": 240,
"splits": {
"train": 168,
"validation": 36,
"test": 36
},
"provenance": {
"source": "synthetic",
"generator": "illustrative-example",
"seed": 1234
}
}| Field | Type | Required | Description |
|---|---|---|---|
| dataset_id | string | Yes | Stable identifier for the dataset across versions. |
| dataset_version | string (semver) | Yes | Version of this dataset release; change it whenever samples, labels or splits change. |
| created_at | string (ISO 8601, UTC) | Yes | When this version was packaged. |
| license | string (SPDX id) | Yes | Terms under which the dataset may be used. |
| task_type | string | Yes | The AI task the labels support, e.g. object-detection. |
| annotation_format | string | Yes | How labels are stored, e.g. coco-json or png-indexed-masks. |
| classes | string[] | Yes | Ordered class names; mask values and class ids index into this list. |
| scenario_reference | string | No | Scenario configuration used to generate synthetic samples, if any. |
| sample_count | integer | Yes | Total number of samples; must equal the sum of the splits. |
| splits | object | Yes | Sample counts per split. Splits must not share source scenes or patients. |
| provenance | object | Yes | Where samples came from (real, synthetic, mixed), how they were made, and the seed. |
A standard-library Python validator for this manifest is in the code examples.
Next steps
- Data Engine GuideRepresentations, annotation formats, manifests and quality assurance.
- Simulation Engine GuideScenario configuration, sensors and sim-to-real challenges.
- Evaluation Engine GuideMetrics, experiment configuration, reports and reproducibility.
- ArchitectureHow the proposed layers and engines fit together.