Working with the Synthex Data Engine
Intended data generation, annotation, curation and quality assurance workflows, data representations, annotation formats and dataset manifests.
Page type: Explains a workflow step by step.
Overview
The Data Engine is designed to turn raw or generated samples into datasets you can trust and reference. Its intended responsibilities:
- prepare annotations in open formats;
- validate quality before data is used;
- curate, version and package datasets with manifests;
- record provenance so every sample can be traced.
Data workflow
- 01Dataset requirements
- 02Scenario configuration
- 03Data generation or acquisition
- 04Annotation preparation
- 05Quality validation
- 06Dataset curation
- 07Dataset packaging
- 08Model training or evaluation
Data representations
ImagesProposed
RGB frames stored as lossless PNG, with consistent naming so labels and metadata can be matched by file name.
Segmentation masksProposed
One mask per image, pixel-aligned. Indexed masks store a class index per pixel.
Bounding boxesProposed
Axis-aligned boxes per object, in pixels (COCO-style) or normalised centres (YOLO-style).
Classification labelsProposed
One class per image, typically a CSV of file name and class.
Dataset metadataProposed
The manifest: identity, version, task, classes, splits, licence and provenance.
Scenario metadataProposed
Per-sample parameters such as lighting, growth stage and seed, so results can be broken down by condition.
Experiment referencesProposed
Links from experiments to the exact dataset id and version they used.
Annotation formats
{
"images": [
{
"id": 17,
"file_name": "images/field_000017.png",
"width": 640,
"height": 480
}
],
"categories": [
{
"id": 1,
"name": "crop"
},
{
"id": 2,
"name": "weed"
}
],
"annotations": [
{
"id": 301,
"image_id": 17,
"category_id": 2,
"segmentation": [
[
212,
96,
231.5,
94,
238,
110.5,
220,
118,
209.5,
108
]
],
"bbox": [
209.5,
94,
28.5,
24
],
"area": 471.25,
"iscrowd": 0
}
]
}# images/street_000042.txt — one object per line
# class_id x_center y_center width height (normalised to 0–1)
0 0.512 0.634 0.081 0.243
2 0.274 0.588 0.196 0.137{
"sample_id": "example-maize-weed-seg/000017",
"image": "images/field_000017.png",
"mask": "masks/field_000017.png",
"mask_encoding": "png-indexed",
"class_index": {
"0": "soil",
"1": "crop",
"2": "weed"
},
"scenario": {
"lighting": "overcast",
"growth_stage": "V4",
"seed": 1234
}
}Dataset manifest explorer
Educational manifest explorer
Illustrative values{
"dataset_id": "example-maize-weed-seg",
"dataset_version": "1.0.0",
"created_at": "2026-01-15T09:00:00Z",
"license": "CC-BY-4.0",
"task_type": "semantic-segmentation",
"annotation_format": "png-indexed-masks",
"classes": [
"soil",
"crop",
"weed"
],
"scenario_reference": "scenarios/maize-weed-overcast-v1.json",
"sample_count": 240,
"splits": {
"train": 168,
"validation": 36,
"test": 36
},
"provenance": {
"source": "synthetic",
"generator": "illustrative-example",
"seed": 1234
}
}Explain each field
- dataset_id
- Stable identifier for the dataset across versions.
- dataset_version
- Version of this dataset release; change it whenever samples, labels or splits change.
- created_at
- When this version was packaged.
- license
- Terms under which the dataset may be used.
- task_type
- The AI task the labels support, e.g. object-detection.
- annotation_format
- How labels are stored, e.g. coco-json or png-indexed-masks.
- classes
- Ordered class names; mask values and class ids index into this list.
- scenario_reference
- Scenario configuration used to generate synthetic samples, if any.
- sample_count
- Total number of samples; must equal the sum of the splits.
- splits
- Sample counts per split. Splits must not share source scenes or patients.
- provenance
- Where samples came from (real, synthetic, mixed), how they were made, and the seed.
Quality assurance
Missing files
Every image has its label and metadata file, and nothing is orphaned.
Annotation mismatches
Label dimensions match the image; boxes and polygons lie inside the frame.
Invalid classes
Every class id or mask value exists in the manifest's class list.
Mask consistency
Masks are single-channel, aligned, and use only expected values.
Duplicate records
No repeated samples or near-duplicates across splits, which would leak test data into training.
Metadata consistency
Required fields are present, types are correct, and counts match the manifest.
Dataset versioning
Any change to samples, labels or splits produces a new version.
Provenance tracking
Each sample records whether it is real or synthetic and how it was produced.
Reproducibility
A result is only meaningful if you can say exactly which data produced it. Manifests give each dataset an identity and version. Controlled configurations (scenario versions and fixed seeds) let a dataset be regenerated. Versioning ensures that two experiments claiming the same dataset really used the same files.
In practice, never overwrite a released dataset version. Record the dataset id and version in every experiment configuration. Keep generation configurations alongside the data.