Research Problem Definition
State a specific, falsifiable question and the result that would answer it — including what would count as 'no effect'.
- Written hypothesis
- Pre-agreed success criteria
- Defined scope and exclusions
Research Methodology
How Synthex approaches synthetic data and AI evaluation research: documented data, controlled comparisons, reproducible experiments and clearly stated limitations.
Synthetic data is evaluated for usefulness; it is never assumed to improve model performance.
Core commitment
Synthetic data must earn its place.
Synthetic datasets are evaluated for their usefulness against real-data baselines on real held-out data. They are never assumed to improve model performance, and results where synthetic data does not help are reported.
Research lifecycle
Each stage writes to a shared record — the experiment manifest — so any result can be traced back to its question, data and settings.
Methodology
State a specific, falsifiable question and the result that would answer it — including what would count as 'no effect'.
Know where every sample came from, under which licence, and who may use it for what.
Design experiments so the effect of one change can be isolated from everything else.
Generate data with recorded parameters, and treat its usefulness as a question to test.
Check labels before they are used to train or judge a model.
Compare against the right baselines, including simpler alternatives that might explain the same gain.
Choose metrics that fit the task and report them per condition, with uncertainty.
Measure how performance changes when conditions change, not just how high it is.
Any reported result must be re-creatable from its manifest.
Report what the work cannot show, including negative results, and avoid claims beyond the evidence.
Methodological practices
Dataset cards describing source, licence, composition and known gaps.
Every run tied to exact data, code and model versions.
Change one thing at a time against a fixed baseline.
Splits fixed before experimentation and never re-drawn after results.
Patient-, scene- or field-level splits so related samples never cross splits.
Defined criteria and recorded decisions for expert review.
Scenario parameters, provenance and review status stored per sample.
Seeds recorded and varied; results reported across seeds.
Failures grouped and inspected, not just counted.
Every report states what it does not show.
On synthetic data
Experiment manifest
The fields every experiment would record. This is a conceptual schema to explain the method — not evidence that a working experiment platform exists.
| Field | Type | Purpose |
|---|---|---|
| experiment_id | string | Unique, immutable identifier for the run. |
| dataset_version | string | Exact dataset version, including split definitions. |
| model_version | string | Architecture and weights version, or training code commit. |
| evaluation_protocol | string | Named protocol fixing metrics, thresholds and test sets. |
| random_seed | integer | Seed for training and sampling, recorded per run. |
| scenario_config | string | Versioned scenario or generation configuration. |
| metric_definitions | string[] | Exact metric definitions used, to avoid ambiguity. |
| result_artifact | string | Pointer to the stored results and report. |
manifest.json
Conceptual schema{
"experiment_id": "<experiment id>",
"dataset_version": "<dataset version>",
"model_version": "<model version>",
"evaluation_protocol": "<evaluation protocol>",
"random_seed": <integer>,
"scenario_config": "<scenario config>",
"metric_definitions": ["<metric definition>", "…"],
"result_artifact": "<result artifact>"
}Placeholder values only. This schema explains the method; it is not evidence that a working experiment platform exists.
Rigorous evaluation
If rigorous, reproducible evaluation matters to your team, we would like to hear about the questions you need answered.