Skip to content

Research Methodology

Rigour Before Results

How Synthex approaches synthetic data and AI evaluation research: documented data, controlled comparisons, reproducible experiments and clearly stated limitations.

Synthetic data is evaluated for usefulness; it is never assumed to improve model performance.

Core commitment

Synthetic data must earn its place.

Synthetic datasets are evaluated for their usefulness against real-data baselines on real held-out data. They are never assumed to improve model performance, and results where synthetic data does not help are reported.

  • Fixed baselines
  • Real held-out evaluation
  • Leakage-safe splits
  • Negative results reported

Research lifecycle

Every stage is versioned

Each stage writes to a shared record — the experiment manifest — so any result can be traced back to its question, data and settings.

Research lifecycle of eight stages arranged in a loop around the experiment manifest, which records every stage.
  1. 01Research questionFalsifiable question and success criterion.
  2. 02Datasets and scenariosLicensed sources, provenance recorded.
  3. 03Experiment designBaselines, splits and metrics fixed up front.
  4. 04Generation or processingVersioned, parameters stored.
  5. 05Annotation validationChecks and expert review.
  6. 06Training or evaluationFixed configs, documented seeds.
  7. 07Results analysisPer-condition results and errors.
  8. 08Reproducibility reviewRe-run from the manifest.

Methodology

Ten principles for credible research

  1. 01

    Research Problem Definition

    State a specific, falsifiable question and the result that would answer it — including what would count as 'no effect'.

    • Written hypothesis
    • Pre-agreed success criteria
    • Defined scope and exclusions
  2. 02

    Dataset Provenance and Governance

    Know where every sample came from, under which licence, and who may use it for what.

    • Dataset cards
    • Licence and usage records
    • De-identification checks
    • Access governance
  3. 03

    Scenario and Experimental Design

    Design experiments so the effect of one change can be isolated from everything else.

    • One variable at a time
    • Scenario specifications under version control
    • Coverage plans
  4. 04

    Synthetic Data Generation

    Generate data with recorded parameters, and treat its usefulness as a question to test.

    • Parameters stored per sample
    • Generator versioning
    • Synthetic data kept out of evaluation sets
  5. 05

    Annotation and Quality Assurance

    Check labels before they are used to train or judge a model.

    • Automated mask checks
    • Inter-annotator agreement
    • Expert review queues
    • Recorded review decisions
  6. 06

    Experimental Controls and Baselines

    Compare against the right baselines, including simpler alternatives that might explain the same gain.

    • Real-data baselines
    • Conventional augmentation baselines
    • Identical training budgets
  7. 07

    Model Evaluation

    Choose metrics that fit the task and report them per condition, with uncertainty.

    • Task-appropriate metrics
    • Per-condition breakdowns
    • Multiple seeds and confidence intervals
  8. 08

    Robustness and Distribution Shift

    Measure how performance changes when conditions change, not just how high it is.

    • Held-out conditions
    • External test data
    • Degradation analysis
  9. 09

    Reproducibility and Versioning

    Any reported result must be re-creatable from its manifest.

    • Versioned data, code and models
    • Documented random seeds
    • Recorded environments
  10. 10

    Limitations and Responsible AI

    Report what the work cannot show, including negative results, and avoid claims beyond the evidence.

    • Limitations sections
    • Negative results reported
    • Domain expert review
    • No clinical or safety claims from technical metrics

Methodological practices

Practices we apply to every study

  • Dataset documentation

    Dataset cards describing source, licence, composition and known gaps.

  • Versioned experiments

    Every run tied to exact data, code and model versions.

  • Controlled comparisons

    Change one thing at a time against a fixed baseline.

  • Training/validation/test separation

    Splits fixed before experimentation and never re-drawn after results.

  • Leakage prevention

    Patient-, scene- or field-level splits so related samples never cross splits.

  • Human-review procedures

    Defined criteria and recorded decisions for expert review.

  • Metadata tracking

    Scenario parameters, provenance and review status stored per sample.

  • Random seed documentation

    Seeds recorded and varied; results reported across seeds.

  • Error analysis

    Failures grouped and inspected, not just counted.

  • Clear limitations

    Every report states what it does not show.

On synthetic data

Synthetic datasets are evaluated for their usefulness against real-data baselines on real held-out data. They are never assumed to improve model performance, and results where synthetic data does not help are reported.

Experiment manifest

A conceptual experiment record

The fields every experiment would record. This is a conceptual schema to explain the method — not evidence that a working experiment platform exists.

Conceptual experiment manifest fields
FieldTypePurpose
experiment_idstringUnique, immutable identifier for the run.
dataset_versionstringExact dataset version, including split definitions.
model_versionstringArchitecture and weights version, or training code commit.
evaluation_protocolstringNamed protocol fixing metrics, thresholds and test sets.
random_seedintegerSeed for training and sampling, recorded per run.
scenario_configstringVersioned scenario or generation configuration.
metric_definitionsstring[]Exact metric definitions used, to avoid ambiguity.
result_artifactstringPointer to the stored results and report.

manifest.json

Conceptual schema
{
  "experiment_id": "<experiment id>",
  "dataset_version": "<dataset version>",
  "model_version": "<model version>",
  "evaluation_protocol": "<evaluation protocol>",
  "random_seed": <integer>,
  "scenario_config": "<scenario config>",
  "metric_definitions": ["<metric definition>", "…"],
  "result_artifact": "<result artifact>"
}

Placeholder values only. This schema explains the method; it is not evidence that a working experiment platform exists.