Skip to content

Autonomous Perception Research

Researching Simulation-Driven Autonomous Perception

Investigating how controlled virtual environments and challenging scenarios can support perception-model development and evaluation.

Research direction. No simulator is integrated yet, and simulation alone does not establish safety.

One scenario · three sensorsIllustrative

scenario: urban.clear.dusk · not simulator output

Illustrative urban scene at dusk shown as an RGB camera image, a depth map and a LiDAR-style point pattern. Not simulator output.

Research problems

Why perception testing needs controlled scenarios

The difficulties simulation-based research is meant to make measurable — and the ones it introduces.

Evidence: Conceptual · R-009
  • Rare events

    Critical situations are too rare in recorded data to test reliably.

  • Adverse weather

    Rain and fog change both appearance and sensor returns.

  • Low visibility

    Night, glare and dust limit what sensors capture.

  • Sensor noise

    Noise, calibration drift and mounting change model inputs.

  • Unusual object configurations

    Occluded, fallen or out-of-place objects break assumptions.

  • Domain gaps

    Simulated data differs from real data in ways models can exploit.

  • Repeatability

    Physical-world experiments are hard to reproduce exactly.

Scenario research pipeline

From hypothesis to scenario refinement

The intended research loop. Capture and annotation generation are conceptual until a simulator integration exists.

  1. 01

    Research Hypothesis

    e.g. detection of partially occluded pedestrians degrades faster in fog than in rain.

  2. 02

    Scenario Specification

    Actors, positions, conditions and variation ranges, written down and versioned.

  3. 03

    Virtual Environment Configuration

    Map, assets and environment parameters for the scenario.

  4. 04

    Sensor Data CaptureConceptual

    RGB, depth or LiDAR-like data from modelled sensors.

  5. 05

    Annotation GenerationConceptual

    Labels derived from scene ground truth.

  6. 06

    Perception Model Evaluation

    Per-scenario, per-condition evaluation with fixed settings.

  7. 07

    Failure Analysis

    Group failures by cause and inspect representative frames.

  8. 08

    Scenario Refinement

    Design new variations around the failures found.

Refinement loop · returns to step 01

Failure analysis defines the next scenario variations; every loop is versioned so results stay comparable.

Interactive scenario explorer

What a scenario implies for data and evaluation

Pick an environment, weather, lighting and sensor to see the scenario's metadata, data outputs, evaluation objectives and perception challenges. Static illustration.

Illustrative Scenario Explorer — Not a Live Simulator

Illustrative Demonstration
Environment
Weather
Lighting
Sensor
ILLUSTRATION · NOT A SIMULATION OUTPUT

Scenario metadata

scenario_id:
urban.fog.dusk
environment:
Urban Street
weather:
Fog
lighting:
Dusk
sensor:
RGB Camera
ground_truth:
boxes, masks

Possible data outputs

  • RGB frames
  • 2D boxes and instance masks
  • Per-frame scenario metadata

Evaluation objective

Detection and segmentation accuracy per condition, comparing fog / dusk against clear daytime conditions.

Potential perception challenges

  • Dense occlusion from parked vehicles and street furniture
  • Contrast falls with distance; range shrinks
  • Low sun glare and long shadows
  • Appearance changes with weather and light

Static illustration. It does not execute CARLA, Unreal Engine, Unity, AirSim or any simulator, and it produces no data. The LiDAR view is a concept drawing, not a sensor model.

Perception tasks

Tasks the research covers

Each task needs different labels and different failure analysis.

  • Platform Vision

    Object Detection

    Locate and classify vehicles, people and obstacles.

    Data required

    • 2D/3D boxes
    • Occlusion attributes
  • Platform Vision

    Semantic Segmentation

    Label every pixel, including drivable space.

    Data required

    • Class masks
    • Free-space maps
  • Platform Vision

    Depth Estimation

    Estimate distance from camera images.

    Data required

    • Per-pixel depth ground truth
  • Research Direction

    Sensor Fusion Research

    Combine camera and range sensors robustly.

    Data required

    • Time-aligned multi-sensor frames
  • Platform Vision

    Edge-Case Evaluation

    Test deliberately on rare, hard situations.

    Data required

    • Scenario-tagged failure reports

Simulation technology

Possible research integrations

No simulator is integrated today: a code search of the repository found no integration (evidence R-010). These are technologies under consideration, with no vendor partnership implied.

Evidence: Unavailable · R-010

Synthex scenario layer (planned)

A backend-independent scenario specification so the same experiment can be expressed across simulators and compared.

Platform Vision
  • CARLA

    Under exploration

    Open-source simulator for autonomous driving research. Code MIT-licensed, assets CC-BY; built on a fork of Unreal Engine.

  • Unreal Engine

    Under exploration

    Real-time 3D engine on which CARLA is built; candidate for photorealistic custom environments.

  • Unity

    Under exploration

    Real-time 3D platform; candidate for custom scenes and sensor tooling.

  • Other frameworks

    Under exploration

    Robotics- or domain-specific simulators, chosen per research question.

CARLA, Unreal Engine and Unity are names or trademarks of their respective owners, mentioned only to describe technologies under consideration.

CARLA description verified against its public project page: carla.org (evidence R-011).

Robustness evaluation

How robustness would be analysed

An illustrative analysis interface. It shows the structure of a robustness study; no Synthex measurements exist, and no values are shown.

Robustness analysis · illustrative

No measurements shown

A coverage plan states which environment × condition combinations a study includes, so gaps are explicit rather than discovered later. This example plan has no measurements.

Example coverage plan by environment and condition
EnvironmentClear dayRainFogNight
UrbanPlannedPlannedPriorityPriority
RuralPlannedNot yet coveredPlannedPriority
IndustrialPlannedNot yet coveredPriorityNot yet covered

Research limitations

What simulation cannot establish on its own

  • Simulation realism limitations

    Rendering, materials and physics simplify the world in ways that matter to perception.

  • Sim-to-real gaps

    Models can learn simulator artefacts; gains may not transfer to real sensors.

  • Sensor model fidelity

    Noise, blooming, rain on lenses and LiDAR returns are hard to model faithfully.

  • Scenario representativeness

    Chosen scenarios reflect designers' assumptions, not the full real world.

  • Need for real-world validation

    Conclusions must be checked against held-out real data and real tests.

  • Not safety certification

    Simulation testing alone does not establish real-world safety or meet certification requirements.

Simulation is evidence, not proof

Simulation results are one input. Conclusions about real-world behaviour need validation on real, held-out data and real-world testing under appropriate safety processes.

Simulation research

Explore Research in Autonomous AI Evaluation

We are interested in research collaborations on scenario design, synthetic perception data and honest robustness evaluation.

Simulation-based testing does not by itself establish real-world safety or certification.