ADAS Testing using Surrogate Models and Hill Climbing

This project presents a novel approach for testing Advanced Driver Assistance Systems (ADAS) in a simulated environment using multi-objective search combined with surrogate modeling based on neural networks. The primary goal is to generate test cases for crash scenarios efficiently, even with limited computational resources.

Table of Contents

• Abstract
• Introduction
• Dataset
• Methodology
  • Fitness Functions
  • Surrogate Model
  • Search with Hill Climbing
• Evaluation
• How to Run
• Conclusion
• References

Abstract

Testing ADAS software is crucial for autonomous vehicles to ensure safety and reliability. This project explores a combination of surrogate modeling and a hill climbing algorithm to generate test cases that simulate potential crash scenarios between a car and a pedestrian. Using CARLA, an open-source driving simulator, and neural networks, we demonstrate how surrogate models can speed up scenario evaluation. Our optimization algorithm, hill climbing, is compared with random search to assess efficiency in finding critical test cases.

Introduction

Autonomous vehicles depend on software systems for safe operation, making effective testing vital. This project proposes solutions to:

1. Automate simulation scenario generation to reveal faults.
2. Reduce the computational cost of simulation scenarios using surrogate models.

Challenges

• Lack of intelligent and automated simulation tools.
• High computational cost of physics-based simulations.

Approach

We leverage CARLA, an open-source simulator, to create synthetic data for a crash scenario between a car and a pedestrian. A neural network-based surrogate model is trained to approximate simulation results, reducing the need for expensive physics-based simulations. This model is then used in conjunction with a hill climbing optimization algorithm to generate critical crash scenarios.

Dataset

The dataset used is generated by the CARLA Simulator, which allows creating realistic driving scenarios. It consists of:

• Pedestrian Position
• Vehicle Speed
• Road Type (fixed)
• Weather Conditions (fixed)

These features are used to simulate interactions between vehicles and pedestrians in potential crash scenarios.

Methodology

Fitness Functions

We employ two fitness functions to generate critical test cases:

1. Minimum Distance between the car and the pedestrian.
2. Minimum Time to Collision.

Both functions aim to capture the severity of a crash scenario.

Surrogate Model

A surrogate model replaces costly simulations with neural networks. We experimented with several models and chose the MLPRegressor due to its ability to model complex patterns in the data. The surrogate model was trained on synthetic data generated by CARLA and used to evaluate crash scenarios in real-time.

Search with Hill Climbing

The hill climbing algorithm starts with a random population of potential solutions (test cases). It iteratively improves the solutions by exploring neighboring scenarios based on the fitness functions. We also implemented random search as a baseline for comparison.

Evaluation

The project was tested on a machine with the following specifications:

• Processor: Intel Core i7 (8th gen, 2.2 GHz)
• Memory: 16 GB RAM
• Operating System: Windows 11
• Programming Language: Python (with scikit-learn and CARLA)

Research Questions (RQs)

1. How does hill climbing compare with random search? We compared the performance of both algorithms on the task of minimizing the distance and time to collision.

2. What is the quality of the test cases generated by hill climbing vs. random search? Hill climbing consistently found better solutions by leveraging the gradient of the fitness function, while random search wasted resources on non-optimal solutions.

Figures show that hill climbing converges faster to critical crash scenarios than random search.

How to Run

Follow the steps below to run the project:

1. Clone the Repository:

git clone https://github.com/NimaMeghdadi/Carla_scenario_test_HillClimbing.git
cd Carla_scenario_test_HillClimbing

2. Install Dependencies: Make sure you have Python 3.x installed. Install the required dependencies using pip:

pip install -r requirements.txt

3. Install CARLA 0.9.13: Download and install CARLA version 0.9.13 from the official CARLA website: CARLA Download

4. Install CARLA Python Package: Install the CARLA Python API to communicate with the simulator:

pip install carla==0.9.13

5. Download Additional CARLA Maps: If required, you can download additional maps for CARLA from the official CARLA repository:

cd /opt/carla-simulator/Import
./ImportAssets.sh

6. Run CARLA Simulator: Start the CARLA simulator on your machine:

./CarlaUE4.sh

7. Run the Main Python Script: After starting the CARLA simulator, execute the main Python script to begin the scenario testing:

python main.py

Conclusion

This project successfully combines neural networks and optimization algorithms to improve the efficiency of ADAS testing. The surrogate model allows for faster evaluation of test cases, while hill climbing outperforms random search in finding critical crash scenarios.