Efficiently Ensuring Traffic Rule Compliance of Motion Plans by Incorporating Scenario Knowledge – Evaluation Scripts

This repository contains the evaluation scripts for the paper "Efficiently Ensuring Traffic Rule Compliance of Motion Plans by Incorporating Scenario Knowledge". The implementation of the algorithms presented in the paper can be found here:

• General LTL augmentation
• Application to traffic rules

📁 Overview

The repository contains the following files and directories:

.
├── configurations/    # Configuration files for the experiments
│
├── data/
│   ├── output/        # Raw results of the experiments presented in the paper
│   └── plots/         # Scenario plots shown in the paper
│
├── scenarios/         # CommonRoad scenarios for the motivating example and experiments
│
├── src/               # Source code for setting up the evaluation environment
│
└── *.py files         # Main scripts for the evaluation: see below for details

Due to license restrictions, we cannot provide the CommonRoad scenarios generated from the exiD and inD datasets. For non-commercial purposes, you can obtain the original datasets for free from the following links:

• exiD: https://www.exid-dataset.com
• inD: https://www.ind-dataset.com

Then you can use the CommonRoad dataset converter to convert the datasets to CommonRoad scenarios.

💻 Getting Started

1. Pull the Docker image efficiently-ensuring-traffic-rule-compliance from the GitHub container registry (Docker may require sudo root privileges if your user is not in the docker group):

docker pull ghcr.io/commonroad/efficiently-ensuring-traffic-rule-compliance/efficiently-ensuring-traffic-rule-compliance:latest

2. After pulling the image, make sure that you are in the root directory of the repository (the directory containing this file) and start the container with:

docker run -it --rm -v $(pwd):/workspace ghcr.io/commonroad/efficiently-ensuring-traffic-rule-compliance/efficiently-ensuring-traffic-rule-compliance:latest /bin/bash

The command above mounts the current working directory into the Docker container. Thus, modifications to the code from inside the container will also be reflected on your local machine. Moreover, it allows you to access the results even after the container has stopped.

Note: If you are using Windows, you have to replace $(pwd) by %cd%.

3. Install the evaluation Python package in the Docker container:

pip install .

If the package is installed successfully, you can run the experiments as described below.

Note: It can happen that you do not have permissions to access the files created inside the container from the host machine. In this case, you can change the owner of the files using sudo chown -R $USER <directory>.

▶️ Replicating the Results from the Paper

All results from the paper can be replicated with the evaluation scripts in this repository. We describe the purpose of each script in detail. To reduce the runtime of the scripts, you can lower the number of repetitions in the configuration files.

run_experiments.py

This script runs the experiments on exiD and inD scenarios. It writes its measurements to the output directory in YAML format. You can run the script as follows

python run_experiments.py

To show advanced options for the script run

python run_experiments.py --help

analyze_results.py

This script produces the statistics presented in the paper from the results of run_experiments.py. By default, it expects the YAML files to be in the output directory. You can run the script as follows

python analyze_results.py

To show advanced options for the script run

python analyze_results.py --help

Note: It can happen that the monitoring algorithm produces inconsistent verdicts for some scenarios. The reason for this seems to be some non-determinism in the environment mode. Thus, if this happens, we suggest rerunning the scenarios in question with the reuse_world configuration option enabled. You can add the scenario names to the list scenarios_with_inconsistent_monitor_results in run_experiments.py which will automatically rerun these scenarios with the aforementioned option.

plot_example_scenarios.py

This script plots the two example scenarios shown in the paper.

motivating_example.py

This script produces the figure used as motivating example in the paper.