Revive driving scene simulation by simulator-conditioned generative models
Yunsong Zhou, Michael Simon, Zhenghao Peng, Sicheng Mo, Hongzi Zhu, Minyi Guo, and Bolei Zhou
- Presented by MetaDriverse, GenForce, and Shanghai Jiao Tong University
- 📬 Primary contact: Yunsong Zhou ( [email protected] )
- arXiv paper | Blog TODO | Slides
🔥 The first simulator-conditioned generative model for controllable driving scene generation with appearance and layout diversity.
🌟 SimGen addresses simulation to reality (Sim2Real) gaps via cascade diffusion paradigm, and follows layout guidance from simulators and cues of the rich text prompts to realistic driving scenarios.
📊 DIVA dataset comprises 147.5 hours of web videos and synthesized data for diverse scene generation and advancing Sim2Real research.
[2024/06]SimGem paper released.[2024/06]DIVA dataset subset released.
- Release DIVA dataset
- Release SimGen code
- Toolkits for novel scene generation
You could install simgen package to enable simulator-conditioned generation.
# You don't have to create new environment, the only requirement is python>=3.10.
conda create -n simgen python=3.10
conda activate simgen
# Install this package
cd ~/SimGen
pip install -e .
# Test torch (expect True)
python -c "import torch; print(torch.cuda.is_available())"
# Install MetaDrive
cd ~/
git clone https://github.com/metadriverse/metadrive.git
cd ~/metadrive
pip install -e .After installation, you could run the following script to test simgen pipeline. Model checkpoint will automatically be downloaded from huggingface:SichengMo-UCLA/SimGen.
python test/test.pyYou can use simgen with metadrive by running the following script. The script will generate synthetic driving videos with the help of metadrive simulator and save the video to the repository root directory for each episode.
python metadrive_simgen.pyFlag for half-precision / float16 inference:
python metadrive_simgen.py -p fp16Otherwise, default setting will be float32.
DIVA-Real. It collects driving videos from YouTube, covering a worldwide range of geography, weather, scenes, and traffic elements and preserving the appearance diversity of a wide range of traffic participants. Here we provide a sample of 🔗 YouTube video list we used. For privacy considerations, we are temporarily keeping the complete data labels private.
DIVA-Sim. The Sim2Real data is induced from the same real-world scenarios, in which we can obtain real-world map topology, layout, and raw sensor data. It also includes hazardous driving behaviors through interactions introduced by adversarial traffic generation. The digital twins (on nuScenes dataset) and safety-critical scenarios (on Waymo Open dataset) can be obtained through this 🔗data link.
All assets and code in this repository are under the Apache 2.0 license unless specified otherwise. The annotation data is under CC BY-NC-SA 4.0. Other datasets (including nuScenes, Waymo, and MetaDrive) inherit their own distribution licenses. Please consider citing our paper and project if they help your research.
@article{zhou2024simgen,
title={SimGen: Simulator-conditioned Driving Scene Generation},
author={Zhou, Yunsong and Simon, Michael and Peng, Zhenghao and Mo, Sicheng and Zhu, Hongzi and Guo, Minyi and Zhou, Bolei},
journal={arXiv preprint arXiv:2406.09386},
year={2024}
}We acknowledge all the open-source contributors for the following projects to make this work possible:
You are welcome to follow other related work from 
- ELM | OpenScene | DriveAGI
- ScenarioNet | CAT | FreeControl