init commit

Author: yanwei-li

.gitignore

@@ -0,0 +1,134 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+*.ipynb
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+
+# cython generated cpp
+data
+.vscode
+.idea
+
+# custom
+*.pkl
+*.pkl.json
+*.log.json
+work_dirs/
+exps/
+*~
+mmdet3d/.mim
+
+# Pytorch
+*.pth
+
+# demo
+*.jpg
+*.png
+data/s3dis/Stanford3dDataset_v1.2_Aligned_Version/
+data/scannet/scans/
+data/sunrgbd/OFFICIAL_SUNRGBD/
+*.obj
+*.ply
+
+# Waymo evaluation
+mmdet3d/core/evaluation/waymo_utils/compute_detection_metrics_main

README.md

@@ -0,0 +1,90 @@
+
+# UVTR
+[![arXiv](https://img.shields.io/badge/arXiv-Paper-<COLOR>.svg)]()
+![visitors](https://visitor-badge.glitch.me/badge?page_id=dvlab-research/UVTR)
+
+**Unifying Voxel-based Representation with Transformer for 3D Object Detection**
+
+Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, Jiaya Jia
+
+[[`arXiv`]()] [[`BibTeX`](#CitingUVTR)]
+
+<div align="center">
+  <img src="projects/docs/uvtr.png"/>
+</div><br/>
+
+This project provides an implementation for the paper "[Unifying Voxel-based Representation with Transformer for 3D Object Detection]()" based on [mmDetection3D](https://github.com/open-mmlab/mmdetection3d). UVTR aims to unify multi-modality representations in the voxel space for accurate and robust single- or cross-modality 3D detection.
+
+## Preparation
+This project is based on [mmDetection3D](https://github.com/open-mmlab/mmdetection3d), which can be constructed as follows.
+* Install PyTorch [v1.7.1](https://pytorch.org/get-started/previous-versions/) and mmDetection3D [v0.17.3](https://github.com/open-mmlab/mmdetection3d/tree/v0.17.3) following [the instructions](https://github.com/open-mmlab/mmdetection3d/blob/v0.17.3/docs/getting_started.md).
+* Copy our project and related files to installed mmDetection3D:
+```bash
+cp -r projects mmdetection3d/
+cp -r extra_tools mmdetection3d/
+```
+* Prepare the nuScenes dataset following [the structure](https://github.com/open-mmlab/mmdetection3d/blob/v0.17.3/docs/data_preparation.md).
+* Generate the unified data info and sampling database for nuScenes dataset:
+```bash
+python3 extra_tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes_unified
+```
+
+## Training
+You can train the model following [the instructions](https://github.com/open-mmlab/mmdetection3d/blob/v0.17.3/docs/datasets/nuscenes_det.md).
+You can find the pretrained models [here](https://drive.google.com/drive/folders/1KvG7tBYhmFQCiF_pAZc3Aa3H_D__-Jqh?usp=sharing) if you want to train the model from scratch.
+For example, to launch UVTR training on multi GPUs,
+one should execute:
+```bash
+cd /path/to/mmdetection3d
+bash extra_tools/dist_train.sh ${CFG_FILE} ${NUM_GPUS}
+```
+or train with a single GPU:
+```bash
+python3 extra_tools/train.py ${CFG_FILE}
+```
+
+## Evaluation
+You can evaluate the model following [the instructions](./docs/GETTING_STARTED.md).
+For example, to launch UVTR evaluation with a pretrained checkpoint on multi GPUs,
+one should execute:
+```bash
+bash extra_tools/dist_test.sh ${CFG_FILE} ${CKPT} ${NUM_GPUS} --eval=bbox
+```
+or evaluate with a single GPU:
+```bash
+python3 extra_tools/test.py ${CFG_FILE} ${CKPT} --eval=bbox
+```
+## nuScenes 3D Object Detection Results
+We provide results on nuScenes *val* set with pretrained models.
+|                                             | NDS (%) | mAP (%) | mATE&darr; | mASE&darr; | mAOE&darr; | mAVE&darr; | mAAE&darr; | download | 
+|---------------------------------------------|:-------:|:-------:|:-------:|:---------:|:---------:|:---------:|:---------:|:---------:|
+| Camera-based |
+| [UVTR-C-R50-H5](projects/configs/uvtr/camera_based/camera/uvtr_c_r50_h5.py) | 40.1 | 31.3 | 0.810 | 0.281 | 0.486 | 0.793 | 0.187 | [GoogleDrive](https://drive.google.com/file/d/1gomNuo5--I5bdDiuiJxnhUbSw4GqE4VO/view?usp=sharing) |
+| [UVTR-C-R50-H11](projects/configs/uvtr/camera_based/camera/uvtr_c_r50_h11.py) | 41.8 | 33.3 | 0.795 | 0.276 | 0.452 | 0.761 | 0.196 | [GoogleDrive](https://drive.google.com/file/d/1ZCwzpsByd5ZulgHltGQCIzoOZmI8FC12/view?usp=sharing) |
+| [UVTR-C-R101](projects/configs/uvtr/camera_based/camera/uvtr_c_r101_h11.py) | 44.1 | 36.1 | 0.761 | 0.271 | 0.409 | 0.756 | 0.203 | [GoogleDrive](https://drive.google.com/file/d/1Mc3ZDGDPqc5uqZvrJswTn4TQdsEwtnAP/view?usp=sharing) |
+| [UVTR-CS-R50](projects/configs/uvtr/camera_based/camera_sweep/uvtr_cs5_r50_h11.py) | 47.2 | 36.2 | 0.756 | 0.276 | 0.399 | 0.467 | 0.189 | [GoogleDrive](https://drive.google.com/file/d/1BHsUzTuColqtHEIXczhgC7SsWi_0mA69/view?usp=sharing) |
+| [UVTR-CS-R101](projects/configs/uvtr/camera_based/camera_sweep/uvtr_cs4_r101_h11.py) | 48.3 | 37.9 | 0.739 | 0.267 | 0.350 | 0.510 | 0.200 | [GoogleDrive](https://drive.google.com/file/d/1JcNbnIBfp5us2CaEktr1-4t5jWOFLldA/view?usp=sharing) |
+| [UVTR-L2C-R101](projects/configs/uvtr/camera_based/knowledge_distill/uvtr_l2c_r101_h11.py) | 45.0 | 37.2 | 0.735 | 0.269 | 0.397 | 0.761 | 0.193 | [GoogleDrive](https://drive.google.com/file/d/1Knc9EHeOjXtAkRzRAN0jPUFSiqK2t1Ac/view?usp=sharing) |
+| [UVTR-L2CS3-R101](projects/configs/uvtr/camera_based/knowledge_distill/uvtr_l2cs3_r101_h11.py) | 48.8 | 39.2 | 0.720 | 0.268 | 0.354 | 0.534 | 0.206 | [GoogleDrive](https://drive.google.com/file/d/1Q5f-fESCKje9q98mj7v6pC9r_yYUw1-4/view?usp=sharing) |
+| LiDAR-based |
+| [UVTR-L-V0075](projects/configs/uvtr/lidar_based/uvtr_l_v0075_h5.py) | 67.6 | 60.8 | 0.335 | 0.257 | 0.303 | 0.206 | 0.183 | [GoogleDrive](https://drive.google.com/file/d/11wepYo4alFifpEEOtnmRJg6-plLE1QD8/view?usp=sharing) |
+| Multi-modality |
+| [UVTR-M-V0075-R101](projects/configs/uvtr/lidar_based/uvtr_l_v01_h5.py) | 70.2 | 65.4 | 0.333 | 0.258 | 0.270 | 0.216 | 0.176 | [GoogleDrive](https://drive.google.com/file/d/1dlxXIS4Cuv6ePxuxMRIaxpG_b1Pk8sqO/view?usp=sharing) |
+## Acknowledgement
+We would like to thank the authors of [mmDetection3D](https://github.com/open-mmlab/mmdetection3d) and [DETR3D](https://github.com/WangYueFt/detr3d) for their open-source release.
+
+## License
+`UVTR` is released under the [Apache 2.0 license](LICENSE).
+
+## <a name="CitingUVTR"></a>Citing UVTR
+
+Consider cite UVTR in your publications if it helps your research.
+
+```
+@article{li2022uvtr,
+  title={Unifying Voxel-based Representation with Transformer for 3D Object Detection},
+  author={Li, Yanwei and Chen, Yilun and Qi, Xiaojuan and Li, Zeming and Sun, Jian and Jia, Jiaya},
+  booktitle={arXiv},
+  year={2022}
+}
+```

extra_tools/create_data.py

@@ -0,0 +1,103 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+import argparse
+from os import path as osp
+
+from data_converter import nuscenes_converter as nuscenes_converter
+from data_converter.create_unified_gt_database import create_groundtruth_database
+
+
+def nuscenes_data_prep(root_path,
+                       info_prefix,
+                       version,
+                       dataset_name,
+                       out_dir,
+                       max_sweeps=10):
+    """Prepare data related to nuScenes dataset.
+
+    Related data consists of '.pkl' files recording basic infos,
+    2D annotations and groundtruth database.
+
+    Args:
+        root_path (str): Path of dataset root.
+        info_prefix (str): The prefix of info filenames.
+        version (str): Dataset version.
+        dataset_name (str): The dataset class name.
+        out_dir (str): Output directory of the groundtruth database info.
+        max_sweeps (int): Number of input consecutive frames. Default: 10
+    """
+    nuscenes_converter.create_nuscenes_infos(
+        root_path, info_prefix, version=version, max_sweeps=max_sweeps)
+
+    if version == 'v1.0-test':
+        info_test_path = osp.join(root_path, f'{info_prefix}_infos_test.pkl')
+        nuscenes_converter.export_2d_annotation(
+            root_path, info_test_path, version=version)
+        return
+
+    info_train_path = osp.join(root_path, f'{info_prefix}_infos_train.pkl')
+    info_val_path = osp.join(root_path, f'{info_prefix}_infos_val.pkl')
+    nuscenes_converter.export_2d_annotation(
+        root_path, info_train_path, version=version)
+    nuscenes_converter.export_2d_annotation(
+        root_path, info_val_path, version=version)
+    create_groundtruth_database(dataset_name, root_path, info_prefix,
+                                f'{out_dir}/{info_prefix}_infos_train.pkl')
+
+
+parser = argparse.ArgumentParser(description='Data converter arg parser')
+parser.add_argument('dataset', metavar='nuscenes', help='name of the dataset')
+parser.add_argument(
+    '--root-path',
+    type=str,
+    default='./data/nuscenes',
+    help='specify the root path of dataset')
+parser.add_argument(
+    '--version',
+    type=str,
+    default='v1.0',
+    required=False,
+    help='specify the dataset version, no need for nuscenes')
+parser.add_argument(
+    '--max-sweeps',
+    type=int,
+    default=10,
+    required=False,
+    help='specify sweeps of lidar per example')
+parser.add_argument(
+    '--out-dir',
+    type=str,
+    default='./data/nuscenes',
+    required='False',
+    help='name of info pkl')
+parser.add_argument('--extra-tag', type=str, default='nuscenes')
+parser.add_argument(
+    '--workers', type=int, default=4, help='number of threads to be used')
+args = parser.parse_args()
+
+if __name__ == '__main__':
+    if args.dataset == 'nuscenes' and args.version != 'v1.0-mini':
+        train_version = f'{args.version}-trainval'
+        nuscenes_data_prep(
+            root_path=args.root_path,
+            info_prefix=args.extra_tag,
+            version=train_version,
+            dataset_name='NuScenesDataset',
+            out_dir=args.out_dir,
+            max_sweeps=args.max_sweeps)
+        test_version = f'{args.version}-test'
+        nuscenes_data_prep(
+            root_path=args.root_path,
+            info_prefix=args.extra_tag,
+            version=test_version,
+            dataset_name='NuScenesDataset',
+            out_dir=args.out_dir,
+            max_sweeps=args.max_sweeps)
+    elif args.dataset == 'nuscenes' and args.version == 'v1.0-mini':
+        train_version = f'{args.version}'
+        nuscenes_data_prep(
+            root_path=args.root_path,
+            info_prefix=args.extra_tag,
+            version=train_version,
+            dataset_name='NuScenesDataset',
+            out_dir=args.out_dir,
+            max_sweeps=args.max_sweeps)

extra_tools/create_data.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+
+set -x
+export PYTHONPATH=`pwd`:$PYTHONPATH
+
+PARTITION=$1
+JOB_NAME=$2
+CONFIG=$3
+WORK_DIR=$4
+GPUS=${GPUS:-1}
+GPUS_PER_NODE=${GPUS_PER_NODE:-1}
+SRUN_ARGS=${SRUN_ARGS:-""}
+JOB_NAME=create_data
+
+srun -p ${PARTITION} \
+    --job-name=${JOB_NAME} \
+    --gres=gpu:${GPUS_PER_NODE} \
+    --ntasks=${GPUS} \
+    --ntasks-per-node=${GPUS_PER_NODE} \
+    --kill-on-bad-exit=1 \
+    ${SRUN_ARGS} \
+    python3 -u tools/create_data.py kitti \
+            --root-path ./data/kitti \
+            --out-dir ./data/kitti \
+            --extra-tag kitti