This package contains panoptic_mapping, a general framework for semantic volumetric mapping. We provide, among other, a submap-based approach that leverages panoptic scene understanding towards adaptive spatio-temporally consistent volumetric mapping, as well as regular, monolithic semantic mapping.
Multi-resolution 3D Reconstruction, active and inactive panoptic submaps for temporal consistency, online change detection, and more.
Credits
Setup
Examples
Other
If you find this package useful for your research, please consider citing our paper:
- Lukas Schmid, Jeffrey Delmerico, Johannes Schönberger, Juan Nieto, Marc Pollefeys, Roland Siegwart, and Cesar Cadena. "Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency" in IEEE International Conference on Robotics and Automation (ICRA), 2022.
[ArXiv | Video]
@inproceedings{schmid2022panoptic, title={Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency}, author={Schmid, Lukas and Delmerico, Jeffrey and Sch{\"o}nberger, Johannes and Nieto, Juan and Pollefeys, Marc and Siegwart, Roland and Cadena, Cesar}, booktitle={2022 IEEE International Conference on Robotics and Automation (ICRA)}, year={2022} }
For a short overview explaining the approach check out our video on youtube:
Installation instructions for Linux. The repository was developed and tested on Ubuntu 18.04 with ROS melodic and Ubuntu 20.04 with ROS noetic.
Ubuntu 18.04 + ROS Melodic.
Prerequisites
-
If not already done so, install ROS (Desktop-Full is recommended).
-
If not already done so, create a catkin workspace with catkin tools:
sudo apt-get install python-catkin-tools
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws
catkin init
catkin config --extend /opt/ros/melodic
catkin config --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo
catkin config --merge-devel
Installation
- Install system dependencies:
sudo apt-get install python-wstool python-catkin-tools autoconf libtool git
- Move to your catkin workspace:
cd ~/catkin_ws/src
- Download repo using SSH or HTTPS:
git clone [email protected]:ethz-asl/panoptic_mapping.git # SSH
git clone https://github.com/ethz-asl/panoptic_mapping.git # HTTPS
- Download and install package dependencies using ros install:
- If you created a new workspace.
wstool init . ./panoptic_mapping/panoptic_mapping_ssh.rosinstall # SSH
wstool init . ./panoptic_mapping/panoptic_mapping_https.rosinstall # HTTPS
wstool update
- If you use an existing workspace. Notice that some dependencies require specific branches that will be checked out.
wstool merge -t . ./panoptic_mapping/panoptic_mapping.rosinstall
wstool update
- Compile and source:
catkin build panoptic_mapping_utils
source ../devel/setup.bash
Ubuntu 20.04 + ROS Noetic.
Prerequisites
-
If not already done so, install ROS (Desktop-Full is recommended).
-
If not already done so, create a catkin workspace with catkin tools:
sudo apt-get install python3-catkin-tools
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws
catkin init
catkin config --extend /opt/ros/noetic
catkin config --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo
catkin config --merge-devel
Installation
- Install system dependencies:
sudo apt-get install python3-pip python3-wstool python3-catkin-tools autoconf libtool git
pip3 install osrf-pycommon
- Move to your catkin workspace:
cd ~/catkin_ws/src
- Download repo using SSH or HTTPS:
git clone [email protected]:ethz-asl/panoptic_mapping.git # SSH
git clone https://github.com/ethz-asl/panoptic_mapping.git # HTTPS
- Download and install package dependencies using ros install:
- If you created a new workspace.
wstool init . ./panoptic_mapping/panoptic_mapping_ssh.rosinstall # SSH
wstool init . ./panoptic_mapping/panoptic_mapping_https.rosinstall # HTTPS
wstool update
- If you use an existing workspace. Notice that some dependencies require specific branches that will be checked out.
wstool merge -t . ./panoptic_mapping/panoptic_mapping.rosinstall
wstool update
- Compile and source:
catkin build panoptic_mapping_utils
source ../devel/setup.bash
The datasets described in the paper and used for the demo can be downloaded from the ASL Datasets.
A utility script is provided to directly download the flat data:
roscd panoptic_mapping_utils
export FLAT_DATA_DIR="/home/$USER/Documents" # Or whichever path you prefer.
chmod +x panoptic_mapping_utils/scripts/download_flat_dataset.sh
./panoptic_mapping_utils/scripts/download_flat_dataset.sh
To run the RIO demos, the original dataset needs to be downloaded and augmented with our supplementary data. Instructions on which scenes to download and how to combine them are found on our dataset page.
This example explains how to run the Panoptic Multi-TSDF mapper on the flat dataset.
-
First, download the flat dataset:
export FLAT_DATA_DIR="/home/$USER/Documents" # Or whichever path you prefer. chmod +x panoptic_mapping_utils/scripts/download_flat_dataset.sh ./panoptic_mapping_utils/scripts/download_flat_dataset.sh
-
Replace the data
base_path
inlaunch/run.launch (L10)
andfile_name
inconfig/mapper/flat_groundtruth.yaml (L15)
to the downloaded path. -
Run the mapper:
roslaunch panoptic_mapping_ros run.launch
-
You should now see the map being incrementally built:
-
After the map finished building, you can save the map:
rosservice call /panoptic_mapper/save_map "file_path: '/path/to/run1.panmap'"
-
Terminate the mapper pressing Ctrl+C. You can continue the experiment on
run2
of the flat dataset by changing thebase_path
-ending inlaunch/run.launch (L10)
torun2
, andload_map
andload_path
inlaunch/run.launch (L26-27)
totrue
and/path/to/run1.panmap
, respectively. Optionally, you can also change thecolor_mode
inconfig/mapper/flat_groundtruth.yaml (L118)
tochange
to better highlight the change detection at work.roslaunch panoptic_mapping_ros run.launch
-
You should now see the map being updated based on the first run:
Panoptic Mapping supports also the monolithic use case. This example explains how to run the Panoptic Single-TSDF mapper on the flat dataset.
-
If not already done so, download the flat dataset:
export FLAT_DATA_DIR="/home/$USER/Documents" # Or whichever path you prefer. chmod +x panoptic_mapping_utils/scripts/download_flat_dataset.sh ./panoptic_mapping_utils/scripts/download_flat_dataset.sh
-
Replace the data
base_path
inlaunch/run.launch (L10)
andfile_name
inconfig/mapper/single_tsdf.yaml (L15)
to the downloaded path. -
To use the single-TSDF mapper and real segmentation predictions, set
use_detectron
inlaunch/run.launch (L6)
to true andconfig
inlaunch/run.launch (L22)
to 'single_tsdf'. -
Run the mapper:
roslaunch panoptic_mapping_ros run.launch
-
You should now see the map being incrementally built.
-
Varying visualization modes are supported, the
classes
(default) will color the mesh according to the predicted semantic class. Other modes, such asclassification
will show the confidence of the aggregated predictions.
Predicted classes (left) and corresponding fusion confidence (right, low to high in red to green).
This example explains how to run the Panoptic Multi-TSDF mapper on the RIO dataset.
-
First, download the original dataset and our supplementary data.
-
Set
use_rio
inlaunch/run.launch (L5)
to true, replace thedata_path
in(L16)
andfile_name
inconfig/mapper/rio_groundtruth.yaml (L15)
to the downloaded path. -
Run the mapper:
roslaunch panoptic_mapping_ros run.launch
-
You should now see the map being incrementally built:
(Left in reconstructed color, right colored by submap)
panoptic_mapping is an open-source project, any contributions are welcome!
For issues, bugs, or suggestions, please open a GitHub Issue.
To add to this repository:
- Please employ the feature-branch workflow.
- Setup our auto-formatter for coherent style (we follow the google style guide):
# Download the linter cd <linter_dest> git clone [email protected]:ethz-asl/linter.git -b feature/noetic cd linter echo ". $(realpath setup_linter.sh)" >> ~/.bashrc bash roscd panoptic_mapping/.. init_linter_git_hooks # You're all set to go!
- Please open a Pull Request for your changes.
- Thank you for contributing!