SeqLoc: Visual Localization with an Image Sequence

Student Project @ 3DV ETH Zurich 2022

Students: Liyuan Zhu, Jingyan Li, Han Sun

Supervisors: Iro Armeni, Daniel Barath

Poster | Video | Report

In SeqLoc, we propose localization with a short image sequence to leverage the redundant information in the sequence. Instead of establishing 2D-3D correspondences, we solve the pose estimation problem by point cloud registration. The proposed method also provides a simple and efficient 3D-3D correspondence generation algorithm to solve the transformation between two SfM-based point clouds. On top of the point cloud registration based localization, we add a global bundle adjustment module to refine the pose estimate with additional constraints from the sequence.

Proposed pipeline(adapted from hloc)

Generation of 3D-3D correspondences

Localization by Point Cloud Registration(TEASER++)

Installation

SeqLoc requires Python >=3.7, PyTorch >=1.1 and open3D==0.10.0.0

git clone --recursive https://github.com/Zhu-Liyuan/SeqLoc
cd SeqLoc/
python -m pip install -e .

Then we install TEASER++ with pybind

cd third_party/TEASER-plusplus
sudo apt install cmake libeigen3-dev libboost-all-dev
mkdir build && cd build
cmake -DTEASERPP_PYTHON_VERSION=3.6 .. && make teaserpp_python
cd python && pip install .

Then build colmap and modified pycolmap from source

cd ../../third_party/pycolmap
pip install .

Run seqloc Pipeline

You can run the seqloc pipeline step by step by using the following commands:

1. SFM to reconstruct local point cloud for image sequences:

python -m pcr.sfm_pipeline --proj_dir="PATH_TO_QUERY_FOLDER"

2. Extract 2d-to-3d correspondence:

python -m pcr.pairs_3d_from_2d --db_model="PATH_TO_DATABASE_FOLDER" --query_model="PATH_TO_QUERY_FOLDER"

3. Register local point cloud to the global model:

python -m pcr.poses_from_pcr --db_model="PATH_TO_DATABASE_FOLDER" --query_model="PATH_TO_QUERY_FOLDER"

Or you can run the seqloc pipeline as a whole by calling:

python -m pcr.seqloc_pipeline --config="pcr/config/seqloc.yaml"

Don't forget to change the Path in seqloc.yaml !!!

Data Structure for database

├── data
    ├── Outputs
        ├── database.db
        ├── point_feat_*.h5(point features)
        ├── matches_*.h5(feature matches)
        ├── flobal_feats_*.h5
        ├── pairs_netlad.txt
        ├── point_cloud.ply
        ├── sfm
            ├── points3D.bin
            ├── cameras.bin
            ├── images.bin
            ├── database.bin
            ├── models
    ├── images
        ├── Image1.JPG
        ├── Image2.JPG
        .
        .
        .
        ├── ImageX.JPG
        

Some useful python scripts to manipulate colmap database and project files

database.py - To manipulate colmap database.

visualize_model.py - Contains visualization function and defines a class for colmap data(points3D,cameras,images)

read_write_model.py - Parses (points3D,cameras,images) into numpy data structures.

More info about the data structure of colmap can be found at https://colmap.github.io/format.html

Contact

Feel free to contact us if you are interested in our project or have any questions.

Liyuan Zhu [email protected]

Jingyan Li [email protected]

Han Sun [email protected]