OSTRA is an One Stop 3D multi-target reconstruction and segmentation framework. It takes the multview images (or videos) and segmentation of targets in each image as the input, generates targets’ 3D models embedded with rich multi-scale segmentation information. We also provide a pipeline to segment objects consistently. Users can choose the Segment-Anything Model (SAM) or the video object segmentation (VOS) approach. SAM can generate good segmentation most of time, but for challenging tasks such as plant panicle segmentation, we suggeste to use a trained SAM-Adapter.
VOS: We use the methods from Yang and Cheng as the foundation for VOS. The VOS approach enables the generation of continuous masks across video frames. Users can select suitable VOS models tailored to their specific tasks.
You can check detailed tutorials here!
SAM-Adapter We use Tianrun Chen's work as the backbone for SAM-Adapter. The adapter here can improve SAM's segmentation performance on rice panicles by fine-tuning it with downstream training. Users can train their own adapter for specific objects.
You can check detailed tutorials here!
This project is tested under python3.9, cuda11.5 and pytorch1.11.0. An equivalent or higher version is recommended.
#Clone OSTRA
git clone --recursive https://github.com/ganlab/OSTRA.git
cd OSTRA
#Install dependencies:
pip install -r requirements.txtOur reconstruction process is based on Colmap. Please follow the instruction and install Colmap first.
Colmap pipeline
python reconstruction_Colmap.pyColmap + OpenMVS pipline
python reconstruction_OpenMVS.pyTwo samples of complex object segmentation:
Please considering cite our paper if you find this work useful!
@misc{xu2023stop,
title={A One Stop 3D Target Reconstruction and multilevel Segmentation Method},
author={Jiexiong Xu and Weikun Zhao and Zhiyan Tang and Xiangchao Gan},
year={2023},
eprint={2308.06974},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Free for non-profit research purposes. Please contact authors otherwise. The program itself may not be modified in any way and no redistribution is allowed. No condition is made or to be implied, nor is any warranty given or to be implied, as to the accuracy of OSTRA, or that it will be suitable for any particular purpose or for use under any specific conditions, or that the content or use of OSTRA will not constitute or result in infringement of third-party rights.
This work is based on Segment Anything, Track Anything, Segment and Track Anything, Colmap, Open3D and SAM-Adapter-backbone. The software is developed by following author(s) and supervised by Prof. Xiangchao Gan([email protected])
Authors:
Jiexiong Xu [email protected] work: framework and reconstruction module
Weikun Zhao [email protected] work: VOS module