Runsen Xu  Weiyao Wang  Hao Tang  Xingyu Chen  Xiaodong Wang  Fu-Jen Chu 
Dahua Lin  Matt Feiszli  Kevin J. Liang 
FAIR, Meta The Chinese University of Hong Kong 

• [2025-05-22] We release the paper of Multi-SpatialMLLM and the codes of our data engine. 🎉

• 📦 Data Engine
• 🏋️‍♂️ Model Training
• 🔗 Citation
• 📄 License
• 👥 Contributing

To set up the Conda environment for our data engine, please follow these steps:

1. Ensure you have Anaconda or Miniconda installed.
2. Clone this repository.

   git clone https://github.com/facebookresearch/Multi-SpatialMLLM.git
   cd Multi-SpatialMLLM

3. Create the Conda environment using the provided YAML file:

   conda env create -f requirements/data_engine.yaml

4. Activate the newly created environment:

   conda activate data_engine

Please follow spatial_engine/utils/scannet_utils/README.md to download and process the ScanNet data.

Follow TAPVid-3D to download the data. We only use the ADT and PStudio subsets. You need to download the version with camera extrinsics annotation according to this.

Finally, we assume the structure of the data is like:

data/tapvid3d_dataset
├── adt
│   ├── "scene_id".npz
├── pstudio
│   ├── "scene_id".npz

We generate the data based on the conversation format of InternVL. You could easily change the generated jsonl file to the format of your own.

1. Run python spatial_engine/camera_movement/calculate_frames_relations.py to calculate the spatial relations between frames, e.g. their overlap ratios. After running this script, a parquet containing this spatial information will be generated in training_data/camera_movement and evaluation_data/camera_movement.

2. Then run python spatial_engine/camera_movement/camera_movement_engine_train_val.py to generate the training and evaluation data.

1. Run python spatial_engine/utils/scannet_utils/make_visibility_info.py to compute the visibility information for each frame. Note that when using this information, loading the file takes a long time, about several minutes.
2. Run python spatial_engine/depth_perception/depth_estimation_dot_engine.py to generate the training and evaluation data for visual-based depth estimation.
3. Run python spatial_engine/depth_perception/depth_estimation_coor_engine.py to generate the training and evaluation data for coordinate-based depth estimation.
4. Run python spatial_engine/depth_perception/depth_comparison_dot_engine.py to generate the training and evaluation data for visual-based depth comparison.
5. Run python spatial_engine/depth_perception/depth_comparison_coor_engine.py to generate the training and evaluation data for coordinate-based depth comparison.

1. Run python spatial_engine/visual_correspondence/visual_correspondence_qa_engine_dot_2_multichoice.py to generate the training and evaluation data for visual correspondence in dot-based multichoice format.
2. Run python spatial_engine/visual_correspondence/visual_correspondence_qa_engine_coor_2_coor.py to generate the training and evaluation data for visual correspondence in coordinate-based format.

1. Run python spatial_engine/object_perception/compute_object_visibility.py to compute the visibility information for each object. After running this script, a pkl file containing this visibility information will be saved to training_data/object_perception and evaluation_data/object_perception.
2. Run bash find_object_coverage.sh to compute the coverage information for each object in each scene. You could check spatial_engine/object_perception/single_object_coverage_finder.py to modify the parameters and run it with several processes.
3. After generating all the coverage information for each scene, run python spatial_engine/object_perception/merge_object_coverage.py to merge the coverage information.
4. Run python spatial_engine/object_perception/single_object_perception_engine.py to generate the training and evaluation data for object perception.

1. Run python spatial_engine/object_movement/single_object_movement_engine_coord.py to generate the training and evaluation data for object movement in coordinate-based format. After running this script, images will be extracted from the npz file and saved to data/my_tapvid3d_images.
2. Run python spatial_engine/object_movement/single_object_movement_engine_dot.py to generate the training and evaluation data for object movement in dot-based format.

We use the InternVL-2 models for experiments in our paper. You could follow their official instructions to easily fine-tune the models with the generated data and reproduce our results. Other VLMs can also be used. Below are some training details used in our experiments, and more can be found in our paper.

• All images should be resized to H*W=1296*968 for training.
• Different from the original InternVL setting of dynamically allocating 12 image tiles to all images, we make sure each image can use up to 6 image tiles for training and evaluation. Please change this line to max_num=self.max_dynamic_patch. Pay attention to GPU OOM issues, and you may change the --max_seq_length to 8192.
• The training config used for our main paper is in data/configs/mix3M.json. Note that this config only uses 3M training samples, and we use LoRA training for research efficiency. You could use more data and fully fine-tune the whole model to get much better performance.
• To preserve the original ability of the model, some general instruction-following data should be added to the training data.

If you find our work and this codebase helpful, please consider starring this repo 🌟 and cite:

@article{xu2025multi,
  title={Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models},
  author={Xu, Runsen and Wang, Weiyao and Tang, Hao and Chen, Xingyu and Wang, Xiaodong and Chu, Fu-Jen and Lin, Dahua and Feiszli, Matt and Liang, Kevin J.},
  journal={arXiv preprint arXiv:2505.17015},
  year={2025}
}

Shield:

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

See contributing and the code of conduct.