Skip to content

Commit f86c782

Browse files
gauravpradeepgauravpradeep
committed
adding support for multiscan
1 parent b9d92fa commit f86c782

28 files changed

+1571
-12
lines changed

‎DATA.md

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ We list the available data used in the current version of CrossOver in the table
1010
| ------------ | ----------------------------- | ----------------------------------- | -------------------------- | -------------------------- |
1111
| ScanNet | `[point, rgb, cad, referral]` | `[point, rgb, floorplan, referral]` | ❌ | ✅ |
1212
| 3RScan | `[point, rgb, referral]` | `[point, rgb, referral]` | ✅ | ✅ |
13+
| MultiScan | `[point, rgb, referral]` | `[point, rgb, referral]` | ❌ | ✅ |
1314

1415

1516
We detail data download and release instructions for preprocessing with scripts for ScanNet + 3RScan.
@@ -110,4 +111,32 @@ Scan3R/
110111
| │ ├── objectsDataMultimodal.pt -> object data combined from data1D.pt + data2D.pt + data3D.pt (for easier loading)
111112
| │ └── sel_cams_on_mesh.png (visualisation of the cameras selected for computing RGB features per scan)
112113
| └── ...
113-
```
114+
```
115+
116+
### MultiScan
117+
Here we refer to the contents of the folder `processed_data/MultiScan` on GDrive. The data structure is the following:
118+
119+
```
120+
MultiScan/
121+
├── objects_chunked/ (object data chunked into hdf5 format for instance baseline training)
122+
| ├── train_objects.h5
123+
| └── val_objects.h5
124+
├── scans/
125+
| ├── scene_00000_00/
126+
| │ ├── gt-projection-seg.pt -> 3D-to-2D projected data consisting of framewise 2D instance segmentation
127+
| │ ├── data1D.pt -> all 1D data + encoded (object referrals + BLIP features)
128+
| │ ├── data2D.pt -> all 2D data + encoded (RGB + floorplan + DinoV2 features)
129+
| │ ├── data2D_all_images.pt (RGB features of every image of every scan)
130+
| │ ├── data3D.pt -> all 3D data + encoded (Point Cloud + I2PMAE features - object only)
131+
| │ ├── object_id_to_label_id_map.pt -> Instance ID to NYU40 Label mapped
132+
| │ ├── objectsDataMultimodal.pt -> object data combined from data1D.pt + data2D.pt + data3D.pt (for easier loading)
133+
| │ └── sel_cams_on_mesh.png (visualisation of the cameras selected for computing RGB features per scan)
134+
| └── ...
135+
```
136+
137+
#### Running preprocessing scripts
138+
Adjust the path parameters of `MultiScan` in the config files under `configs/preprocess`. Run the following (after changing the `--config-path` in the bash file):
139+
140+
```bash
141+
$ bash scripts/preprocess/process_multiscan.sh
142+
```

‎README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,8 @@ See [DATA.MD](DATA.md) for detailed instructions on data download, preparation a
117117
| ------------ | ----------------------------- | ----------------------------------- | -------------------------- | -------------------------- |
118118
| Scannet | `[point, rgb, cad, referral]` | `[point, rgb, floorplan, referral]` | ❌ | ✅ |
119119
| 3RScan | `[point, rgb, referral]` | `[point, rgb, referral]` | ✅ | ✅ |
120+
| MultiScan | `[point, rgb, referral]` | `[point, rgb, referral]` | ❌ | ✅ |
121+
120122

121123
> To run our demo, you only need to download generated embedding data; no need for any data preprocessing.
122124
@@ -133,7 +135,7 @@ Various configurable parameters:
133135
- `--database_path`: Path to the precomputed embeddings of the database scenes downloaded before (eg: `./release_data/embed_scannet.pt`).
134136
- `--query_modality`: Modality of the query scene, Options: `point`, `rgb`, `floorplan`, `referral`
135137
- `--database_modality`: Modality used for retrieval. Same options as above.
136-
- `--ckpt`: Path to the pre-trained scene crossover model checkpoint (details [here](#checkpoints)), example_path: `./checkpoints/scene_crossover_scannet+scan3r.pth/`).
138+
- `--ckpt`: Path to the pre-trained scene crossover model checkpoint (details [here](#checkpoints)), example_path: `./checkpoints/scene_crossover_scannet+scan3r.pth/`.
137139

138140
For embedding and pre-trained model download, refer to [generated embedding data](DATA.md#generated-embedding-data) and [checkpoints](#checkpoints) sections.
139141

‎TRAIN.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ $ bash scripts/train/train_instance_crossover.sh
2121
```
2222

2323
#### Train Scene Retrieval Pipeline
24-
Adjust path/configuration parameters in `configs/train/train_scene_crossover.yaml`. You can also add your customised dataset or choose to train on Scannet & 3RScan or either. Run the following:
24+
Adjust path/configuration parameters in `configs/train/train_scene_crossover.yaml`. You can also add your customised dataset or choose to train on Scannet, 3RScan & MultiScan or any combination of the same. Run the following:
2525

2626
```bash
2727
$ bash scripts/train/train_scene_crossover.sh

‎configs/evaluation/eval_instance.yaml

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,13 +43,23 @@ data :
4343
max_object_len : 150
4444
voxel_size : 0.02
4545

46+
MultiScan:
47+
base_dir : /media/sayan/Expansion/data/datasets/MultiScan
48+
process_dir : ${data.process_dir}/MultiScan
49+
processor3D : MultiScan3DProcessor
50+
processor2D : MultiScan2DProcessor
51+
processor1D : MultiScan1DProcessor
52+
avail_modalities : ['point', 'cad', 'rgb', 'referral']
53+
max_object_len : 150
54+
voxel_size : 0.02
55+
4656
task:
4757
name : InferenceObjectRetrieval
4858
InferenceObjectRetrieval:
4959
val : [Scannet]
5060
modalities : ['rgb', 'point', 'cad', 'referral']
5161
scene_modalities : ['rgb', 'point', 'referral', 'floorplan']
52-
ckpt_path : /drive/dumps/multimodal-spaces/runs/release_runs/instance_crossover_scannet+scan3r.pth
62+
ckpt_path : /drive/dumps/multimodal-spaces/runs/release_runs/instance_crossover_scannet+scan3r+multiscan.pth
5363

5464

5565
inference_module: ObjectRetrieval

‎configs/evaluation/eval_scene.yaml

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,13 +43,23 @@ data :
4343
max_object_len : 150
4444
voxel_size : 0.02
4545

46+
MultiScan:
47+
base_dir : /media/sayan/Expansion/data/datasets/MultiScan
48+
process_dir : ${data.process_dir}/MultiScan
49+
processor3D : MultiScan3DProcessor
50+
processor2D : MultiScan2DProcessor
51+
processor1D : MultiScan1DProcessor
52+
avail_modalities : ['point', 'cad', 'rgb', 'referral']
53+
max_object_len : 150
54+
voxel_size : 0.02
55+
4656
task:
4757
name : InferenceSceneRetrieval
4858
InferenceSceneRetrieval:
4959
val : [Scannet]
5060
modalities : ['rgb', 'point', 'cad', 'referral']
5161
scene_modalities : ['rgb', 'point', 'referral', 'floorplan'] #, 'point']
52-
ckpt_path : /drive/dumps/multimodal-spaces/runs/release_runs/scene_crossover_scannet+scan3r.pth
62+
ckpt_path : /drive/dumps/multimodal-spaces/runs/release_runs/scene_crossover_scannet+scan3r+multiscan.pth
5363

5464
inference_module: SceneRetrieval
5565
model:

‎configs/preprocess/process_1d.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,14 @@ data:
2525
label_filename : labels.instances.align.annotated.v2.ply
2626
skip_frames : 1
2727

28+
MultiScan:
29+
base_dir : /media/sayan/Expansion/data/datasets/MultiScan
30+
process_dir : ${data.process_dir}/MultiScan
31+
processor3D : MultiScan3DProcessor
32+
processor2D : MultiScan2DProcessor
33+
processor1D : MultiScan1DProcessor
34+
skip_frames : 1
35+
2836
Shapenet:
2937
base_dir : /drive/datasets/Shapenet/ShapeNetCore.v2/
3038

‎configs/preprocess/process_2d.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,14 @@ data:
2727
label_filename : labels.instances.align.annotated.v2.ply
2828
skip_frames : 1
2929

30+
MultiScan:
31+
base_dir : /media/sayan/Expansion/data/datasets/MultiScan
32+
process_dir : ${data.process_dir}/MultiScan
33+
processor3D : MultiScan3DProcessor
34+
processor2D : MultiScan2DProcessor
35+
processor1D : MultiScan1DProcessor
36+
skip_frames : 1
37+
3038
modality_info:
3139
1D :
3240
feature_extractor:

‎configs/preprocess/process_3d.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,14 @@ data:
2424
processor1D : Scan3R1DProcessor
2525
label_filename : labels.instances.align.annotated.v2.ply
2626

27+
MultiScan:
28+
base_dir : /media/sayan/Expansion/data/datasets/MultiScan
29+
process_dir : ${data.process_dir}/MultiScan
30+
processor3D : MultiScan3DProcessor
31+
processor2D : MultiScan2DProcessor
32+
processor1D : MultiScan1DProcessor
33+
skip_frames : 1
34+
2735
modality_info:
2836
1D :
2937
feature_extractor:

‎configs/preprocess/process_multimodal.yaml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,15 @@ data:
2828
skip_frames : 1
2929
avail_modalities : ['point', 'rgb', 'referral']
3030

31+
MultiScan:
32+
base_dir : /media/sayan/Expansion/data/datasets/MultiScan
33+
process_dir : ${data.process_dir}/MultiScan/
34+
chunked_dir : ${data.process_dir}/MultiScan/objects_chunked
35+
processor3D : Scan3R3DProcessor
36+
processor2D : Scan3R2DProcessor
37+
processor1D : Scan3R1DProcessor
38+
avail_modalities : ['point', 'rgb', 'referral']
39+
3140
modality_info:
3241
1D :
3342
feature_extractor:

‎configs/train/train_instance_baseline.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,17 @@ data :
4444
max_object_len : 150
4545
voxel_size : 0.02
4646

47+
MultiScan:
48+
base_dir : /media/sayan/Expansion/data/datasets/Multiscan
49+
process_dir : ${data.process_dir}/MultiScan/
50+
chunked_dir : ${data.process_dir}/MultiScan/objects_chunked
51+
processor3D : MultiScan3DProcessor
52+
processor2D : MultiScan2DProcessor
53+
processor1D : MultiScan1DProcessor
54+
avail_modalities : ['point', 'rgb', 'referral']
55+
max_object_len : 150
56+
voxel_size : 0.02
57+
4758
task:
4859
name : ObjectLevelGrounding
4960
ObjectLevelGrounding :

0 commit comments

Comments
 (0)