StructDiffusion

Pytorch implementation for RSS 2023 paper StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects. [PDF] [Video] [Website]

StructDiffusion combines a diffusion model and an object-centric transformer to construct structures given partial-view point clouds and high-level language goals, such as “set the table”.

Installation

conda create -n StructDiffusion python=3.8
conda activate StructDiffusion
pip install -r requirements.txt
pip install -e .

If the correct version of some dependencies are not installed, try the following.

pip uninstall torch torchaudio torchvision
conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=11.3 -c pytorch
pip install chardet

Notes on Dependencies

• Use the compatibility matrix to match pytorch lightning and pytorch

• torch: After installation, check if pytorch can use .cuda().

• h5py==2.10: this specific version is needed.

• If AttributeError: module 'numpy' has no attribute 'typeDict' is encountered, try uninstall numpy and install numpy==1.21.

• Tested on Ubuntu 18.04 with RTX 3090

Data and Assets

• Training Rearrangement Sequences
• Testing Rearrangement Scenes
• Pairwise Collision Data
• Object Models
• Pretrained Models

Quick Starts

Set up data and models:

• Required data: Testing Rearrangement Scenes
• Required models: Pretrained Models
• Modify the config file base.yaml based on where the testing data and pretrained model are stored. Specifically, modify base_dirs.testing_data and base_dirs.wandb_dir in the config file.

There are two options:

• Running the diffusion model on testing scenes using infer.py:

# in ./scripts/
python infer.py

• Running the diffusion model and collision discriminator on testing scenes using infer.py:

# in ./scripts/
python infer_with_discriminator.py

Training

Training the Conditional Pose Diffusion Model

• Required data: Training Rearrangement Sequences
• Modify the config file base.yaml based on where the training data is stored and where you want to store the trained model.
• You can change params of the model in conditional_pose_diffusion.yaml
• Train the model with train_generator.py. The training progress can be monitored with wandb

# in ./scripts/
python train_generator.py

Training the Pairwise Collision Discriminator

• Required data: Pairwise Collision Data
• Modify the config file base.yaml based on where the training data is stored and where you want to store the trained model.
• Note that training this model requries both Training Rearrangement Sequences and Pairwise Collision Data. We will use partial object point clouds from the rearrangement sequences and then use the query poses and groundtruth collision labels from the collision data.
• You can change params of the model in pairwise_collision.yaml
• Train the model with train_discriminator.py. The training progress can be monitored with wandb.

# in ./scripts/
python train_discriminator.py

Citation

If you find our work useful in your research, please cite:

@inproceedings{structdiffusion2023,
    title     = {StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects},
    author    = {Liu, Weiyu and Du, Yilun and  Hermans, Tucker and Chernova, Sonia and Paxton, Chris},
    year      = {2023},
    booktitle = {RSS 2023}
}