SegmentStyler [Project's Report]

Part-aware language-based mesh texture editing

SegmentStyler is a framework for part-aware mesh texture editing that uses natural language and allows for separate customization of different mesh segments based on multiple text prompts, resulting in a coherent texture with the text input.

rainbow pattern arm glowing iron leg guy fawkes logo back camo pattern seat	persian carpet arm swiss cheese leg desert sand back fleur-de-lis pattern seat	watermelon pattern arm red leather leg graffiti back colorful crochet seat	snowflake pattern arm metallic leg honeycomb back wooden seat

Note

This project was implemented during a guided research practical course at TUM in the Winter Semester of 2022/2023 by Maximilian Winter and Murilo Bellatini. For more information, please refer to the project's report, which can be found here.

Getting Started

Installation

1. Run setup.sh.

sh ./setup.sh

2. Setup PartGlot

   1. Download data and weights here.
   2. Add data files and weights to data/PartGlotData folder.

3. Setup Data (Note: this step can be skipped if you only want to try out the pipeline with the one shape that's already given in data/)

   1. Download ShapeNetCore.v2 here
   2. Download the part annotations here
   3. Start pre-processing (remeshing and fixing normals) by running

   python src/helper/get_new_normals_from_sdf.py
   

   For the pre-processing, make sure to have the following packages installed

   • trimesh
   • mesh_to_sdf
   • skimage To avoid conflicts, it is recommended to install them (especially the last two) in a separate environment.

4. Activate environment.

conda activate dl3d

4. Confirm Weights & Biases installation. More info: click here.

pip install wandb
wandb login

5. Enjoy ;)

Run examples

To see an example, you can run

bash example_run.sh

To gain more insights, you can have a look at

• evaluation_baseline.sh
• ablation_study_b_to_g.sh which were used for performing our ablation study.

Miscellaneous

The notebooks folder contains code snippets that we used during our development process. You can find a comprehensive index of these notebooks here. However, please note that the documentation within each notebook is currently limited and may not be up-to-date. We apologize for any inconvenience and appreciate your understanding.

Reference Projects

1. Text2Mesh
2. PartGlot