LEGO_StableDiffusion

Overview

The project aim was to fine-tuned stable diffusion model in order to generate images in the LEGO style based on the prompt.

Fine-tuned model is created on top of Stable Diffusion text to image model.

To run app install requirements by typing: pip3 install -r requirements, then run the main app file python3 app/main.py, making sure you are in the root repo directory.

Database was created by our own by scraping the LEGO pictures from websites such as gettyimages.com, istockphoto.com and brickset.com (scripts available in the Preprocessing directory). LEGO database with mixed figures and constructions images is available here.
Image captioning was done using BLIP model. Image captioning script (image_captioning.py) is available in the Preprocessing/BLIP project directory.
App is created using gradio library. In order to change GPU index or models' names do it in the app/main.py file.

Models are available here: https://www.kaggle.com/datasets/michasakowicz/lego-dataset-models.

Results

Images:

Rubber Duck Aliens visiting the Earth for the first time	A small cabin on top of a snowy mountain in the style of Disney	A high tech solarpunk utopia in the Amazon rainforest	An insect robot preparing a delicious meal

App: