AutoPresent: Designing Structured Visuals From Scratch

This is the code for the paper AutoPresent: Designing Structured Visuals From Scratch by Jiaxin Ge*, Zora Zhiruo Wang*, Xuhui Zhou, Yi-Hao Peng, Sanjay Subramanian, Qinyue Tan, Maarten Sap, Alane Suhr, Daniel Fried, Graham Neubig, Trevor Darrell.

Quick Start

🌟 Try Our online Huggingface Demo!

Click here to try our demo on Huggingface.

Notebook

Coming soon

Contents

This repository contains the code for AutoPresent.

• Install
• SlidesBench
• Create Slides via Programs
  • Generate Programs
  • Refinement
  • Execution
• AutoPresent: Training Slide Generation Models
  • Model Weights
  • Model Training
• Evaluation
  • Reference-based Evaluation
  • Reference-free Evaluation

Install

conda create -n slides python=3.10
conda activate slides

pip install -r requirements.txt
pip insatll . # to install SlidesLib
sudo apt-get install unoconv  # for converting pptx to jpg/pdf

You need to have an OpenAI API key to run the code. We recommend:

conda env config vars set OPENAI_API_KEY=your_key

SlidesBench

The slidesbench directory includes code for:

1. creating examples for any given slide deck in PPTX format, and
2. auto-generate reference python code to create the slides.

See slidesbench/README.md for more details.

Create Slides via Programs

Generate Programs

cd generate

To generate programs to create slide for the entire slide deck:

# sufficient instruction setting
python create_slide_deck.py --slide_deck "art_photos" --setting "perfect"

# visual absence setting
python create_slide_deck.py --slide_deck "art_photos" --setting "visual"

# creative generation setting
python create_slide_deck.py --slide_deck "art_photos" --setting "creative"

If you want to use SlidesLib, add --use_library.

Refinement

You will need the 1 st pass model outputs to start the refinement.

To do refinement on a specific slide, you can provide the instruction, the existing slide, and the code used to generate the slide. The model will generate the refined code and save it to the refined_code_path. For example, you can do the refinement on the model output slide (in slidesbench/examples/art_photos/slide_1/slide.png) as follows:

cd refinement
python refine.py --existing_slide slidesbench/examples/art_photos/slide_1/slide.png --existing_code slidesbench/examples/art_photos/slide_1/original_code.py --instruction_path instruction.txt --refined_code_path refined_code.py --refinement_type fully_specified

You can also run refinement on all the slides generated by the baseline model by running:

cd refinement
python refine_all.py --base_path examples/art_photos/slide_1 --refinement_type fully_specified

Note: you need to have the baseline model outputs to start the refinement.

Execution

To execute the python files generated by the baseline model, use the files under the script folder. For Llama or AutoPresent generated results, you might need to clean the code files first so that they are directly exectable.

cd scripts

# Clean up the code first if needed. You may need to modify this file for different baseline models. 
python clean_generated_code.py --base_dir "examples/" --target_file "Llama.py"

# Execute the python files.
python execute_each_file.py --base_dir "examples/" ----cleaned_file_name "cleaned_Llama.py"

AutoPresent: Training Slide Generation Models

To install SlidesLib, run:

pip install .

This will ensure that SlidesLib is installed.

Model Weights

We release the Autopresent model weights for the three challenges:

Detailed Instructions with Images

Detailed Instructions Only

High Level Instructions

Model Training

We provide the training code in the folder autopresent. In the train.py, set the "your-hf-token" to your own Huggingface token to push the trained model to your Huggingface account. An example to run the code:

cd autopresent

# Train on fully specified task
python train.py --data_path "../slidesbench/dataset_train.csv" --model_name "meta-llama/Llama-3.1-8B-Instruct" --finetuned_model_name "Your_Huggingface_Account/Slide_llama_3.1_fully_specified"

# Train on visual absense task
python train.py --data_path "../slidesbench/dataset_train_no_img.csv" --model_name "meta-llama/Llama-3.1-8B-Instruct" --finetuned_model_name "Your_Huggingface_Account/Slide_llama_3.1_visual_absence"

# Train on creative generation task
python train.py --data_path "../slidesbench/dataset_train_high_level.csv" --model_name "meta-llama/Llama-3.1-8B-Instruct" --finetuned_model_name "Your_Huggingface_Account/Slide_llama_3.1_creative_generation"

Then, to use the trained model to generate the code on SlidesBench challenges, run:

cd autopresent

# Generate Code on fully specified task
python generate.py --path "path_to_your_trained_model" ----save_output_pth "presenter_fully_specified.py" --dataset_path "../slidesbench/dataset_test_fully_specified.csv"

# Generate Code on visual absence task
python generate.py --path "path_to_your_trained_model" ----save_output_pth "presenter_visual_absence.py" --dataset_path "../slidesbench/dataset_test_visual_absence.csv"

# Generate Code on creative generation task
python generate.py --path "path_to_your_trained_model" ----save_output_pth "presenter_creative_generation.py" --dataset_path "../slidesbench/dataset_test_high_level.csv"

For each slide deck, this will save a python file named save_output_pth under the according folder (eg. slidesbench/examples/art_photos/slide_1/presenter.py)

Evaluation

We support two types of evaluation: reference-based evaluation and reference-free evaluation.

cd evaluate
python evaluate_all.py --slide_name "art_photos" # evaluate all pages in the slide deck

Reference-based Evaluation

To evaluate how similar the model-generated slide is to the reference slide, to propose a set of metrics to regarding the spatial, textual, coloring similarities.

To evaluate a single page slide:

python page_eval.py \
--generated_pptx ../slidesbench/examples/art_photos/slide_1/gpt-4o-mini_model.pptx --generated_page 0 \
--reference_pptx ../slidesbench/examples/art_photos/art_photos.pptx --reference_page 0

To evaluate on the entire slide deck:

python slides_eval.py \
--example_dir ../slidesbench/examples/art_photos \
--predict_name "gpt-4o-mini_model"  # name of the pptx files

Reference-free Evaluation

Independently to the the reference slide, we can also evaluate the generated slide based on varied design principles, including content, formatting, and overall presentation.

To evaluate a single page slide:

python reference_free_eval.py \
--image_path ../baseline/examples/art_photos/slide_1/gpt_4o_mini.jpg