Multi modal rag demo #698

init27 · 2024-10-01T20:36:39Z

MM RAG Demo:

3 Part Series:

Notebook 1: Data Prep and Synethetic Labelling using 11B model
WIP: Notebook 2: Cleaning up Synthetic Labels using 1B or 3B model and creating DB with
WIP: Notebook 3: MM-RAG using lance-db

Scripts:

Script 1: Annotating Dataset with multi-GPUs
Script 2: Cleaning up Descriptions and making the db
Script 3: Llama 3.2 + lance db gradio demo

Not working-Validating idea for MM RAG

Data Prep Notebook

Notebook 1: I will add MD details later (as usual)

Script for Multi-GPU labelling with TQDM

Added corrupt image handling

Improve TQDM

Small Bug

Added llama to progress bar

WIP Cleanup_notebook

Finalised Notebook 2

recipes/quickstart/Multi-Modal-RAG/README.md

HamidShojanazeri

Thanks @init27 for the PR! left some comments inline.

recipes/quickstart/Multi-Modal-RAG/README.md

HamidShojanazeri · 2024-11-20T13:27:52Z

recipes/quickstart/Multi-Modal-RAG/README.md

+1. Please grab your HF CLI Token from [here](https://huggingface.co/settings/tokens)
+2. git clone [this dataset](https://huggingface.co/datasets/Sanyam/MM-Demo) inside the Multi-Modal-RAG folder: `git clone https://huggingface.co/datasets/Sanyam/MM-Demo`
+3. Launch jupyter notebook inside this folder
+4. We will also run two scripts after the notebooks


this seems unnecessary to be part of the requirements?

@HamidShojanazeri actually we need to clone the dataset from Huggingface, so im pointing that out.

Do you think I should move it inside a notebook?

recipes/quickstart/Multi-Modal-RAG/README.md

recipes/quickstart/Multi-Modal-RAG/notebooks/Part_1_Data_Preperation.ipynb

Co-authored-by: Hamid Shojanazeri <[email protected]>

…ration.ipynb Co-authored-by: Hamid Shojanazeri <[email protected]>

…llama-recipes into Multi-Modal-RAG-Demo

init27 added 3 commits October 1, 2024 10:54

Notebook for Part 3

aaadefb

Not working-Validating idea for MM RAG

Create Part_1_Data_Preperation.ipynb

02d53f1

Data Prep Notebook

Update Part_1_Data_Preperation.ipynb

730bc6f

Notebook 1: I will add MD details later (as usual)

facebook-github-bot added the cla signed label Oct 1, 2024

init27 added 14 commits October 1, 2024 13:59

Create label_script.py

1773feb

Script for Multi-GPU labelling with TQDM

Update label_script.py

6dd384c

Added corrupt image handling

Update label_script.py

4a297e5

Improve TQDM

Update label_script.py

849684b

Small Bug

Update label_script.py

7a9e595

Added llama to progress bar

Create Part_2_Cleaning_Data_and_DB

66cb619

WIP Cleanup_notebook

Update Part_2_Cleaning_Data_and_DB

f7e687b

Finalised Notebook 2

Add Gradio App and Re-fact Part 3 nb

abe2c5a

add drop down menu

fde6ebe

Add methods

f1c1f6d

Fix Methods and fix prompts

089e0b1

Add high level ReadMe

9bbf3ef

Update final_demo.py

c8b1a2d

Fix Folders, Add readme

ca7cdce

init27 marked this pull request as ready for review October 17, 2024 20:01

init27 added 11 commits October 18, 2024 10:32

Remove nb

0fb47b6

fix nbs

480fb94

Fix Spelling Mistakes

181e841

Fix link

30018f8

Part 1 Readme

2fde71d

Fix Readme

bdb05c1

Spell Check Round 2

e76e148

Round 3

fe115fb

Rename

532618e

Credit

d7cf320

Nb 1

d984730

init27 added 8 commits October 18, 2024 15:46

Part 1-fin

d77aa30

nb-2

3adf211

nb2-final

a013bc8

Nb-3 complete

9926fb2

Fix spelling

1687587

Rebase (#789)

cdaf80b

Update README.md

1a7e4a0

Update README.md

26f0096

IgorKasianenko reviewed Nov 18, 2024

View reviewed changes

recipes/quickstart/Multi-Modal-RAG/README.md Outdated Show resolved Hide resolved

init27 added 12 commits November 19, 2024 21:13

Update README.md

03ffc5b

Update README.md

3996d06

Update README.md

1a2f8f8

Update README.md

8764d5f

Update README.md

1c02602

Update README.md

565ed73

Update Part_1_Data_Preperation.ipynb

cecfe46

Update Part_2_Cleaning_Data_and_DB.ipynb

7b67aa5

Update Part_3_RAG_Setup_and_Validation.ipynb

4640c83

Update Part_3_RAG_Setup_and_Validation.ipynb

ec23508

Update label_script.py

57ee65a

rebase (#794)

7676268

HamidShojanazeri reviewed Nov 20, 2024

View reviewed changes

recipes/quickstart/Multi-Modal-RAG/notebooks/Part_1_Data_Preperation.ipynb Outdated Show resolved Hide resolved

recipes/quickstart/Multi-Modal-RAG/notebooks/Part_1_Data_Preperation.ipynb Outdated Show resolved Hide resolved

init27 and others added 7 commits November 20, 2024 08:43

Update recipes/quickstart/Multi-Modal-RAG/README.md

4d35c8c

Co-authored-by: Hamid Shojanazeri <[email protected]>

Update recipes/quickstart/Multi-Modal-RAG/README.md

c9df1be

Co-authored-by: Hamid Shojanazeri <[email protected]>

address comments

9950b75

Update recipes/quickstart/Multi-Modal-RAG/notebooks/Part_1_Data_Prepe…

51a5ae8

…ration.ipynb Co-authored-by: Hamid Shojanazeri <[email protected]>

Merge branch 'Multi-Modal-RAG-Demo' of https://github.com/meta-llama/…

78bdd63

…llama-recipes into Multi-Modal-RAG-Demo

Update Part_1_Data_Preperation.ipynb

0284c66

move to use-cases

62c1fa4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi modal rag demo #698

Multi modal rag demo #698

init27 commented Oct 1, 2024 •

edited

Loading

HamidShojanazeri left a comment

HamidShojanazeri Nov 20, 2024

init27 Nov 20, 2024

Multi modal rag demo #698

Are you sure you want to change the base?

Multi modal rag demo #698

Conversation

init27 commented Oct 1, 2024 • edited Loading

HamidShojanazeri left a comment

Choose a reason for hiding this comment

HamidShojanazeri Nov 20, 2024

Choose a reason for hiding this comment

init27 Nov 20, 2024

Choose a reason for hiding this comment

init27 commented Oct 1, 2024 •

edited

Loading