Skip to content

Commit

Permalink
Add the obtained results
Browse files Browse the repository at this point in the history
  • Loading branch information
bryant1410 committed Jun 18, 2023
1 parent 9b429a2 commit 46b4b00
Show file tree
Hide file tree
Showing 4 changed files with 6,299 additions and 9 deletions.
27 changes: 18 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,28 @@

[[Paper]](https://arxiv.org/pdf/2305.18786.pdf)

In this work, we propose a simple and effective method to **probe vision-language models**.
In this work, we propose a simple and effective method to **probe vision-language models** (VLMs).

Our method is **scalable**, as it does not require data annotation and makes use of existing datasets.
With our method, we analyzed the performance of **CLIP**, a popular state-of-the-art multi-modal model, on the **SVO-Probes** benchmark.
With our method, we analyzed the performance of [CLIP](https://openai.com/research/clip), a popular state-of-the-art
multi-modal model, on the [SVO-Probes](https://github.com/deepmind/svo_probes) benchmark.

We hope our work contributes to ongoing efforts to discover the limitations of multi-modal models and help build more robust and reliable systems.
Our framework can be easily used to analyze other benchmarks, features, and multi-modal models
![A description of our probing method, showing 2 images being input to CLIP, then 3 scores being computed. Different
kind of features are used to compute their correlation with each of the scores.](images/task_overview.png)

<p style="text-align:center">
<img src="images/task_overview.png" alt="A description of our probing method, showing 2 images being input to clip, then 3 scores being computed. Different kind of features are used to compute their correlation with each of the scores.">
</p>
We hope our work contributes to ongoing efforts to discover the limitations of multi-modal models and help build more
robust and reliable systems. Our framework can be easily used to analyze other benchmarks, features, and multi-modal
models.

## Setup
## Obtained Results

Under [results/](results) you can find the detailed results obtained with our method for the 3 different scores tested
(read the paper for details). They come from the output of running the code in this repository (see below to reproduce
it).

## Reproducing the Results

### Setup

With Python >= 3.8, run the following commands:

Expand All @@ -26,7 +35,7 @@ huggingface-cli login
mkdir data
```

**We will post more instructions soon.**
**We'll write more instructions soon.**

## Citation

Expand Down
Loading

0 comments on commit 46b4b00

Please sign in to comment.