Caption-Anything is a versatile image processing tool that combines the capabilities of Segment Anything, Visual Captioning, and ChatGPT. Our solution generates descriptive captions for any object within an image, offering a range of language styles to accommodate diverse user preferences. It supports visual controls (mouse click) and language controls (length, sentiment, factuality, and language).

• Visual controls and language controls for text generation
• Chat about selected object for detailed understanding
• Interactive demo

Along the River During the Qingming Festival (清明上河图)

Updates

• 2023/04/13: add Colab Tutorial
• 2023/04/12: add Hugging Face demo
• 2023/04/11: Release code

Demo

Explore the interactive demo of Caption-Anything, which showcases its powerful capabilities in generating captions for various objects within an image. The demo allows users to control visual aspects by clicking on objects, as well as to adjust textual properties such as length, sentiment, factuality, and language.

---

---

---

Getting Started

Linux

# Clone the repository:
git clone https://github.com/ttengwang/caption-anything.git
cd caption-anything

# Install dependencies:
pip install -r requirements.txt

# Download the [SAM checkpoints](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth).
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth ./segmenter/sam_vit_h_4b8939.pth

# Configure the necessary ChatGPT APIs
export OPENAI_API_KEY={Your_Private_Openai_Key}

# Run the Caption-Anything gradio demo.
python app.py --segmenter huge --captioner blip2 --port 6086 # requires 11.7G GPU memory
#python app.py --segmenter base --captioner blip2 # requires 8.5G GPU memory
#python app.py --segmenter base --captioner blip # requires 5.5G GPU memory

Windows(powershell)

Tested in Windows11 using Nvidia 3070-8G.

# Clone the repository:
git clone https://github.com/ttengwang/caption-anything.git
cd caption-anything

# Install dependencies:
pip install -r requirements.txt

# Download the [base SAM checkpoints](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth).
Invoke-WebRequest https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth -OutFile ./segmenter/sam_vit_b_01ec64.pth

# Configure the necessary ChatGPT APIs
$env:OPENAI_API_KEY = '{Your_Private_Openai_Key}'

# Run the Caption-Anything gradio demo.
python app.py --captioner blip --port 6086 --segmenter base

Usage

from caption_anything import CaptionAnything, parse_augment
args = parse_augment()
visual_controls = {
    "prompt_type":["click"],
    "input_point":[[500, 300], [1000, 500]],
    "input_label":[1, 0], # 1/0 for positive/negative points
    "multimask_output":"True",
}
language_controls = {
    "length": "30",
    "sentiment": "natural", # "positive","negative", "natural"
    "imagination": "False", # "True", "False"
    "language": "English" # "Chinese", "Spanish", etc.
}
model = CaptionAnything(args, openai_api_key)
out = model.inference(image_path, visual_controls, language_controls)

Acknowledgements

The project is based on Segment Anything, BLIP/BLIP-2, ChatGPT. Thanks for the authors for their efforts.

Contributor

Our project wouldn't be possible without the contributions of these amazing people! Thank you all for making this project better.

Teng Wang @ Southern University of Science and Technology & HKU & Tencent ARC Lab
Jinrui Zhang @ Southern University of Science and Technology
Junjie Fei @ Xiamen University
Zhe Li @ Southern University of Science and Technology
Yunlong Tang @ Southern University of Science and Technology
Mingqi Gao @ Southern University of Science and Technology & University of Warwick
Hao Zheng @ Southern University of Science and Technology