Classifier-guided CLIP Distillation for Unsupervised Multi-label Classification (CVPR 2025)

Official pytorch implementation of "Classifier-guided CLIP Distillation for Unsupervised Multi-label Classification"

Dongseob Kim^*,1 , Hyunjung Shim^2
1 Samsung Electronics, and ² Korea Advanced Institute of Science & Technology

Abstract Multi-label classification is crucial for comprehensive image understanding, yet acquiring accurate annotations is challenging and costly. To address this, a recent study suggests exploiting unsupervised multi-label classification leveraging CLIP, a powerful vision-language model. Despite CLIP's proficiency, it suffers from view-dependent predictions and inherent bias, limiting its effectiveness. We propose a novel method that addresses these issues by leveraging multiple views near target objects, guided by Class Activation Mapping (CAM) of the classifier, and debiasing pseudo-labels derived from CLIP predictions. Our Classifier-guided CLIP Distillation (CCD) enables selecting multiple local views without extra labels and debiasing predictions to enhance classification performance. Experimental results validate our method's superiority over existing techniques across diverse datasets.

Updates

20 April, 2025: Initial upload

Installation

Step 0. Install PyTorch and Torchvision following official instructions, e.g.,

pip install torch torchvision
# FYI, we're using torch==1.13.1 and torchvision==0.14.1
# We used docker image pytorch:1.13.1-cuda11.6-cudnn8-devel
# or later version will be okay

Step 1. Install packages.

pip install ftfy regex tqdm munch 
# FYI, we're using mmcv-full==1.4.0 

Step 2. Download CLIP pretrained model.

mkdir pretrained
curl https://openaipublic.azureedge.net/clip/models/be1cfb55d75a9666199fb2206c106743da0f6468c9d327f3e0d0a543a9919d9c/RN50x64.pt --output pretrained/RN50x64.pt

Step 3. Install CARB.

git clone https://github.com/k0u-id/CCD.git

Dataset Preparation & Pretrained Checkpoint

In our paper, we experiment with pascal VOC, MS COCO, and Nuswide. please refer data/README.md

• Example directory hierarchy

  CCD
  |--- data
  |    |--- pascal
  |    |    |---Annotations
  |    |    |---ImageSets
  |    |    |---JPEGImages
  |    |    | ...
  |    |--- coco
  |    |    | ...
  |    |--- nuswide
  |    |    | ...
  |--- metadata
  |    |--- voc12
  |    |    |---formatted_train_images.npy
  |    |    |---formatted_train_labels.npy
  |    |    |---formatted_val_images.npy
  |    |    | ...
  |    |--- voc07
  |    |    | ...
  |    |--- coco
  |    |    | ...
  |    |--- nuswide
  |    |    | ...
  |--- results
  |    |--- output_dirs (excuted-date&time)
  |    | ...
  | ...
  

metadata

• voc12
• voc07
• coco
• nuswide

Pretrained Checkpoint

• voc12
• voc07
• coco
• nuswide

training CCD

CCD trains multi-label classification model with label updating.

# Please see this file for the detail of execution.
# You can change detailed configuration by changing config files (e.g., CARB/configs/carb/cityscapes_carb_dual.py)
bash train_gpu#0.sh 

Inference CCD

# uncomment run_test in main_multi.py
# change line 59 in test.py
# we will refine inference code soon

Acknoledgement

This code is highly borrowed from Multi-Label Learning from Single Positive Labels, BridgeGapExplanationPAMC.

Citation

If you use CCD or this code base in your work, please cite

@article{kim2025classifier,
  title={Classifier-guided CLIP Distillation for Unsupervised Multi-label Classification},
  author={Kim, Dongseob and Shim, Hyunjung},
  journal={arXiv preprint arXiv:2503.16873},
  year={2025}
}