We release paper and code for SwAV, our new self-supervised method. SwAV pushes self-supervised learning to only 1.2% away from supervised learning on ImageNet with a ResNet-50! It combines online clustering with a multi-crop data augmentation.
We also present DeepCluster-v2, which is an improved version of DeepCluster (ResNet-50, better data augmentation, cosine learning rate schedule, MLP projection head, use of centroids, ...). Check out DeepCluster-v2 code.
This code implements the unsupervised training of convolutional neural networks, or convnets, as described in the paper Deep Clustering for Unsupervised Learning of Visual Features.
Moreover, we provide the evaluation protocol codes we used in the paper:
- Pascal VOC classification
- Linear classification on activations
- Instance-level image retrieval
Finally, this code also includes a visualisation module that allows to assess visually the quality of the learned features.
- a Python installation version 2.7
- the SciPy and scikit-learn packages
- a PyTorch install version 0.1.8 (pytorch.org)
- CUDA 8.0
- a Faiss install (Faiss)
- The ImageNet dataset (which can be automatically downloaded by recent version of torchvision)
We provide pre-trained models with AlexNet and VGG-16 architectures, available for download.
- The models in Caffe format expect BGR inputs that range in [0, 255]. You do not need to subtract the per-color-channel mean image since the preprocessing of the data is already included in our released models.
- The models in PyTorch format expect RGB inputs that range in [0, 1]. You should preprocessed your data before passing them to the released models by normalizing them:
mean_rgb = [0.485, 0.456, 0.406];std_rgb = [0.229, 0.224, 0.225]Note that in all our released models, sobel filters are computed within the models as two convolutional layers (greyscale + sobel filters).
You can download all variants by running
$ ./download_model.sh
This will fetch the models into ${HOME}/deepcluster_models by default.
You can change that path in the environment variable.
Direct download links are provided here:
- AlexNet-PyTorch
- AlexNet-prototxt + AlexNet-caffemodel
- VGG16-PyTorch
- VGG16-prototxt + VGG16-caffemodel
We also provide the last epoch cluster assignments for these models. After downloading, open the file with Python 2:
import pickle
with open("./alexnet_cluster_assignment.pickle", "rb") as f:
b = pickle.load(f)
If you're a Python 3 user, specify encoding='latin1' in the load fonction.
Each file is a list of (image path, cluster_index) tuples.
Finally, we release the features extracted with DeepCluster model for ImageNet dataset. These features are in dimension 4096 and correspond to a forward on the model up to the penultimate convolutional layer (just before last ReLU). In you plan to cluster the features, don't forget to normalize and reduce/whiten them.
Unsupervised training can be launched by running:
$ ./main.sh
Please provide the path to the data folder:
DIR=/datasets01/imagenet_full_size/061417/train
To train an AlexNet network, specify ARCH=alexnet whereas to train a VGG-16 convnet use ARCH=vgg16.
You can also specify where you want to save the clustering logs and checkpoints using:
EXP=exp
During training, models are saved every other n iterations (set using the --checkpoints flag), and can be found in for instance in ${EXP}/checkpoints/checkpoint_0.pth.tar.
A log of the assignments in the clusters at each epoch can be found in the pickle file ${EXP}/clusters.
Full documentation of the unsupervised training code main.py:
usage: main.py [-h] [--arch ARCH] [--sobel] [--clustering {Kmeans,PIC}]
[--nmb_cluster NMB_CLUSTER] [--lr LR] [--wd WD]
[--reassign REASSIGN] [--workers WORKERS] [--epochs EPOCHS]
[--start_epoch START_EPOCH] [--batch BATCH]
[--momentum MOMENTUM] [--resume PATH]
[--checkpoints CHECKPOINTS] [--seed SEED] [--exp EXP]
[--verbose]
DIR
PyTorch Implementation of DeepCluster
positional arguments:
DIR path to dataset
optional arguments:
-h, --help show this help message and exit
--arch ARCH, -a ARCH CNN architecture (default: alexnet)
--sobel Sobel filtering
--clustering {Kmeans,PIC}
clustering algorithm (default: Kmeans)
--nmb_cluster NMB_CLUSTER, --k NMB_CLUSTER
number of cluster for k-means (default: 10000)
--lr LR learning rate (default: 0.05)
--wd WD weight decay pow (default: -5)
--reassign REASSIGN how many epochs of training between two consecutive
reassignments of clusters (default: 1)
--workers WORKERS number of data loading workers (default: 4)
--epochs EPOCHS number of total epochs to run (default: 200)
--start_epoch START_EPOCH
manual epoch number (useful on restarts) (default: 0)
--batch BATCH mini-batch size (default: 256)
--momentum MOMENTUM momentum (default: 0.9)
--resume PATH path to checkpoint (default: None)
--checkpoints CHECKPOINTS
how many iterations between two checkpoints (default:
25000)
--seed SEED random seed (default: 31)
--exp EXP path to exp folder
--verbose chatty
To run the classification task with fine-tuning launch:
./eval_voc_classif_all.sh
and with no finetuning:
./eval_voc_classif_fc6_8.sh
Both these scripts download this code.
You need to download the VOC 2007 dataset. Then, specify in both ./eval_voc_classif_all.sh and ./eval_voc_classif_fc6_8.sh scripts the path CAFFE to point to the caffe branch, and VOC to point to the Pascal VOC directory.
Indicate in PROTO and MODEL respectively the path to the prototxt file of the model and the path to the model weights of the model to evaluate.
The flag --train-from allows to indicate the separation between the frozen and to-train layers.
We implemented voc classification with PyTorch.
Erratum: When training the MLP only (fc6-8), the parameters of scaling of the batch-norm layers in the whole network are trained. With freezing these parameters we get 70.4 mAP.
You can run these transfer tasks using:
$ ./eval_linear.sh
You need to specify the path to the supervised data (ImageNet or Places):
DATA=/datasets01/imagenet_full_size/061417/
the path of your model:
MODEL=/private/home/mathilde/deepcluster/checkpoint.pth.tar
and on top of which convolutional layer to train the classifier:
CONV=3
You can specify where you want to save the output of this experiment (checkpoints and best models) with
EXP=exp
Full documentation for this task:
usage: eval_linear.py [-h] [--data DATA] [--model MODEL] [--conv {1,2,3,4,5}]
[--tencrops] [--exp EXP] [--workers WORKERS]
[--epochs EPOCHS] [--batch_size BATCH_SIZE] [--lr LR]
[--momentum MOMENTUM] [--weight_decay WEIGHT_DECAY]
[--seed SEED] [--verbose]
Train linear classifier on top of frozen convolutional layers of an AlexNet.
optional arguments:
-h, --help show this help message and exit
--data DATA path to dataset
--model MODEL path to model
--conv {1,2,3,4,5} on top of which convolutional layer train logistic
regression
--tencrops validation accuracy averaged over 10 crops
--exp EXP exp folder
--workers WORKERS number of data loading workers (default: 4)
--epochs EPOCHS number of total epochs to run (default: 90)
--batch_size BATCH_SIZE
mini-batch size (default: 256)
--lr LR learning rate
--momentum MOMENTUM momentum (default: 0.9)
--weight_decay WEIGHT_DECAY, --wd WEIGHT_DECAY
weight decay pow (default: -4)
--seed SEED random seed
--verbose chatty
You can run the instance-level image retrieval transfer task using:
./eval_retrieval.sh
We provide two standard visualisation methods presented in our paper.
First, it is posible to learn an input image that maximizes the activation of a given filter. We follow the process described by Yosinki et al. with a cross entropy function between the target filter and the other filters in the same layer. From the visu folder you can run
./gradient_ascent.sh
You will need to specify the model path MODEL, the architecture of your model ARCH, the path of the folder in which you want to save the synthetic images EXP and the convolutional layer to consider CONV.
Full documentation:
usage: gradient_ascent.py [-h] [--model MODEL] [--arch {alexnet,vgg16}]
[--conv CONV] [--exp EXP] [--lr LR] [--wd WD]
[--sig SIG] [--step STEP] [--niter NITER]
[--idim IDIM]
Gradient ascent visualisation
optional arguments:
-h, --help show this help message and exit
--model MODEL Model
--arch {alexnet,vgg16}
arch
--conv CONV convolutional layer
--exp EXP path to res
--lr LR learning rate (default: 3)
--wd WD weight decay (default: 10^-5)
--sig SIG gaussian blur (default: 0.3)
--step STEP number of iter between gaussian blurs (default: 5)
--niter NITER total number of iterations (default: 1000)
--idim IDIM size of input image (default: 224)
I recommand you play with the hyper-parameters to find a regime where the visualisations are good. For example with the pre-trained deepcluster AlexNet, for conv1 using a learning rate of 3 and 30.000 iterations works well. For conv5, using a learning rate of 30 and 3.000 iterations gives nice images with the other parameters set to their default values.
Finally, we provide code to retrieve images in a dataset that maximally activate a given filter in the convnet.
From the visu folder, after having changed the fields MODEL, EXP, CONV and DATA, run
./activ-retrieval.sh
We have proposed another unsupervised feature learning paper at ICCV 2019. We have shown that unsupervised learning can be used to pre-train convnets, leading to a boost in performance on ImageNet classification. We achieve that by scaling DeepCluster to 96M images and mixing it with RotNet self-supervision. Check out the paper and code.
You may find out more about the license here.
If you use this code, please cite the following paper:
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. "Deep Clustering for Unsupervised Learning of Visual Features." Proc. ECCV (2018).
@InProceedings{caron2018deep,
title={Deep Clustering for Unsupervised Learning of Visual Features},
author={Caron, Mathilde and Bojanowski, Piotr and Joulin, Armand and Douze, Matthijs},
booktitle={European Conference on Computer Vision},
year={2018},
}