Full-Range Self-Attention

This folder contains the implementation of Full-Range Self-Attention based on DeiT and PVT models for image classification.

Dependencies

• Python 3.9
• PyTorch == 1.11.0
• torchvision == 0.12.0
• numpy
• timm == 0.4.12
• einops
• yacs

pip install -r requirements.txt

Data preparation

The ImageNet dataset should be prepared as follows:

$ tree data
imagenet
├── train
│   ├── class1
│   │   ├── img1.jpeg
│   │   ├── img2.jpeg
│   │   └── ...
│   ├── class2
│   │   ├── img3.jpeg
│   │   └── ...
│   └── ...
└── val
    ├── class1
    │   ├── img4.jpeg
    │   ├── img5.jpeg
    │   └── ...
    ├── class2
    │   ├── img6.jpeg
    │   └── ...
    └── ...

Train Models from Scratch

• To train FSA-DeiT/FSA-PVT on ImageNet from scratch, run:

python -m torch.distributed.launch --nproc_per_node=8 main.py --cfg <path-to-config-file> --data-path <imagenet-path> --output <output-path>