POSE

Progressive Open Space Expansion for Open-Set Model Attribution

Tianyun Yang, Danding Wang, Fan Tang, Xinying Zhao, Juan Cao, Sheng Tang, CVPR 2023 Paper, Slide

Abstract

Despite the remarkable progress in generative technology, the Janus-faced issues of intellectual property protection and malicious content supervision have arisen. Efforts have been made to manage synthetic images by attributing them to a set of potential source models. However, the closed-set classification setting limits the application in real-world scenarios for handling contents generated by arbitrary models. In this work:

1. We tackle an important challenge for applying model attribution to open scenarios, the Open-Set Model Attribution (OSMA) problem, which attributes images to known models and identifies images from unknown ones.
   
2. We propose a novel Progressive Open Space Expansion (POSE) solution, which simulates the potential open space of unknown models progressively by a set of lightweight augmentation models and consequently reduces open space risk.
   
3. We construct an OSMA benchmark simulating the real-world scenarios, on which extensive experiments prove the superiority of POSE compared with existing GAN attribution methods and off-the-shelf OSR methods.

Prerequisites

• Linux
• NVIDIA GPU + CUDA 11.1
• Python 3.7.13
• PyTorch 1.10.0

Dataset

The whole dataset is hosted here. Download, unzip, and put the dataset into the directory ./dataset/.

As the size of the whole dataset is up to 50GB. We provide a small subset of the dataset in DatasetSubset. You can use this for the evaluation and visualization demos below. Also, download and put the subset into the directory ./dataset/.

The annotation files are in ./dataset and can be downloaded along with the project. The organization is as follows:

dataset
├── $split{id}_test
│   └── annotations
│       ├── $split{id}_test.txt
│       ├── $split{id}_test_out.txt
│       ├── $split{id}_test_out_seed.txt
│       ├── $split{id}_test_out_arch.txt
│       └── $split{id}_test_out_dataset.txt
├── $split{id}_train
│   └── annotations
│       └── $split{id}_train.txt
└── $split{id}_val
    └── annotations
        └── $split{id}_val.txt

where split{id}_train.txt, split{id}_val.txt, split{id}_test.txt are the annotation files for training, validation, closed-set testing. split{id}_test_out.txt is the annotation file for all open-set/unknown data. split{id}_test_out_seed.txt, split{id}_test_out_arch.txt, split{id}_test_out_dataset.txt are annotation files for unseen seed, unseen architecture, and unseen dataset respectively.

Training

• Run the following script:

sh ./script/run_train.sh

• Following is a training script example for split1:

python3 main.py --config_name progressive --mode POSE --device cuda:0 --data split1

Checkpoints

• We provide checkpoints trained on five splits of the dataset in Checkpoints. Download and put files into the directory ./checkpoints. Each file split#/model.pth corresponds to the weight trained on split#. The directory also includes augmentation models and model weights for split1 and split2 during the progressive training progress.

Demo

• Demo1 for open-set evaluation, including metric calculation, confidence histogram, ROC curve, and detailed results for each unknown class.

• Demo2 for t-SNE visualization of known, unknown, and augmented data. (Fig.4 in our paper)

• Demo3 for t-SNE visualization of the progressive open space expansion process. (Fig.5 in our paper)

Testing

• For result reproduction, you may download the whole dataset and run the following:

  sh ./script/run_test.sh
  
• Following is a testing script example for split1:

  python3 test.py --model_path ./checkpoints/split1/model.pth --device cuda:0 -data split1
  

Acknowledgement

We sincerely appreciate the following repositories where generative models in our dataset are mainly from:

https://github.com/ningyu1991/GANFingerprints
https://github.com/TuBui/image_attribution
https://github.com/kwotsin/mimicry
https://github.com/yunjey/stargan
https://github.com/LynnHo/AttGAN-Tensorflow
https://github.com/tkarras/progressive_growing_of_gans
https://github.com/NVlabs/stylegan
https://github.com/NVlabs/stylegan2
https://github.com/NVlabs/stylegan3
https://github.com/POSTECH-CVLab/PyTorch-StudioGAN
https://github.com/ondyari/FaceForensics
https://github.com/Rudrabha/Wav2Lip
https://github.com/YuvalNirkin/fsgan
https://github.com/richarduuz/Research_Project/tree/master/ModelC

How to Cite

@inproceedings{yang2023progressive,
  title={Progressive Open Space Expansion for Open-Set Model Attribution},
  author={Yang, Tianyun and Wang, Danding and Tang, Fan and Zhao, Xinying and Cao, Juan and Tang, Sheng},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={15856--15865},
  year={2023}
}