Towards Efficient and Effective Adversarial Training

This repository contains code for the implementation of our NeurIPS 2021 paper Towards Efficient and Effective Adversarial Training. Accompanying resources can be found here: [video] [poster]

• We propose single-step Nuclear Norm Adversarial Training (NuAT), and a two step variant of the same (NuAT2), coupled with a novel cyclic-step learning rate schedule, to achieve state-of-the-art results amongst efficient training methods, and results comparable to the effective multi-step training methods.
• We demonstrate improved performance by using exponential averaging of the network weights coupled with cyclic learning rate schedule (NuAT-WA).
• We propose a two-step variant NuAT2-WA, that utilizes stable gradients from the weight-averaged model for stable initialization in the first attack step, yielding improved results.
• We improve the stability and performance of the single-step defense (NuAT) further at a marginal increase in computational cost, by introducing a hybrid approach (NuAT-H) that switches adaptively between single-step and two-step optimization for attack generation.

Trained model checkpoints can be found here.

Nuclear Norm Adversarial Training (NuAT)

In this work, we propose a novel Nuclear Norm regularizer to improve the adversarial robustness of Deep Networks through the use of single-step adversarial training. Training with the proposed Nuclear Norm regularizer enforces function smoothing in the vicinity of clean samples by incorporating joint batch-statistics of adversarial samples, thereby resulting in enhanced robustness.

Nuclear-Norm based Attack: In a given minibatch, we consider to be the matrix composed of vectorized pixel values (arranged row-wise) of each image, to be a matrix of the same dimension as consisting of independently sampled Bernoulli noise, and to be the matrix containing the corresponding ground truth one-hot vectors. The following loss function which utilizes the pre-softmax values is maximized for the generation of single-step adversaries:

In single-step Nuclear Norm Adversarial Training (NuAT), the following loss function is minimized during training:

The first term in the above equation corresponds to the cross-entropy loss on clean samples, and the second term corresponds to the Nuclear-Norm of the difference in pre-softmax values of clean images and their corresponding single-step adversaries .

Results on CIFAR-10

We summarise the robust white-box evaluations on the CIFAR-10 dataset below. We kindly refer the reader to our paper for further details on the two-step adversarial training method (NuAT2), exponential weight averaging (NuAT-WA) and Hybrid Nuclear Norm Adversarial Training (NuAT-H).

Environment Settings

• Python: 3.6.8
• PyTorch: 1.6.0+cu101
• TorchVision: 0.7.0+cu101

Citing this work

@inproceedings{
sriramanan2021towards,
title={Towards Efficient and Effective Adversarial Training},
author={Gaurang Sriramanan and Sravanti Addepalli and Arya Baburaj and Venkatesh Babu Radhakrishnan},
booktitle={Advances in Neural Information Processing Systems},
editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},
year={2021},
url={https://openreview.net/forum?id=kuK2VARZGnI}
}