Code for the paper Nonsmooth Implicit Differentiation: Deterministic and Stochastic Convergence Rates by Riccardo Grazzi, Massimiliano Pontil and Saverio Salzo (ICML 2024).
Install the packages in requirements.txt.
Check out elastic_net_toy.ipynb for an illustrative comparison between deterministic AID and ITD derivative approximation methods for (nonsmooth) elastic net.
Run one of the following files:
- elastic_net_deterministic.py for the experiments comparing AID and ITD on computing the derivative with respect to the hyperparameters of elastic net.
- elastic_net_stochastic.py for the experiments comparing AID-FP and NSID and SID on computing the derivative with respect to the hyperparameters of elastic net.
- data_poisoning_stochastic.py for the experiments comparing (N)SID with constant and decreasing step-size schedules on computing the derivative with respect to the corruption noise of the data poisoning with elastic net regularization. The MNIST dataset will be automatically downloaded in the data folder when first run.
When the dataset is large (e.g. for data poisoning) using the GPU can speed up the computation. All Experiments artifacts will be saved in the exps folder inside the project directory
Use the notebooks in the analyse_results folder to generate plots from the data of previously run experiments.
See hypertorch for more details on the AID and ITD hypergradient approximation methdos and some examples on how to incorporate them in a project.
nonsmooth_implicit_diff/stoch_hg.py contains the code for the NSID method to approximate the derivaive of a fixed point which is a composition of an outer map and an inner map accessible only through a stochastic unbiased estimator.
Details on the experimental settings can be found in our paper.
If you use this code, please cite our paper.
@article{grazzi2024nonsmooth,
title={Nonsmooth implicit differentiation: Deterministic and stochastic convergence rates},
author={Grazzi, Riccardo and Pontil, Massimiliano and Salzo, Saverio},
journal={arXiv preprint arXiv:2403.11687},
year={2024}
}