Deep Generative Models course, OzonMasters, 2022

Description

The course is devoted to modern generative models (mostly in the application to computer vision).

We will study the following types of generative models:

• autoregressive models,
• latent variable models,
• normalization flow models,
• adversarial models,
• diffusion models.

Special attention is paid to the properties of various classes of generative models, their interrelationships, theoretical prerequisites and methods of quality assessment.

The aim of the course is to introduce the student to widely used advanced methods of deep learning.

The course is accompanied by practical tasks that allow you to understand the principles of the considered models.

Materials

#	Date	Description	Slides
0	February, 4	Logistics and intro.	slides
1	February, 8	Lecture: Motivation. Divergence minimization framework. Autoregressive modelling.	slides
		Seminar: Introduction. Density estimation in 1D. MADE theory.	notebook
2	February, 15	Lecture: Autoregressive models (WaveNet, PixelCNN, PixelCNN++). Bayesian Framework. Latent Variable Models.	slides
		Seminar: MADE practice. PixelCNN implementation hints. Bayesian inference intro, conjugate distributions.	notebook
3	February, 22	Lecture: Variational lower bound. EM-algorithm, amortized inference. ELBO gradients, reparametrization trick.	slides
		Seminar: Mean field approximation.	notebook
4	March, 1	Lecture: Variational Autoencoder (VAE). Posterior collapse and decoder weakening. Tighter ELBO (IWAE).	slides
		Seminar: EM-algorithm. VAE theory. Automatic differentiation through random graph.	---
5	March, 8	Lecture: Flow models definition. Forward and reverse KL divergence. Linear flows (Glow). Residual flows (Planar/Sylvester flows).	slides
		Seminar: IWAE theory. IWAE variational posterior. VAE vs Normalizing flows.	---
6	March, 15	Lecture: Autoregressive flows (MAF/IAF). Coupling layer (RealNVP).	slides
		Seminar: Planar flows. Forward vs Reverse KL.	notebook
7	March, 22	Lecture: Uniform and variational dequantization. ELBO surgery and optimal VAE prior. Flows-based VAE posterior vs flow-based VAE prior.	slides
		Seminar: VAE prior (VampPrior). SurVAE. RealNVP hints.	---
8	March, 29	Lecture: Disentanglement learning (beta-VAE, DIP-VAE + summary). Likelihood-free learning. GAN theorem.	slides
		Seminar: GAN vs VAE vs NF. GAN in 1d coding.	notebook
9	April, 5	Lecture: Vanishing gradients and mode collapse, KL vs JSD. Adversarial Variational Bayes. Wasserstein distance.	slides
		Seminar: GAN vs VAE theory. KL vs JS divergences.	notebook
10	April, 12	Lecture: Wasserstein GAN. WGAN-GP. Spectral Normalization GAN. f-divergence minimization.	slides
		Seminar: WGAN: practice. Optimal transport task. SN-GAN: practice.	notebook
11	April, 19	Lecture: GAN evaluation (Inception score, FID, Precision-Recall, truncation trick). GAN models (Self-Attention GAN, BigGAN, PGGAN, StyleGAN).	slides
		Seminar: StyleGAN: implementation hints.	notebook
12	April, 26	Lecture: 12. Discrete VAE latent representations. Vector quantization, straight-through gradient estimation (VQ-VAE). Gumbel-softmax trick (DALL-E). Neural ODE.	slides
		Seminar: NeuralODE explanation.	---
13	May, 17	Lecture: Adjoint method. Continuous-in-time NF (FFJORD, Hutchinson's trace estimator). Kolmogorov-Fokker-Planck equation and Langevin dynamic. SDE basics.	slides
		Seminar: TBA	TBA
14	May, 24	Lecture: Score matching. Noise conditioned score network (NCSN). Denoising diffusion probabilistic model (DDPM).	slides
		Seminar: TBA	TBA
	May, 31	Oral exam	TBA

Homeworks

Homework	Date	Deadline	Description	Link
1	February, 13	February, 27	Theory (MADE, Mixture of Logistics). PixelCNN on MNIST. PixelCNN autocomplete and receptive field.
2	February, 27	March, 13	Theory (log-derivative trick, IWAE theorem). VAE on 2D data. VAE on CIFAR10.
3	March, 13	March, 27	Theory (Sylvester flows). RealNVP on 2D data. RealNVP on CIFAR10.
4	March, 27	April, 10	Theory (MI in ELBO surgery). VAE with AR decoder on MNIST. VAE with AR prior on CIFAR10.
5	April, 10	April, 24	Theory (IW dequantization, LSGAN). WGAN/WGAN-GP on 2D data. WGAN-GP on CIFAR10.
6	April, 24	May, 15	Theory (Neural ODE backprop). SN-GAN on CIFAR10. FID and Inception Score.

Game rules

• 6 homeworks each of 13 points = 78 points
• oral cozy exam = 26 points
• maximum points: 78 + 26 = 104 points

Final grade: floor(relu(#points/8 - 2))

Previous episodes

• MIPT, autumn 2021
• OzonMasters, spring 2021
• MIPT, autumn 2020

Author, feel free to contact :)

• telegram: @roman_isachenko
• e-mail: [email protected]