Course material for TDT15 at NTNU, a course on Deep Generative Models (3.75 ECTS credits)

Teachers

Oskar Jørgensen
Prof. Helge Langseth

Course Content

In this course, you will learn the theory behind several popular deep generative models. In addition to a theoretical foundation, you will also implement and train these models yourself. We will talk about autoregressive models, variational autoencoders, generative adversarial nets, diffusion models, and flow-based models.

The grade will consist of 2 exercises (in the form of Jupyter Notebooks) and a final project. For the final project you will need to conduct a mini-research project where you train and evaluate a generative model. You will both submit the code, and hold a short presentation where you show your findings.

To truly understand these models, students will need a strong foundation in mathematics and statistics. For the exercises and the final project, knowledge of Python and PyTorch (or some other deep learning framework) is highly recommended. The course will be based on the book Deep Generative Modeling by Jakub M. Tomczak.

Course Grading

Notebooks: 30%
Final Project & Presentation: 70%

Lectures

The course will consist of several seminars, as well as a session where you will present your projects (room & time will be announced soon)

• Seminar 1: Introduction + Autoencoders (29.08, 12:15-14:00, F3, Gamle Fysikk)
• Seminar 2: Autoregressive models
• Seminar 3: Diffusion Models  + Generative Adversarial Nets
• Seminar 4: Flow-based models

Weeks where there is no lecture, there will be an optional exercise class where you can get help with the exercises or final project. The first exercise class will be in the first week of September.

Recommended Prerequisites

Calculus and linear algebra (TMA4400 or similar)

Statistics (TMA4245 or similar)

Machine Learning (TDT4172 or similar)

Python & PyTorch (or other Deep Learning framework)

Course Material

The course will be based on the book Deep Generative Modeling by Jakub M. Tomczak. Alternatively, much of the information can be found in the following papers, which can also serve as an inspiration when choosing your own mini-research project.

Papers

Autoregressive models

[Vaswani et al. 17] Attention is All You Need, NeurIPS 2017. This paper introduced the transformer architecture, which is used everywhere in deep learning. Autoregressive transformers in particular, are very common in generative models. They typically need discrete data to operate.
[Tian et al. 24] Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction, NeurIPS 2024. This paper showed that autoregressive models performing next-scale prediction can outperform diffusion models on image generation.

Autoencoders

[Kingma and Welling 14] Auto-Encoding Variational Bayes, ICLR 2014. Introduces the VAE.
[van den Oord et al. 17] Neural Discrete Representation Learning, NeurIPS 2017. This paper introduced the VQVAE, an extension to the VAE that allowed it to operate on discrete data.

Generative Adversarial Networks

[Goodfellow et al. 14] Generative Adversarial Nets, NIPS 2014. Introduces Generative Adversarial Networks (GANs).
[Arjovsky et al. 17] Wasserstein Generative Adversarial Networks, ICML 2017. Introduces Wasserstein-GANs, which improve stability during training.

Diffusion Models

[Ho et al. 20] Denoising Diffusion Probabilistic Models, NeurIPS 2020. Introduces DDPM models.
[Song et al. 20] Denoising Diffusion Implicit Models, ICLR 2021. Introduces DDIM models.

Flow models

[Lipman et al. 22] Flow Matching for Generative Modeling, ICLR 2023. One of three papers at ICLR 2023 that introduced flow matching.
[Zhai et al. 24] Normalizing Flows are Capable Generative Models, ICML 2025. Shows that normalizing flows also can scale well and generate high quality images.