Neural Networks and Related Models

(Please note: This course is co-taught by Dr Susan Wei)

This course is an introduction to deep learning as well as some probabilistic models involving neural networks (flow-based models and deep generative models).

Part I: Deep Learning Basics and Models

• An Introduction to Neural Networks: key components of DL pipeline, multilayer perception, forward/backward propagation, computational graphs
• Stochastic Optimization and Extensions
• The Art of Model Training and Regularization: Model selection, weight decay, dropout, initialization
• Convolutional Neural Networks and Recurrent Neural Networks

Part II: Variational Inference, Normalizing Flows, and Deep Generative Models

• An Introduction to Variational Inference
• Normalizing Flows (for both Variational Inference and Density Estimation)
• Deep Generative Models: Variational Autoencoders & Generative Adversarial Networks.

Computational Demonstrations will use the Python package PyTorch. The latter part of the course will involve demonstrations using the probabilistic programming language Pyro, which is based on PyTorch and has additional features.

Pre-requisites

Prior Mathematical Knowledge: Calculus, Linear Algebra, Probability, and Statistics at an advanced undergraduate level.

Prior Computational Experience: The computational aspects of this course use the programming language Python. Some familiarity with Python would be advantageous. Basic familiarity with programming is required.

Pre-reading

• For mathematical background required: https://d2l.ai/chapter_appendix-mathematics-for-deep-learning/index.html
• For a brief introduction to Python with a focus on Data Science: Appendix D of Data Science and Machine Learning: Mathematical and Statistical Methods, by Kroese et. Al. (2020)
  (note that the book is made available for free online by the authors)
• For those familiar with NumPy (array and linear algebra package in in Python) and wish to learn a bit about PyTorch prior to the course, see Tensor Basics.