CSCI 378: Deep Learning

REED COLLEGE, SPRING 2019

This course is an introduction to deep neural architectures and their training. Beginning with the fundamentals of regression, optimization, and regularization, the course will then survey a variety of architectures and their associated applications. Students will develop projects that implement deep learning systems to perform various tasks. Prerequisites: Mathematics 201, 202, and Computer Science 221. 

BASIC INFO

Professor: Mark Hopkins, hopkinsm@reed.edu

Class Schedule: MWF 3:10-4pm, Physics 240A

Office Hours: MW 4:10-6pm, Th 10am-noon in Library 314 (M 4:10-5 and Th 10-11am are reserved for this class; the remainder are shared with CSCI 121).

Textbook (optional): Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.

Syllabus: downloadable here

LECTURE NOTES

  1. Machine Learning: A Whirlwind Guide (pdf)
  2. Gradient Descent (pdf)
  3. Gradient Descent: Now in 2D! (pdf)
  4. A Quick Review of Basic Probability Formulas (pdf)
  5. Bayesian Networks (pdf)
  6. How to Read a Bayesian Network (pdf)
  7. Functional Causal Models (pdf)
  8. Exponential Distributions (pdf)
  9. Regression Problems (pdf)
  10. Argmax and Monotonic Functions (pdf)
  11. Linear Regression: MLE (pdf)
  12. Linear Regression: MAP (pdf)
  13. Logistic Regression: MLE (pdf)
  14. Chain Rule of Partial Derivatives (pdf)
  15. Regression: A Neural View (pdf)
  16. Feature Discovery Networks (pdf)
  17. Training Feature Discovery Networks (pdf)  
  18. Neural Networks and Backpropagation (pdf)
  19. Backpropagation: A Matrix Formulation (pdf)
  20. Minibatch Gradient Descent (pdf)
  21. Multiway Classification (pdf)
  22. Convolutional Neural Networks (pdf)
  23. Padding and Pooling (pdf)
  24. Activation Functions (pdf)
  25. CNNs for Images (pdf)

 

LECTURE CODE

  1. Introduction to Tensors (Wednesday, Jan. 30): https://classroom.github.com/a/h3tOU2km
  2. Solution to Rubik (Monday, Feb. 4): https://github.com/Mark-Hopkins-at-Reed/csci-378/blob/master/code/rubik/rubik.py
  3. Introduction to Autograd (Friday, March 22): https://classroom.github.com/a/dNYuHSx7

 

HOMEWORK AND PROJECT LINKS

  1. HW1 (“Memoize”, due Friday, Feb. 1): https://classroom.github.com/a/WqP7Oqmf
  2. HW2 (“Rubik”, due Monday, Feb. 4): https://classroom.github.com/a/yaOkb2-4
  3. Project 1 (“Descent”, due Friday, Feb. 8): https://classroom.github.com/a/zidPHJ4R
  4. Project 2 (“Lines”, due Friday, Mar. 8): https://classroom.github.com/a/Ewo2k4xc
  5. Project 3 (“Logistics”, due Friday, Mar. 15): https://classroom.github.com/a/EZrBrI8n
  6. HW9 (due Monday, Apr. 1): https://github.com/Mark-Hopkins-at-Reed/csci-378/blob/master/hw/hw9.pdf
  7. Project 4, Part 1 (“Colonels”, due Friday, Apr. 5): https://classroom.github.com/a/veJR40RC
  8. Project 4, Part 2 (“Basic Training”, due Friday, Apr. 12): https://classroom.github.com/a/o8q4BTjw
  9. Project 4, Part 3 (“Kernels”, due Friday, Apr. 26): https://classroom.github.com/a/jX3RW4jL

 

SCHEDULE

Jan 28: Introduction

Jan 30: Class Infrastructure/Introduction to Torch

  • Reading: None
  • Assignment (due Friday, Feb. 1): HW1 (link above, under Homework Links).

Feb 1: A Whirlwind Guide to Machine Learning

  • Reading: None
  • Assignment (due Monday, Feb. 4): HW2 (link above, under Homework Links).

Feb 4: Gradient Descent

  • Reading: Review Machine Learning: A Whirlwind Guide lecture notes.
  • Assignment (suggested): Find out where in the world Chekhov’s Sun might be (i.e. complete Descent1.ipynb).

Feb 6: Gradient Descent

  • Reading: Review first part of Gradient Descent lecture notes.
  • Optional Reading: Goodfellow, Chapter 8.3 and 8.5
  • Assignment: Project 1 (link above, under Homework and Project Links)

Feb 8: Bayesian Networks

Feb 11: Reading a Bayesian Network

Feb 13: D-Separation/Causal Models

Feb 15: Causal Models

Feb 18: Exponential Distributions

  • Optional Reading: Goodfellow, Chapter 3.9.3 and 3.9.4
  • Assignment: Complete handout distributed in class.

Feb 20: Regression Problems

Feb 22: Linear Regression and Maximum Likelihood

  • Reading: Review lecture notes on Regression Problems.
  • Optional Reading: Goodfellow, Chapter 5.1, 5.5
  • Assignment (due Monday): Complete handout distributed in class.

Feb 25: Linear Regression and Maximum Likelihood

  • Reading: Nothing additional, but review lecture notes on Regression Problems if you haven’t already.
  • Assignment: Nothing to hand in, but try to get Questions 1 and 2 of Project 2 finished.

Feb 27: Linear Regression and Maximum Likelihood

Mar 1: Ridge Regression

  • Reading: Review lecture notes on Linear Regression: MLE. In particular, make sure to understand how we compute the “closed form” solution for ordinary linear regression (i.e. slides 10-12).
  • Assignment: Complete slide 16 from Linear Regression: MAP notes.

Mar 4: Lasso Regression

Mar 6: Midterm 1

  • Assignment: Project 2 is due on Friday.

Mar 8: Logistic Regression

  • Reading: Review lecture notes on Regression Problems (pdf), particularly the part on Logistic Regression.
  • Assignment: None.

Mar 11: Logistic Regression/The Chain Rule of Partial Derivatives

  • Reading: Review lecture notes on The Chain Rule of Partial Derivatives
  • Assignment: None, but remember that Project 3 is due on Friday.

Mar 13: Regression: A Neural View

  • Reading: None.
  • Assignment: Project 3 is due Friday.

Mar 15: Feature Discovery Networks

  • Reading: Review lecture notes on Regression: A Neural View.
  • Assignment: None.

Mar 18: Neural Networks and Backpropagation

  • Reading: Review lecture notes on Feature Discovery Networks and Training Feature Discovery Networks.
  • Assignment: Complete the handout distributed in class.

Mar 20: Backpropagation: A Matrix Formulation

  • Reading: Review the lecture notes on Neural Networks and Backpropagation.
  • Assignment: Bring laptops to class on Friday.

Mar 22: Torch and Autograd

  • Reading: Review the lecture notes on Backpropagation: A Matrix Formulation.
  • Assignment: Complete HW9.

Apr 1: Minibatch Gradient Descent

  • Reading: None.
  • Assignment: None, but Project 4 (Part 1) is due on Friday.

Apr 3: Multiway Classification

  • Reading: Review the notes on Minibatch Gradient Descent.
  • Assignment: Project 4 (Part 1) is due Friday.

Apr 5: Intro to Convolutional Neural Networks

  • Reading: Review the notes on Multiway Classification.
  • Assignment: None, but Project 4 (Part 2) has been assigned and is due next Friday.

Apr 8: Convolutional Neural Networks

  • Reading: None.
  • Assignment: None, but remember that Project 4 (Part 2) is due Friday.

Apr 10: Padding and Pooling

Apr 12: Midterm 2 Review

  • Reading: Review the lecture notes on Padding and Pooling.
  • Assignment: Midterm 2 is on Monday in class. It will be 80 minutes, from 3:10pm to 4:30pm.

Apr 15: Midterm 2

  • Reading: Study for the midterm!
  • Assignment: None.

Apr 17: CNNs for Images

  • Reading: None.
  • Assignment: Answer the question posed in Slide 13 of the CNNs for Images notes.

Apr 19: Computing Convolutions