**CSCI 378: Deep Learning**

###### REED COLLEGE, SPRING 2019

This course is an introduction to deep neural architectures and their training. Beginning with the fundamentals of regression, optimization, and regularization, the course will then survey a variety of architectures and their associated applications. Students will develop projects that implement deep learning systems to perform various tasks. Prerequisites: Mathematics 201, 202, and Computer Science 221.

### BASIC INFO

**Professor:** Mark Hopkins, hopkinsm@reed.edu

**Class Schedule:** MWF 3:10-4pm, Physics 240A

**Office Hours:** MW 4:10-6pm, Th 10am-noon in Library 314 (M 4:10-5 and Th 10-11am are reserved for this class; the remainder are shared with CSCI 121).

**Textbook (optional):** *Deep Learning *by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.

**Syllabus:** downloadable here

### LECTURE NOTES

**Machine Learning: A Whirlwind Guide**(pdf)**Gradient Descent**(pdf)**Gradient Descent: Now in 2D!**(pdf)**A Quick Review of Basic Probability Formulas**(pdf)**Bayesian Networks**(pdf)**How to Read a Bayesian Network**(pdf)**Functional Causal Models**(pdf)**Exponential Distributions**(pdf)**Regression Problems**(pdf)**Argmax and Monotonic Functions**(pdf)**Linear Regression: MLE**(pdf)**Linear Regression: MAP**(pdf)**Logistic Regression: MLE**(pdf)**Chain Rule of Partial Derivatives**(pdf)**Regression: A Neural View**(pdf)**Feature Discovery Networks**(pdf)**Training Feature Discovery Networks**(pdf)**Neural Networks and Backpropagation**(pdf)**Backpropagation: A Matrix Formulation**(pdf)**Minibatch Gradient Descent**(pdf)**Multiway Classification**(pdf)**Convolutional Neural Networks**(pdf)**Padding and Pooling**(pdf)**Activation Functions**(pdf)**CNNs for Images**(pdf)

### LECTURE CODE

https://classroom.github.com/a/h3tOU2km**Introduction to Tensors (Wednesday, Jan. 30):****Solution to Rubik (Monday, Feb. 4):**https://github.com/Mark-Hopkins-at-Reed/csci-378/blob/master/code/rubik/rubik.py**Introduction to Autograd (Friday, March 22):**https://classroom.github.com/a/dNYuHSx7

### HOMEWORK AND PROJECT LINKS

https://classroom.github.com/a/WqP7Oqmf**HW1 (“Memoize”, due Friday, Feb. 1):****HW2 (“Rubik”, due Monday, Feb. 4):**https://classroom.github.com/a/yaOkb2-4**Project 1 (“Descent”, due Friday, Feb. 8):**https://classroom.github.com/a/zidPHJ4R**Project 2 (“Lines”, due Friday, Mar. 8):**https://classroom.github.com/a/Ewo2k4xc**Project 3 (“Logistics”, due Friday, Mar. 15):**https://classroom.github.com/a/EZrBrI8n**HW9 (due Monday, Apr. 1):**https://github.com/Mark-Hopkins-at-Reed/csci-378/blob/master/hw/hw9.pdf**Project 4, Part 1 (“Colonels”, due Friday, Apr. 5):**https://classroom.github.com/a/veJR40RC**Project 4, Part 2 (“Basic Training”, due Friday, Apr. 12):**https://classroom.github.com/a/o8q4BTjw**Project 4, Part 3 (“Kernels”, due Friday, Apr. 26):**https://classroom.github.com/a/jX3RW4jL

### SCHEDULE

**Jan 28: Introduction**

**Reading:**None**Assignment:**- Sign up for a Github account if you don’t already have one (www.github.com).
- Go through the following setup, if desired (no need to use the Anaconda setup if you prefer another one, but make sure you install torch and go through the Github Classroom tutorial): https://github.com/Mark-Hopkins-at-Reed/csci-378/blob/master/admin/setup.md
- Bring your laptop on Wednesday!

**Jan 30: Class Infrastructure/Introduction to Torch**

**Reading:**None**Assignment (due Friday, Feb. 1):**HW1 (link above, under Homework Links).

**Feb 1: A Whirlwind Guide to Machine Learning**

**Reading:**None**Assignment (due Monday, Feb. 4):**HW2 (link above, under Homework Links).

**Feb 4: Gradient Descent**

**Reading:**Review Machine Learning: A Whirlwind Guide lecture notes.**Assignment (suggested):**Find out where in the world Chekhov’s Sun might be (i.e. complete Descent1.ipynb).

**Feb 6: Gradient Descent**

**Reading:**Review first part of Gradient Descent lecture notes.**Optional Reading:**Goodfellow, Chapter 8.3 and 8.5**Assignment:**Project 1 (link above, under Homework and Project Links)

**Feb 8: Bayesian Networks**

**Reading:**Review full Gradient Descent and Gradient Descent: Now in 2D! notes.**Assignment:**None

**Feb 11: Reading a Bayesian Network**

**Reading:**Review lecture notes on Probability and Bayesian Networks .**Assignment (due Wednesday):**Complete handout distributed in class.

**Feb 13: D-Separation/Causal Models**

**Reading:**Review lecture notes on How to Read a Bayesian Network.**Assignment (due Friday):**Complete handout distributed in class.

**Feb 15: Causal Models**

**Reading:**None**Assignment:**Nothing to hand in, but please review lecture notes on Functional Causal Models.

**Feb 18: Exponential Distributions**

**Optional Reading:**Goodfellow, Chapter 3.9.3 and 3.9.4**Assignment:**Complete handout distributed in class.

**Feb 20: Regression Problems**

**Reading:**Review lecture notes on Exponential Distributions.**Assignment:**None.

**Feb 22: Linear Regression and Maximum Likelihood**

**Reading:**Review lecture notes on Regression Problems.**Optional Reading:**Goodfellow, Chapter 5.1, 5.5**Assignment (due Monday):**Complete handout distributed in class.

**Feb 25: Linear Regression and Maximum Likelihood**

**Reading:**Nothing additional, but review lecture notes on Regression Problems if you haven’t already.**Assignment:**Nothing to hand in, but try to get Questions 1 and 2 of Project 2 finished.

**Feb 27: Linear Regression and Maximum Likelihood**

**Reading:**Review lecture notes on Argmin and Monotonic Functions.**Assignment:**Nothing to hand in, but try to get Question 3 of Project 2 finished.

**Mar 1: Ridge Regression**

**Reading:**Review lecture notes on Linear Regression: MLE. In particular, make sure to understand how we compute the “closed form” solution for ordinary linear regression (i.e. slides 10-12).**Assignment:**Complete slide 16 from Linear Regression: MAP notes.

**Mar 4: Lasso Regression**

**Reading:**Review lecture notes on Linear Regression: MAP.**Assignment:**Study for Wednesday’s exam!

**Mar 6: Midterm 1**

**Assignment:**Project 2 is due on Friday.

**Mar 8: Logistic Regression**

**Reading:**Review lecture notes on Regression Problems (pdf), particularly the part on Logistic Regression.**Assignment:**None.

**Mar 11: Logistic Regression/The Chain Rule of Partial Derivatives**

**Reading:**Review lecture notes on The Chain Rule of Partial Derivatives**Assignment:**None, but remember that Project 3 is due on Friday.

**Mar 13: Regression: A Neural View**

**Reading:**None.**Assignment:**Project 3 is due Friday.

**Mar 15: Feature Discovery Networks**

**Reading:**Review lecture notes on Regression: A Neural View.**Assignment:**None.

**Mar 18: Neural Networks and Backpropagation**

**Reading:**Review lecture notes on Feature Discovery Networks and Training Feature Discovery Networks.**Assignment:**Complete the handout distributed in class.

**Mar 20: Backpropagation: A Matrix Formulation**

**Reading:**Review the lecture notes on Neural Networks and Backpropagation.**Assignment:**Bring laptops to class on Friday.

**Mar 22: Torch and Autograd**

**Reading:**Review the lecture notes on Backpropagation: A Matrix Formulation.**Assignment:**Complete HW9.

**Apr 1: Minibatch Gradient Descent**

**Reading:**None.**Assignment:**None, but Project 4 (Part 1) is due on Friday.

**Apr 3: Multiway Classification**

**Reading:**Review the notes on Minibatch Gradient Descent.**Assignment:**Project 4 (Part 1) is due Friday.

**Apr 5: Intro to Convolutional Neural Networks**

**Reading:**Review the notes on Multiway Classification.**Assignment:**None, but Project 4 (Part 2) has been assigned and is due next Friday.

**Apr 8: Convolutional Neural Networks**

**Reading:**None.**Assignment:**None, but remember that Project 4 (Part 2) is due Friday.

**Apr 10: Padding and Pooling**

**Reading:**Review the lecture notes on Convolutional Neural Networks.**Assignment:**Project 4 (Part 2) is due Friday.

**Apr 12: Midterm 2 Review**

**Reading:**Review the lecture notes on Padding and Pooling.**Assignment:**Midterm 2 is on Monday in class. It will be 80 minutes,**from****3:10pm to 4:30pm**.

**Apr 15: Midterm 2**

**Reading:**Study for the midterm!**Assignment:**None.

**Apr 17: CNNs for Images**

**Reading:**None.**Assignment:**Answer the question posed in Slide 13 of the CNNs for Images notes.

**Apr 19: Computing Convolutions**

**Reading:**Review the notes on CNNs for Images.