CSCI 381: Deep Learning

UPCOMING DEADLINES

• Thursday, April 25: Lab 7 (Dispatches)
• Monday, May 6: Final Project

LECTURE SLIDES

1. Machine Learning: (pdf)
2. Gradient Descent: (pdf)
3. Variants of Gradient Descent: (pdf)
4. Multidimensional Gradient Descent: (pdf)
5. Argmin and Monotonicity: (pdf)
6. Linear Regression: (pdf)
7. Regularization: (pdf)
8. Logistic Regression: (pdf)
9. Function Graphs: (pdf)
10. Optimizing Logistic Regression: (pdf)
11. Limitations of Linear Models: (pdf)
12. Neural Networks: (pdf)
13. Activation Functions: (pdf)
14. Backpropagation: (pdf)
15. Minibatch Gradient Descent: (pdf)
16. Parameter Initialization: (pdf)
17. Dropout: (pdf)
18. Convolutional Neural Networks: (pdf)
19. Higher Order Kernels: (pdf)
20. Padding: (pdf)
21. Pooling: (pdf)
22. Computing Convolutions: (pdf)
23. Token Embeddings: (pdf)
24. Recurrent Neural Networks: (pdf)
25. Encoder-Decoder Models: (pdf)
26. Attention: (pdf)
27. Parallel Sequence Encoding: (pdf)
28. Transformer Decoders: (pdf)
29. Batch and Layer Normalization: (pdf)
30. Residual Connections: (pdf)
31. Multihead Attention: (pdf)
32. Foundation Models: (pdf)

LECTURE CODE/VIDEOS

1. Github Classroom Walkthrough: (mp4)
2. Introduction to Tensors: (repo)
3. Module Training: (simple.py) (circle.py)
4. Dropout: (dropout.py)
5. Digitize!: (repo)
6. Bitpatterns: (repo)
7. Image Convolutions: (convolve.py)
8. Word Embeddings: (analogy.py)
9. RNNs: (rnndemo.py)
10. RNN Decoding With Attention: (reverso.py)
11. Batch Normalization: (batchnorm.py)

LAB LINKS

• Lab 0 (Rubik): (classroom)
• Lab 1 (Descent): (classroom)
• Lab 2 (Lines): (classroom)
• Lab 3 (Logistics): (classroom)
• Lab 4 (Colonels): (classroom)
• Lab 5 (Basic Training): (classroom)
• Lab 6 (Kernels): (classroom)
• Lab 7 (Dispatches): (classroom)

SCHEDULE

Feb 2: Introduction

• Homework: (assignment) (solution)
• Reminders: 
  • Sign up for a Github account if you don’t already have one (www.github.com). 
  • Install pytorch in your Python environment: https://pytorch.org/get-started/locally/
  • Go through the Github Classroom walkthrough
  • Bring your laptop on Monday!

Feb 5: Torch and Tensors

• Classroom Activity: (repo)

Feb 7: Supervised Learning/Gradient Descent

Feb 9: Multidimensional Gradient Descent

• Homework: (assignment) (solution)

Feb 12: Argmin and Monotonic Functions

• Homework: (assignment) (solution)

Feb 14: Linear Regression

Feb 17: Carnaval d’Hiver

Feb 19: Regularization

• Homework: (assignment) (solution)

Feb 21: Logistic Regression/Function Graphs

• Homework: (assignment) (solution)

Feb 23: Optimizing Logistic Regression

Feb 26: “Logistics” Lab Session

• Homework: (assignment)

Feb 28: Neural Networks

Mar 1: Activation Functions/Backpropagation

• Homework: (assignment) (solution)

Mar 4: Minibatch Gradient Descent

Mar 6: Autograd/Torch Modules

Mar 8: Autograd/Torch Modules

• Lecture Code: (simple.py) (circle.py)

Mar 11: Parameter Initialization

Mar 13: Dropout

• Lecture Code: (dropout.py)

Mar 15: Digitize!

• Lecture Code: (repo)

Apr 1: Convolutional Neural Networks

• Homework: (assignment)

Apr 3: Higher Order Kernels

• Homework: (assignment)
• Lecture Code (Bitpatterns): (repo)
• Lecture Code (Image Convolutions): (convolve.py)

Apr 5: Padding and Pooling

Apr 10: Token Embeddings

• Lecture Code (Word Embeddings): (analogy.py)

Apr 12: Recurrent Neural Networks

• Lecture Code (RNNs): (rnndemo.py)

Apr 15: Encoder-Decoder Models

Apr 17: Attention

• Lecture Code (RNN Decoding with Attention): (reverso.py)

Apr 19: Transformers

Apr 22: Batch and Layer Normalization

• Lecture Code (Batch Normalization): (batchnorm.py)

Apr 24: Residual Connections and Multihead Attention