CSCI 381: Deep Learning
WILLIAMS COLLEGE, SPRING 2024
This course is an introduction to deep neural architectures and their training. Beginning with the fundamentals of regression, optimization, and regularization, the course will then survey a variety of architectures and their associated applications. Students will develop projects that implement deep learning systems to perform various tasks. Prerequisites: Computer Science 136 and the Discrete Math Proficiency Requirement.
BASIC INFO
Professor: Mark Hopkins, mh24@williams.edu
Class Schedule: MWF 11-1150am (Section 1), noon-1250pm (Section 2)
Office Hours: Tu 1-230pm, W 2-330pm, Th 2-330pm (TCL 309)
Syllabus: downloadable here
UPCOMING DEADLINES
- Thursday, April 25: Lab 7 (Dispatches)
- Monday, May 6: Final Project
LECTURE SLIDES
- Machine Learning: (pdf)
- Gradient Descent: (pdf)
- Variants of Gradient Descent: (pdf)
- Multidimensional Gradient Descent: (pdf)
- Argmin and Monotonicity: (pdf)
- Linear Regression: (pdf)
- Regularization: (pdf)
- Logistic Regression: (pdf)
- Function Graphs: (pdf)
- Optimizing Logistic Regression: (pdf)
- Limitations of Linear Models: (pdf)
- Neural Networks: (pdf)
- Activation Functions: (pdf)
- Backpropagation: (pdf)
- Minibatch Gradient Descent: (pdf)
- Parameter Initialization: (pdf)
- Dropout: (pdf)
- Convolutional Neural Networks: (pdf)
- Higher Order Kernels: (pdf)
- Padding: (pdf)
- Pooling: (pdf)
- Computing Convolutions: (pdf)
- Token Embeddings: (pdf)
- Recurrent Neural Networks: (pdf)
- Encoder-Decoder Models: (pdf)
- Attention: (pdf)
- Parallel Sequence Encoding: (pdf)
- Transformer Decoders: (pdf)
- Batch and Layer Normalization: (pdf)
- Residual Connections: (pdf)
- Multihead Attention: (pdf)
- Foundation Models: (pdf)
LECTURE CODE/VIDEOS
- Github Classroom Walkthrough: (mp4)
- Introduction to Tensors: (repo)
- Module Training: (simple.py) (circle.py)
- Dropout: (dropout.py)
- Digitize!: (repo)
- Bitpatterns: (repo)
- Image Convolutions: (convolve.py)
- Word Embeddings: (analogy.py)
- RNNs: (rnndemo.py)
- RNN Decoding With Attention: (reverso.py)
- Batch Normalization: (batchnorm.py)
LAB LINKS
- Lab 0 (Rubik): (classroom)
- Lab 1 (Descent): (classroom)
- Lab 2 (Lines): (classroom)
- Lab 3 (Logistics): (classroom)
- Lab 4 (Colonels): (classroom)
- Lab 5 (Basic Training): (classroom)
- Lab 6 (Kernels): (classroom)
- Lab 7 (Dispatches): (classroom)
SCHEDULE
Feb 2: Introduction
- Homework: (assignment) (solution)
- Reminders:
- Sign up for a Github account if you don’t already have one (www.github.com).
- Install pytorch in your Python environment: https://pytorch.org/get-started/locally/
- Go through the Github Classroom walkthrough
- Bring your laptop on Monday!
Feb 5: Torch and Tensors
- Classroom Activity: (repo)
Feb 7: Supervised Learning/Gradient Descent
Feb 9: Multidimensional Gradient Descent
- Homework: (assignment) (solution)
Feb 12: Argmin and Monotonic Functions
- Homework: (assignment) (solution)
Feb 14: Linear Regression
Feb 17: Carnaval d’Hiver
Feb 19: Regularization
- Homework: (assignment) (solution)
Feb 21: Logistic Regression/Function Graphs
- Homework: (assignment) (solution)
Feb 23: Optimizing Logistic Regression
Feb 26: “Logistics” Lab Session
- Homework: (assignment)
Feb 28: Neural Networks
Mar 1: Activation Functions/Backpropagation
- Homework: (assignment) (solution)
Mar 4: Minibatch Gradient Descent
Mar 6: Autograd/Torch Modules
Mar 8: Autograd/Torch Modules
Mar 11: Parameter Initialization
Mar 13: Dropout
- Lecture Code: (dropout.py)
Mar 15: Digitize!
- Lecture Code: (repo)
Apr 1: Convolutional Neural Networks
- Homework: (assignment)
Apr 3: Higher Order Kernels
- Homework: (assignment)
- Lecture Code (Bitpatterns): (repo)
- Lecture Code (Image Convolutions): (convolve.py)
Apr 5: Padding and Pooling
Apr 10: Token Embeddings
- Lecture Code (Word Embeddings): (analogy.py)
Apr 12: Recurrent Neural Networks
- Lecture Code (RNNs): (rnndemo.py)
Apr 15: Encoder-Decoder Models
Apr 17: Attention
- Lecture Code (RNN Decoding with Attention): (reverso.py)
Apr 19: Transformers
Apr 22: Batch and Layer Normalization
- Lecture Code (Batch Normalization): (batchnorm.py)
Apr 24: Residual Connections and Multihead Attention