Introduction to Machine Learning
Virginia Tech, Electrical and Computer EngineeringFall 2016: ECE 5424 / 4424 - CS 5824 / 4824
Course Information
Have you ever wondered how Siri understands voice commands? How Netflix recommends movies to watch? How Kinnect recognizes full-body gestures? How Watson defeated Jeopardy champions? The answer is Machine Learning -- the study of algorithms that learn from large quantities of data, identify patterns and make predictions on new instances.
- Instructor
- Stefan Lee (steflee -at- vt)
- Office Hours
- Friday 3-5PM, Whittemore 468
- Lead TA
- Aroma Mahendru (aroma.mahendru -at- vt)
- Office Hours
- Wednesday 3-5PM, Whittemore 266
- Class meets
- Tues, Thurs 2:00 - 3:15, DURH 261
- Homeworks
- Scholar Assignments
- Q&A and Discussions
- Scholar Forum
- Class Kaggle Links
-
Released with Homeworks
- HW Kaggle
Challenge Curators - Viraj Prabhu, Deshraj Yadav,
Tejas Khot, Qing Sun
- Supervised Learning (Classification/Regression)
- Nearest Neighbour
- Naive Bayes
- Logistic Regression
- Support Vector Machine
- Neural Networks (Multi-layer Perceptrons, Convolutional Neural Networks, Recurrrent Neural Nets)
- Decision Trees
- Unsupervised Learning
- Clustering, Kmeans
- Dimensionality Reduction (Principal Component Analysis, Linear Discriminant Analysis)
- Other Learning Setting and Advanced Topics
- Weakly-Supervised and Semi-Supervised Learning
- Reinforcement Learning
- Learning Theory (VC dimensions, generalization error bounds, PAC Theory)
- Structured Learning (Bayes Nets, Markov Random Fields, Structured SVMs)
Frequently Asked Questions
- I want to register for this course but it is full. What do I do?
- This class is filled to capacity every year. How many (if any) additional students we can accommodate depends on how many TAs we get and if there is a bigger room available. Best advice is to see the advisors in 340 Whittemore about force-adding the course and come to the first day of class. Many student tend to drop the first week after HW0 so you may be able to join the class.
- How should I contact the instructors and TAs with questions about the class?
- We will primarily use the Scholar forum for discussion. If you are confused about something, it is likely others might have similar doubts. Posting and responding to questions on the course forum helps us address everyone at once. Of course if something is private, just email the instructor.
- How do I do well in the course?
- Make sure you have the proper prerequisite skills to succeed. We list some general skills you will need and HW0 will give you a chance to test yours before drop/add week finishes. If you decided to stick with the class, the best advice is to pay attention in lecture, ask / answer questions on the forum, start early on assignments, and most importantly of all - start early on assignments. Seriously. Start early. Start early. Start early.
Schedule (Tentative)
- General Setup: learning from data
- Your First Classifier: 1/k Nearest Neighbour
- (Finish) 1/k Nearest Neighbour
- Kernel regression/classification
- (Finish) Kernel regression/classification
- Curse of Dimensionality
- Probability Review
- Statistical Estimation (MLE, MAP, Bayesian)
- Gaussians
- Linear Models
- Regression
- Regression
- (Finish) Regression
- Model Selection
- Error Decomposition
- Cross-validation
- (Finish) Error Decomposition
- Bias-Variance Tradeoff
- Classification: Naive Bayes
- (Finish) Naive Bayes
- Classification: Logistic Regression
- (Finish) Logistic Regression
- NB & LR connections
- Support Vector Machines
- Linear SVMs
- Hard & soft-margin SVMs
- Support Vector Machines
- Lagrangian Duality
- Support Vector Machines
- SVM dual
- Kernel Trick
- (Finish) Support Vector Machines
- Multi-class SVT
- Neural Networks
- Perceptron
- Multilayer Perceptron
- Neural Networks
- Backprop
- Autoencoders
- Convolutional Nets
- Decision Trees
- Mutual Information
- (Finish) Decision Trees
- Bagging
- (Finish) Bagging
- Boosting
- PAC-Learning
- Generalization bounds
- Clustering
- K-Means
- Gaussian Mixture Models
- EM for GMMs
- EM for general unsupervised learning
- (Finish) EM for general unsupervised learning
- Factor Analysis
- PCA
- SVD
- Dimensionality Reduction
- 40% Homework
- 25% Project
- 10% Mid Term
- 20% Final
- 5% Class Participation
- No penalties for medical reasons (bring doctor's note) or emergencies.
- Every student has 5 free late days (5 x 24hour-chunks) for this course; these cannot be used for exams (midterm or final) or the poster session.
- After all free late days are used up, penalty is 25% for each additional late day.
- Study groups are allowed. You can discuss the questions and potential directions for solving them. However, your discussions should be collaborative and not involve one student simply telling the other student the solutions.
- If you worked in a group, please write the names of the other participants on the first page of your assignment.
- Important: you need to write your solutions and code independently, and not as a group activity.
- We often reuse problem set questions from various other sources. We expect students to want to learn and not google for answers. The purpose of problem sets in this class is to help you think about the material, not just give us the right answers. We take Honor Code violations seriously.
- Probability and Statistics
- Distributions, densities, marginalization, moments, typical distributions (Gaussian, Laplacian, etc)
- Calculus and Linear Algebra
- Matrix multiplication, eigenvalues, positive semi-definiteness, multivariate derivates.
- Algorithms
- Dynamic programming, basic data structures, complexity (NP-hardness).
- Programming
- Matlab for HWs. Your language of choice for project.
- No coding / compilation / debugging support.
- Ability to deal with abstract mathematical concepts.
- 10-701 Machine Learning, Carnegie Mellon University
- CIS 520 Machine Learning, UPenn
- CS 229 Machine Learning, Stanford
- CSE 546 Machine Learning, University of Washington
- Machine Learning, Coursera
| Week/Day | Date | Topic | Notes |
|---|---|---|---|
| 1 T | Aug 23 |
Overview of Machine Learning Slides (pptx), Slides (pdf) |
HW0 is out. Due: Aug 25. Readings: Barber Chap 1, 13.1. [Optional] Video: Sam Roweis -- What is Machine Learning? |
| 1 R | Aug 25 | Supervised Learning
|
Reading: Barber Chap 14. [Optional] Video: Pedro Domingos -- The K-NN Algorithm |
| 2 T | Aug 30 |
|
|
| 2 R | Sept 1 |
|
HW1 is out. Due: Sept 14. |
| 3 T | Sept 6 |
Project Details Supervised Statistical Learning |
Readings: Barber 8.1, 8.2. [Optional] Videos: Probability Primer. [Optional] Video: Michael Jordon -- Bayesian or Frequentist: Which Are You? |
| 3 R | Sept 8 |
|
[Optional] Video: Daphne Koller -- Coursera: Probabilistic Graphical Models,
MLE Lecture,
MAP
Lecture. Readings: Barber 8.6, 8.7. |
| 4 T | Sept 13 |
|
Readings: Barber 8.4, 17.1, 17.2. |
| 4 R | Sept 15 |
|
HW2 is out. Due: Oct. 3rd. Readings: 17.1, 17.2. |
| 5 T | Sept 20 |
|
Project proposal is due: Sept 23. |
| 5 R | Sept 22 |
|
Readings: Barber 10.1-3. [Optional] Video: Andrew Ng -- Naive Bayes |
| 6 T | Sept 27 |
|
|
| 6 R | Sept 29 |
|
Readings: Barber 17.4. [Optional] Video: Andrew Ng -- Logistic Regression. Reading: Tom Mitchell -- Book Chapter: Naive Bayes and Logistic Regression |
| 7 T | Oct 4 |
Midterm Review Slides (pptx) , Slides (pdf) |
Readings: Boyd and Vandenberghe
Chap 2 (Convex Sets): 2.1, 2.2, 2.3; Chap 3 (Convex Functions): 3.1, 3.2. [Optional] Video: Stephen Boyd -- Convex Optimization. |
| 7 R | Oct 6 | In class Mid-Term | |
| 8 T | Oct 11 |
|
Readings: Barber 17.5. [Optional] Video: Andrew Ng -- KKT Conditions and SVM Duality. [Optional] Video: Stephen Boyd -- Lagrangian Duality. |
| 8 R | Oct 13 |
|
|
| 9 T | Oct 18 |
|
HW3 is out. Due Nov 7th. |
| 9 R | Oct 20 |
|
Reading: Murphy 16.5. [Optional] Reading: Hastie, Tibshirani, Friedman -- Chap 11. [Optional] Video: Andrew Ng -- Coursera: Machine Learning, Neural Networks lecture, Backpropagation lecture. |
| 10 T | Oct 25 |
|
Reading: Murphy 16.5. [Optional] Reading: Hastie, Tibshirani, Friedman -- Chap 11. [Optional] Reading: Yann LeCun et al. -- Efficient BackProp |
| 10 R | Oct 27 |
|
Reading: Murphy 16.1-16.2.
[Optional] Reading: Hastie, Tibshirani, Friedman -- Chap 9.2. [Optional] Video: Pedro Domingos -- Coursera: Machine Learning, Decision Trees lecture, |
| 11 T | Nov 1 |
|
|
| 11 R | Nov 3 |
Ensemble Methods
|
Reading: Murphy 16.4. [Optional] Video: Robert Schapire -- Boosting [Optional] Reading: Tom Dietterich -- Ensemble Methods in Machine Learning. |
| 12 T | Nov 8 | In-class project presentations. |
HW4 is out. Due: Nov 21. |
| 12 R | Nov 10 | In-class project presentations. | |
| 13 T | Nov 15 |
CVPR Deadline - No Class |
|
| 13 R | Nov 17 |
Unsupervised Learning |
Readings: Barber 20.1-20.3. [Optional] Video: Andrew Ng -- Clustering, GMT |
| 14 T | Nov 22 | Thanksgiving Break - No Class | |
| 14 R | Nov 24 | Thanksgiving Break - No Class | |
| 15 T | Nov 29 |
|
|
| 15 R | Dec 1 |
|
Readings: Barber 15.1-15.4. [Optional] Video: Andrew Ng -- PCA |
| Dec 6 | Project Poster+Demo Presentation | Location TBA | |
| Dec 14 | Final Exam | In class (DURH 261); 2:05 - 4:05 pm |
Grading
Late policy for deliverables
Collaboration Policy
Project
Team project: 1-3 people. Description and grading policy (proposal, midsem presentation, final poster + demo presentation).Textbooks
None required. Reference Books:[On Library Reserve]
Machine Learning: A Probabilistic Perspective, Kevin Murphy
[Free PDF from the book webpage]
The Elements of Statistical Learning, Hastie, Tibshirani, and Friedman
[Free PDF from author's webpage]
Bayesian Reasoning and Machine Learning, David Barber
[Available in the Library]
Pattern Recognition and Machine Learning, Chris Bishop
Prerequisites
Related Classes / Online Resources
Note to people outside VT
Feel free to use the slides and materials available online here.If you use our slides, an appropriate attribution is requested.
Please email the instructor with any corrections or improvements.
© 2016 Virginia Tech
Webpage CSS courtesy Bootstrap and Polo Chau.